We're unable to sign you in at this time. Please try again in a few minutes.
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Investigation |

Reliability of Risk-Adjusted Outcomes for Profiling Hospital Surgical Quality

Robert W. Krell, MD1; Ahmed Hozain, BS2; Lillian S. Kao, MD, MS3; Justin B. Dimick, MD, MPH1
[+] Author Affiliations
1Department of Surgery, University of Michigan Health System, Ann Arbor
2Department of Surgery, Michigan State University College of Human Medicine, East Lansing
3Department of Surgery, The University of Texas at Houston Medical School, Houston
JAMA Surg. 2014;149(5):467-474. doi:10.1001/jamasurg.2013.4249.
Text Size: A A A
Published online

Importance  Quality improvement platforms commonly use risk-adjusted morbidity and mortality to profile hospital performance. However, given small hospital caseloads and low event rates for some procedures, it is unclear whether these outcomes reliably reflect hospital performance.

Objective  To determine the reliability of risk-adjusted morbidity and mortality for hospital performance profiling using clinical registry data.

Design, Setting, and Participants  A retrospective cohort study was conducted using data from the American College of Surgeons National Surgical Quality Improvement Program, 2009. Participants included all patients (N = 55 466) who underwent colon resection, pancreatic resection, laparoscopic gastric bypass, ventral hernia repair, abdominal aortic aneurysm repair, and lower extremity bypass.

Main Outcomes and Measures  Outcomes included risk-adjusted overall morbidity, severe morbidity, and mortality. We assessed reliability (0-1 scale: 0, completely unreliable; and 1, perfectly reliable) for all 3 outcomes. We also quantified the number of hospitals meeting minimum acceptable reliability thresholds (>0.70, good reliability; and >0.50, fair reliability) for each outcome.

Results  For overall morbidity, the most common outcome studied, the mean reliability depended on sample size (ie, how high the hospital caseload was) and the event rate (ie, how frequently the outcome occurred). For example, mean reliability for overall morbidity was low for abdominal aortic aneurysm repair (reliability, 0.29; sample size, 25 cases per year; and event rate, 18.3%). In contrast, mean reliability for overall morbidity was higher for colon resection (reliability, 0.61; sample size, 114 cases per year; and event rate, 26.8%). Colon resection (37.7% of hospitals), pancreatic resection (7.1% of hospitals), and laparoscopic gastric bypass (11.5% of hospitals) were the only procedures for which any hospitals met a reliability threshold of 0.70 for overall morbidity. Because severe morbidity and mortality are less frequent outcomes, their mean reliability was lower, and even fewer hospitals met the thresholds for minimum reliability.

Conclusions and Relevance  Most commonly reported outcome measures have low reliability for differentiating hospital performance. This is especially important for clinical registries that sample rather than collect 100% of cases, which can limit hospital case accrual. Eliminating sampling to achieve the highest possible caseloads, adjusting for reliability, and using advanced modeling strategies (eg, hierarchical modeling) are necessary for clinical registries to increase their benchmarking reliability.

Figures in this Article

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal


Place holder to copy figure label and caption
Mean Reliability of Risk-Adjusted 30-Day Outcomes by Hospital Caseload Tercile and Procedure Type

A, Mortality; the mortality rate for laparoscopic gastric bypass was zero for all hospital caseloads. B, Severe morbidity. C, Any morbidity.

Graphic Jump Location





You need to register in order to view this quiz.
Submit a Comment


Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 7

Sign in

Purchase Options

• Buy this article
• Subscribe to the journal

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections

The Rational Clinical Examination: Evidence-Based Clinical Diagnosis
Evidence to Support the Update

The Rational Clinical Examination: Evidence-Based Clinical Diagnosis
Original Article: Does This Patient Have an Instability of the Shoulder or a Labrum Lesion?