0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Original Article |

Evaluating Popular Media and Internet-Based Hospital Quality Ratings for Cancer Surgery FREE

Nicholas H. Osborne, MD; Amir A. Ghaferi, MD; Lauren H. Nicholas, PhD; Justin B. Dimick; MD MPH
[+] Author Affiliations

Author Affiliations: Michigan Surgical Collaborative for Outcomes Research and Evaluation, Department of Surgery, University of Michigan, Ann Arbor.


Arch Surg. 2011;146(5):600-604. doi:10.1001/archsurg.2011.119.
Text Size: A A A
Published online

Objective  We sought to determine whether “best hospitals,” as defined by the US News & World Report or HealthGrades, have lower mortality rates than all other American hospitals for cancer surgery.

Design  Retrospective cross-sectional study.

Setting  Medicare database (2005-2006).

Patients  All patients with a diagnosis of malignancy who underwent pancreatectomy, esophagectomy, and colectomy (n = 82 724).

Main Outcomes Measures  Risk-adjusted mortality rates at best hospitals according to both the US News & World Report and HealthGrades, was compared with all other US hospitals, adjusting for differences in patient factors and surgical acuity. Risk-adjusted mortality rates between best hospitals and all other hospitals was compared after controlling for differences in hospital volume.

Results  Risk-adjusted mortality was significantly lower in US News & World Report best hospitals for all 3 procedures: pancreatectomy (odds ratio [OR], 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 0.55-0.86). Risk-adjusted mortality was significantly lower in HealthGrades best hospitals for colectomy (OR, 0.79; 95% CI, 0.65-0.95). However, after accounting for hospital volume, risk-adjusted mortality was only significantly lower at the US News & World Report best hospitals for colectomy (OR, 0.79; 95% CI, 0.62-0.99) and was not significantly lower at HealthGrades best hospitals for any of the 3 oncologic procedures.

Conclusions  Publicly marketed hospital rating systems of surgical quality such as the US News & World Report “America's Best Hospitals” and HealthGrades “Best Hospitals” may identify high-quality hospitals for some oncologic surgeries. However, these ratings fail to identify other high-volume hospitals of equal quality.

Figures in this Article

The US News & World Report “America's Best Hospitals” and the HealthGrades “Best Hospitals” are two of the most recognized publicly available hospital ratings. The US News & World Report “America's Best Hospitals” is an annual rating of the “fifty best” hospitals across 16 medical and surgical specialties including the 50 best cancer hospitals. HealthGrades markets hospital report cards to health care consumers on the internet, reporting the “quality” of health care of 5000 American hospitals.1

Both the US News & World Report and HealthGrades hospital quality ratings receive significant publicity. Hospitals frequently use these ratings in advertising, increasing the influence of the ratings in public perception of hospital quality.2 The US News & World Report is widely available, boasting a paper circulation of more than 2 million readers and an online readership with free access to all reports free of charge. Similarly, HealthGrades reports more than 5 million visits monthly to it's consumer Web site and markets itself as the nation's “leading healthcare ratings organization.”1

Although these quality ratings are marketed as a consumer aid for choosing hospitals, it is unclear whether these ratings appropriately identify high-quality hospitals. Previous studies have shown that competing hospital quality ratings frequently provide inconsistent results.3 No previous studies have addressed the question of whether the “best hospitals” have better cancer surgery outcomes than other hospitals in the United States.

We sought to determine independently whether these two quality ratings truly identify the best hospitals for cancer surgery. We used the 2005-2006 MedPar data set to compare the 30-day mortality in the best hospitals and all other hospitals for 3 cancer surgery procedures.

US NEWS & WORLD REPORT “AMERICA'S BEST HOSPITALS”

The 50 top-ranked cancer hospitals were determined from the 2008 publication of the US News & World Report “America's Best Hospitals.”4 These hospitals are determined by a combination of 3 equally weighted measures: hospital infrastructure, hospital reputation among subspecialists, and 30-day mortality rate. Only hospitals that are either a teaching hospital or have a minimum number of advanced technologies are considered eligible for the ratings. In addition, hospitals must meet a minimum discharge and surgical volume threshold for 12 subspecialties (cancer; ear, nose, and throat; endocrinology; gastrointestinal disorders; geriatric care; gynecology; heart and heart surgery; kidney disease; neurology and neurosurgery; orthopedics; respiratory disorders; and urology). The US News & World Report evaluates hospital infrastructure using measures from the American Hospital Association annual survey, including hospital teaching status, Nurse Magnet status, staff-to-bed ratios, hospital volume, and the presence of advanced technologies. The most subjective portion of the rating system, reputation of the hospitals among subspecialists, is based on a survey sent out to fewer than 100 members of each subspecialty to rank the 5 best hospitals. Risk-adjusted mortality rates for each hospital are calculated using Medicare claims data from the previous year. Using a composite score of these 3 measures, hospitals are ranked and the top 50 hospitals for each specialty are published.

HEALTHGRADES “BEST HOSPITALS”

The HealthGrades “America's 50 Best Hospitals” were collected directly from the HealthGrades Web site in October 2008.1In contrast to the US News & World Report ratings, the HealthGrades ratings are based solely on Medicare data. HealthGrades generates hospital ratings using a proprietary method, comparing predicted with observed mortality rates for all hospitals. The predicted 30-day mortality rate is derived from Medicare Part A billing data (MedPar 2005-2006). These predicted mortality rates were then compared with the observed mortality rates for each hospital, generating an observed to expected ratio. The hospitals are then rated based on a ratio of observed to expected mortality rate for 27 procedures and diagnoses. The 50 hospitals with the best observed to expected ratios are then included in the final list available on the HealthGrades Web site.

STUDY SAMPLE

All patients who underwent the 3 cancer operations, pancreatectomy, esophagectomy, and colectomy, were identified from the MedPar database from 2005 through 2006 using International Statistical Classification of Diseases, 10th Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes (n = 82 724). A total of 6455 patients underwent surgery in the 50 US News & World Report “America's Best Cancer Hospitals,” compared with 76 369 patients in the remaining 4445 hospitals nationwide. A total of 3452 patients from the 50 HealthGrades “Best Hospitals” were compared with 79 112 patients from the remaining 4445 hospitals nationwide. A total of 639 hospitals that do not participate in the American Hospital Association Annual Survey were not included in the analysis. All of the best hospitals from the US News & World Report “America's Best Cancer Hospitals” and the HealthGrades “Best Hospitals” participate in the American Hospital Association Annual Survey.

STATISTICAL ANALYSIS

Given the lack of agreement between the two quality ratings, the 50 best hospitals in each rating were compared with all other hospitals separately for each individual procedure. Membership in the Council of Teaching Hospitals was obtained from the 2003 American Hospital Association Annual Survey. Patient age, race, comorbidities, surgical acuity, and hospital characteristics were compared between best hospitals and all other hospitals using χ2 statistics, t tests, and analyses of variance.

Using all patients who underwent each of these 3 operations, we generated expected mortality rates using logistic regression models, adjusted for comorbidity burden, patient demographics, and emergent operation. Our independent variable was whether patients received care in a best hospital or all other hospitals. Random-effects logistic regression was used to provide robust adjustment for the effect of hospital clustering.

Next, to assess the relative effect of procedural volume on mortality in best hospitals, we created quintiles of procedure-specific hospital volume. Using random-effects logistic regression, we compared mortality following each of the 3 operations between best hospitals and all other hospitals, adjusting for patient covariates and hospital volume quintile. Using the relative attenuation of the odds ratio (OR), we estimated the relative effect of hospital volume on mortality in the best hospitals compared with all other hospitals.5

Comparing the US News & World Report “America's Best Cancer Hospitals” and the HealthGrades “Best Hospitals,” only 2 hospitals are considered best hospitals in both ratings.

Patient characteristics were compared between the 50 best hospitals in each rating and all other hospitals for each individual operation and each rating system (Table 1). The patients treated in the best hospitals were significantly different from the patients in all other hospitals. Across all procedures, US News & World Report “America's Best Hospitals” were less likely to operate emergently or urgently. HealthGrades “Best Hospitals” treated fewer black patients and were less likely to operate emergently or urgently. Patient comorbidity burden did not significantly differ between best hospitals from either rating system and all other hospitals.

Table Graphic Jump LocationTable 1. Patient Characteristics of US News & World Report “America's Best Cancer Hospitals,” HealthGrades “Best Hospitals,” and All Other Hospitals (2005-2006)

The characteristics of hospitals in each rating differed significantly from all other hospitals (Table2). US News & World Report “America's Best Cancer Hospitals” were significantly less likely to be very-low-volume hospitals for all 3 operations. “America's Best Cancer Hospitals” were also more likely to be teaching hospitals as compared with all other hospitals. Similarly, HealthGrades “Best Hospitals” were less likely to be very-low-volume for any of the cancer operations. HealthGrades “Best Hospitals” were also more likely to be teaching hospitals as compared with all other hospitals.

Table Graphic Jump LocationTable 2. Hospital Characteristics of US News & World Report “America's Best Cancer Hospitals,” HealthGrades “Best Hospitals,” and All Other Hospitals (2005-2006)

Adjusting for patient characteristics and surgical acuity, 30-day mortality rates were lower in US News & World Report “America's Best Cancer Hospitals” for all 3 operations: pancreatectomy (OR, 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 0.55-0.86). In contrast, risk-adjusted mortality was significantly lower in HealthGrades “Best Hospitals” following 1 of 3 procedures: colectomy (OR, 0.79; 95% CI, 0.65-0.95). The risk-adjusted mortality rates of the US News & World Report and HealthGrades rated hospitals and all other hospitals are shown in Figure 1 and Figure 2, respectively.

Place holder to copy figure label and caption
Figure 1.

Risk-adjusted mortality rates following cancer surgery in HealthGrades “Best Hospitals” and all other hospitals (2005-2006).

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Risk-adjusted mortality rates following cancer surgery in US News & World Report “America's Best Cancer Hospitals” and all other hospitals (2005-2006).

Graphic Jump Location

After accounting for hospital volume, US News & World Report “America's Best Cancer Hospitals” 30-day mortality rates were significantly lower for only 1 of 3 procedures: colectomy (OR, 0.69; 95% CI, 0.62-0.99). In comparison, accounting for hospital volume, mortality rates did not differ significantly between HealthGrades “Best Hospitals” and all other hospitals for any of the 3 procedures (Table2).

Using the relative attenuation of the OR to determine how much of the differences in mortality can be explained by volume, a significant proportion of the differences in mortality between “Best Hospitals” and all other hospitals in both rating systems can be attributed to hospital volume (Table 3). As much as 71% of the observed differences in mortality following pancreatectomy between US News & World Report “Best Hospitals” and other hospitals is owing to differences in hospital volume.

Table Graphic Jump LocationTable 3. Odds Ratios of 30-Day Mortality Rate Following Cancer Surgery for HealthGrades “Best Hospitals” and US News & World Report “America's Best Hospitals” Compared With All Other Hospitals (2005-2006)

Current consumer-marketed ratings of hospital quality may identify hospitals with lower surgical mortality rates for 3 cancer surgery procedures. “America's Best Cancer Hospitals” and HealthGrades “Best Hospitals” had significantly lower risk-adjusted mortality rates for all 3 and 1 of 3 operations, respectively. Previous studies of hospital ratings have shown that both HealthGrades and US News & World Report best hospitals have lower mortality rates for patients admitted with acute myocardial infarction and some cardiovascular procedures.68 Interestingly, these differences in mortality rates did not exist for all operations in both rating systems. Of note, HealthGrades rating system did not identify high-quality hospitals for patients who underwent high-risk cancer surgery, pancreatectomy, and esophagectomy.

Although these ratings systems may identify hospitals with lower-than-average mortality, a significant amount of the differences in surgical mortality are attributable to differences in procedural volume. In this analysis, procedural volume completely explained the differences in mortality for pancreatectomy and esophagectomy. These differences in procedural volume have been previously reported as important predictors of risk-adjusted mortality, especially for less-common, high-morbidity surgical procedures such as esophagectomy and pancreatectomy.9 Similarly, when comparing hospitals performing cardiovascular procedures, volume has a significant effect on quality ratings.8 Although both the US News & World Report and the HealthGrades quality ratings rely on risk-adjusted mortality rates, neither rating controls for differences in procedural volume. Our findings show that both ratings fail to identify equally well-performing hospitals of similar volume. Mortality rates between best hospitals and hospitals of similar volume were not significantly different.

This study has important implications for patients trying to choose safe hospitals for oncologic surgery. While we found a mortality rate benefit to choosing these highly rated hospitals, patients may equally benefit from going to a similarly high-volume hospital closer to their home. Pervious study by Birkmeyer and colleagues has shown that most patients live close to a high-volume hospital. In an analysis of patients who underwent pancreatectomy or esophagectomy, more than one-quarter of patients lived closest to a high-volume hospital and nearly 75% of patients would travel fewer than 30 minutes to reach a high-volume hospital.10 Because these ratings list only a selected number of best hospitals, these consumer aids may dissuade patients from seeking care at closer high-volume, equal-quality hospitals.

This study has several limitations. First, the study assessed the validity of these hospital ratings using the outcome of 30-day mortality. Mortality represents just one domain of surgical quality; other outcomes are equally important such as readmission rates, complications, and cost. In addition, in the Donabedian structure-process-outcomes model of quality,11 structural and process measures may play important roles in measurement of surgical quality. Unfortunately, current databases do not include robust measures of these domains. Although this analysis was limited to mortality rate, this is a key measure used in the methodologies of both hospital ratings. Second, our study relies on Medicare claims data, limiting our study population to patients older than 65 years. However, of elderly persons represent most patients undergoing these procedures, and elderly patients have the highest risk associated with these surgeries. In addition, administrative data lack robust clinical detail to entirely account for differences in case-mix and patient illness severity that may confound the differences in mortality rate between hospitals. We have controlled for available patient risk factors including demographics, comorbidities, and acuity of admission/surgery. Although differences in patient factors may exist between hospitals, there is a lack of clinical data to adequately account for such differences. More robust patient and hospital data will likely improve our ability to accurately rate the quality of hospitals.

This study has significant policy implications. As politicians, patients, and payers continue to demand accessible quality measurements of medical care, accurate hospital quality ratings must be developed. Reliable and accessible methods of rating hospital quality will require a collaboration of not only independent ratings organizations, but hospitals and payers as well. Unfortunately, current quality ratings may confuse patients and dissuade them from receiving care in high-quality hospitals close to home. Future methods of rating hospital quality should consider not only quality but access to those quality hospitals.

Both US News & World Report and HealthGrades may identify high-quality hospitals for receiving cancer surgery. However, these ratings fail to identify other high-volume hospitals with equivalent outcomes. Further research into the measurement of surgical quality will be necessary to help aid patient decision making.

Correspondence: Nicholas H. Osborne, MD, 6312 Medical Science Building I, 1150 W Medical Center Dr, SPC5604, Ann Arbor, MI 48109-5604 (nichosbo@umich.edu).

Accepted for Publication: April 9, 2010.

Author Contributions:Study concept and design: Osborne and Dimick. Acquisition of data: Osborne, Ghaferi, Nicholas, and Dimick. Analysis and interpretation of data: Osborne, Ghaferi, Nicholas, and Dimick. Drafting of the manuscript: Osborne, Ghaferi, Nicholas, and Dimick. Critical revision of the manuscript for important intellectual content: Osborne, Ghaferi, Nicholas, and Dimick. Statistical analysis: Osborne, Ghaferi, Nicholas, and Dimick. Obtained funding: Osborne. Study supervision: Dimick.

Financial Disclosure: None reported.

 HealthGrades Web site. http://www.healthgrades.com/. Accessed November 24, 2008
Larson  RJSchwartz  LMWoloshin  SWelch  HG Advertising by academic medical centers. Arch Intern Med 2005;165 (6) 645- 651
PubMed Link to Article
Leonardi  MJ McGory  MLKo  CY Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg 2007;142 (9) 863- 869
PubMed Link to Article
 US News & World Report America's Best Hospitals 2008. US News & World Report Web site. http://health.usnews.com/best-hospitals. Accessed March 30, 2009
Finlayson  EVGoodney  PPBirkmeyer  JD Hospital volume and operative mortality in cancer surgery: a national study. Arch Surg 2003;138 (7) 721- 726
PubMed Link to Article
Chen  JRadford  MJWang  YMarciniak  TAKrumholz  HM Do “America's Best Hospitals” perform better for acute myocardial infarction? N Engl J Med 1999;340 (4) 286- 292
PubMed Link to Article
Krumholz  HMRathore  SSChen  JWang  YRadford  MJ Evaluation of a consumer-oriented internet health care report card: the risk of quality ratings based on mortality data. JAMA 2002;287 (10) 1277- 1287
PubMed Link to Article
Osborne  NHNicholas  LHGhaferi  AAUpchurch  GR  JrDimick  JB Do popular media and internet-based hospital quality ratings identify hospitals with better cardiovascular surgery outcomes? J Am Coll Surg 2010;210 (1) 87- 92
PubMed Link to Article
Birkmeyer  JDDimick  JBStaiger  DO Operative mortality and procedure volume as predictors of subsequent hospital performance. Ann Surg 2006;243 (3) 411- 417
PubMed Link to Article
Birkmeyer  JDSiewers  AEMarth  NJGoodman  DC Regionalization of high-risk surgery and implications for patient travel times. JAMA 2003;290 (20) 2703- 2708
PubMed Link to Article
Donabedian  A The quality of medical care. Science 1978;200 (4344) 856- 864
PubMed Link to Article

Figures

Place holder to copy figure label and caption
Figure 1.

Risk-adjusted mortality rates following cancer surgery in HealthGrades “Best Hospitals” and all other hospitals (2005-2006).

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Risk-adjusted mortality rates following cancer surgery in US News & World Report “America's Best Cancer Hospitals” and all other hospitals (2005-2006).

Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1. Patient Characteristics of US News & World Report “America's Best Cancer Hospitals,” HealthGrades “Best Hospitals,” and All Other Hospitals (2005-2006)
Table Graphic Jump LocationTable 2. Hospital Characteristics of US News & World Report “America's Best Cancer Hospitals,” HealthGrades “Best Hospitals,” and All Other Hospitals (2005-2006)
Table Graphic Jump LocationTable 3. Odds Ratios of 30-Day Mortality Rate Following Cancer Surgery for HealthGrades “Best Hospitals” and US News & World Report “America's Best Hospitals” Compared With All Other Hospitals (2005-2006)

References

 HealthGrades Web site. http://www.healthgrades.com/. Accessed November 24, 2008
Larson  RJSchwartz  LMWoloshin  SWelch  HG Advertising by academic medical centers. Arch Intern Med 2005;165 (6) 645- 651
PubMed Link to Article
Leonardi  MJ McGory  MLKo  CY Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg 2007;142 (9) 863- 869
PubMed Link to Article
 US News & World Report America's Best Hospitals 2008. US News & World Report Web site. http://health.usnews.com/best-hospitals. Accessed March 30, 2009
Finlayson  EVGoodney  PPBirkmeyer  JD Hospital volume and operative mortality in cancer surgery: a national study. Arch Surg 2003;138 (7) 721- 726
PubMed Link to Article
Chen  JRadford  MJWang  YMarciniak  TAKrumholz  HM Do “America's Best Hospitals” perform better for acute myocardial infarction? N Engl J Med 1999;340 (4) 286- 292
PubMed Link to Article
Krumholz  HMRathore  SSChen  JWang  YRadford  MJ Evaluation of a consumer-oriented internet health care report card: the risk of quality ratings based on mortality data. JAMA 2002;287 (10) 1277- 1287
PubMed Link to Article
Osborne  NHNicholas  LHGhaferi  AAUpchurch  GR  JrDimick  JB Do popular media and internet-based hospital quality ratings identify hospitals with better cardiovascular surgery outcomes? J Am Coll Surg 2010;210 (1) 87- 92
PubMed Link to Article
Birkmeyer  JDDimick  JBStaiger  DO Operative mortality and procedure volume as predictors of subsequent hospital performance. Ann Surg 2006;243 (3) 411- 417
PubMed Link to Article
Birkmeyer  JDSiewers  AEMarth  NJGoodman  DC Regionalization of high-risk surgery and implications for patient travel times. JAMA 2003;290 (20) 2703- 2708
PubMed Link to Article
Donabedian  A The quality of medical care. Science 1978;200 (4344) 856- 864
PubMed Link to Article

Correspondence

CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 4

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections
PubMed Articles