Predicting In-Hospital Mortality in Patients Undergoing Complex Gastrointestinal Surgery:  Determining the Optimal Risk Adjustment Method FREE

Administrative databases are being increasingly used to evaluate outcomes following multiple surgical procedures.^1- 2 These databases possess a comprehensive set of comparable information, including patient demographics, primary and secondary diagnoses, procedures, and details regarding hospital stay. These data are readily available in electronic form. The databases offerrelatively easy access to a large sample size. Through comparison with clinical databases, some administrative databases have been validated for researching comorbidity-influenced outcomes in some patient populations. This was done for multiple diagnoses, codes, and databases.^3- 18 However, administrative databases are not designed for research purposes and consequently their limitations must be considered. For example, studies of cardiac surgery using administrative databases have observed coding inaccuracies and a failure to record potentially important comorbidities.^19- 20 Because outcomes in longitudinal studies can be significantly altered by comorbid conditions, a number of metrics to assess burden of disease and comorbid health conditions have been derived.^1,21- 26

To our knowledge, these comorbidity indexes have not yet been validated specifically for patients who have undergone major abdominal surgical resections. Consequently, it is not known how accurately these various algorithms can adjust in these populations.

The objective of our study was to assess and compare the 4 most common algorithms with regard to their ability to predict in-hospital mortality in patients undergoing major gastric, hepatic, and pancreatic resections. Secondarily, we compared the association between scores and total hospital charges as well as length of stay (LOS).

All data were extracted from the Healthcare Cost and Utilization Project Nationwide Inpatient Sample database for 2002 to 2007.²⁷ The Nationwide Inpatient Sample is the largest all-payer database of national hospital discharges (about 8 million patients per annum) maintained by the Agency for Healthcare Research and Quality. It represents a 20% stratified sample of nonfederal acute care hospitals in the United States including community, general, and academic centers, but not long-term care facilities. Each discharge abstract includes a patient identifier, demographic data, hospital transfer status, admission type (emergency, urgent, or elective), primary and secondary diagnoses (up to 15), procedures (up to 15), health insurance status, total hospital charges, and LOS. Since each record in the Nationwide Inpatient Sample is for a single hospitalization (ie, not a person), there could be multiple records for an individual if that individual had several hospitalizations. Nationwide Inpatient Sample data compare favorably with the National Hospital Discharge Survey, supporting the validity of this database.²⁸

International Classification of Diseases, Ninth Version, Clinical Modification (ICD-9-CM) procedures codes (43.4-43.9, 50.2-50.5, 52.2, and 52.5-52.8) were used to identify adult patients (≥18 years) hospitalized with a primary or secondary procedure of gastric, hepatic, or pancreatic resection between 2002 and 2007. Because our primary outcome was to measure in-hospital mortality, we excluded patients with missing mortality data or those transferred to other institutions. Our secondary outcomes were LOS and total hospital charges adjusted for inflation to 2007 dollars using the US Consumer Price Index for medical care.²⁹

In our study, we assessed 4 commonly used risk adjustment measures: the Deyo adaptation of the Charlson comorbidity algorithm,²⁵ the Elixhauser comorbidity algorithm,²⁶ the Disease Staging algorithm (Thomson Medstat Inc, Ann Arbor, Michigan),³⁰ and the All Patient Refined Diagnosis-Related Groups (APR-DRGs) algorithm (3M Health Information Systems, Wallingford, Connecticut).³¹ These methods use information obtained in the hospital discharge abstract including demographics, diagnosis, and procedure codes. Both the Charlson/Deyo and Elixhauser algorithms are nonproprietary, can be routinely applied to administrative data using widely available computer algorithms, and identify 17 and 30 categories of comorbidities, respectively, using ICD-9-CM diagnosis codes.^25- 26

Both the Charlson/Deyo and Elixhauser algorithms contain a variable for liver disease as a comorbidity. Because we studied hepatic resections, we excluded these variables from our hepatic resection primary analysis.

We also examined the Disease Staging and APR-DRGs algorithms, which are proprietary risk adjustment methods with logic unavailable for outside scrutiny. Results of these algorithms were provided at no cost by the Agency for Healthcare Research and Quality. In Disease Staging, severity is defined as the likelihood of death or organ failure resulting from disease progression and independent of the treatment process.³⁰ Disease progression is measured using 4 stages (with additional substages) of increasing complexity (stage 1 = no complications or problems of minimal severity; stage 2 = problems limited to a single organ or system, significantly increased risk of complications; stage 3 = multiple site involvement, generalized site involvement, poor prognosis; and stage 4 = death). Disease Staging uses age, sex, admission and discharge status, and diagnoses to generate a predictive scale for mortality.³⁰

The APR-DRGs algorithm is a clinical model that expands on the basic DRG structure designed to group patients into approximately 500 categories with similar clinical features and resource use.³¹ The APR-DRGs algorithm includes the addition of 4 subclasses to each DRG category to identify minor, moderate, major, or extreme risk of mortality, defined as the extent of physiologic decompensation or organ system loss of function. The process of classifying a patient consists of assessing the level of each secondary diagnosis; determining the base subclass for the patient based on all of his or her secondary diagnoses; and, finally, determining the final subclass of the patient by incorporating the impact of the principal diagnosis, age, procedures, and combinations of categories of secondary diagnoses.³¹

Logistic regression analyses were used to assess the contributions of the individual Charlson/Deyo and Elixhauser comorbidities to predicted in-hospital mortality. We also categorized these comorbidities (as 0, 1, 2, or ≥3 present), as commonly performed in the literature,³² to determine any impact on risk adjustment compared with the complete list of comorbidities. Similar models were generated for the Disease Staging Mortality Prediction Scale and the APR-DRGs risk of mortality subclass. Each regression model contained a set of independent variables adjusting for sociodemographic and clinical differences between patients including age, sex, health insurance (private, Medicaid, Medicare, self-pay, and other/unknown), race (white, African American, Hispanic, Asian/Pacific Islander, and other/unknown), emergency admission (emergency vs urgent and elective combined), and transfer status (transferred in vs not transferred in). We also adjusted for hospital characteristics including location and teaching status (rural, urban nonteaching, and urban teaching) and region of the United States (Northeast, Midwest, South, and West).

Linear regression models also determined the impact of the Charlson/Deyo and Elixhauser comorbidities on LOS and hospital charges, which were logarithmically transformed because of their skewed distributions. We present exponentiation of the coefficients from these models to reflect the percentage of change in resource use from having a particular comorbidity independent of other patient characteristics.

To assess the performance of the risk adjustment methods for predicting in-hospital mortality, we calculated the C statistic, an approximation of the area under the receiver operating characteristic curve and a measure of model discrimination.³³ The C statistic ranges from 0.5 to 1.0, with 1.0 indicating perfect discrimination and 0.5 indicating no ability to discriminate. The C statistic is commonly used to compare risk adjustment methods.^34- 37 Ninety-five percent confidence intervals (CIs) and comparisons between C statistics were calculated using the method of DeLong et al.³⁸ Models were analyzed for each cohort of patients (gastric, hepatic, and pancreatic) separately.

Using sensitivity analyses, we analyzed subgroups of each cohort. Those subgroups were (1) limited to 1 type of surgical resection (excluding resections of more than 1 anatomic position, eg, having hepatic and pancreatic resection in the same admission, as those patients would have special characteristics); (2) admissions where the resection procedure of interest was the primary procedure; and (3) admissions where the resection procedure of interest was a secondary procedure.

For hepatic and pancreatic cohorts, we excluded transplant patients from our primary analysis (patients with ICD-9-CM procedure code 50.5 for liver transplant and 52.8 for pancreatic transplant). In supplemental analyses, we assessed the performance of the risk adjustment models in cohorts including those transplant patients.

These supplemental analyses allowed us to evaluate the utility of these methods in a broad spectrum of patients with the surgical resections of interest.

Based on our multivariate models, we computed deciles of predicted probabilities of death for each risk adjustment method. As described by Iezzoni et al,³⁴ deciles were arranged in 10 × 10 tables for each of the 6 pairwise comparisons between methods. Our goal was to illustrate the magnitude of the differences in predicted probabilities of death between risk adjustment methods. Based on differences in the average probabilities of death between deciles, we considered a difference of 3 or more deciles between measures as clinically important (median 5.0% difference in mortality rates). For each comparison, we calculated the proportion of patients with “similar” and “different” predicted likelihoods of death (probabilities calculated by both methods ≤2 vs ≥3 deciles apart).³⁴

All statistical analyses were performed using SAS version 9.1.3 software (SAS Institute Inc, Cary, North Carolina).

In total, 40 746 gastric resections, 15 279 hepatic resections, and 13 236 pancreatic resections met our inclusion criteria. The studied procedure was the primary diagnosis in 69% among gastric resections, 85% among hepatic resections, and 88% among pancreatic resections. The median age of the cohorts ranged from 58 to 63 years, 50% to 54% of patients were female, 52% to 56% were white, and 37% to 52% had private health insurance. Overall, in-hospital mortality was 5.9% in gastric and pancreatic resections and 4.4% in hepatic resections; median LOS ranged from 5 days (interquartile range [IQR], 3-8 days) for hepatic resections to 10 days (IQR, 7-17 days) for pancreatic resections, and median total hospital charges were lowest for gastric resections ($37 770 [IQR, $21 332-$71 644]) and highest for pancreatic resections ($64 522 [IQR, $41 592-$109 980]) (Table 1).

Table 2 and Table 3 provide the prevalence and independent effects of each comorbidity on in-hospital mortality, LOS, and hospital charges in each cohort.

In the gastric resection cohort, any malignancy (29.7%), peptic ulcer (22.5%), diabetes mellitus (18.6%), and chronic pulmonary disease (18.1%) were the most prevalent comorbidities. Neurological disorder (odds ratio [OR], 4.99; 95% CI, 3.23-7.71), peptic ulcer (OR, 3.87; 95% CI, 3.52-4.25), moderate and severe liver disease (OR, 3.49; 95% CI, 2.85-4.28), renal disease (OR, 2.89; 95% CI, 2.52-3.31), and congestive heart failure (OR, 2.10; 95% CI, 1.88-2.35) were associated with at least doubling of the adjusted odds of in-hospital mortality when using the Charlson/Deyo algorithm. In contrast, hypertension (45.5%), fluid and electrolyte disorders (20.7%), diabetes mellitus (17.8%), chronic pulmonary disorders (17.2%), and deficiency anemia (16.2%) were the more prevalent comorbidities in the Elixhauser algorithm. Coagulopathy (OR, 4.47; 95% CI, 3.89-5.13), renal failure (OR, 3.38; 95% CI, 2.88-3.96), weight loss (OR, 2.39; 95% CI, 2.09-2.74), neurological disorders (paralysis) (OR, 2.10; 95% CI, 1.43-3.09), and cardiac arrhythmias (OR, 1.99; 95% CI, 1.78-2.22) were associated with the highest odds of in-hospital mortality when using the Elixhauser algorithm.

In the hepatic resection cohort, any malignancy (40.1%) or metastatic tumor (48.8%) and diabetes mellitus (14.0%) were the most prevalent comorbidities. Cerebrovascular disease (OR, 10.0; 95% CI, 6.47-15.44), neurological disorder (OR, 6.56; 95% CI, 2.43-17.67), congestive heart failure (OR, 2.14; 95% CI, 1.49-3.08), myocardial infarction (OR, 2.09; 95% CI, 1.44-3.05), and renal disease (OR, 2.03; 95% CI, 1.33-3.10) were each associated with more than a 2-fold increased risk of in-hospital mortality in the Charlson/Deyo algorithm. Conversely, hypertension (35.9%), metastatic cancer (14.8%), and diabetes (13.8%) were the more prevalent comorbidities in the Elixhauser algorithm. Paralysis (OR, 4.65; 95% CI, 1.99-10.84), coagulopathy (OR, 3.99; 95% CI, 3.18-5.00), renal failure (OR, 2.86; 95% CI, 1.81-4.51), fluid and electrolyte disorders (OR, 2.98; 95% CI, 2.46-3.60), neurological diseases other than paralysis (OR, 2.59; 95% CI, 1.66-4.02), and cardiac arrhythmias (OR, 2.00; 95% CI, 1.55-2.58) were associated with the highest odds of in-hospital mortality in the Elixhauser algorithm.

In the pancreatic resection cohort, any malignancy (51.9%) or metastatic tumor (25.8%) and diabetes mellitus (20.6%) were the most prevalent comorbidities. AIDS (OR, 5.28; 95% CI, 1.59-17.56), renal failure (OR, 3.39; 95% CI, 2.46-4.69), liver disease (OR, 3.38; 95% CI, 2.05-5.58), congestive heart failure (OR, 3.13; 95% CI, 2.43-4.02), and cerebrovascular disease (OR, 2.21; 95% CI, 1.29-3.77) had the highest association with in-hospital mortality in the Charlson/Deyo algorithm. In contrast, hypertension (39.1%), metastatic cancer (24.0%), fluid and electrolyte disorders (20.6%), and diabetes (19.8%) were the more prevalent comorbidities in the Elixhauser algorithm.

Coagulopathy (OR, 6.04; 95% CI 4.82-7.57), renal failure (OR, 4.38; 95% CI 3.00-6.38), fluid and electrolyte disorders (OR, 2.34; 95% CI, 1.98-2.76), and congestive heart failure (OR, 2.67; 95% CI 2.01-3.55) were the most important independent predictors of the in-hospital mortality in the latter algorithm.

As shown in eTable 1, mortality, together with LOS and hospital charges, increased with the number of comorbidities according to both the Elixhauser and Charlson/Deyo algorithms in each of the 3 cohorts. The majority (62%-68%) of patients in the 3 cohorts also had 3 or more Elixhauser or Charlson/Deyo comorbidities.

For the gastric resection cohort, LOS and charges increased proportionally with the number of comorbidities. The mortality rate ranged from 1.6% to 2% for patients without comorbidities to 8.7% to 10.3% for patients with 3 or more comorbidities. Similar trends were observed in the pancreatic resection cohorts.

Interestingly, for hepatic resections, there was no significant clinical difference for LOS and charges. However, mortality was inversely proportional to the number of comorbidities in the Charlson/Deyo algorithm.

Additional analyses were performed to assess the relationships between Disease Staging (the Predicted Mortality Scale score quartile) and the 4 APR-DRGs risk of mortality subclasses with clinical outcomes (eTable 2). As observed in the nonproprietary algorithms, increased mortality risk or subclass score (Disease Staging and APR-DRGs, respectively) was associated with older age, longer LOS, higher hospital charges, and mortality. For the 3 cohorts, those outcomes were more pronounced among higher values of subclass scores. For example, mortality rates according to APR-DRGs ranged from 0.2% to 0.4% for the lowest quartile to 43.2% to 55.2% for the highest quartile. Using the Disease Staging scale, mortality rates ranged from 0.2% to 1.3% in the lowest quartile to 13.0% to 20.8% in the highest quartile.

Table 4 provides C statistics for in-hospital mortality for each of the risk adjustment methods in the different resection cohorts. All risk-adjustment models were adjusted for age, sex, race, insurer, admission status, and hospital characteristics.

The C statistics were 0.792 (95% CI, 0.783-0.801), 0.847 (95% CI, 0.839-0.855), 0.903 (95% CI, 0.896-0.909), and 0.941 (95% CI, 0.937-0.946) for the Charlson/Deyo algorithm, Elixhauser algorithm, Disease Staging, and APR-DRGs risk of mortality subclasses, respectively. Similar findings were observed in patients whether gastric resection was their primary or secondary indication of admission. In the 3 groups of patients, the highest accuracy was observed in the APR-DRGs scale. The Elixhauser model was more predictive than the Charlson/Deyo algorithm (P < .001 for all comparisons).

The C statistics were 0.757 (95% CI, 0.736-0.777), 0.810 (95% CI, 0.792-0.828), 0.857 (95% CI, 0.841-0.872), and 0.937 (95% CI, 0.927-0.947) for the Charlson/Deyo algorithm, Elixhauser algorithm, Disease Staging, and APR-DRGs risk of mortality subclasses, respectively. The same trend of the C statistics was observed whether hepatic resection was the primary or secondary indication for admission or in transplant patients. The APR-DRGs algorithm was the most accurate scale and as compared with the Charlson/Deyo algorithm, the Elixhauser model was more accurate in all subgroups.

For the whole cohort, the C statistics were 0.741 (95% CI, 0.722-0.759), 0.811 (95% CI, 0.794-0.828), 0.868 (95% CI, 0.855-0.881), and 0.922 (95% CI, 0.911-0.932) for the Charlson/Deyo algorithm, Elixhauser algorithm, Disease Staging, and APR-DRGs risk of mortality subclasses, respectively. Similar trends were observed in subgroup analyses (admissions with primary or secondary indications or in transplant patients). As observed in the other 2 types of resections, the APR-DRGs scale was the most accurate scale, and the Elixhauser index had higher accuracy than the Charlson/Deyo algorithm.

Table 5 shows the percentage of patients with similar, and very different, predicted probabilities of death for pairs of risk adjustment methods, as well as the percentage of patients who died within each group. Agreement between methods relied on the type of measures (proprietary vs nonproprietary). In gastric resections, agreement between the Charlson/Deyo algorithm and both Disease Staging and APR-DRGs was 72.9% and 70.3%, while for the Elixhauser algorithm, agreement with the latter 2 measures was 66.5% and 66.4%, respectively. Agreement between the Elixhauser and Charlson/Deyo algorithms was 69.6% while the highest agreement was between Disease Staging and APR-DRGs (77.0%). The agreement between the Charlson/Deyo and Elixhauser algorithms was higher than the agreement between either of them and proprietary measures in hepatic and pancreatic resections (63.9% vs 43.8%-59.9% and 77.1% vs 57.7%-69.5%, respectively).

Patients viewed as more sickly by the nonproprietary algorithms had a lower death rate than those using the proprietary measures. For example, among gastric resections, 15.9% of patients were viewed as “sicker” by the Elixhauser algorithm (compared with the APR-DRGs model); only 0.8% died. Similarly, of the 17.7% of patients viewed as more sickly by APR-DRGs, 5.6% died. Similar trends were observed in the other 2 cohorts.

It is important to assess prognosis for each individual patient prior to major surgical procedures, particularly if outcomes differ depending on patient comorbidities. In our study, we described the ability of the 4 most commonly used risk adjustment algorithms to predict in-hospital mortality in patients after gastric (40 000), hepatic (15 000), and pancreatic (13 000) resections. After assessment of the performance of each algorithm, we were able to validate each of the 4 methods for this patient population, as their overall C statistics for all 3 types of resections were more than 0.73. A C statistic approximating 0.75 is considered acceptable for discrimination and validation of methods for ongoing use.

When we compared the performance of each method, the proprietary comorbidity indexes (Disease Staging and APR-DRGs) outperformed the nonproprietary ones (Charlson/Deyo and Elixhauser). These results were generally consistent across subgroups. These proprietary methods may also include some complications of care, in contrast to the other methods that include only comorbidities. This issue likely biases any comparison in favor of the proprietary methods. Additionally, although these 2 scores have consistently outperformed other algorithms in multiple studies evaluating various primary diagnoses,^1,3- 4,20 their cost makes them less accessible to researchers with limited budgets. It is therefore important to consider the 2 widely available nonproprietary methods that are based on simple computer algorithms separately from the proprietary methods. We primarily wanted to scrutinize the nonproprietary scores, and all comparisons between these 2 groups were added for informative purposes.

Comparison of the 2 proprietary methods showed that APR-DRGs was more accurate than Disease Staging. Both methods had remarkably high C statistics (0.836-0.945) in all subgroups. In the comparison of nonproprietary methods, we demonstrate that even though both scores have been validated for use with these patient populations, the Elixhauser algorithm showed significantly higher accuracy among all subgroups.

In our study, we did not perform any clinical data abstraction from medical records and so we did not validate the quality of the administrative database comparing it with the “gold standard” for this patient population. In this article, we are therefore only able to compare the 4 comorbidity indexes in their relative performance.

In addition to the comparison of risk adjustment methods, we also described how single comorbidities influence outcome in this patient population. For all 3 types of resections, renal disease, congestive heart failure, coagulopathy, and fluid and electrolyte disorders were associated with at least a 2-fold increase in odds of in-hospital mortality, as well as moderate to severe liver disease in gastric and pancreatic resection cohorts (this comorbidity was removed from the adjusted model for hepatic resection). The fluid and electrolyte disturbances were also among the most prevalent diseases, occurring in 12% to 20% of patients. Another important predictor of increased mortality risk, LOS, and hospital charges was cerebrovascular disease, which displayed a large variance in ORs among the cohorts. In gastric and pancreatic resections, cerebrovascular disease was associated with ORs of 1.69 and 2.21, respectively, but patients with cerebrovascular disease undergoing hepatic resections experienced a 10-fold increase in their risk of dying. When we compared the Charlson/Deyo and Elixhauser indexes, the biggest discrepancy was peptic ulcer disease in patients who underwent gastric resection. This was identified as a comorbidity in 22.5% of patients in the Charlson/Deyo method (OR, 3.87) but only 1.2% of patients with the Elixhauser method (OR, 0.96). This difference is likely due to the exclusion of gastric bleeding in the Elixhauser method.

Other conditions were also associated with prolonged hospital stay yet decreased in-hospital mortality rates. As described by others,^33,39- 40 these findings likely represent an inverse effect of patient severity on coding of certain common but relatively unthreatening diseases. For example, a seriously ill patient with multiple serious diseases concurrent to a diagnosis of obesity is less likely to have it listed among his or her comorbidities in the discharge abstract compared with an otherwise healthy obese patient.

On comparison of the relative predicted probabilities of death, methods agreed in only 43.8% to 74.6% of scenarios. In all cohorts, patients viewed as sicker by the nonproprietary algorithms had lower rates of death compared with the evaluation of sick patients by the proprietary algorithms. A succinct rationale for this observation is currently unknown.

Our secondary end points, total hospital charges and length of hospital stay, also increased proportionally with the burden of comorbidities. This is an expected outcome as sicker patients could reasonably be expected to spend a longer time in care and incur more expenses while in the hospital.

In conclusion, we found that the selection of a particular method for comorbidity risk adjustment after gastric, hepatic, and pancreatic resections has a significant impact on analysis of in-hospital mortality. This is explained by the reality that different methods classify the same patients into different risk categories. As a result, provider-specific outcomes may be evaluated differently depending on the method selected. Finally, the importance of a particular comorbidity can also be misinterpreted because of inadequate risk adjustment.