Quality Assessment in High-Acuity Surgery:  Volume and Mortality Are Not Enough FREE

Variation in health care quality is not a new phenomenon. More than 30 years ago, Wennberg and Gittelsohn¹ demonstrated significant differences in resource utilization and expenditures in 13 hospital service areas in Vermont. They concluded that these differences affected the quality and cost of medical care delivered and suggested that causes of variation could explain the performance of any given health care system.

Today, quality improvement analyses have emerged and have contributed to selective regionalization of high-acuity surgery.^2- 8 Birkmeyer and others⁷ demonstrated that surgical outcomes are affected by operative volume. In a landmark study,⁷ the 30-day hospital mortality rate was correlated with hospital volume for 14 high-acuity operations, including pancreatectomy, esophagectomy, certain cardiovascular procedures, and others. Higher-volume hospitals had significantly lower operative mortality rates for these operations, although the relative importance of volume varied according to the type of procedure performed. For example, pancreatectomy demonstrated the most dramatic difference in mortality (4-fold) and carotid endarterectomy the least. The authors suggested that certain high-acuity operations should be performed at high-volume medical centers to optimize postoperative outcomes.

Although these studies have made important contributions to our understanding of the quality of surgical care, some researchers believe that other factors, beyond hospital volume, ultimately affect surgical performance. It is recognized that surgical mortality rates vary across high- and low-volume hospitals, but the impact of baseline patient risk on this process is poorly understood. Other critics of volume-outcome studies regard the mortality rate as a crude metric of surgical quality. Some question whether the demonstrated outcomes accurately reflect hospital quality or the proficiency of its surgeons. Still others believe that these sorts of analyses, often using national or regional administrative databases, may be inadequate for assessing specific differences in health care outcomes. We lack a standardized definition of “high volume” for any given procedure, which further intimates that quality may plateau when operative volume exceeds a certain threshold. Imprecise measures of quality and variations in patient morbidity risk preclude objective comparisons of surgical outcomes that are based on volume alone.^9- 15

To address these problems, surgical risk–adjusted scoring systems have been proposed and analyzed. Among them is POSSUM (Physiologic and Operative Severity Score for the Enumeration of Mortality and Morbidity), which was developed in the early 1990s to appraise and audit the quality of surgical care in institutions.¹⁶ This system adjusts for patients' preoperative risks and for the nature of the operation performed. Twelve physiologic and 6 operative variables are analyzed, and predictions for morbidity are calculated and assigned values from 0% to 100%. More recently, this scoring system has been applied, and validated in multiple complex operations, as an objective quality measure to assess differences between surgeons and to monitor their performance across time.^17- 20

In this study, we describe a versatile model that objectively evaluates and links variations in preoperative morbidity with actual clinical and economic outcomes for a high-acuity operation (pancreatectomy) in a high-volume medical center. Furthermore, we suggest that this model can reliably examine the underlying impact of patient acuity on health care outcomes and the contribution of various process improvement measures at the level of the practice or health care professional.

Two surgeons performed 296 consecutive pancreatic resections between October 18, 2001, and August 24, 2006, including 209 pancreaticoduodenectomies and 80 subtotal and 7 central pancreatectomies. Preoperative diagnoses included suspected periampullary or neuroendocrine tumors, intraductal papillary mucinous neoplasms, cysts, and pancreatitis. Final pathological examination revealed that most patients had malignancies (52%), primarily pancreatic ductal adenocarcinoma (n = 89) or other periampullary malignancies (n = 42). Other pathologic conditions encountered include cystic neoplasms (n = 57), pancreatitis (n = 50), neuroendocrine tumors (n = 17), and other various benign (n = 35) or malignant (n = 6) conditions. Patients undergoing pancreaticoduodenectomy most often had ductal adenocarcinoma (n = 77; 37%), whereas those subjected to distal pancreatectomy largely had cystic disease (n = 32; 40%). Central pancreatectomy was most frequently performed for intraductal papillary mucinous neoplasms (n = 3) or pancreatitis (n = 2).

Through the collaborative efforts of a dedicated multiservice and specialty team, a standardized care path for pancreatic resection was implemented at Beth Israel Deaconess Medical Center in February 2004. This pathway was designed to outline a standardized management approach for all patients. It provides a guide for preoperative planning, thromboembolic and antibiotic prophylaxis, perioperative pain management, and the removal of central venous catheters, nasogastric tubes, urinary catheters, and intra-abdominal drains. Furthermore, this clinical pathway standardizes patients' perioperative fluid resuscitation, alimentation, and diagnostic testing. Finally, the pathway incorporates several patient-centered initiatives and provides psychosocial counseling, geriatric consultation, input from medical pancreatologists, and early rehabilitation planning when indicated.

In accordance with guidelines for human subjects research, approval was obtained from the institutional review board of Beth Israel Deaconess Medical Center. Data on preoperative, intraoperative, and postoperative care were prospectively collected for each case and retrospectively reviewed. Preoperative variables include patient demographics (ie, age, sex, and comorbidities), presenting symptoms (eg, jaundice, weight loss, diarrhea, and pain), cardiac or respiratory symptoms, laboratory tests, previous imaging studies, and any other diagnostic or therapeutic studies performed (ie, endoscopic ductal stenting or sphincterotomy). Baseline laboratory values were obtained within 1 to 2 weeks of the operation and include white blood cell count, hemoglobin level, hematocrit level, serum sodium and potassium concentrations, and blood urea nitrogen level. Pulse, blood pressure, and electrocardiogram readings were also obtained preoperatively. Intraoperative variables include total operative time, blood loss, fluid resuscitation, blood transfusions, gland characteristics, surgical technique, and the use of drains, stents, somatostatin analogues, or adhesive sealants. Final disease pathology was determined after each case. Postoperative events and clinical outcomes were recorded by an independent reviewer and confirmed by the operating surgeon, including results of laboratory, imaging, and diagnostic studies; therapeutic interventions; requirements for nutritional support; incidence and type of complications; intensive care unit (ICU) utilization and duration; hospitalization duration; discharge disposition; hospital readmissions; repeated operations; and the 30-day hospital mortality rate. All economic data were collected and analyzed using Casemix TSI (Department of Health Care Quality, Harvard Medical School). Total hospital costs are defined as costs from the initial operation to hospital discharge, plus any costs accrued during hospital readmissions within 30 days postoperatively.

Clinical and economic outcomes for patients undergoing pancreatic resection in the pancreaticobiliary surgical practice at Beth Israel Deaconess Medical Center were examined. In addition to common indicators of quality—complications, hospitalization duration, and operative mortality rates—we explored the following clinical outcomes that have emerged as other measures of surgical quality: ICU utilization, blood transfusions, patient discharge disposition, hospital readmission, and repeated surgical intervention. Similarly, overall cost-efficiency was examined through analysis of total hospital costs accrued after surgical intervention. Table 1 defines these clinical and economic outcomes.

Beyond this traditional assessment, an additional approach was examined to determine the impact of preoperative morbidity risk on outcomes in a high-volume specialty center. Annual variations in patient acuity were directly correlated with observed surgical outcomes. Specifically, we explored whether the actual number of complications vs the predicted number of complications—the observed-expected (O/E) morbidity ratio—could accurately assess temporal differences in the quality of surgical care.

Expected morbidity was predicted for each of the 296 consecutive patients in the following manner. In accordance with POSSUM, 12 physiologic and 6 operative variables were prospectively recorded for each patient undergoing pancreatic resection.¹⁶ Physiologic variables included patient age, Glasgow coma score, the presence of cardiac and respiratory symptoms, vital signs (systolic blood pressure and pulse), serum biochemistry evaluation (urea nitrogen, sodium, and potassium levels), hematologic investigation (white blood cell count and hemoglobin level), and electrocardiographic and chest radiographic findings. Operative variables included the magnitude of the operation, the number of operations performed within 30 days, intraoperative blood loss, the degree of peritoneal contamination, the presence or absence of malignancy, and the timing of surgical intervention. Scores for each variable were divided into 4 grades based on exponentially increasing levels of severity (grades 1, 2, 4, and 8). Physiologic and operative severity scores were then calculated, with the former potentially ranging from 12 to 88 and the latter from 6 to 48. Applying these severity scores, the risk of developing a postoperative complication (from 0% to 100%) was predicted for each individual patient using the following POSSUM equation (EM indicates expected morbidity):

Log_eEM/(1−EM) = −5.91 + (0.16 × Physiologic Score) + (0.19 × Operative Score).

The expected morbidity for our practice overall, and that for each year, was then estimated by calculating the mean POSSUM score for the timeframe assessed.

The incidence of postoperative complications (observed morbidity) was also determined for all patients undergoing pancreatic resection in our practice and was broken down annually. To standardize reporting of actual morbidity, the incidence and severity of postoperative complications were graded according to the Clavien complication scheme.²¹ This reliable tool for quality assessment has been analyzed and validated in a large international series of more than 6000 patients spanning a variety of operations and with more than 600 pancreatic resections in a separate analysis. This system describes 5 grades of clinical severity based not on duration of stay in the hospital but rather on the distinction of escalating levels of therapeutic interventions required to treat adverse events.

In addition to general postoperative occurrences, procedure-related complications specific to pancreatic resection were considered: ileus, delayed gastric emptying, pancreatic leak, biliary leak, gastrointestinal bleed, abscess, wound complications, sepsis, pneumonia, respiratory failure, myocardial infarction, acute renal failure, pulmonary embolism, respiratory distress, pneumonia, urinary tract infection, and neurologic complications.

Using the incidence of postoperative complications and the mean POSSUM score, O/E morbidity ratios were calculated. In the proposed quality assessment model, comparisons of surgical performance are evaluated through analysis of changes in O/E morbidity. A ratio equal to 1.00 demonstrates expected performance. Ratios greater than 1.00 suggest that outcomes are worse than expected. Conversely, ratios less than 1.00 suggest that outcomes achieved are better than expected.

The O/E morbidity ratio was examined further by correlating it with various systematic process improvement measures originally implemented with the intention of improving various aspects of care along the continuum of the preoperative, intraoperative, and postoperative periods. Specific protocols were evaluated and include initiatives to improve the administration of thromboembolic and antibiotic prophylaxis at least 30 minutes before anesthetic induction and approaches to optimize perioperative glycemic control. Compliance with these measures was evaluated and compared with the O/E morbidity ratio for each year.

In addition, the O/E morbidity ratio was also assessed vis-à-vis patient-centered outcomes to determine whether improvements in these initiatives resulted in improved surgical performance. These protocols include reducing intraoperative blood loss, initiating early oral intake, and achieving better patient-reported pain control. Adherence to these patient-centered goals was similarly correlated with the annual O/E morbidity ratio.

Annual outcomes were compared using the χ² statistic, analysis of variance, and test for independent samples. Factors associated with morbidity were calculated based on cross-tabulations using the χ² statistic and the Pearson correlation test. Statistical significance was accepted at <.05. Factors contributing to clinical and economic outcomes underwent univariate and multivariate analyses using the stepwise logistic regression model. Factors with ≤.25 were retained for multivariate analysis. All statistical computations were performed using a software program (SPSS 14.0 for Windows; SPSS Inc, Chicago, Ill).

Postoperative outcomes were evaluated for all patients undergoing pancreatic resection in our practice during the 5-year period. Although the number of pancreatic resections was already regarded as “high volume” at the practice's inception, operative volume steadily increased and by 2005 (76 cases) was 5 times the commonly accepted threshold for this operation (≥16 cases per year). Although the overall rate of complications was 54.1% (n = 160), only 16.6% of patients had major complications (Clavien III-V): those requiring surgical, endoscopic, or radiologic intervention or resulting in organ dysfunction or death. Similarly, few patients required blood transfusions (15.9%) or subsequent management in intensive care settings (4.7%), and 4 deaths (1.4%) occurred overall (3 after pancreaticoduodenectomy and 1 after subtotal pancreatectomy). The median hospitalization duration for all the patients was 8 days (range, 1-48 days). At hospital discharge, 86.0% of the patients returned directly home and 14.0% required additional care in rehabilitation facilities. Rates of hospital readmissions and repeated operations were similarly infrequent (9.8% and 4.7%, respectively). These clinical outcomes were effectively delivered for this high-acuity operation with an overall median total hospital cost of $21 620 per patient.

Further examination of postoperative outcomes, scrutinized on a yearly basis, indicates that general improvement occurred progressively (Table 2). Rates of complications steadily fell from 67.6% in 2001 to 40.8% in 2006 ( = .05); yet, all other clinical outcomes described previously herein remained relatively equal during this span. Median hospitalization duration was 8 days, and it transiently declined to 7 days between 2002 and 2004, but this was not statistically significant ( = .38). Similarly, rates of ICU utilization, hospital readmissions, andrepeated operations demonstrated limited variability and lacked statistical significance.

Although overall clinical outcomes between 2001 and 2006 met or exceeded benchmark standards for pancreatic resection, deeper analysis demonstrates that considerable improvements in quality still occurred temporally. For this complete 5-year period, expected morbidity (mean POSSUM score) for all patients undergoing pancreatic resection in our practice equaled 55.1%. Therefore, preoperative risk assessment predicted that 163 patients would develop any postoperative complication. This prediction was accurate, as 160 patients (54.1%) actually developed a complication. Consequently, the overall O/M morbidity ratio was 0.98, approximately equal to 1.00.

Further analysis indicates that expected morbidity varied substantially across time, increasing from 50.6% in 2001 to a high of 60.5% in 2004 (recently, patient acuity has decreased slightly). Mean expected morbidity was higher for Whipple resections (57.0% for all 5 years) than for distal resections (48.4%). Despite sicker patients overall, observed morbidity rates declined from 68% in 2001 to its current rate of 41% (Table 3). Mean observed morbidity for Whipple resections was 57.3%; for distal resections it was 51.0%. The O/E morbidity ratios for the entire cohort subsequently fell from 1.34 in 2001 to 0.81 currently. The O/E ratios decreased at an equivalent pace for proximal and distal resections. This manifested as a 40% decrease (from 1.29 to 0.78) for Whipple resections and a 43% decline (from 1.50 to 0.85) for distal resections.

This decrease in O/E morbidity ratios was associated with better cost-efficiency across time (Figure 1). Total hospital costs per patient declined annually from $31 541 in 2001 to $18 829 in 2006. This decline was more pronounced for Whipple resections (from $37 805 to $19 907; 47% change) than for distal resections (from $21 712 to $16 478; 24% change). Linear regression analysis not only demonstrated that the O/E morbidity ratio was strongly associated ( = .001) with total hospital costs per patient but also indicated that a 0.10 decrease in the O/E morbidity ratio equates to a $2549 cost savings per patient, with a strong ² correlation coefficient of 0.95 (Figure 2). Based on this same premise, cost differentials of $3726 per Whipple resection (² = 0.89;  = .005) and $781 per distal resection (² = 0.89;  = .005) can be predicted.

A detailed breakdown of costs shows that individual cost centers varied in their savings across time. Significantly lower laboratory (from $1120 to $253) and pharmacy (from $1405 to $251) costs and substantially decreased radiology costs (from $1223 to $356) had the most positive impact on the overall savings. General room and board, ICU, and transfusion costs were unchanged across time.

To identify factors that contributed most to this improving surgical performance in our pancreaticobiliary surgery practice, we sought to examine various process improvement measures implemented between 2001 and 2006, including timely thromboembolic and antibiotic prophylaxis, optimal glycemic control, reduced intraoperative blood loss, early oral intake, and improved pain management. Overall compliance with these measures, and achievement of these goals, was correlated with O/E morbidity ratios on a yearly basis.

Significant improvements were observed during each period of the perioperative continuum (Table 4). Preoperative administration of thromboembolic prophylaxis increased from 3% in 2001 to 100% in 2006 (<.001). Intraoperative, blood loss fell from 425 to 300 mL ( = .03). Postoperative rates of early oral intake also improved across time. In 2001, 55.9% of the patients tolerated a soft solid diet within 5 days of the initial operation compared with nearly all the patients (94.7%) in 2006 (<.001). Other measures exhibited increased compliance but at less impressive rates. Preoperative administration of antibiotic prophylaxis increased from 82.4% to 98.7%, and the proportion of patients who achieved optimal glycemic control increased from 52.9% to 75.0%. Although most patients reported only minor or moderate pain, average pain severity scores remained unchanged between 2001 and 2006 ( = .33).

A stepwise regression analysis was performed to determine the association of the O/E morbidity ratio with these measures. Patient age, level of acuity (individual POSSUM score), operating time, type of pathologic abnormality, surgeon, and annual operative volume did not demonstrate statistical significance when univariate analysis was performed. However, thromboembolic and antibiotic prophylaxis, intraoperative blood loss, glycemic control, early oral intake, and overall rates of complications all were individually associated with the O/E morbidity ratio ( = .048). Multivariate analysis indicated that thromboembolic prophylaxis, median blood loss, and early oral intake had the strongest correlations with annual O/E morbidity ratios (<.001). Collectively, these process improvement measures helped mitigate the impact of increasing patient acuity on postoperative outcomes and contributed to lower O/E morbidity ratios.

Recently, expectations for improved surgical quality have risen. Complications and hospital costs have assumed the interests of patients, payers, and health care professionals and are increasingly scrutinized by the general and medical communities.^11,22- 29 A 2001 report by the Institute of Medicine characterized the failures of the American health care system, lamenting its inadequacies by stating that the current system is harmful to patients, ineffective in delivery of care, and too costly.²⁵ These notions, and those of others, have piqued interest in comparative health quality assessment.

Assessment of surgical quality improvement has, to date, focused on volume-outcome correlations. The association between hospital volume and operative mortality has been investigated since the 1990s and is well-documented in the literature. Nearly all the studies^2- 8 demonstrate that high-volume medical centers consistently achieve lower mortality rates across a large spectrum of operations. Although few investigators debate these findings, many criticize their specific applicability, and others propose that global policy development based on these studies is misguided.^10- 14 These critics contend that volume-mortality correlations alone do not provide adequate objective comparisons of institutions or surgical practices, particularly when systematic variations in patients' preoperative risk profiles exist among surgeons, practices, and hospitals. And although the relationship between volume and mortality suggests that hospital resources, surgical expertise, or both contribute to better outcomes, this ignores other significant outcomes, such as morbidity and costs. Furthermore, this correlation is an inadequate measure of surgical performance when high-volume medical centers are directly compared and in settings where intrapractice outcomes are analyzed. Advancements in perioperative care have made death a rare event in most elective high-acuity operations. For example, in the field of pancreatic surgery, operative mortality as high as 50% in the 1960s has declined considerably to less than 5% presently.^30- 34 These concepts indicate that volume and mortality are not enough and should not be the absolute benchmarks for surgical quality assessment.^35- 36

Contemporary discussions have proposed that morbidity is a more appropriate measure of health care quality than is mortality. In many surgical series, complications are often the primary outcome. The impact of complications is well-described, and it is increasingly apparent that postoperative morbidity is the leading contributor to longer hospital stays and increased hospital costs.^22,27,35 However, historical comparative assessments using postoperative morbidity rates have not always considered the effects of variance in preoperative patient risk. Today, our evolving understanding of the effect of preoperative morbidity emphasizes the need to evaluate the impact of predictive risk factors.

In the 1990s, POSSUM was developed to appraise and audit the quality of surgical care in institutions.¹⁶ It gained wide acceptance in the United Kingdom and Europe and has been validated as a satisfactory method for predicting surgical complications across multiple disciplines and across various levels of surgical procedure complexity. Numerous studies^20,37- 38 demonstrate its application as a means of surgical appraisal, and they indicate that increasing aggregate POSSUM scores reflect poor clinical outcomes.

In the United States, the Veteran Affairs National Surgical Quality Improvement Program (NSQIP) has fast become the standard model for quality assessment. Similar to POSSUM, it provides a validated measure of surgicalperformance for each institution by using the O/E morbidity ratio concept.^15,24,36 However, NSQIP analyses often depend on administrative data that are randomly accrued and that generate broad estimates of surgical performance that do not provide procedure-specific outcome measures.³⁵ Therefore, although the NSQIP may be valuable in comparing institutional quality, it has limitations in assessing personal or practice outcomes for particular procedures.

Given the inherent limitations of generalized quality assessment models, we sought to develop a procedure-specific model for a major (high-acuity) abdominal operation, an application that uses clinical detail and predictive risk assessment to correlate expected outcomes with actual events. It is intended to supplement the recently adopted analyses performed by the NSQIP at Beth Israel Deaconess Medical Center. The proposed model is versatile and assesses the interplay among surgical performance (morbidity and mortality), institutional-based systems improvements (costs), and patient-centered outcomes (satisfaction). Together, these concepts establish what we refer to as the Quality Triangle (Figure 3).

The model is based on principles of POSSUM. Numerous variables are analyzed to estimate an individual's preoperative risk of developing a postoperative complication. For a given cohort, overall expected morbidity is determined by the mean POSSUM score. Expected morbidity is then linked with observed outcomes to determine the O/E morbidity ratio. This ratio assesses whether outcomes are better or worse than expected (O/E ratio less than or greater than 1.00, respectively). Annual O/E morbidity ratios are correlated with various process improvement initiatives to identify specific elements that contribute to overall health care quality. We define quality according to each component of the Quality Triangle, with the surgeon, hospital, and patient as equal stakeholders (Figure 3). A decreasing O/E morbidity ratio reflects improved surgical performance; better cost-efficiency (ie, reduced hospital costs) reflects overall process improvement; and compliance with patient-centered initiatives reflects improvements in patient satisfaction.

In our practice, the quality of performance improved overall during the past 5 years. Although patient acuity rose, rates of complications declined. This translated into considerable improvements in the O/E morbidity ratio, as observed outcomes continue to be better than expected. This decline in the O/E morbidity ratio was concurrently associated with various process improvements in our practice. Between 2001 and 2006, protocols to increase thromboembolic prophylaxis, reduce intraoperative blood loss, and begin early oral intake contributed to better surgical performance overall. Collectively, these initiatives helped mitigate the impact of increasing patient acuity on postoperative outcomes. Consequently, these improvements resulted in better cost efficiency, as total hospital costs declined considerably. The cost reductions were primarily achieved not through decreasing duration of hospital or ICU stay (room and board costs) but rather through substantial decreases in pharmacy, radiology, and laboratory costs, modalities critical in the management of complications. Regression analysis of this trend also predicts that any further decrease in the O/E morbidity ratio will reproducibly equate to substantial cost savings.

Three concepts may explain these trends in surgical performance. First, between 2002 and 2003, the O/E morbidity ratio decreased dramatically from 1.34 to the 1.00 threshold. Although this improvement represents (in this series) the largest decrease in the O/E morbidity ratio (a 0.34 decline) and the greatest cost savings per patient ($9690), it also reflects the initial development period of our practice. An experienced, high-volume surgeon initiated and built a specialty approach with an emphasis on systems integration. While operative volume increased during this period, more patients received thromboembolic and antibiotic prophylaxis, intraoperative blood loss decreased considerably, and optimal glycemic control was frequently achieved. Patients benefited by recovering bowel function earlier, tolerating oral intake sooner, and reporting lower pain scores overall. Although, it is impossible to discern whether these improvements were directly attributable to higher operative volume or better process management, we believe that volume alone cannot explain these achievements, particularly when higher operative volumes in subsequent years did not result in such striking declines in the O/E morbidity ratio.

Second, based on increasing volume demands between 2001 and 2003, a fellowship-trained junior partner was hired in late 2003. As a result, operative volume increased 1.5 times (from 44 to 66 cases) in the first year. Although clinical outcomes remained stable during this period, observed and expected morbidity rates both increased. This, however, did not result in a change in the O/E morbidity ratio, which remained level at 1.00. Similarly, a marginal increase in median hospital costs per patient was observed and probably reflects the higher patient acuity during this period, the initial clinical learning curve of a new junior attending surgeon, or both.

Finally, in 2004, a detailed clinical pathway for postoperative management was implemented in our practice. This care path outlines a standardized management approach for all patients undergoing pancreatic resection and integrates multiple disciplines invested in the care of the patient. Based on the tenants of evidence-based medicine, it provides a simple guide for the management and timely removal of central venous catheters, nasogastric tubes, urinary catheters, and intra-abdominal drains. Furthermore, this clinical pathway standardizes perioperative fluid resuscitation, antibiotic drug administration, diagnostic testing, and initiation of oral intake. This coincided with an institutional initiative in “team building” among the operating room personnel. With rigorous adherence to these paradigms, considerable cost savings were observed immediately after its implementation, and by the second year, the O/E morbidity ratio had fallen well below the 1.00 threshold.

In summary, the results of this study suggest that mortality does not continue to correlate with volume in an already “high-volume” practice. Although patient acuity increased, morbidity rates declined and contributed to substantial cost savings. Furthermore, despite traditional outcomes that may already meet or exceed benchmark outcomes for a given major surgical procedure, improved process management can mitigate the impact of preoperative risk and effectively deliver quality advances. We propose that this quality assessment approach for high-acuity surgery enables accurate and objective comparisons of surgical outcomes. This model shows promise in evaluating clinical and economic performance, the impact of process improvements, and variations between surgeons and within practices. This quality assessment model can be customized to analyze or predict factors that affect the Quality Triangle.

Charles M. Vollmer, Jr, MD, Department of Surgery, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Stoneman 9, Boston, MA 02215.

December 13, 2006.

: Vollmer, Pratt, and Callery. : Vollmer, Pratt, Vanounou, Maithel, and Callery. : Vollmer, Pratt, Vanounou, Maithel, and Callery. : Vollmer, Pratt, and Callery. : Vollmer, Pratt, Vanounou, Maithel, and Callery. : Pratt and Maithel. : Vollmer and Callery. : Vollmer, Vanounou, and Callery.

None reported.

This research was conducted with support from the Clinical Research Fellowship Program at Harvard Medical School, offered by the Doris Duke Charitable Foundation, and the Harvard PASTEUR Program and the Office of Enrichment Programs.

This paper was presented at the 87th Annual Meeting of the New England Surgical Society; September 15, 2006; Groton, Conn; and is published after peer review and revision. The discussions that follow this article are based on the originally submitted manuscript and not the revised manuscript.

The one thing that I noticed was that in 2001, when you started your data set, you had very few cases. The question that comes up is whether that was the inception of team formation that progressed through 2005, when your data set was complete? There has been a lot of observational literature, not as mathematical as yours, that indicates that when a new unit is established or a unit becomes a closed unit as in critical care there are improvements independent of some of the things that you pointed out, like antibiotic and glucose control. I wonder if you could comment on that.

That O/E value for the first year was a partial year value and so the numbers were for 4 months for that particular hatch mark, so I think it is a bit of an aberration that the O/E ratio was so high at that point in time. If the data were reanalyzed as 5 distinct 12-month periods, the steepness of that first year's decline would be dampened (this is actually reflected in the data presented in the manuscript submission). That was the point when the practice started and, to get to your query, that was when a new surgeon came into a new hospital and developed a new plan for taking care of these very complex patients over the complete spectrum of the perioperative period (preoperatively, intraoperatively, and postoperatively). I think that what you are seeing there in that first hatch mark (during the first year) and then beyond into the second year is the development of a team system, as you have alluded to.

I think what you are implying a little more specifically is the potential effect of streamlined team care among anesthesia, nursing, and surgeons, specifically in the operating room. That sort of paradigm did, in fact, take place in our institution, but later in the time period, around 2004, so I would not necessarily attribute the early changes to that process improvement.

I actually had 2 related questions. One is who decides that they had a complication and how do you track that? Second and a related question is how you collected these data. Was the (POSSUM) score collected prospectively, or is this all retrospective analysis? And then, finally, how often are you supplying this information back to yourselves? How often do you review it? Do you review it on a quarterly basis or a yearly basis? How has that been done to help you gain these improvements?

First of all, we have just come upon the value of this analysis within the past 4 to 5 months, but your point is very important: that the whole purpose of this is to use it as an audit of the practice. Therefore, we are eager to use this prospectively down the line.

Second, the POSSUM score is evaluated after the operation because there is an operative component to it. We have analyzed this risk assessment tool in depth in terms of pancreatic operations over the past 5 years, finding excellent fidelity and predictive value of the POSSUM score, and we have a separate manuscript ready to go out on that. In that analysis we found that the operative component of POSSUM was less important and less relevant than the preoperative physiologic score, and that is probably because some of the contribution factors for calculating the operative score do not apply to the specific operation or operations that we do. There are only 2 or 3 of those components that actually figure into it.

Finally, as far as data collection goes, this is a very important point because there are 2 ends of the spectrum to this. If you look at an administrative database, you look at imprecision in the ability to determine what these complications are. If you look, as we have, at a surgeon- or practice-derived labeling of the complications, you then introduce bias and certainly that would be the case if we were to allow everyone in their own practice to determine what is and what is not a complication. I think the thing that helps us get around that is the nature of the Clavien scoring system for complications. It is not a matter of determining specifically how many fistulae you have or anastomotic leaks, etc. Instead it is basically broken down by the severity of the complication and the ultimate impact upon the patient.

I have 2 questions. One is a follow-up to a previous question. Do you have an independent person doing this or is it self-reported? For instance, in the NSQIP, an independent nurse gathers the data. And, two, you start a new program, you have 2 fellowship-trained surgeons out of fellowship, how do you know that this is not simply a learning curve for an attending, which I believe there certainly is, so how do you explain that?

Let me address that first. First of all, the 2 surgeons in this practice are composed of a mid-career senior surgeon with much experience with pancreatic resections. The second is a junior attending—well, that would be me. I started 3 years ago and clearly we have actually used this method to go back and compare the 2 of us. What we found is, yes, I as a junior surgeon have had a steep change in the O/E over the first 1 to 2 years but then have plateaued out at the same level as my partner at this point. As far as the full data set goes, we are dealing with a very experienced surgeon when the practice began and we are not realizing a learning curve on his part.

The second point was about who is collating the data. We have had a dedicated research student who independently accrues the data regularly for us and corroborates with me as to the classification of the complications and such if necessary. Although we do have NSQIP at our institution and we have started to employ that at the institutional and departmental levels, we do not have an independent nurse or data filing agent.

I have 2 comments. The title of your manuscript seems to suggest that your data might refute the well-known data presented by Dr John Berkmeyer, highlighted in your early slide, that volume is highly predictive of operative mortality. In reality you have shown that you also are a high-volume center with a very low mortality rate for a high-risk procedure. While it is certainly admirable, I do not think you have shown that volume is an adequate indicator because you are a high-volume center. My questions for you are in reference to one of your last slides, regarding your O/E morbidity and mortality ratio. Maybe the best use of this O/E ratio is for medium- to low-volume centers to see if they are falling within an appropriate O/E ratio compared with high-volume centers. If lower-volume centers perform within an acceptable ratio, it would seem they should be allowed to perform these high-acuity surgeries. I wonder if you would comment on that.

I will say that by no means did we want to refute the Berkmeyer findings but rather to build upon them. We wanted to find out how we did by comparison. One of the things I brought up there is, “Do we really understand all the outcomes in a high-volume setting?” And, “If you advance volume in an already high-volume setting, are you going to be able to achieve another mortality threshold or not?” I think our data indicate that perhaps not in this case, although if you look at our numbers we are many, many fold over what is defined as a high-volume institution for pancreatic surgery.

Regarding your second comment, I think the O/E attacks a different question and can be applied to all levels of volume. It really is going to get at scenarios like, “If you are a small-volume surgeon and you have few complications, is it because you are tackling cases that have a low propensity for complications over time?” This is going to be a way to analyze these low- and mid-volume surgeons/hospitals. This is what they are doing in Europe, where POSSUM was used originally as an audit system for the UK, and it is really taking prominence in Europe as far as comparing practice profiles. Through this technique, we can compare both practices and individual surgeons based on the types of cases they are handling.

None reported.