Association Between Implementation of a Medical Team Training Program and Surgical Morbidity FREE

Considerable efforts to reduce risk associated with operations have been promoted over the past decade.¹ Although surgical mortality dominates the headlines, it is, in fact, the more common nonfatal surgical complications that could provide the earliest signals of failures in the medical system. In a previous article, we² studied the effect that Medical Team Training (MTT) had on surgical mortality rates. In the present study, we devoted our attention to surgical morbidity rates to contribute to the growing body of research regarding methods of reducing surgical errors.^3- 8

In 2006, the Veterans Health Administration (VHA) implemented a nationwide MTT program to facilitate structured interactive communication in surgical care and improve surgical outcomes. The VHA MTT program requires preoperative briefings and postoperative debriefings guided by a checklist⁹ and using cognitive aids.^10- 11 The goal of this study was to compare the annual surgical morbidity rates for facilities that received the VHA MTT program with those that had not.

Quiz Ref IDThe MTT program^12- 13 aims to improve the communications within the operating room (OR) and help the staff to work better as a team. It allows staff to identify potential trouble spots and any obstacles to a safe and successful operation through checklist-guided preoperative briefings and postoperative debriefings. The program began with 2 months of preparation and planning with an implementation team at the training VHA facility. This was followed by a 1-day learning session held on site. The facilities included were provided sample checklists and briefing/debriefing tools that facilities adapted for their needs.⁹ The MTT program is based on crew resource management theory from aviation.¹⁴ Everyone in the OR is trained to work as a team and challenge each other when they identify safety risks. After the learning session, 4 quarterly follow-up telephone interviews are conducted with the team to provide support, coaching, and assessment of implementation of the MTT program. Details of the MTT program have been published.²

This was a retrospective health services cohort study using a contemporaneous control group. The VHA Surgical Quality Improvement Program (VASQIP) annual surgical morbidity rates from 3 fiscal years—2006, 2007, and 2008—were used.

The training list contained 130 facilities. Facilities on the training list but without all 3 years of VASQIP data were excluded from the analysis (n = 14). Facilities trained during the pilot phase of the study were also excluded (n = 10). Facilities trained in 2008 (n = 32) were excluded because data for the 2009 fiscal year were unavailable. In total, 74 facilities were analyzed.

Quiz Ref IDConsidered as the criterion standard for measuring surgical quality and adopted by the American College of Surgeons,¹⁵ VASQIP provides observed and risk-adjusted 30-day morbidity rates for major noncardiac operations.^16- 19 Surgical specialties included general, orthopedic, urology, vascular, neurosurgery, otolaryngology, noncardiac thoracic, and plastic. Current Procedural Terminology^18(p495) codes of procedures with known low morbidity and mortality rates or transurethral resections of the prostate, transurethral resections of the bladder tumor, and herniorrhaphies exceeding the limit of 5 per week were excluded.Quiz Ref IDUsing VASQIP data, we included annual surgical morbidity rates as our primary end points for each of the 74 facilities for fiscal years from 2006 to 2008. Because MTT is a facility-level intervention, aggregated morbidity rates at the facility level were outcome measures.

Although not studied as an outcome here because it was the focus of a previous study,² mortality data were used as a covariate in statistical modeling. Unless otherwise stated, all morbidity and mortality rates were annual, and both observed and risk-adjusted rates were studied. All years were meant to be VHA fiscal years. For example, year 2008 indicates fiscal year 2008 (from October 1, 2007, to September 30, 2008). The VASQIP methods have been described in detail in other studies.^16- 19 Detailed definitions and descriptions of outcomes can be found in the eAppendix.

To examine baseline characteristics of the MTT and non-MTT facilities, we compared the following: rural or urban status, complexity, VASQIP surgical volume, and baseline annual surgical morbidity as well as mortality rate (Table 1). The VHA 2005 Complexity Model²⁰ categorizes VHA facilities into 3 levels: level 1 is high complexity; level 2, medium complexity; and level 3, low complexity. These designations are based on a composite involving the number of patients evaluated, patient risk, number of physician specialists, teaching status, research funding, and intensive care unit capability. The VHA urban/rural/highly rural classification is based partly on census tract and partly on population density. Facilities located in US census tracts designated as urban are considered urban. All others are considered rural, except for facilities located in a county with a population density of less than 7 people per square mile, which are considered highly rural.²¹

Annual surgical morbidity rate was defined as the number of VASQIP surgical morbidities divided by the number of procedures during the fiscal year. The primary outcome measure was change in morbidity rate from 2006 (1 year before facilities enrolled in the MTT program) to 2008 (1 year after facilities enrolled in the MTT program). Continuous variables not distributed normally were compared using the Mann-Whitney test; their medians and ranges, as well as means are presented. Pearson χ² tests were used to compare proportions (Table 1 and Table 2). Comparisons of morbidity rates before (2006) and after (2008) were made separately for MTT and non-MTT groups, using paired Wilcoxon signed rank test (Table 3). Multivariable Poisson generalized estimating equations^22- 23 (GEEs) were used to assess MTT associations with outcome while adjusting for secular trends as well as propensity scores (Table 3).

Rate ratios (RRs) and accompanying 95% CIs were calculated to represent the strength of association between MTT intervention and morbidity rates. The GEE method was used to account for the longitudinal nature of the yearly data. We controlled for baseline characteristics that included complexity, surgical volume, urbanicity, and baseline morbidity and mortality rates. Along with the main effect of MTT and time (unit in years), an interaction term, MTT × time, was entered into the model to study whether the trajectories of the 2 groups differed.

We compared not only MTT program facilities with those not yet trained but also morbidity rates in the same facility after receiving vs before receiving MTT. As a result, some facilities served as their own controls as well as controls for others.

Because MTT was not randomly assigned in this study, potential confounding and selection biases were accounted for by developing a propensity score for MTT.^24- 25 The propensity score was based on the following variables measured at baseline: morbidity and mortality rates, the average number of sampled procedures per facility, hospital complexity, and urbanicity. These were summarized as 1 propensity score through a full nonparsimonious logistic model. For each facility, a propensity score was calculated and facilities then were stratified by their propensity scores into 3 strata. Within these strata, covariates in MTT and non-MTT groups were similarly distributed, and stratifying by propensity score could remove more than 90% of the overt bias due to the covariates used to estimate the score.²⁶ Once groups were stratified by propensity score, they were again separated into MTT and non-MTT groups to determine whether there were differences in baseline variables, such as morbidity rates, to suggest an imbalance. Propensity scores could not remove hidden biases except to the extent that unmeasured variables were correlated with the measured covariates used to compute the score.^24- 26

Propensity score matching was used to select non-MTT program facilities that were similar to MTT program facilities with respect to propensity score and other covariates, thereby matching on many confounders simultaneously.²⁷ As a result, in each propensity score stratum defined by this procedure, the covariates were balanced and the assignment to MTT could be considered random. At that time, within each stratum, a full Poisson GEE analysis was performed to compute the effect of the MTT program. The total effect of the MTT program was finally obtained as a summary of the effect estimate of each stratum. This provided a valid estimate of the MTT program effect because the GEE analyses compared facilities with similar baseline characteristics. All analyses were performed using commercial software (STATA, version 10.0; StataCorp, College Station, Texas). All P values were 2-sided at a significance level of .05.

The study was approved by the Research and Development Committee at the VA Medical Center in White River Junction, Vermont, and considered exempt by the Dartmouth College institutional review board.

A total of 119 383 sampled procedures from 74 facilities were analyzed. The MTT program was implemented in 42 facilities (57%), and 32 facilities not-yet trained (43%) served as the contemporaneous control group. Before-and-after analyses were performed for the MTT group and the non-MTT group, and the changes in the 2 groups were compared. The MTT program was implemented at 42 facilities in 2007. Thirty-two facilities were not trained in the MTT program during 2006-2008. The baseline (before) for all 74 facilities was 2006. Because 2008 (after) was the year following the 2007 implementation year for the MTT program facilities, we chose 2008 as the follow-up measure for all 74 facilities. The 2 groups did not differ significantly (Table 1) in morbidity or mortality rates, surgical volume, surgical complexity,²⁴ or urbanicity.²⁵

Because facilities that underwent MTT were not randomly selected, we tried to correct potential imbalances between the trained and the nontrained groups through propensity score matching. Table 2 displays the 3 groups of facilities stratified by propensity scores for each baseline variable. Mean MTT selection propensity scores ranged from 0.54 to 0.78 across propensity tertiles, with excellent discrimination between MTT and non-MTT groups (C statistic = 0.79). The distribution of key potential confounders—morbidity rates at baseline—was similar within propensity tertiles for facilities with and without the MTT program. This indicates that none of the 3 propensity score strata showed significant differences in morbidity rates between the groups at baseline. In short, we were not able to discern any overt selection bias in our model after matching on propensity score.

The change in morbidity rate from baseline was examined: facilities in the MTT program (n = 42) had a statistically significant decrease of 17% in observed morbidity rate (RR, 0.83; 95% CI, 0.79-0.88; P = .01). Facilities not in the program (n = 32) had a decrease of 6% in observed morbidity; however, this was not statistically significant (RR, 0.94; 95% CI, 0.86-1.05; P = .11). After risk adjustment, both groups showed statistically significant improvement—a decrease of 15% for those in the MTT program and a decrease of 10% for those not yet in the MTT program (Table 3)—unlike the findings without risk adjustment. Quiz Ref IDThe magnitude of the difference between trained and untrained groups was also smaller after risk adjustment (a 5–percentage point difference vs 11–percentage point difference). To put this in another way, 88% of the MTT program facilities (37 of 42) improved their risk-adjusted morbidity rates vs 69% of the non-MTT group (22 of 32; P = .04). Although risk-adjusted morbidity rates declined in both groups, the decline was 20% steeper in the MTT group (RR, 1.20; 95% CI, 1.19-1.22; P = .001) after further adjustment for morbidity and mortality rates, procedure volumes, and propensity scores (Table 3).

Wilcoxon matched-pairs signed rank tests were performed for other outcome measures, although only total morbidity was risk adjusted by VASQIP. Nevertheless, significant drops in event rates were observed in all but one outcome (pulmonary embolism) for the MTT group. Decreases were also observed in the non-MTT group, albeit smaller, and none was statistically significant. Because of the low event rates for deep venous thrombosis and pulmonary embolism, and because they were clinically linked, additional analysis was performed in which these morbidities were combined as a single outcome, and combined infection end points were also analyzed (Table 3). Propensity score–matched, surgical risk–adjusted GEE analyses were similarly applied to specific nonfatal complications. Because we observed a decline trend in both the MTT and non-MTT groups, we also found that the decline in the MTT group was steeper than that in the non-MTT group by 15% (RR, 1.15) to 28% (RR, 1.28; Table 3).

The VHA MTT program was associated with a statistically significant reduction in the annual surgical morbidity rate. Meanwhile, the ongoing VASQIP program uses a robust system of data feedback, support, and follow-up to address processes of care in each facility. Because of this, the results described in this study occurred within the larger context of the VASQIP's active quality improvement program.

Quiz Ref IDThorough preparation before an operation and effective communication during the operation are promising methods of reducing surgical errors.^3- 8 However, studies^3- 8 in this area have been lacking in measurable medical outcomes (eg, length of survival, as well as rates of complications and recurrences), and many studies lack a control group or baseline performance data. Although checklists have been shown^7- 8 to significantly reduce surgical complications, the VHA MTT program focuses more broadly on teamwork and on the use of briefings as the main tool to enhance proactive communication among the clinical staff. A checklist, although important, is only one component of the overall strategy to improve communication, as a static checklist cannot anticipate and explicitly include every possible factor. Briefings aided by a checklist, however, are able to facilitate and improve communication throughout an operation, which is a dynamic process that invariably has unforeseen complications.

Another difference between the VA MTT program and other crew resource management–based programs¹⁴ is the 2-month preparation and planning phase in which MTT faculty work with an implementation team consisting of surgical staff at the facility to assess problem areas in the OR. During this time, the OR staff gains a deeper understanding of local communication problems at their facility that could be addressed by MTT. In addition, the 12-month follow-up helps the OR staff to implement the communication tools learned in the training, solve unanticipated problems, and provide data to the MTT faculty regarding their progress. Finally, quarterly telephone calls keep pressure on the OR teams to continue to improve and disseminate the program throughout their services. Further research is needed to determine which aspects of the VA MTT program account for the change in surgical morbidity.

Details regarding degrees of briefing and debriefing were collected during the 4 follow-up telephone interviews conducted during the year after the MTT program was instituted.² Based on these interviews, we hypothesize that preoperative checklist–driven briefings provide an opportunity to correct problems before they surface. Surgical teams have shared stories, such as discovering that a patient required cardiac clearance, and others identified during the briefing the need for additional equipment or implants.²

Teams also shared the value of voicing problems in the debriefing. Examples included fixing broken equipment or instruments, ordering extra or back-up sets of instruments to prevent intraoperative delays, and improving collaboration with the radiology service for quicker response times. The resolution of such issues likely also prevented adverse events.

This study has several limitations. First, it was not randomized; consequently, unmeasured potential confounders are likely to exist. One concern is the imbalance between the intervention (MTT) and control (non-MTT) groups at baseline. Another is potential bias in the formation of study groups. For example, the first facilities to complete the MTT program may also be those with the greatest likelihood to improve.

We addressed these concerns because propensity score matching approximates an unbiased design. We grouped the facilities according to their propensity scores based on their baseline characteristics. As a result, some groups had limited sample sizes. We then conducted analyses within these groups to control for potential confounders. Before propensity score matching, the difference between the MTT and non-MTT groups in reduction of the risk-adjusted morbidity rate was 50% (15% in the MTT group vs 10% in the non-MTT group). After using propensity score matching, the estimated difference was 20% (RR, 1.20).

Our limited sample size and observational study design might partially or fully account for the observed association; thus, we caution against a simplistic causal interpretation of our model. However, studies with different designs, populations, and research groups appear to have arrived at a similar estimate of the impact of surgical checklists on morbidity.^7- 8

We adopted the more flexible and versatile longitudinal GEE model to analyze the data because it could ameliorate some of the shortfalls in study design. For example, each facility served as its own control, thus enabling us to remove some extraneous, but unavoidable, sources of variability among the facilities, such as location, size, and structure. Although the VHA introduced the MTT program because facilities needed to improve communication, it is nevertheless possible that facilities could have started to implement some aspects of the MTT program before their enrollment. The best way that we could address this limitation was to use facilities as their own controls in the analysis and thus remove the effect of heterogeneity from this source.

Because many factors can reduce surgical morbidity, the inclusion in our design of a contemporaneous control group from the study period that was similar to the groups at the MTT facilities after being matched on propensity scores should have lessened the potential for confounding due to existing secular trends.

We observed a general secular trend of decreasing surgical complications in the VHA during our study period (2006 vs 2008) for non-MTT facilities. Many contributing factors are likely: the active improvement program of VASQIP, the cost of complications in an era of health care cost containment, continuous efforts to improve the quality and safety of care for veterans, modified indications for operations, the advancement of medical science and medical training, and many other efforts to reduce medical errors (eg, better system design, equipment design, and medication design [packaging, labeling, administering, and tracking]). We found that the MTT program, beyond these general efforts and secular trend, resulted in significant additional reduction in surgical morbidity. Finally, the applicability of this study to the general population may be limited because VHA patients have been found¹⁶ to differ from patients in the private sector.

In conclusion, participation in the VHA MTT program, which emphasizes communication and teamwork during an operation through checklist-driven briefings and debriefings, is associated with lower surgical morbidity.