0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Poster Session |

Failure of a Web-Based Educational Tool to Improve Residents' Scores on the American Board of Surgery In-Training Examination FREE

Charles M. Ferguson, MD; Andrew L. Warshaw, MD
[+] Author Affiliations

Author Affiliations: Department of Surgery, Massachusetts General Hospital and Harvard Medical School, Boston.


Arch Surg. 2006;141(4):414-416. doi:10.1001/archsurg.141.4.414.
Text Size: A A A
Published online

Hypothesis  Usage of a Web-based educational tool will improve residents' scores on the American Board of Surgery In-Training Examination (ABSITE).

Design  Before and after trial.

Setting  Academic surgical residency in a tertiary care university hospital.

Participants  Nineteen postgraduate year 1 (PGY1) and PGY2 residents.

Intervention  All PGY1 and PGY2 residents were given unlimited access to a Web-based educational tool aimed at early-level residents in surgery. The educational tool covers the basic science of surgery and basic concepts of clinical surgery and uses a variety of educational techniques (readings, problem-based learning, case-based learning, and practice tests). Although residents were encouraged to complete the learning package, they were not required to do so.

Main Outcome Measures  The ABSITE scores and changes in ABSITE scores were compared with the amount of time residents spent using the program and the number of tutorials completed.

Results  There was no significant difference in ABSITE scores before or after the use of the Web-based educational tool. There was no significant relationship between use of the tool (either in total time or total tutorials) and ABSITE score. For PGY2 residents, there was a negative relationship between total time spent on the program and ABSITE score (linear regression analysis best-fit slope, −0.20 ± 0.1; r2 = 0.33) and total tutorials completed and ABSITE score (linear regression analysis best-fit slope, −0.34 ± 0.22; r2 = 0.25).

Conclusion  There was no demonstrable beneficial effect of a Web-based educational tool on cognitive knowledge of residents as measured by ABSITE scores.

Figures in this Article

Attainment of cognitive knowledge is a crucial part of residency education. Surgical residencies have used a combination of lectures, conferences, reading programs, and informal discussions to help residents obtain the cognitive knowledge base required to be independent surgeons. With the recent mandate to decrease resident work hours, there has been interest in using innovative methods to teach basic knowledge in surgery. Specifically, web-based modules have been advocated as a method to increase cognitive knowledge, which can be used while at work or on off-duty hours. These educational tools use a combination of reading material, video clips, simulated patients, and question-and-answer format to transmit knowledge. The Association of Program Directors in Surgery (Bethesda, Md) is currently embarking on a project to produce a web-based multimedia educational tool for residents. BeST Resident (Harvard Medical International, Cambridge, Mass) is a commercial Web-based education program aimed at residents in the early stages of postgraduate surgical education. Through a variety of methods (textbook readings, problem-based learning, case-based learning, and practice tests), participants are exposed to the basic scientific knowledge expected of a junior-level resident and covered by examinations such as the American Board of Surgery In-Training Examination (ABSITE) and the Fellowship of the Royal College of Surgeons examination.

Methods to assess an adequate knowledge base have included individual residency written and oral examinations, evaluation by preceptors on clinical rotations of a resident's knowledge base, and use of the ABSITE. The ABSITE has become widely accepted by surgical residencies as a primary method of evaluating residents' cognitive knowledge base and has been used by our residency since its inception in 1975 as part of the evaluation process in assessing residents' cognitive knowledge. This study is an attempt to see whether use of BeST Resident affected residents' scores on the ABSITE.

Beginning in June 2004, unlimited access to BeST Resident was provided to postgraduate year 1 (PGY1) and PGY2 residents. Residents were told that they were expected to complete all tutorials in the program during the 2004 through 2005 academic year. Resident usage of the program was monitored by the program director, and emails were sent to residents with low usage reminding them of the importance of studying for the ABSITE and the availability of this program, but no actual requirement of usage was made, and no adverse consequences of nonusage were outlined. On the day of the ABSITE, each resident's total hours of usage of BeST Resident and number of tutorials completed were tabulated. Nineteen PGY1 and PGY2 residents took the ABSITE in 2005, all of whom had unlimited access to BeST Resident. Nine of these residents were PGY2 and had taken the ABSITE in 2004, allowing evaluation of the change in score with BeST Resident.

The ABSITE scores were reported by Standard Score from the ABS; percent correct and percentile score were not used in any of the analyses. Statistical analyses were performed using GraphPad Prism (GraphPad Software, Inc, San Diego, Calif). Correlation of total hours of BeST Resident usage or tutorials completed with ABSITE score or change in score was by linear regression analysis.

Mean ABSITE scores for PGY1 residents rose from 471 to 481 with the institution of BeST Resident, but this change was not statistically significant. Likewise, although mean ABSITE scores for PGY2 residents fell (510 to 495), this difference was not significant. Of note, we were unable to demonstrate a change in ABSITE score related to decreased work hours (Table).

Table Graphic Jump LocationTable. Mean ABSITE Scores by Postgraduate Year

Nineteen residents had unlimited access to BeST Resident and used the program for a mean of 13.62 hours (median, 11.5 hours; range, 0-58.5 hours), completing a mean of 30.31 tutorials (median, 20; range, 0-100). Linear regression analysis of ABSITE score vs total time spent on BeST Resident revealed a best-fit slope of −0.03 ± 0 (r2 = 0.03). Linear regression analysis of ABSITE score vs total tutorials completed on BeST Resident reveals a best-fit slope of 0.00 ± 0.09, (r2 = 0.00). Thus, there was no significant relationship between BeST Resident usage (as measured by time or total tutorials) and ABSITE score.

Nine residents had unlimited access to BeST Resident and had taken the ABSITE in 2004 and 2005, thus allowing evaluation of change in standard score. Changes in these residents' scores ranged from −70 to +137, with a mean of +12.9 standard score points. These residents used BeST Resident a mean of 16.22 hours (median, 10.5 hours; range, 0-58.5 hours) and completed a mean of 35.9 tutorials (median, 15; range, 0-100). Linear regression analysis of change in score vs total time spent on the program revealed a best-fit slope of −0.20 ± 0.1 (r2 = 0.33) (Figure 1). Linear regression analysis of change in score vs total tutorials completed revealed a best-fit slope of −0.34 ± 0.22 (r2 = 0.25) (Figure 2). Thus, there was a negative relationship between BeST Resident usage and change in ABSITE score.

Place holder to copy figure label and caption
Figure 1.

Change in American Board of Surgery In-Training Examination (ABSITE) score vs total time on program.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Change in American Board of Surgery In-Training Examination (ABSITE) score vs total tutorials.

Graphic Jump Location

The process of learning surgery is long and arduous. It involves the acquisition of cognitive knowledge, judgmental skills, technical skills, and interpersonal skills. With the mandate to decrease the number of hours spent in the hospital by residents, we have become interested in techniques to shift some of this learning to nonwork, out-of-hospital hours. Specifically, cognitive knowledge has historically been an area of surgical learning relegated largely to off-hours time and left to the discretion of the individual resident. Cognitive knowledge thus seems to be a good area to try newer teaching techniques such as a Web-based teaching tool. Such a tool provides a method to access various teaching techniques (readings, video clips, simulated patients, practice tests) outside the confines of the normal work/educational environment and hours yet maintain the ability for faculty to monitor progress in usage.

Measuring the acquisition of this knowledge is difficult. Although the ABSITE is certainly an imperfect measure of cognitive knowledge, it is widely used as 1 measure of this acquisition, and it is standardized and reproducible.1 Thus, we chose to use it as our measure of whether a web-based education module would help improve our residents' knowledge base and improve their scores on this examination. We could find no relationship between BeST Resident usage and ABSITE score but did demonstrate a weak negative relationship between such usage and the change in residents' score between PGY1 and PGY2. Of concern is the low level of usage of the program by our residents. Residents stated that they felt that the web-based program was not as comprehensive nor as detailed as textbook review, and thus the majority of residents elected to use textbook review rather than BeST Resident as their primary method of study. Perhaps a different web-based tool would be of more value to our residents, but based on our experience, we cannot presently recommend such educational methods because it is not completely clear whether it is the content or the format of the program that residents found unappealing.

Scores on the ABSITE are commonly used as measures of cognitive knowledge of surgery, and they have been used as a means to evaluate various teaching methods. Previous studies have demonstrated a correlation of ABSITE scores to residents' study habits2 and to programmed reading.3,4 Although some studies have shown a relationship with conference attendance,5,6 others have not.7 Resident-performed preparation and presentation of educational lectures has been shown to improve scores on the Thoracic Surgery in Training Examination,8 reinforcing the concept that it is the individual resident's effort in self-education that most affects performance on the ABSITE. Previous authors have shown improvements in ABSITE score with work-hour limitations9: this was not the case for our residents, whose scores actually decreased following the implementation of work-hour limitations. Finally, we have previously found that residents with a history of poor performance on the ABSITE improved their scores by 50 standard points (half of a standard deviation) following attendance at the Association of Program Directors in Surgery basic science course (C.M.F., unpublished data, 2000). Unfortunately, expense makes this a very limited option.

Several limitations of our study need to be recognized. First, this is a rather small sample to judge the validity of a specific learning tool. A larger, cooperative study would be more reliable in showing the effectiveness of a web-based teaching tool. Second, the actual usage of BeST Resident by our residents was modest. Residents were not threatened with negative action for nonusage and thus were not truly “required” to complete the program or even to try it. Although this may seem an unconventional policy, it is in keeping with the educational philosophy of surgical residency: we provide the tools for education, but the resident must use them. Most of our residents felt that this was not a valuable tool for them, hence the low usage.

It is possible that had residents been required to complete the entire educational package of BeST Resident, then scores may have increased, but this is pure speculation. In fact, the individual in our residency who had the greatest total time (58.5 hours) and tutorials completed (100) had the greatest decrease in ABSITE score (−70 standard score points). In summary, we could not demonstrate a beneficial effect of a Web-based educational tool on cognitive knowledge of our residents as measured by ABSITE scores.

Correspondence: Charles M. Ferguson, MD, Massachusetts General Hospital, 15 Parkman St, Boston, MA 02114 (cferguson@partners.org).

Accepted for Publication: October 4, 2005.

Berry  RE ABSITE: uses and abuses. Curr Surg 1984;41329- 334
PubMed
Derossis  AMDa Rosa  DSchwartz  AHauge  LSBordage  G Study habits of surgery residents and performance on American Board of Surgery In-Training Examinations. Am J Surg 2004;188230- 236
PubMed Link to Article
Hirvela  ERBecker  DR Impact of programmed reading on ABSITE performance. Am J Surg 1991;162487- 490
PubMed Link to Article
De Virgilio  CStabile  BELewis  RJBrayack  C Significantly improved American Board of Surgery In-Training Examination scores associated with weekly assigned reading and preparatory examinations. Arch Surg 2003;1381195- 1197
PubMed Link to Article
Godellas  CVHuang  R Factors affecting performance on the American Board of Surgery In-Training Examination. Am J Surg 2001;181294- 296
PubMed Link to Article
Godellas  CVHauge  LSHuang  R Factors affecting improvement on the American Board of Surgery In-Training Exam (ABSITE). J Surg Res 2000;911- 4
PubMed Link to Article
Itani  KMRMiller  CCChurch  HMMcCollum  CH Impact of a problem-based learning conference on surgery residents' in training exam (ABSITE) scores. J Surg Res 1997;7066- 68
PubMed Link to Article
Bull  DAStringham  JCKarwande  SVNeumayer  LA Effect of a resident self-study and presentation program on performance on the Thoracic Surgery In-Training Examination. Am J Surg 2001;181142- 144
PubMed Link to Article
Barden  CBSpecht  MCMcCarter  MDDaly  JMFahey  TJ  III Effects of limited work hours on surgical training. J Am Coll Surg 2002;195531- 538
PubMed Link to Article

Figures

Place holder to copy figure label and caption
Figure 1.

Change in American Board of Surgery In-Training Examination (ABSITE) score vs total time on program.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Change in American Board of Surgery In-Training Examination (ABSITE) score vs total tutorials.

Graphic Jump Location

Tables

Table Graphic Jump LocationTable. Mean ABSITE Scores by Postgraduate Year

References

Berry  RE ABSITE: uses and abuses. Curr Surg 1984;41329- 334
PubMed
Derossis  AMDa Rosa  DSchwartz  AHauge  LSBordage  G Study habits of surgery residents and performance on American Board of Surgery In-Training Examinations. Am J Surg 2004;188230- 236
PubMed Link to Article
Hirvela  ERBecker  DR Impact of programmed reading on ABSITE performance. Am J Surg 1991;162487- 490
PubMed Link to Article
De Virgilio  CStabile  BELewis  RJBrayack  C Significantly improved American Board of Surgery In-Training Examination scores associated with weekly assigned reading and preparatory examinations. Arch Surg 2003;1381195- 1197
PubMed Link to Article
Godellas  CVHuang  R Factors affecting performance on the American Board of Surgery In-Training Examination. Am J Surg 2001;181294- 296
PubMed Link to Article
Godellas  CVHauge  LSHuang  R Factors affecting improvement on the American Board of Surgery In-Training Exam (ABSITE). J Surg Res 2000;911- 4
PubMed Link to Article
Itani  KMRMiller  CCChurch  HMMcCollum  CH Impact of a problem-based learning conference on surgery residents' in training exam (ABSITE) scores. J Surg Res 1997;7066- 68
PubMed Link to Article
Bull  DAStringham  JCKarwande  SVNeumayer  LA Effect of a resident self-study and presentation program on performance on the Thoracic Surgery In-Training Examination. Am J Surg 2001;181142- 144
PubMed Link to Article
Barden  CBSpecht  MCMcCarter  MDDaly  JMFahey  TJ  III Effects of limited work hours on surgical training. J Am Coll Surg 2002;195531- 538
PubMed Link to Article

Correspondence

CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 10

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...