Author Affiliations: Surgical Computing and Imaging Research Group, Department of Surgical Oncology and Technology, Imperial College, London, St Mary[[rsquo]]s Hospital, London, United Kingdom.
Objective assessment of surgical skill has recently been shown to be possible through the use of dexterity-based and video analysis systems. The aim of this study was to synchronize these 2 modalities to produce a comprehensive surgical assessment tool.
The Imperial College Surgical Assessment Device is a dexterity-based motion analysis device that has been developed in the Department of Surgical Oncology and Technology by the Surgical Computing and Imaging Research Group. Further advances to this system have been made to enable synchronized acquisition of hand kinematics and video from real procedures, and their concurrent analysis. To test the feasibility of the system, 10 laparoscopic cholecystectomies performed by 5 different surgeons on consenting patients were recorded. Analysis focused on the entire procedure and also on specific parts of the operation such as the clipping and cutting of the cystic duct and artery.
Dexterity analysis was performed using the objective measures of time path length, number of movements, velocities, and trajectories. Comparative analysis of a surgeon’s dexterity was carried out on the whole procedure and by using the synchronized zoom facility in the software. Kinematic signals revealed rapid changes in velocity caused by alternating between different instruments or occurring after complications such as bleeding.
This new motion analysis system has been shown to be an effective tool for the comprehensive assessment of operative procedures.
Recent high-profile cases of medical mishaps have increased public and subsequent political pressure of the need to objectively assess the competence of all surgeons.1 However, this has traditionally been performed in a subjective manner according to the master-apprenticeship model. This mode of assessment is known to be biased, and recent literature has commented on the fact that surgical assessment needs to be made more objective.2 Furthermore, Kaufman et al3 and Reznick et al4 have remarked on the need for objective feedback during the process of surgical training. Without feedback, learning is inadequate, as deficiencies in skills cannot be properly addressed.
The lack of feedback in the acquisition of surgical skill was clearly evident from the large number of unnecessary complications that occurred during the introduction of laparoscopic surgery.5 Without standardized and assessed methods of training, it is difficult to assure the public of the credentials of the surgeons operating on them. To achieve this, courses were introduced to teach the basic psychomotor skills required for laparoscopic surgery. Inexperienced surgeons had to perform tasks in the skills laboratory and were assessed subjectively by expert observers.6 More recently, there has been an influx of tools enabling the objective assessment of psychomotor skills acquisition in both open and laparoscopic surgery. These can be broadly divided into motion analysis systems such as the Imperial College Surgical Assessment Device (ICSAD)7 and Advanced Dundee Endoscopic Psychomotor Tester (ADEPT),8 and video-based assessment tools such as Objective Structured Assessment of Technical Skill (OSATS).9
The ICSAD makes it possible to quantifiably assess surgical dexterity by comparing the performance of experienced and novice surgeons on a simple surgical task. An experienced surgeon would be able to complete the task by making fewer movements and with a lower path length than an inexperienced trainee.10- 12 During training sessions, it would also be possible for the trainee to view a benchmark to be achieved in terms of dexterity. It is also possible to objectively assess dexterity for criteria-based progression from task performance such as targeting, transfer, and dissection in laparoscopic surgery, through to procedural performance that integrates these skills into a complete laparoscopic cholecystectomy.13
However, the ICSAD trace does not provide any information regarding the quality of the operation performed nor its outcome. Video-based assessment tools do enable qualitative assessment of skill14 in an objective manner though the process is labor intensive and must involve expert surgeons to perform the ratings. The development of technology that could integrate computerized dexterity analysis with video-based assessment would perhaps be a more efficient and comprehensive method of surgical skills assessment. It is the aim of this article to describe the development of such a tool and to confirm the feasibility of using it as a skills assessment device in the real operating theater.
The term “movement” is defined as a significant velocity peak, made up of an acceleration followed by a deceleration. In ICSAD, a movement is calculated from the time-stamped Cartesian positional data derived from the electromagnetic sensors, each one attached to the dorsum of the surgeon’s hands. Previous studies have validated this concept11,15- 17 in combination with qualitative assessment methods.
A movement was defined as an instant acceleration from one local static point followed by a deceleration to another static point. An a priori condition stated that each movement is counted when the physical movement high-peak velocity value exceeds a predefined velocity tolerance. Regardless of wrist movement or upper limb repositioning, it is the movements of the sensors placed on the dorsal surface of the hands that are recorded. As long as the a priori condition is satisfied, the parameter movement is counted at a specific time stamp.
In motion analysis of hand movement, it is essential to separate a deliberate movement from human hand tremor and measurement error, which form the background noise. Therefore, a calculation of the velocity threshold was integrated into the software. To set this tolerance, we measured the background noise by placing both trackers on the bench (not attached to hands) to record only noisy velocity values. A value of 12 mm/s was found to be the noise threshold. Moreover, the elimination of small insignificant movements due to hand tremor was achieved by attaching the receivers to hands, which were placed on the experimental bench. It was found that a value of 15 mm/s could adequately remove the background noise and hand tremor, discriminating insignificant from significant movements. In addition to the number of movements made by each hand, the system also calculated the time taken for the task, the distance traveled by each hand, and the average speed.
The ICSAD system consists of 3 components. A commercially available electromagnetic tracking system (Isotrak II; Polhemus Inc, Colchester, Vt) connected to a portable computer through a standard RS-232 (serial) port, independent motion acquisition software, and custom-made analysis software used for converting the positional data of the trackers into dexterity measures. The electromagnetic tracking system (Figure 1) consists of an electromagnetic field generator and 2 sensors that are attached to the dorsum of the surgeon’s hands.
Imperial College Surgical Assessment Device (ICSAD). The ICSAD system consists of 3 components. A commercially available electromagnetic tracking system (Isotrak II; Polhemus Inc, Colchester, Vt) connected to a portable computer through a standard RS-232 (serial) port, an independent motion acquisition software, and bespoke analysis software for converting the positional data of the trackers into dexterity measures. The Isotrak II system consists of an electromagnetic field generator and 2 sensors that were attached to the dorsum of the surgeon’s hands.
The ICSAD device was integrated into a large-scale project involving the assessment of model-based tasks and real procedures carried out using robotic, conventional laparoscopic, as well as open surgery. New software, the Robotic Video and Motion Analysis Software (ROVIMAS), was developed to provide concurrent qualitative and quantitative functionalities.18 This is based on the same framework as the original ICSAD analysis software and enables compatibility when comparing old and new data sets. All ICSAD components (acquisition and data analysis software) were integrated into ROVIMAS, with data retrieval and analysis automatically performed by the new software. Additional visual functionality includes Cartesian coordinates and distance and velocity graphs, as well as hand direction, polar graph, and trajectories. Moreover, a new efficient graphical method called “density analysis” provides information regarding which part of the surgical scene was the focus of activity, the active scene surface, and the total task surface.
The zooming functionality is of significant importance as it allows isolation of various parts of procedures to be broken down according to the particular assessment process, saving valuable time. Each time zooming occurs, the number of movements and of hands’ path length are shown and recalculated with all graphs being updated accordingly.
The ROVIMAS software also includes a video frame–capturing facility (Figure 2). Motion and video data are captured in a time-synchronous manner achieving synchronization accuracy of less than 40 milliseconds.19 Video frames are compressed in real time using the Microsoft VKI codecs (Microsoft Corporation, Redmond, Wash) with a pixel size of 320×240. It is also possible to capture single full-sized bitmap pictures while recording a task.
Imperial College Surgical Assessment Device (ICSAD) data and video acquisition component. On the left side information regarding the commercially available electromagnetic tracking system (Isotrak II; Polhemus Inc, Colchester, Vt) is displayed along with the tracking status. On the right the video capturing window displays the endoscopic surgical scene. Both motion and video streams are recorded using the same time stamp succeeding in perfect synchronization.
The main advantage of this system is that through video synchronization it is possible to assess the operation using the ROVIMAS-integrated video player (Figure 3) and to concurrently study the corresponding motion signal trace. An application that identifies high-velocity movements and associates them with erroneous movements was developed previously. A playback line bar has been implemented on the signal trace to enable searching for a particular part of the procedure, and the dexterity graph will change in synchrony. Alternatively, one can zoom into the desired signal regions and the video will play only the required part and not the whole operation. ROVIMAS also uses the same concept of motion and video synchronization when data are acquired from the da Vinci telemanipulator system (Mountain View, Calif).
Snapshot of the Robotic Video and Motion Analysis Software (ROVIMAS) showing the crown pattern during a laparoscopic cholecystectomy case in which each pair of 2 high-velocity peaks represents extraction-insertion of the scissors-stapler instrument. Users can browse these data using the “browsing bar” and watch the corresponding video frames from the video player. P indicates path.
The ROVIMAS system was used to record and analyze 10 laparoscopic cholecystectomies performed by 5 surgeons: 1 experienced surgeon (A.D.) (>100 laparoscopic cases) and 4 inexperienced surgeons (<50 cases) between October 1, 2003, and February 28, 2004. Ethics approval for this study was obtained from the local research ethics committee.
Our intention was not to produce assessment results from real procedures but rather to explore the feasibility of using this system in the operating room. This was to ensure that data could be reliably recorded for the whole and parts of the procedure, that the operative environment (instruments, anesthetic equipment, diathermy, and others) would not cause signal disturbance, and, most importantly, that the electromagnetic field would be safe from a patient’s perspective. Data recording began with insertion of the endoscopic camera into the abdominal cavity and terminated once the gallbladder had been extracted from the abdomen.
Ten laparoscopic cholecystectomies were recorded, and accurate data were available for all operations. Measurements illustrating the entire procedure and breakdown into cystic duct dissection/clipping and cutting from 10 recorded cholecystectomies are given in the Table. Objective data parameters (number of movements, path length, and time taken) were calculated for entire operations and for dissecting/clipping and cutting of the cystic duct. Left movement (LM) and right movement (RM) represent the left-hand and right-hand movements, respectively. Left-path length (LPL) and right-path length (RPL) characterize the left-hand and right-path hand length, respectively. Additionally, it would have been possible to perform qualitative analysis on the whole procedure, or selected parts, using a validated objective qualitative method of assessment, such as OSATS, and correlate it with the quantitative data.
The recorded procedures presented different levels of difficulty. It was our intention to display the widest possible range of operative difficulty possible, thus demonstrating a wide variety of results. Surgeons had to deal with complications such as acute cholecystitis or a large left lobe of the liver. The combination of operative difficulty and surgical expertise demonstrates a variation in operative duration of between 735.1 and 3080.8 seconds. Notwithstanding duration as a significant factor, it cannot be used as a stand-alone criterion to determine surgical dexterity.
Figure 3 shows a snapshot of the dissection of the cystic duct. Data from the red signal correspond to the right hand while those from the green signal correspond to the left hand. The “crown pattern” illustrated can be recognized in all procedures, where each pair of 2 high-velocity peaks represents extraction-insertion of the scissors-stapler instrument. By zooming in only on in that part between 110 and 311.4 second, the number of movements was found to be 240 for the left hand and 380 for the right hand, indicating greater activity for the right hand (dissection part of the procedure). Figure 4 shows the clipping and cutting of the cystic duct where the time spent was 13 and 17 seconds, respectively. For the right hand, the numbers of movements were 29 and 32 and the path length was 0.898 and 1.137 m.
Velocity data (right hand) for clipping (A) and cutting (B) the cystic duct as appear in the Robotic Video and Motion Analysis Software (ROVIMAS). Screenshot shows higher hand activity during cutting (denser velocity peaks) compared with clipping that requires more precise movements (wider peaks). P indicates path.
Listed in the Table is analysis of data from cases 1 and 6 in which the 2 surgeons had similar operative times, though they differed in their number of movements for each hand. In case 1, the experienced surgeon equally exploited his left and right hands with 811 and 911 (left/right) movements, resulting in a deficit of 100 movements (right/left movements) and a rate of 0.98 and 1.10 movements per second, respectively. The ratio of movements per second was used to eliminate the time duration caused by various reasons during operation. The parameter movement deficit was used to compare the activity of the right hand with that of the left hand. Low ratios are preferred as they indicate greater bimanual dexterity. The inexperienced surgeon performing the surgery in case 6 made many more movements with the right hand that resulted in a deficit of 293 movements and a rate of 1.29 movements per second for the right hand.
During dissection of the cystic duct the experienced surgeon made 0.51 movement per second and even fewer during clipping and cutting of the duct (0.39 movements per second). The junior surgeon produced a similar performance of dexterity during cystic duct dissection (0.69 movements per second), though was making more movements during clipping and cutting of the duct (0.84 movements per second).
The surgeon in case 9 had to deal with a case of acute cholecystitis, and the entire procedure was completed in 3080.8 seconds. Despite the difficulty, low rates of 0.39 and 0.89 movements per second (left/right) were still achieved.
To develop a competency-based system of progression in surgical training, it is necessary to incorporate validated tools of assessment within the program. This pilot study has shown that it is feasible to synchronize dexterity-based and video analysis systems into 1 platform and that this may lead to greater efficiency during the assessment of surgical skill. The ultimate goal is, of course, to completely automate assessment in surgery, though this is a distant notion. Rosen et al20,21 have discussed the introduction of probabilistic methods to surgical skills assessments through the use of mathematical models, with encouraging preliminary results.
However, one of the main problems with objectification of any type of surgical skills assessment is the complexity of the procedure and variability in different surgical styles and approaches. That is why it is necessary for expert surgeons to analyze videos of the procedure, ensuring each surgeon is progressing in a safe and purposeful manner. Current distinctions between experienced and novice surgeons involve numbers of procedures performed, though further validation of objective assessment tools can enable the use of dexterity and quality parameters to discriminate between surgeons. It may then be possible to define progression in terms of improvement in dexterity and quality, rather than on the number of procedures performed, or complications experienced.
In terms of concurrent hand motion and video analysis as a method of approaching surgical skills assessment, hand movement analysis has been used extensively in other areas of science, including engineering and military applications. In hand sign recognition, hand tracking is used to record path sequences of spatial data and to train decision-making systems. These can be integrated into home appliances and execute commands according to hand signs.22,23 A number of studies have been conducted to track parts of the human body and investigate the relationship between kinematics and human anatomy.24 For example, an electromagnetic device has been used to measure lumbar spinal and pelvic motion of elite level rowers on a rowing ergometer to quantify and discriminate between good and bad rowing styles.25 Further studies may make it possible to define “ideal” movement sequences to improve performance.
Hand motion analysis has also been applied in robotics to understand and mimic natural human hand movements,26- 28 aeronautics,29,30 biomechanics,31 and handwritten recognition.32,33 Movement tracking sensors have been applied to digitally record handwritten scripts and recognize their correspondent alphanumeric characters using probabilistic stochastic approaches such as hidden Markov models and neural networks.
Data retrieval by this system is reliable and instantaneous and may be able to reduce the time taken for video-based assessment of technical skills by focusing in on key areas of the operation. Furthermore, parts of the operation that have a more movements-per-unit time may indicate a complication such as bleeding. Our preliminary results have demonstrated the potential use of this system as a promising tool in a real environment. It has been shown that physical and technical constraining factors in an operating theater do not affect data acquisition or data analysis. Therefore, it is our belief that this application should continue to be explored, thus pursuing the maximal potential of this technology.
Correspondence: Aristotelis Dosis, MSc, Surgical Computing and Imaging Research Group, Department of Surgical Oncology and Technology, Imperial College London, 10th Floor, Queen Elizabeth, Queen Mother Wing, St Mary’s Hospital, London W2 1NY, United Kingdom (email@example.com).
Accepted for Publication: September 21, 2004.
Funding/Support: The BUPA Foundation, London, England, contributed financially to this research.
Acknowledgments: We thank all of the past research fellows of the department for their contribution to the development and validation of ICSAD.
Thank you for submitting a comment on this article. It will be reviewed by JAMA Surgery editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.
Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.
* = Required Field
Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.
Register and get free email Table of Contents alerts, saved searches, PowerPoint downloads, CME quizzes, and more
Subscribe for full-text access to content from 1998 forward and a host of useful features
Activate your current subscription (AMA members and current subscribers)
Purchase Online Access to this article for 24 hours
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 44
Customize your page view by dragging & repositioning the boxes below.
and access these and other features:
Enter your username and email address. We'll send you a link to reset your password.
Enter your username and email address. We'll send instructions on how to reset your password to the email address we have on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.