Knowing the operative game plan: a novel tool for the assessment of surgical procedural knowledge
=================================================================================================

* Jacques Balayla
* Simon Bergman
* Gabriela Ghitulescu
* Liane S. Feldman
* Shannon A. Fraser

## Abstract

**Background:** What is the source of inadequate performance in the operating room? Is it a lack of technical skills, poor judgment or a lack of rocedural knowledge? We created a surgical procedural knowledge (SPK) assessment tool and evaluated its use.

**Methods:** We interviewed medical students, residents and training program staff on SPK assessment tools developed for 3 different common general surgery procedures: inguinal hernia repair with mesh in men, laparoscopic cholecystectomy and right hemicolectomy. The tools were developed as a step-wise assessment of specific surgical procedures based on techniques described in a current surgical text. We compared novice (medical student to postgraduate year [PGY]-2) and expert group (PGY-3 to program staff) scores using the Mann–Whitney *U* test. We calculated the total SPK score and defined a cut-off score using receiver operating characteristic analysis.

**Results:** In all, 5 participants in 7 different training groups (*n* = 35) underwent an interview. Median scores for each procedure and overall SPK scores increased with experience. The median SPK for novices was 54.9 (95% confidence interval [CI] 21.6–58.8) compared with 98.05 (95% CP 94.1–100.0) for experts (*p* = 0.012). The SPK cut-off score of 93.1 discriminates between novice and expert surgeons.

**Conclusion:** Surgical procedural knowledge can reliably be assessed using our SPK assessment tool. It can discriminate between novice and expert surgeons for common general surgical procedures. Future studies are planned to evaluate its use for more complex procedures.

All surgical procedures share, among others, 3 fundamental components: a cognitive factor, a technical element and a judgment component. The cognitive factor consists of the knowledge of the theoretical steps of the procedure in question. The technical element takes into account the theoretical steps and translates them into the performance of the operation. The judgment component comes from the procedural component and from surgical experience, allowing a surgeon to determine the appropriate course of action for patients on a case-by-case basis. From this paradigm emerges a fundamental question: What is the source of inadequate performance in the operating room (OR)? Is it a lack of technical skills, poor judgment, or a lack of knowledge and familiarity with the procedure?

Whereas technical examination measures have been validated,1–6 procedural knowledge assessments in surgery have not been well documented. With advances in technology, changes in surgical equipment, the development of new surgical techniques and the rotation of new trainees in surgical departments, assessing a surgeon’s procedural competence becomes a valuable tool to ensure adequate performance in the OR. Furthermore, in the current climate of diminishing working hours and shorter training periods, the surgical profession must address the complex issue of how to teach surgery and how to assess an individual’s competence.7

An ideal, single objective measurement tool for the assessment of technical and procedural skills continues to elude us.8 Indeed, while trying to identify the nontechnical skills required by surgeons in the OR and assessing behavioural marker systems that have been developed for rating surgeons’ nontechnical skills, Yule and colleagues9 concluded that further work is required to develop a valid taxonomy for the appraisal of an individual surgeon’s skills.

Adequate acquisition of surgical competence throughout training is fundamental for successful patient care. If technical skills are acquired at a rate proportional to surgical exposure and education, then the knowledge of the steps of the surgical procedure should theoretically increase as well, and we expect that this ability should be similar among surgical trainees of the same rank. The purpose of the present study was to develop an operative knowledge assessment tool to quantify the trend in knowledge acquisition of the procedural steps and to establish the tool’s reliability to investigate the practical applications it may bring to future educational research.

## Methods

We defined surgical procedural knowledge (SPK) as “the ability to enumerate the steps of a given surgical procedure in the order that they occur in the OR.” Based on this definition, we developed an SPK assessment tool for 3 different common general surgical procedures: inguinal hernia repair with mesh in men, laparoscopic cholecystectomy and open right hemicolectomy (Appendix 1, available online at [www.cma.ca/cjs](http://www.cma.ca/cjs)).

Education behaviourist theory suggests that competence is best assessed by precise measures of performance generally documented by checklists.10 Therefore, our SPK tools were developed as step-wise assessments of each operation based on Fisher’s *Mastery of Surgery*, 5th edition.11 The SPK tool lists all of the steps of each procedure, from opening to closure in chronological order and in checklist form. These checklists are then used as a marking grid by each investigator to grade participants.

We interviewed medical students, residents and staff in McGill University’s general surgery program about the SPK tools, asking them to enumerate in chronological order the steps that they were familiar with for each operation. A score was subsequently assigned per procedure, and errors of both commission (unnecessarily added or incorrectly ordered steps) and omission (missed steps) were discounted from the final score. To evaluate inter-rater reliability, interviews were recorded and scored by 2 observers separately. One observer scored the participant over the course of a face-to-face interview. To reduce performance anxiety, time constraints, distractions, overall stress and other potential confounding biases, this observer was chosen from outside of the surgical program and was previously unfamiliar to the study participants. The second observer scored the interview while listening to its audio recording at a later time. A total SPK score was calculated by adding the total score of each procedure out of 100 and dividing by 3.

### Statistical analysis

The SPK scores for “novice” (medical student, residents in postgraduate years [PGY] 1 and 2) were compared with “expert” surgeons (residents in PGY 3–5, program staff) using the Mann–Whitney *U* test. Finally, an SPK cut-off score was defined by receiver operating characteristic (ROC) analysis of SPK scores between novice and expert surgeons.

## Results

In all, 5 participants in 7 different training groups (*n* = 35) underwent an interview assessment. Median scores for each procedure and overall SPK scores increased with surgical experience (Table 1, Fig. 1). In addition, variability (standard deviation [SD]) within training levels decreased as surgical experience increased (Table 1). Among novice surgeons, inguinal hernia repair assessments generated the lowest scores. Laparoscopic cholecystectomy scores were consistently higher in every training group. Residents in PGY-5, fellows and program staff demonstrated perfect scores across all procedures (Fig. 1).

![Fig. 1](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/55/4_Suppl_2/S158/F1.medium.gif)

[Fig. 1](http://canjsurg.ca/content/55/4_Suppl_2/S158/F1)

Fig. 1 
Overall and procedure-specific median surgical procedural knowledge (SPK) scores. PGY = postgraduate year; SD = standard deviation.

View this table:
[Table 1](http://canjsurg.ca/content/55/4_Suppl_2/S158/T1)

Table 1 
Procedure-specific and overall surgical procedural knowledge scores

Overall, the median SPK for novices was 54.9 (95% confidence interval [CI] 21.6–58.8) compared with 98.05 (95% CI 94.1–100) for experts (*p* = 0.012). The SPK cutoff score of 93.1, calculated by ROC analysis, discriminates between novice and expert surgeons. The frequency distribution of SPK scores for novice and expert surgeons is illustrated in Figure 2, with the ROC cut-off score demonstrating the discrimination between the groups. The interclass correlation coefficient for the total SPK was 0.99 (95% CI 0.98–0.995) between the 2 observers.

![Fig. 2](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/55/4_Suppl_2/S158/F2.medium.gif)

[Fig. 2](http://canjsurg.ca/content/55/4_Suppl_2/S158/F2)

Fig. 2 
Frequency and receiver operating characteristic cut-off score for novice versus expert overall surgical procedural knowledge scores.

## Discussion

Addressing the question of SPK is a key component in the development of curricula and assessment tools for surgical trainees. Given that an ideal, single objective measurement tool for the assessment of surgical procedural skills continues to elude the surgical community, we took on this project to provide some insight into this question. Spencer12 reported that 75% of the important events in an operation are related to making decisions, whereas only 25% are related to manual skills. This is not a new concept: Frederick Treves wrote in 1891, “The actual manipulative part of surgery requires no very great skill and many an artisan shows infinitely more adeptness in his daily work [...] It is in the mental processes involved in an operation that not a few fail. There is some lack in […] the capacity for forming a ready judgment, which must follow each movement of the surgeon’s scalpel.”13

Our results confirm our expectation that SPK should increase proportionally with level of training and surgical experience. These particular procedures were chosen because of their inherent frequency in the OR, as even entry-level residents are exposed to them on a regular basis. After all, barring breast biopsies and appendectomies, these are the most common operations performed at our centre. The steps of an open appendectomy were chosen as examples for the required task at hand for the residents before engaging in this exercise and, despite being more common than an open hemicolectomy, could thus not be used as part of the assessment tool.

Scores for specific procedures and overall SPK scores increased with level of training, supporting the construct validity of our assessment tool. In addition, with advancement in surgical training, within-group variability in overall SPK scores decreased, indicating a levelling of cognitive ability across members of the same academic rank with increased surgical education and exposure. Medical students displayed similar standard deviation estimates as residents in PGY-2 simply owing to poor performance as a whole cohort, thus displaying low variability among individual results. Finally, by PGY-5, surgeons displayed perfect SPK scores. The procedural knowledge for these basic surgical procedures plateaus, as one would hope, early in surgical training. On our assessment tool, residents in PGY-3 displayed similar scores to those of the staff surgeons. Whether this association holds true for more advanced procedures and whether this knowledge plateau will be maintained at the PGY-3 level as work-hour restrictions impact surgical training remains to be investigated.

There are potentially 2 practical applications of the SPK tool. The first is for curricula planning. The SPK tool can identify and address issues with particular procedures that are not being well taught in a program (e.g., inguinal hernia procedures performed by junior residents). As such, should surgical education curricula need to be reassessed, our tool may offer insight with respect to the current status of procedural knowledge of trainees of a given surgical centre and provide areas of focus on which efforts can be applied. Second, normative SPK scores could be established with additional data, and then the tool could be applied at an individual level, allowing identification of trainees who may need remedial attention to develop this aspect of surgical knowledge.

We believe that ours is the first study to attempt to assess the procedural component of surgical knowledge separately from basic clinical knowledge and surgical judgment or technical knowledge among surgical trainees.1–6 We hypothesize that procedural knowledge should be similar among surgical trainees of the same rank but acknowledge that substantial variance may exist among surgical residents in how they acquire procedural skills. The purpose of our SPK tool is to measure that procedural knowledge once acquired, regardless of how it was obtained. Inherent heterogeneity in operative exposure may also exist within a program with multiple sites and among teaching staff, which may affect the acquisition of SPK.

### Limitations

The following limitations must be acknowledged. The sample size of this pilot study was small and restricted to 1 institution. In addition, the operations for which the SPK tool was developed are common, and this may have masked procedural knowledge among trainees for less common, more advanced procedures. Finally, owing to time constraints, our investigation was conducted as a cross-sectional study rather than longitudinally. Thus, we were not able to assess individual evolution in SPK; however, this would be interesting to evaluate in future studies.

## Conclusion

Whereas several studies have addressed measurements of technical knowledge among surgical trainees,1–6 no other study, to our knowledge, has attempted to evaluate their procedural ability as defined here. The SPK tool was used to measure procedural knowledge at each recognized level of surgical training, including data from medical students, and attending staff, which provided information about our transformation throughout surgical training. Finally, as the SPK tool is retrieved from a well established surgical textbook, it is a standardized tool that can be used without alteration at other surgical centres, thus allowing for comparison with other training sites or among different generations of surgeons within the same program.

The true purpose of creating a tool that measures the procedural knowledge of surgical trainees is not to discriminate against and decrease OR time for trainees who score poorly, but rather to accurately identify whether SPK is an area where we need to focus our resources to improve surgical education. Given the results of our pilot study, we intend to pursue future projects to correlate these results with technical in-training examination scores and validated intraoperative, procedure-specific performance checklist scores. In addition, validation of the SPK tool across multiple institutions and for more complex procedures is required.

The current state of surgical nontechnical skills assessment is in its infancy,9 and our SPK tool may provide a reliable instrument to evaluate how well we are teaching our trainees in the face of evolving training schedules and curricula while distinguishing between novice and expert surgeons for basic procedures.

## Footnotes

*   Abstract presented at the Canadian Surgery Forum 2010, Québec, Que., Sept. 2–5, 2010.

*   **Competing interests:** None declared.

*   **Contributors:** All authors contributed to study design, reviewed the article and approved its publication. J. Balayla and S.A. Fraser acquired and analyzed the data, and wrote the article.

*   Accepted November 16, 2011.

## References

1.  Palter VN, MacRae HM, Grantcharov TP.Development of an objective evaluation tool to assess technical skill in laparoscopic colorectal surgery: a Delphi methodology.Am J Surg 2011;201:251–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.amjsurg.2010.01.031&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=20832048&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 

2.  Lodge D, Grantcharov T.Training and assessment of technical skills and competency in cardiac surgery.Eur J Cardiothorac Surg 2011;39:287–93.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.ejcts.2010.06.035&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=20724174&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 

3.  Leblanc F, Delaney CP, Neary PC, et al.Assessment of comparative skills between hand-assisted and straight laparoscopic colorectal training on an augmented reality simulator.Dis Colon Rectum 2010;53:1323–7.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=20706077&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 

4.  van Hove PD, Tuijthof GJM, Verdaasdonk EGG, et al.Objective assessment of technical surgical skills.Br J Surg 2010;97:972–87.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/bjs.7115&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=20632260&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000279116000002&link_type=ISI) 

5.  Zheng B, Hur HC, Johnson S, et al.Validity of using fundamentals of laparoscopic surgery (FLS) program to assess laparoscopic competence for gynecologists.Surg Endosc 2010;24:152–60.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=19517182&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 

6.  Peters JH, Fried GM, Swanstrom LL, et al.Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery.Surgery 2004;135:21–7.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0039-6060(03)00156-9&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=14694297&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000187715900004&link_type=ISI) 

7.  Thomas WE.Teaching and assessing surgical competence.Ann R Coll Surg Engl 2006;88:429–32.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=17002841&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 

8.  Tang B, Hanna GB, Carter F, et al.Competence assessment of laparoscopic operative and cognitive skills: objective structured clinical examination (OSCE) or observational clinical human reliability assessment (OCHRA).World J Surg 2006;30:527–34.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1007/s00268-005-0157-z&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=16547622&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000236618600007&link_type=ISI) 

9.  Yule S, Flin R, Paterson-Brown S, et al.Nontechnical skills for surgeons in the operating room: a review of the literature.Surgery 2006;139:140–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.surg.2005.06.017&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=16455321&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000235358800002&link_type=ISI) 

10. Bhatti NI, Cummings CW.Competency in surgical residency training: defining and raising the bar.Acad Med 2007;82:569–73.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/ACM.0b013e3180555bfb&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=17525542&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000247025600008&link_type=ISI) 

11. 1.  Fischer JE, 
    2.  Bland KI
    
    , eds (2006) Mastery of surgery (Lippincott, Williams & Wilkins, Philadelphia (PA)), 5th ed.
    
    

12. Spencer F.Teaching and measuring surgical techniques: the technical evaluation of competence.Bull Am Coll Surg 1978;63:9–12.
    
    

13. Hall JC, Ellis C, Hamdorf J.Surgeons and cognitive processes.Br J Surg 2003;90:10–6.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/bjs.4020&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access\_num=12520568&link_type=MED&atom=%2Fcjs%2F55%2F4_Suppl_2%2FS158.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000180629700004&link_type=ISI)