The effectiveness of video feedback in the acquisition of orthopedic technical skills

doi:10.1016/j.amjsurg.2003.12.011

The American Journal of Surgery

Volume 187, Issue 3, March 2004, Pages 427-432

https://doi.org/10.1016/j.amjsurg.2003.12.011 Get rights and content

Abstract

Background

The addition of video feedback to bench model training offers residents the opportunity to see themselves perform a surgical task. Videotaped feedback therefore promotes self-evaluation, a critical learning skill, and also has the potential to influence how a resident executes a skill once they have had the opportunity to see themselves perform the task.

Methods

Twenty-nine surgical residents were video recorded while performing three technical skills. They then were randomly assigned to receive either no feedback, video feedback alone, or video feedback with the help of an expert, an orthopedic surgeon. The surgical task was then repeated. Orthopedic surgeons evaluated the videotapes using the global rating scale and technical checklist form.

Results

One-way between-subject analysis of variance comparing the pretest and post-test difference scores on three different measures for each of the three tasks revealed no statistically significant differences. After controlling for rater variance, the global rating scores across the three surgical tasks did not reveal any statistically significant differences.

Conclusions

This study failed to demonstrate an improvement in technical skills based on utilization of video feedback.

Section snippets

Methods

Twenty-nine orthopedic surgical residents across five levels from postgraduate year 1 (PGY1) to PGY5 were recruited. Individual consent was obtained from all participants. Ethics approval was obtained from the Research Ethics Board at Mount Sinai Hospital, Toronto.

Three surgical procedures were taught, practiced and assessed using models in a laboratory environment, over a 4-hour period. An orthopedic surgeon in a standardized fashion demonstrated the performance of each task. Each task was

Results

Twenty-nine of a total of 44 orthopedic surgical residents were available to participate in this study. Of the 15 residents who did not take part in the study, 6 were out of town and 9 had conflicts in scheduling that prevented them from participating. Of the 29 residents participating the breakdown across the five academic year levels was as follows: 8 from PGY1, 6 from PGY2, 7 from PGY3, 3 from PGY4, and 5 from PGY5. The residents were proportionally distributed by year level across the three

Comments

Our study attempted to identify whether videotaped feedback would be a useful addition to bench model teaching as measured by change in the performance on the task. Our findings, revealed no differences attributable to the use of videotape in affecting surgical performance on bench model tasks. The lack of a difference between feedback interventions may have been related to several factors that included: combining experienced residents with junior residents, a small number of residents

References (20)

R. Reznick et al.
Testing technical skill via an innovative “bench station” examination
Am J Surg
(1997)
D.J. Anastakis et al.
Assessment of technical skills transfer from the bench training model to the human model
Am J Surg
(1999)
M.F. Stranc et al.
Video assessment of surgical technique
Br J Surg
(1991)
D.A. Rogers et al.
Computer-assisted learning versus a lecture and feedback seminar for teaching a basic surgical technical skill
Am J Surg
(1998)
D.A. Rogers et al.
The impact of external feedback on compter-assisted learning for surgical technical skill training
Am J Surg
(2000)
Barnes RW. Surgical handicraft: teaching and learning surgical skills. Am J Surg...
P. Winckel et al.
Reliability and construct validity of a structured technical skills assessment form
Am J Surg
(1994)
Heppell J, Beauchamp G, Chollet A. Ten-year experience with a basic technical skills and perioperative management...
J.A. Martin et al.
Objective structured assessment of technical skill (OSATS) for surgical residents
Br J Surg
(1997)
G. Regehr et al.
Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination
Acad Med
(1998)

There are more references available in the full text version of this article.

Cited by (43)

Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation
2022, Journal of Surgical Education
Citation Excerpt :
The intervention in this study randomized the group of 29 residents into 3 groups, each undergoing a single 15-minute coaching session. Any positive effect may have been too small to detect with the limited sample size.37 The studies that demonstrate a positive effect of VBC were a combination of randomized, non-randomized, and observational studies that evaluated the effect of a VBC curriculum.
This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation.
A search was performed on Scopus and PubMed for the terms “operation,” “operating room,” “surgery,” “resident,” “house staff,” “graduate medical education,” “teaching,” “coaching,” “assessment,” “reflection,” “camera,” and “video” on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review.
Original, peer-reviewed studies focused on VBC or assessment.
VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room.
VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.
Asynchronous Unsupervised Video-Enhanced Feedback As Effective As Direct Expert Feedback in the Long-Term Retention of Practical Clinical Skills: Randomised Trial Comparing 2 Feedback Methods in a Cohort of Novice Medical Students
2018, Journal of Surgical Education
Citation Excerpt :
This study was unable to recruit the minimum number of candidates as suggested by the power calculation (46), with many students citing they could not commit to each of the study days due to prior personal or educational obligations and limitations in the research team’s access to recruiting students. Recruitment challenges appear to be a problem in many similar educational studies of this kind where participant numbers are generally low; however, the number of students recruited in this study is comparable to other medical education interventions and significant results were achieved for this to be considered a meaningful study.6,8,9,13–15 Two prior studies demonstrate improvement in performance of a clinical or surgical skill in the group that received video-assisted feedback compared to standard lecture feedback.6,8
To determine the degree of practical clinical skills’ retention over a 7-week period after receiving either video-enhanced direct expert feedback (DEF) or asynchronous unsupervised video-enhanced feedback (UVF).
A prospective single-blinded randomized trial was conducted over a 7-week period with novice medical students. Following a generic introduction, PowerPoint presentation and live demonstration of intravenous cannulation was given by an expert, and candidates performed the same task in isolation while being recorded and were randomized to receive either DEF or UVF. Further, 20 students were randomized to receive UVF and 22 to receive DEF. Candidates returned to repeat the video-recorded task at week 1, 4, and 7, with no further feedback provision on these occasions. Performances were fully anonymized and independently marked by 2 expert assessors.
Newcastle University, Medical School, England, United Kingdom.
A total of 42 novice medical students from the preclinical years were selected, with no prior experience of intravenous cannulation.
No significant difference existed between demographics of either cohort. Good between-assessor score correlation was noted, with an intraclass correlation coefficient (ICC) of 0.89. The DEF arm significantly improved from their prefeedback performance at Day 1 on repeating the skill a week later (p < 0.0001); this improved score was maintained throughout the duration of the study. The UVF arm showed a nonsignificant improvement; however, there was no significant difference in the scores between the UVF and the DEF groups at any point in the trial.
Video-technology may play a significant role in the provision of feedback for practical skills. The ability for experts to provide remotely delivered but individualized feedback can be an efficient use of valuable resources, while students being able to self-assess their own performance alongside an expert video have the potential to provide students with an excellent opportunity to learn clinical skills without requiring intensive educator involvement.
Instant replay: Evaluation of instant video feedback in surgical novices for a laparoscopic gallbladder dissection
2018, American Journal of Surgery
Citation Excerpt :
There was no opportunity for additional practice trials or ongoing feedback. Previous studies using single episodes of video-based feedback have also failed to demonstrate an increase in post-test performance.8,18 Even with a short (20 min) practice period to incorporate feedback, Wittler et al. were unable to demonstrate a significant improvement in ultrasound guided central venous catheter insertion after a single episode of video-based feedback compared to verbal feedback.18
Athletes often use video to improve their technique. We hypothesized that surgical novices given feedback using video-replay would outperform surgical novices given verbal feedback in the performance of a laparoscopic task.
Our study used a prospective, randomized control design. The surgical task involved the laparoscopic dissection of a pig gallbladder. Our participants performed a dissection, pre- and post-traditional or video feedback. Each recording was independently scored by two staff surgeons using the previously validated rating tools.
There was no significant difference between video feedback or traditional feedback groups in their mean overall or task specific scores. Both traditional and video-feedback groups had a trend towards improved performance post-feedback.
No significant difference in performance by both our global assessment metrics or task-specific metrics was observed. Video feedback requires further study to investigate its impact on surgical training.
Personalized video feedback improves suturing skills of incoming general surgery trainees
2018, Surgery (United States)
The American Board of Surgery encourages graduating medical students to prepare for surgical residency before day 1. We sought to determine the impact of personalized video feedback on an advance preparation task.
We conducted a nonrandomized study comparing video feedback versus no feedback. We sent incoming surgical interns a preparatory package 2 months before starting residency. Trainees video-recorded themselves performing a subcuticular wound closure, 3 times at 3-week intervals, and submitted these for appraisal. A staff surgeon provided personalized feedback on each video as a narrated voiceover. The voiced-over videos were then returned to trainees. We compared performance (time and completion rate) on suturing in a multistation assessment against residents from the previous year (no-feedback group).
The feedback group had a higher completion rate for the suturing assessment than the no-feedback group (23/28 [82%] vs. 8/27 [30%], P < .0001). The feedback group also completed the suturing station at a faster rate than those without feedback (hazard ratio 4.9 [95% confidence interval (CI): 2.2,11.2), P < .0001). Global rating scores were significantly higher for the feedback group (mean difference [5-point scale] = 0.7 [95% CI: 0.3, 1.1]). However, Objective Structured Assessment of Technical Skills scores indicated no significant difference between groups (mean difference [5-point scale] = 0.3 [95% CI: 0.0, 0.6]). Within the feedback group, we found significant improvement from baseline to final performances (mean difference = 109 seconds [95% CI: 79, 140]).
Personalized narrated feedback as part of a home-based advance preparation package for incoming residents is associated with higher performance on early objective assessments.
Randomized trial to assess the effect of supervised and unsupervised video feedback on teaching practical skills
2015, Journal of Surgical Education
Citation Excerpt :
They concluded that expert feedback is essential to maximize candidate improvement.22 However, others compared no feedback, video-assisted expert feedback, and video with self-review (which included the trainee watching their own performance with no feedback at all) in orthopedic trainees and demonstrated no improvement in performance in either feedback group when compared with the no feedback control.23 Feedback in any training domain is said to be either internal or external.24
Feedback is a vital component of the learning process; however, great variation exists in the quality, quantity, and method of delivery. Video feedback is not commonly used in the teaching of surgical skills. The aim of this trial was to evaluate the benefit of 2 types of video feedback—individualized video feedback (IVF), with the student reviewing their performance with an expert tutor, and unsupervised video-enhanced feedback (UVF), where the student reviews their own performance together with an expert teaching video—to determine if these improve performance when compared with a standard lecture feedback.
A prospective blinded randomized control trial comparing lecture feedback with IVF and UVF was carried out. Students were scored by 2 experts directly observing the performance and 2 blinded experts using a validated pro forma. Participants were recorded on video when performing a suturing task. They then received their feedback via any of the 3 methods before being invited to repeat the task.
A total of 32 students were recruited between the 3 groups. There was no significant difference in suturing skill performance scores given by those directly observing the students and those blinded to the participant. There was no statistically significant difference between the 2 video feedback groups (p = 1.000), but there was significant improvement between standard lecture feedback and UVF (p = 0.047) and IVF (p = 0.001).
Video feedback can facilitate greater learning of clinical skills. Students can attain a similar level of surgical skills improvement with UVF as with teacher-intensive IVF.
The impact of nontechnical skills on technical performance in surgery: A systematic review
2012, Journal of the American College of Surgeons
Citation Excerpt :
Measures for this included number of errors,19,27,29-32,39,40 errors in technique assessed via OCHRA,35-38 and final product quality.7,8,25,32 Other assessment tools to capture quality of technical performance included global rating scales such as OSATS,18,21-26,33,34,42 and MOSAT.15 Checklists were also used in the form of task-specific checklists,15,22-25,33 essential-item checklist,33 procedure-specific skill,26 procedural problems and errors (NOPES),35,36 and anesthetic checklists (assessing medical knowledge and technical skill of anesthesiologists).41
Failures in nontechnical and teamwork skills frequently lie at the heart of harm and near-misses in the operating room (OR). The purpose of this systematic review was to assess the impact of nontechnical skills on technical performance in surgery.
MEDLINE, EMBASE, PsycINFO databases were searched, and 2,041 articles were identified. After limits were applied, 341 articles were retrieved for evaluation. Of these, 28 articles were accepted for this review. Data were extracted from the articles regarding sample population, study design and setting, measures of nontechnical skills and technical performance, study findings, and limitations.
Of the 28 articles that met inclusion criteria, 21 articles assessed the impact of surgeons' nontechnical skills on their technical performance. The evidence suggests that receiving feedback and effectively coping with stressful events in the OR has a beneficial impact on certain aspects of technical performance. Conversely, increased levels of fatigue are associated with detriments to surgical skill. One article assessed the impact of anesthesiologists' nontechnical skills on anesthetic technical performance, finding a strong positive correlation between the 2 skill sets. Finally, 6 articles assessed the impact of multiple nontechnical skills of the entire OR team on surgical performance. A strong relationship between teamwork failure and technical error was empirically demonstrated in these studies.
Evidence suggests that certain nontechnical aspects of performance can enhance or, if lacking, contribute to deterioration of surgeons' technical performance. The precise extent of this effect remains to be elucidated.

View all citing articles on Scopus

View full text

Surgical educationThe effectiveness of video feedback in the acquisition of orthopedic technical skills

Abstract

Background

Methods

Results

Conclusions

Section snippets

Methods

Results

Comments

Am J Surg

Am J Surg

Br J Surg

Am J Surg

Am J Surg

Reliability and construct validity of a structured technical skills assessment form

Am J Surg

Objective structured assessment of technical skill (OSATS) for surgical residents

Br J Surg

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination

Acad Med

Surgical education
The effectiveness of video feedback in the acquisition of orthopedic technical skills