RT Journal Article
SR Electronic
T1 Evaluating the reliability of surgical assessment methods in an orthopedic residency program
JF Canadian Journal of Surgery
JO CAN J SURG
FD Canadian Medical Association
SP 299
OP 304
DO 10.1503/cjs.010614
VO 58
IS 5
A1 Nicholas Smith
A1 John Harnett
A1 Andrew Furey
YR 2015
UL http://canjsurg.ca/content/58/5/299.abstract
AB Background: Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form.Methods: A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach’s α and Fleiss κ.Results: Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach’s α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval &0.071 to 0.364), demonstrating low interrater reliability.Conclusion: Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.