Quantifying recall bias in surgical safety: a need for a modern approach to morbidity and mortality reviews
===========================================================================================================

* Hamad Alsubaie
* Mitchell Goldenberg
* Teodor Grantcharov

## Abstract

**Background:** Despite recent investments into reducing errors and adverse events in health care, methods for quality improvement in surgery are outdated and ineffective. Most current efforts in this field are centred around morbidity and mortality conferences (MMCs), which have remained unchanged for over 100 years. The present study aimed to quantify the recall bias associated with details from surgical cases.

**Methods:** We gathered immediate postoperative questionnaires from 1 surgeon, 1 fellow and 11 trainees following 25 routine surgical cases. Information elicited included their perceived level of concentration, mental preparedness and assessment of whether the procedure deviated from its expected course, including any intraoperative adverse events. We readministered the questionnaire 7–9 days later to assess participants’ ability to recall important aspects of the procedure.

**Results:** After 1 week, members of the surgical team were universally inaccurate in their recollection of even major details from the operating room. Although most participants felt mentally prepared and perceived no issues with concentration during the case, all participants misclassified operations as having been performed with or without adverse events in almost every included case.

**Conclusion:** Our findings show that recall bias regarding surgical safety events is exceedingly common. This likely has a major impact on the integrity of data presented at MMCs.

It has been long understood by physicians that the morbidity and mortality conference (MMC) serves primarily as an opportunity to learn from each other’s mistakes. Since its introduction in the early 1900s by Codman,1 the MMC has compelled the health care team to carefully dissect adverse medical events or outcomes in order to identify strategies to prevent future errors leading to patient harm. The MMC is a universal tool to evaluate and improve health care quality and safety, and surgeons have long been innovators in this endeavour.

In 2002, the Canadian government budgeted $50 million over 5 years for the creation of the Canadian Patient Safety Institute, and many health care organizations have subsequently initiated efforts to improve patient safety.2 The most widely adopted strategy for quality assessment in academic surgical programs is the departmental MMC, which uses case-finding methods to identify opportunities for improvement in patient outcomes.

Although historically the MMC has held an important educational role, it is known that case-finding strategies for safety and quality assessment have several limitations, including a focus on individual performance rather than organizational processes and attention to individual events rather than patterns of outcomes.3,4 In addition, it has been established that self-reporting behaviour is subject to recall bias. Gaskell and colleagues5 suggested 4 types of recall error: forgotten details or entire events (omission), recall of events that did not occur (commission), reporting that an event happened earlier than it actually did (backward telescoping) and reporting that an event happened more recently than it did (forward telescoping). It has also been recognized that the longer the recall period, the less accurate the reporting becomes.6 However, although the likelihood of recall error increases with longer recall periods, so does the amount of information available, so there is a potential trade-off between recall error and information gathering.7

To better understand these issues of recall bias and knowledge gathering around adverse events in surgery, we questioned members of the surgical team about organizational, situational, communication and team dynamics as well as any perceived deviations from the expected course of the operation, and repeated this questionnaire 1 week postoperatively. Substantial discrepancy in case recall lends justification to the adoption of alternative documentation strategies for surgical safety improvement, such as surgical video integration into the MMC, as means of improving education, patient safety and health care delivery.

## Methods

### Study design

This survey study was conducted at an academic tertiary care centre with a high volume of laparoscopic surgery. The survey was administered at the end of an operative case and was completed by 3 members of the surgical team: the surgeon, the fellow and the medical trainee (resident or medical student). The questionnaire was then readministered 7–9 days postoperatively to test for recall bias. All abdominal and pelvic laparoscopic operative cases were included, ranging from simple procedures such as cholecystectomy and laparoscopic hernia repair to complex upper gastrointestinal procedures such as laparoscopic gastrectomy for cancer, fundoplication and bariatric surgery. Both elective and emergency cases were included. Patient identifiable data were removed.

### Survey

The questionnaire was designed to assess multiple factors that could affect the recall of the operative case (Appendix 1, available at canjsurg.ca/017317-a1). After basic participant demographic information was collected, the surgical team member was asked in a closed-ended way whether the case deviated from the standard operative course. If the response was “Yes,” the participant was given the opportunity to provide a brief description of the deviation, classify it as minor or major, and state whether the team discussed it at the time. Participants were also asked to rank their mental preparedness/readiness for the procedure, as well as the level of interpersonal communication, on Likert-type scales of 1–4. This was done to ensure the absence of factors that could limit the participants’ ability to recall details of the case. Finally, the questionnaire inquired whether the case was scheduled as planned and whether participants felt they were able to adequately concentrate during the procedure.

Seven to 9 days postoperatively, participants were asked whether they could remember the details of the case, and, if they did, the questionnaire was repeated.

Participants provided consent to participate and to provide anonymous survey responses. Owing to the deidentified nature of the study, patients were not asked to provide consent.

## Results

Twenty-five surgical cases were included in the final analysis. All physicians approached to participate in the study agreed. Twenty-five questionnaires were collected from a single staff surgeon and from 1 fellow, and 20 questionnaires were collected from 11 different trainees (6 PGY-1, 7 PGY-4 and 7 PGY-5).

In the immediate postoperative period, the staff surgeon felt that he was able to concentrate adequately and that the procedure had been scheduled as planned in all included study cases. He felt that communication in the operating room team was optimal in 22 cases (88%) and that he was mentally prepared for the operation in all cases (Fig. 1). In 13 cases (52%), the surgeon reported a deviation from the normal course of the operation, major in 8 cases and minor in 5. Interestingly, at 1 week postoperatively, the surgeon could not recall details from 15 cases (60%), misclassified 4 cases (16%) as having minor deviations and incorrectly recalled 5 cases (20%) as being performed without any deviation (Table 1).

![Fig. 1](http://canjsurg.ca/https://www.canjsurg.ca/content/cjs/62/1/39/F1.medium.gif)

[Fig. 1](http://canjsurg.ca/content/62/1/39/F1)

Fig. 1 
Distribution of responses to questions regarding team communication and mental preparedness.

View this table:
[Table 1](http://canjsurg.ca/content/62/1/39/T1)

Table 1 
Participant responses regarding whether the case deviated from the standard operative course postoperatively and 7–9 days later

A similar phenomenon was observed with the fellow. Again, in all included cases, the fellow felt that there were no limitations on his ability to concentrate and that the case was appropriately scheduled as planned. He felt mentally prepared for the operation in 23 cases (92%) and rated the communication in the operating room as optimal in 24 cases (96%) (Fig. 1). There was strong agreement between the faculty surgeon and the fellow in the immediate postoperative period regarding whether a deviation from the planned operative course occurred (22/25 [88%]). The fellow felt that 16 cases (64%) contained deviations in operative course, of which 6 were major and 10 were minor. However, when asked 1 week later, he was unable to recall 18 cases (72%), and he incorrectly classified 3 cases (12%) as involving deviations and 4 (16%) as having occurred without any intraoperative events (Table 1).

Medical trainees felt mentally prepared in 18/20 cases (90%) and felt that communication was optimal in all cases (Fig. 1). They did not readily agree with the faculty surgeon regarding whether a deviation from the planned operative course occurred (7/20 [35%]). Trainees felt that 7 cases (35%) contained deviations in operative course. At the time of follow-up 1 week later, they were unable to recall the details of 13 cases (65%) and misclassified 3 cases (15%) as deviating from the standard procedural course. However, they were able to correctly recall 4 cases (20%) as having no events intraoperatively (Table 1).

## Discussion

We used a survey instrument to quantify the effect of time delay in recalling events regarding patient safety in surgical procedures. This gap in memory has the potential to negatively affect the quality of data presented at the MMC, which may have adverse consequences for education, quality improvement and patient safety. We found that the number of operative cases incorrectly classified as following the expected operative course was high among all members of surgical team; errors included those of omission as well as commission. Although the MMC remains a gold standard for surgical quality improvement and surgical education, there remains a lack of quantitative data on the accuracy of the data presented at these conferences. Efforts to improve the educational value of surgical MMCs remain limited, and previous efforts to assess the MMC have been largely qualitative, with findings inherently difficult to reproduce.8 Other groups have published tools to improve the educational experience of the MMC, generally consisting of rubrics to enhance situational awareness or to provide structured methods of analysis and presentation of data only.8–10

There has been limited investigation surrounding the phenomenon of recall bias as it pertains to patient safety. Our data conclusively show that the surgeon and the fellow were unable to remember pertinent details from operations as few as 7 days earlier, despite reporting immediately postoperatively being mentally prepared and being allowed to adequately concentrate during the operation. Interestingly, the most accurate recall was among the medical trainee cohort. This may suggest that increasing exposure to a given surgical procedure leads to a decreased ability to discriminate between recent cases, further adding to the list of limitations of the MMC.

One possible solution to overcoming recall bias regarding intraoperative safety events is the inclusion of operative video capture as routine practice or incorporation of video data into the medical record. The routine use of prospective media capture in the operating room is currently being investigated as a means of improving the ability to assess surgical safety and to implement quality-improvement strategies in surgery.11 In a study by Bonrath and colleagues,12 analysis of intracorporeal video captured during 38 laparoscopic surgical procedures identified 66 human errors, and in 25 operations they identified an adverse event that required additional rectification by the surgeon. Wauben and colleagues13 showed that, although surgeons performing laparoscopic cholecystectomy recorded having achieved the “critical view of safety” in the medical record in 100% of cases, objective video review of these cases showed that this was done in only 43% of cases. Combined with our findings, these reports suggest that objective analysis of operative video would increase both the accuracy and the educational yield of the MMC, providing specific and detailed examples not only of human errors but also of latent safety threats that could be mitigated through targeted intervention.12,14,15

### Limitations

There are limitations to our study that must be mentioned. First and most notably, a single staff surgeon and a single fellow were surveyed, which may limit the generalizability of the findings. Therefore, it is important that these initial findings be validated with a larger and more heterogeneous surgeon cohort. Second, the absence of video capture from the included cases means that a detailed analysis of the mechanism of error in these cases was impossible. Third, our sample was not large enough to warrant any inferential statistical analysis. However, we feel that our findings are hypothesis-generating and should spur future investigation into the phenomenon of recall bias in surgery and its effects on patient safety and quality improvement. Finally, there was substantial disagreement between medical trainees and the staff surgeon postoperatively regarding whether the case went as planned. However, we feel that this finding adds to the argument that subjective recall of intraoperative events is unreliable and that even greater variability may exist among trainees, who are often tasked with gathering the data and presenting the MMC. We plan to evaluate changes in patient outcomes following the implementation of a formal video capture program and subsequent curricula development and data-enhanced MMC.

## Conclusion

A surgeon, a fellow and medical trainees were unable to accurately recall simple details from surgical cases that were performed only 7–9 days previously. Without improved technology-supported methods of data capture in surgery, crucial information that can benefit both the training of future surgeons and patients’ well-being will continue to be lost.

## Footnotes

*   **Competing interests:** None declared.

*   **Contributors:** H. Alsubaie and T. Grantcharov designed the study. H. Alsubaie acquired the data, which all authors analyzed. M. Goldenberg and T. Grantcharov wrote the article, which all authors reviewed and approved for publication.

*   Accepted April 17, 2018.

## References

1.  Gregor A, Taylor D.Morbidity and mortality conference: its purpose reclaimed and grounded in theory.Teach Learn Med 2016;28:439–47.
    
    

2.  Benassi P, MacGillivray L, Silver I, et al.The role of morbidity and mortality rounds in medical education: a scoping review.Med Educ 2017;51:469–79.
    
    

3.  Laffel G, Blumenthal D.The case for quality management science.JAMA 1989;262:2869–73.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.1989.03430200113036&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=2810623&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1989AZ86500028&link_type=ISI) 

4.  Feldman L, Barkun J, Barkun A, et al.Measuring postoperative complications in general surgery patients using an outcomes-based strategy: comparison with complications presented at morbidity and mortality rounds.Surgery 1997;122:711–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/S0039-6060(97)90078-7&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9347847&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1997YB55000018&link_type=ISI) 

5.  Gaskell GD, Wright DB, O’Muircheartaigh CA.Telescoping of landmark events: implications for survey research.Public Opin Q 2000;64:77–89.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1086/316761&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=10810077&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000088111800006&link_type=ISI) 

6.  Stull DE, Leidy NK, Parasuraman B, et al.Optimal recall periods for patient-reported outcomes: challenges and potential solutions.Curr Med Res Opin 2009;25:929–42.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1185/03007990902774765&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19257798&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 

7.  Kjellsson G, Clarke P, Gerdtham UG.Forgetting to remember or remembering to forget: a study of the recall period length in health care survey questions.J Health Econ 2014;35:34–46.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jhealeco.2014.01.007&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=24595066&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 

8.  Mitchell EL, Lee DY, Arora S, et al.SBAR M&M: a feasible, reliable, and valid tool to assess the quality of surgical morbidity and mortality conference presentations.Am J Surg 2012;203:26–31.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.amjsurg.2011.07.008&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=22088267&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 

9.  Mondoux SE, Frank JR, Kwok ES, et al.Teaching M&M rounds skills: enhancing and assessing patient safety competencies using the Ottawa M&M model.Postgrad Med J 2016;92:631–5.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6InBvc3RncmFkbWVkaiI7czo1OiJyZXNpZCI7czoxMToiOTIvMTA5My82MzEiO3M6NDoiYXRvbSI7czoxNzoiL2Nqcy82Mi8xLzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Mitchell EL, Lee DY, Arora S, et al.Improving the quality of the surgical morbidity and mortality conference.Acad Med 2013;88:824–30.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1097/ACM.0b013e31828f87fe&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=23619062&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000319658500023&link_type=ISI) 

11. Goldenberg MG, Jung J, Grantcharov TP.Using data to enhance performance and improve quality and safety in surgery.JAMA Surg 2017;152:972–3.
    
    

12. Bonrath EM, Gordon LE, Grantcharov TP.Characterising ‘near miss’ events in complex laparoscopic surgery through video analysis.BMJ Qual Saf 2015;24:516–21.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzgvNTE2IjtzOjQ6ImF0b20iO3M6MTc6Ii9janMvNjIvMS8zOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. Wauben LSGL, van Grevenstein WMUV, Goossens RHM, et al.Operative notes do not reflect reality in laparoscopic cholecystectomy.Br J Surg 2011;98:1431–6.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1002/bjs.7576&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21633952&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 

14. Hu Y, Arriaga AF, Roth EM, et al.Protecting patients from an unsafe system: the etiology and recovery of intraoperative deviations in care.Ann Surg 2013;256:203–10.
    
    

15. Guerlain S, Adams RB, Turrentine FB, et al.Assessing team performance in the operating room: development and use of a ‘black-box’ recorder and other tools for the intraoperative environment.J Am Coll Surg 2005;200:29–37.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2004.08.029&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15631917&link_type=MED&atom=%2Fcjs%2F62%2F1%2F39.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000226228600006&link_type=ISI)