Structured assessment format for evaluating operative reports in general surgery

doi:10.1016/j.amjsurg.2007.08.053

The American Journal of Surgery

Volume 195, Issue 1, January 2008, Pages 24-29

Presented at the Annual Meeting of the Association for Surgical Education, Washington, DC, April 11–14, 2007

https://doi.org/10.1016/j.amjsurg.2007.08.053 Get rights and content

Abstract

Background

Despite its multifaceted importance, no validated or reliable tools assess the quality of the dictated operative note. This study determined the construct validity, interrater reliability, and internal consistency of a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in general surgery.

Methods

SAFE-OR was developed by using consensus criteria set forth by the Canadian Association of General Surgeons. This instrument includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a videotaped laparoscopic sigmoid colectomy. Blinded, independent faculty evaluators graded the transcribed reports using SAFE-OR.

Results

Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, t = .001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 ± 4.7 vs 35.9 ± 7.6, t = .006). Interclass correlation coefficients were .98 (95% confidence interval, .96–.99) for structured assessment and .93 (95% confidence interval, .83–.97) for global quality scales. Cronbach α coefficients for internal consistency were .85 for structured assessment and .96 for global quality assessment scales.

Conclusions

SAFE-OR shows significant construct validity, excellent interrater reliability, and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and provide a mechanism for implementing, monitoring, and refining curriculum for dictation skills.

Section snippets

Study Objective

To develop a Structured Assessment Format for Evaluating Operative Reports (SAFE-OR) in gastrointestinal surgery and evaluate it for (1) interrater agreement, (2) construct validity, and (3) internal consistency.

Methods

SAFE-OR was developed by using the Canadian Association of General Surgeons consensus criteria for gastrointestinal operative procedures for both face and content validity [14]. It consists of a structured assessment form (Fig. 1) and a global quality ratings scale (Fig. 2).

Structured assessment items are of 2 types: simple, either correctly described or not (eg, “date of surgery”), or complex, with a spectrum of possible descriptions (eg, “technique of reconstruction”). Simple items are graded

Construct validity

The mean overall structured assessment score (out of 44) was significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, P = .001). The mean overall global quality score (out of 45) was also lower for novice versus experienced residents (25.6 ± 4.7 vs 35.9 ± 7.6, P = .006).

Reliability/internal consistency

The ICC for overall structured assessment scores was .98 (95% confidence interval, .96–.99). For ordinal elements, the ICC ranged from .73 to .90 (Table 1). For nominal structured assessment items,

Comments

“All clinical skills evaluations, regardless of purpose, structure, content or scoring, must be designed so that reliable and valid assessment scores and/or decisions can be made” [25]. Specifically, until now, no mechanism identified has been rigorously shown to measure what it purports to (validity) in a consistent and precise manner (reliability) when examining the quality of an operative report in general surgery.

Only 1 article identified in the surgical education literature shows an

Acknowledgment

Supported by a medical education research grant from the Royal College of Physicians and Surgeons of Canada (RCPSC), Ottawa, Ontario, Canada.

References (26)

R.A. Moore
The dictated operative note: important but is it being taught?
J Am Coll Surg
(2000)
R. Brydges et al.
Computer-assisted assessment of one-handed knot tying skills performed within various contexts: a construct validity study
Am J Surg
(2006)
M.C. Vassiliou et al.
A global assessment tool for evaluation of intraoperative laparoscopic skills
Am J Surg
(2005)
A.A. Gumbs et al.
Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills
J Am Coll Surg
(2007)
H.M. MacRae et al.
Assessment of critical appraisal skills
Am J Surg
(2004)
L. Cyr et al.
Measures of clinical agreement for nominal and categorical data: the kappa coefficient
Comput Biol Med
(1992)
J.E. Frank
The CanMEDS 2005 Physician Competency Framework. Better Standards. Better PhysiciansBetter Care
(2005)
T.F. Lyons et al.
The relationship of physicians’ medical recording performance to their medical care performance
Med Care
(1974)
S. Phelan et al.
Teaching Medical Dictation: Results of a Course for Medical Students
Teach Learn Med
(1993)
Rogers B. Operation notes: an audit of the Royal College of surgeons of England guidelines [Royal College of Surgeons...

R.J.D.B. Baigrie et al.

An audit of the quality of operation notes in two district general hospitalsAre we following Royal College guidelines?

Ann R Coll Surg Engl

(1994)

M.B. Flynn et al.

The operative note as billing documentation: a preliminary report

Am Surg

(2004)

Y.W. Novitsky et al.

Prospective, blinded evaluation of accuracy of operative reports dictated by surgical residents

Am Surg

(2005)

Cited by (33)

Advancement in the quality of operative documentation: A systematic review and meta-analysis of synoptic versus narrative operative reporting
2019, American Journal of Surgery
The operative report is vital for patients and central to surgical quality assessment. Narrative operative reports are often poor quality. Synoptic reporting can improve documentation. The objective was to identify and appraise studies comparing synoptic and narrative operative reporting.
A systematic review of the literature was performed. The primary outcome was completion of critical elements for an operative report. Additional secondary outcomes were measured. Meta-analysis was performed where possible. Quality analysis was performed using Newcastle-Ottawa Scale (NOS).
1471 citations were identified; 16 studies included. Mean NOS was 7.09 out of 9 (+/−- SD 1.73). Meta-analysis demonstrated that synoptic reporting was significantly more complete (SMD 1.70, 95% CI 1.13 to 2.26; I² 98%). Completion time was shorter with synoptic reporting (mean difference −0.86, 95% CI -1.17 to −0.55). Secondary outcomes favoured synoptic reporting.
Synoptic reporting platforms outperform narrative reporting and should be incorporated into surgical practice.
Standardization of Laparoscopic Operative Reporting: Improving Gynaecological Surgeon Communication
2018, Journal of Obstetrics and Gynaecology Canada
Citation Excerpt :
Participants were also asked to obtain six photographs at the beginning of each laparoscopic or robotic procedure corresponding to the six anatomical landmark zones of the pelvis. Cases were collected from the participating surgeons, de-identified, and independently scored by three judges, who also were all gynaecological surgeons, using the SAFE-OR.1 The photographs were printed and submitted to the study coordinator by depositing them into a locked box kept in each operating room.
No standardization of quality of operative reporting currently exists, and this represents a missed opportunity for communication among health care providers. This study proposed a method to improve operative notes by structuring the findings by six anatomical zones of the pelvis. Objective I was to validate the method of documenting six zones of the pelvis by using intraoperative photography. Objective II was to compare this method with dictations from operative reports created before introducing this method.
This retrospective cohort study evaluated pre- and post-intervention results of using six zones to guide operative reporting. Reports were collected from participating surgeons and were scored using a validated scoring tool. Each participant was taught to photograph six zones and use the zones in the operative report. Pre- and post-intervention cases were compared using generalized linear mixed models.
Scores of study participants using the zones were significantly higher than those without (P < 0.0001). Surgeons showed an ability to improve their reporting. The detail illustrated in the cases was qualitatively richer, and the anatomy within the six zones was referenced more frequently.
Compared with reports without the technique, incorporating the six zones greatly enhances operative reporting and likely would improve communication among care providers. More reliable communication of intraoperative findings has the potential to enhance the value of laparoscopy greatly as a diagnostic tool across gynaecological subspecialties.
À l'heure actuelle, il n'existe aucune norme de qualité pour la production de rapports opératoires; pour les fournisseurs de soins de santé, cela constitue une occasion manquée de communiquer efficacement. La présente étude suggère une façon d'améliorer les notes opératoires en organisant les données selon six zones anatomiques prédéterminées dans le pelvis. Le premier objectif était de valider la consignation des données des six zones du pelvis à l'aide de photographies prises durant l'intervention. Le deuxième objectif était de comparer les rapports ainsi produits à des rapports dictés avant la détermination des six zones.
Cette étude de cohorte rétrospective a évalué les rapports produits avant et après la mise en place de la méthode des six zones. Les rapports ont été recueillis auprès de chirurgiens participants, et ont été notés au moyen d'un outil de notation validé. Les participants ont appris à photographier les six zones et à se servir de celles-ci pour produire leurs rapports opératoires. La comparaison des résultats obtenus avant et après la mise en place de la méthode des six zones a été faite au moyen de modèles linéaires mixtes généralisés.
Les notes obtenues par les participants s'étant servis des six zones étaient significativement plus élevées que celles des autres participants (P < 0,0001). Les chirurgiens ont réussi à améliorer la qualité de leurs rapports : les renseignements qu'ils contenaient étaient qualitativement plus riches, et les structures anatomiques à l'intérieur des six zones y étaient plus souvent mentionnées.
Le recours aux six zones améliore grandement les rapports opératoires et améliorerait fort probablement l'efficacité des communications entre les fournisseurs de soins. Une meilleure communication des données peropératoires a le potentiel d'améliorer considérablement la valeur de la laparoscopie comme outil diagnostic dans les sous-spécialités de la gynécologie.
Electronic synoptic reporting: assessing the completeness of synoptic and narrative reports for Roux-en-Y gastric bypass
2017, Surgery for Obesity and Related Diseases
Operative reporting is the foundation of surgical communication. The quality indicators (QIs) contained in these reports can be used to document the performance of processes that affect patient care and may afford quality assurance with improvement in healthcare.
We assessed the degree to which the electronic synoptic report (SR) documents the operative QIs compared with narrative reports (NR) for Roux-en-Y gastric bypass (RYGB). The time to report availability on patient chart was also identified as a secondary measure.
Academic center, public hospital, Canada.
A total of 40 prospectively collected RYGB synoptic reports and 40 case-matched historical NRs were compared against checklist QIs that were established by a national Delphi process. These checklist QIs are validated and have high interrater agreement at our institution. Time from dictation to report availability on patient chart was measured.
SR had a mean completion of 99.7% (±standard deviation [SD] 1.3%) compared with 64.0% (±SD 6.3%) for NR (t = 36.0, P<.0001). All subsections of SR were>99% complete and significantly higher than NR (P<.001). The mean time from NR dictation to report availability was 4.14 days (SD±3.17, range 0–10 d). All SRs were in the operative area after the procedure and were available immediately.
The RYGB synoptic report is superior to the narrative report for inclusion of accepted quality indicators and time to availability. Important elements, including process of care, demographic characteristics, and anatomic-related data, were often missing from the NR. SR is a promising method for improving documentation for RYGB.
Use of a computerized arthroplasty registry to generate operative reports decreases transcription errors
2017, International Journal of Medical Informatics
Citation Excerpt :
The quality of future care delivered may be hampered by poor communication [5]. Despite the clear importance of the operative report in the electronic medical record (EMR), few studies have addressed report quality [3,4,6–11]. Studies from other surgical specialties have shown standardized reports result in more complete and reliable interpretable operative data compared with non-standard operative reports [9].
Despite the clear importance of the operative report in the electronic medical record, few studies have addressed the quality.
We prospectively evaluated 300 consecutive patients undergoing primary total joint arthroplasties for operative report errors utilizing three different forms of documentation (standard dictation vs. templated dictation vs. a computer registry database generated operative report). The three types of reports were evaluated for errors which were classified as either major or minor.
There were significantly more total errors in the standard dictation group compared to both the computer registry database generated (p < 0.001) and the templated operative reports (p < 0.001). Major errors were significantly reduced in the database generated reports compared to the templated (p < 0.001) and standard dictation groups (p < 0.001). There were significantly more minor errors in the standard dictation group (p < 0.001) compared to the other two groups. No statistically significant differences in major errors were noted when comparing standard vs. templated operative reports. There was no difference in minor or total errors between the database generated and templated operative reports.
The use of a computer registry database generated operative report resulted in fewer major errors versus a templated or standard dictated operative report. Further research is warranted in this area to validate these findings across subspecialties and institutions.
Bariatric operative reporting: Quality assessment and perceptions among bariatric surgeons
2017, Surgery for Obesity and Related Diseases
The quality of narrative operative notes is poor. No investigation has previously addressed operative reporting specifically in bariatric surgery.
To evaluate surgeons’ perceptions of the quality of operative reporting in bariatric surgery and compare this to an audit of Roux-en-Y gastric bypass (RYGB) narrative reports using validated quality indicators.
University hospital, Canada.
A Web-based survey was distributed to bariatric surgeons across Canada. Perceptions regarding the quality of reporting were gathered using a Likert scale (modified Structured Assessment Format for Evaluating Operative Reports) and free text fields. Forty RYGB narrative reports were analyzed against established quality indicators and compared to respondent’s perceptions based on themes.
Twenty-four of 34 bariatric surgeons (71%) completed the survey. The most commonly performed procedures were RYGB and sleeve gastrectomy (96% and 100%, respectively). Currently, 70.8% perform a traditional narrative report. The average Structured Assessment Format for Evaluating Operative Reports score for narrative dictations by bariatric surgeons was neutral (27.9/40). The lowest scoring items were the “description of indications” (2.9/5) and “succinctness” (3.3/5). Opinions reflected a need for an immediately generated, standardized, template-based report to improve the quality and accessibility of operative documentation. The quality audit reinforced respondents’ perceptions. Reports included only 62.0%±6.6% of quality indicators, with the lowest scoring areas being “patient details,” “preoperative events,” and “postoperative details” (41.1%, 32.4%, and 31.7%, respectively).
This survey revealed a perception of mediocre quality of narrative dictations. This was reinforced by an audit of RYGB operative reports. Future investigations should focus on improving this form of operative communication.
Development of consensus-derived quality indicators for laparoscopic Roux-en-Y gastric bypass surgery
2017, Surgery for Obesity and Related Diseases
Synoptic operative reporting is a solution to the poor quality of narrative reports.
To develop operative report quality indicators (QI) for laparoscopic Roux-en-Y gastric bypass (LRYGB) to generate validated parameters by which these reports can be evaluated and improved.
University hospital in Canada.
A Delphi protocol was used to determine QIs for LRYGB reporting. Bariatric surgeons across Canada were recruited along with key physician stakeholders to participate via a secure web-based platform. Participants initially submitted potential QIs. These were grouped by theme. Items were rated on 9-point Likert scales in subsequent rounds. Scores of 70% or greater were used for inclusion consensus, and 30% or less denoted exclusion. Elements scoring 30% to 70% were recirculated by runoff in subsequent rounds to generate the final list of QIs.
Four community and 4 academic bariatric surgeons were invited, representing all provinces performing LRYGB. The 4 multidisciplinary invitees included 1 minimally invasive/acute care surgeon, 1 tertiary abdominal radiologist, 1 gastroenterologist performing advanced endoscopy, and 1 general surgeon with expertise in synoptic reporting. Round 1 achieved an 83.3% (10/12) response and identified 91 potential items for consideration. Round 2 had a 100% response, and 69 items reached inclusion consensus. The third round achieved a 100% response and resulted in 75 QIs reaching final inclusion consensus.
This study established consensus-derived multidisciplinary QIs for LRYGB operative reports. This will allow further assessment of the quality of narrative reports and afford the development of a synoptic operative report that may ameliorate identified deficiencies.

View all citing articles on Scopus

View full text

Association for surgical educationStructured assessment format for evaluating operative reports in general surgery

Abstract

Background

Methods

Results

Conclusions

Section snippets

Study Objective

Methods

Construct validity

Reliability/internal consistency

Comments

Acknowledgment

J Am Coll Surg

Am J Surg

Am J Surg

J Am Coll Surg

Am J Surg

Comput Biol Med

The CanMEDS 2005 Physician Competency Framework. Better Standards. Better PhysiciansBetter Care

The relationship of physicians’ medical recording performance to their medical care performance

Med Care

Teaching Medical Dictation: Results of a Course for Medical Students

Teach Learn Med

An audit of the quality of operation notes in two district general hospitalsAre we following Royal College guidelines?

Ann R Coll Surg Engl

The operative note as billing documentation: a preliminary report

Am Surg

Prospective, blinded evaluation of accuracy of operative reports dictated by surgical residents

Am Surg

Association for surgical education
Structured assessment format for evaluating operative reports in general surgery