Abstract
Background: Ultrasonographic features can be used to predict mediastinal lymph node malignancy during endobronchial ultrasonography. Despite the validity of using these features for this purpose, the features are not being widely used in clinical practice. This may be attributable to the absence of educational programs that teach clinicians how to identify the features. To address this knowledge gap, we developed an online educational module to teach clinicians how to correctly interpret ultrasonographic features.
Methods: The module was designed using corrective feedback and test-enhanced learning theories and distributed to clinicians in relevant specialties. The efficacy of the program was determined by comparing the percentages of correctly identified ultrasonographic features as each clinician progressed through the module. Participants were also asked to self-rate their confidence during the module. Analysis of variance was conducted, and a learning curve and descriptive statistics were generated.
Results: Twenty-two of the 29 participants (76%) completed the module. Analysis of variance indicated that the percentage of correctly identified features increased significantly as clinicians completed the module (p = 0.004); this finding is supported by the positive slope of the learning curve. Even though they initially reported some difficulty with identifying certain features, their confidence increased as they progressed through the module. When asked, 86% of participants reported that they found the educational module helpful and 90% reported that they would recommend it to others.
Conclusion: Participating clinicians were receptive to the interactive educational module. It enhances clinician skill and confidence in interpreting ultrasonographic features. The results of this study provide the foundation needed to test the validity of the educational module in clinical settings and to further explore clinician preferences for educational programs.
Lung cancer is the most common cause of cancer-related death worldwide.1 Advances in detection technology have resulted in opportunities for earlier treatment intervention.1,2 One such advance is minimally invasive endobronchial ultrasonography (EBUS). It allows for real-time visualization of the mediastinal lymph nodes (LNs) and provides clinicians with an opportunity to identify ultrasonographic features.3 It has become the standard of care for sampling mediastinal or hilar LNs suspected of malignancy.3
Because EBUS requires substantial technical skill, inconclusive biopsy results do occur. The results of as many as 42% of EBUS procedures are determined to be inconclusive, which by traditional standards necessitates either repeat biopsy or mediastinoscopy.4 Furthermore, the results of 40% of EBUS procedures are negative for malignancy. 5 Given that a large proportion of EBUS biopsies produce either inconclusive or negative results, the use of ultrasonographic features to predict malignancy via a validated score has the potential to reduce procedure times, decrease the need for repeat procedures and prevent the need for biopsy in certain cases.
Previous studies evaluating ultrasonographic features have concluded that the presence of certain observable features predicts malignancy. Predictive scores based on these features have been developed; however, their development has often lacked methodologic rigour, prospective data collection and adequate sample size.6–8 Our laboratory has developed and validated a 4-point score using several ultrasonographic features to accurately predict LN malignancy (specificity 96.34% and negative predictive value 86.50%).9
Despite the possible added clinical benefit and validity of using ultrasonographic features, their use has not been widely adopted in clinical practice. This may be attributable to the lack of formal educational programs teaching clinicians how to identify ultrasonographic features. Recently, EBUS educational programs focusing entirely on biopsy and entry techniques have emerged.10,11 However, these programs do not mention the importance of assessing ultrasonographic features before biopsy. To address this knowledge gap, we endeavoured to assess the feasibility of a novel online educational module that aims to teach clinicians of varying expertise how to correctly identify ultrasonographic features of mediastinal LNs.
Methods
Design
This study focused on the feasibility of an online educational module to teach clinicians how to correctly identify ultrasonographic LN features that predict malignancy. The online educational module was developed using the Research Electronic Data Capture (REDCap) system (Fig. 1). The module focused on interpretation of the following ultrasonographic features: margin description (well defined or poorly defined), echogenicity (heterogeneous or homogeneous), central hilar structure detection (absent or present), shape (round or nonround), small axis length (≥ 10 mm or < 10 mm) and central necrosis (absent or present). The definitions used in the educational module are listed in Box 1.6,7,9
Sample of the ultrasonographic feature educational module. The module had several components, and learners completed it entirely online.
Definitions used in the educational module
Shape
Round shape: defined as the ratio of the long and short axis being less than 1.5
Oval shape: defined as the ratio of the long and short axis being more than 1.5
Margins
Well-defined margins: greater than 50% echogenic line delimiting the LN
Poorly defined margins: less than 50% echogenic line delimiting the LN
Echogenicity
Heterogeneous echogenicity: presence of nonuniform echogenic patterns
Homogeneous echogenicity: presence of uniform echogenic patterning
CHS
Absence of a CHS: missing a flat, central, echogenic structure in the LN
Presence of a CHS: a flat, central, echogenic structure in the LN
Small axis length
Small axis ≥ 10 mm: presence of a small axis length ≥ 10 mm
Small axis < 10 mm: absence of a small axis length < 10 mm
Central necrosis
Presence of central necrosis: presence of a central hypoechoic structure in the LN
Absence of central necrosis: absence of a central hypoechoic structure in the LN
CHS = central hilar structure; LN= lymph node
A didactic approach was used to develop the educational program, which comprised 6 sections: a pre-education assessment, educational videos, ultrasonographic criteria identification with corrective feedback, practice with corrective feedback, a summary of learned concepts, and a posteducation assessment. Original still images and videos of mediastinal LNs from EBUS procedures were selected by an experienced endoscopist to generate a representative sample of malignant and benign LNs to be used in the module. Accompanying the LN videos and images were informative text, descriptive diagrams and interactive features. Two educational theories were used to inform the development of the module: immediate corrective feedback and test-enhanced learning theory.
Corrective feedback
Corrective feedback is the process by which a student is made aware of their errors after responding to a single question or a series of questions. The literature has shown that corrective feedback may rectify any misinformation that the student learns during the learning process.12,13 Corrective feedback may occur immediately or in a delayed manner. The immediate approach follows behaviourism theory, which asserts that corrective feedback is a form of positive reinforcement and therefore should be given immediately after an incorrect response to avoid having the student absorb misinformation during the learning process.12 The delayed approach to corrective feedback follows the interference-preservation theory, which contends that there is a beneficial learning effect when corrective feedback is given toward the end of an educational program.12 Despite the literature discussing the merits of delayed corrective feedback, the inconsistencies in these findings resulted in the immediate approach to corrective feedback being used in the design and development of this educational module.
Test-enhanced learning theory
Testing has traditionally been used as a method of summative and formative assessment for students. The results of tests provide the student and the teacher with an objective measurement of how successful the student’s learning has been.14,15 The cognitive science literature consistently shows that testing also leads to increased retention of content in comparison with studying alone.16 Long-term retention of knowledge is further increased when students receive feedback on incorrect responses.16 The literature indicates that the use of testing as an educational tool improves retention of tested information along with related information. Thus, the combination of testing and repetitive content contribute to optimal learning conditions for participants.
Application of the educational theories within the educational module
Concepts were taken from each of the previously mentioned theories to develop a comprehensive learning tool. Immediate corrective feedback was used during practice portions of the module. Participants were asked how to identify specific ultrasonographic features. If a participant answered a question incorrectly an interactive window immediately appeared on the screen, informing them of the error and providing them with a tip to avoid making the same mistake in the future. This immediate corrective feedback was intended to positively reinforce the newly learned content. Participants were tested with the score developed by Shafiek and colleagues 10 times during the module: at 2 separate points during the module, they scored 5 LNs consecutively.6
Study population and sampling method
This study was approved as exempt by the Hamilton Integrated Research Ethics Board (HiREB). Participants were recruited via email and an informational session was held at the 2017 Annual Meeting of the Canadian Association of Thoracic Surgeons. During the informational session participants were informed of the primary research objectives and provided with details regarding the online educational program. Participants included residents, fellows and staff in respirology and thoracic surgery specialties from institutions across Canada. Participants completed the educational module anonymously.
Statistical analysis
The overall efficacy of the educational module was assessed by comparing the percentages of correctly identified ultrasonographic features as the participant progressed through the module. Statistical comparisons throughout the learning module were completed using a 1-way repeated-measures analysis of variance (ANOVA). The participants’ level of confidence with identifying ultrasonographic features as they progressed through the educational module was also assessed using a 1-way repeated-measures ANOVA. All assumptions for statistical procedures were checked before the procedures were conducted; any failures were corrected with statistical compensatory methods. Statistical analysis was completed using SPSS version 21 (SPSS Inc.). All p values were based on 2-sided hypotheses, and p values < 0.05 were considered statistically significant.
Results
Participant characteristics
In total, 29 clinicians completed at least part of the online educational module between June 2017 and November 2017; 22 of them completed all of the components of the module. Participant demographics are described in Table 1. The average age of participating clinicians was 35 (standard deviation 8) years, 20 (69%) specialized in thoracic surgery and 17 (59%) stated they performed EBUS regularly (defined as completing at least 1 EBUS procedure per week). The level of experience of the participants varied: there were 8 residents (28%), 12 fellows (41%) and 9 staff (31%). The 12 fellows were equally split between respirology (50%) and thoracic surgery specialties (50%). Each participant completed at least 1 of the 10 components of the educational module. Among the 7 participants who did not complete all 10 components of the educational module, the degree of module completion ranged from 10% to 90% (Table 1).
Participant demographics and clinical experience
One-way repeated-measures ANOVA
The results of the 1-way repeated-measures ANOVA assessing the correctly identified ultrasonographic features are shown in Table 2. There was a significant effect for the percentage of correctly identified ultrasonographic features and the number of scoring assessments completed (n = 10, F3.05,57.97 = 9.619, p < 0.001). These results suggest that overall as participants progressed through the educational module the percentage of correctly identified ultrasonographic features increased significantly. A pairwise comparison using the Bonferroni post hoc analysis indicated that correct identification of ultrasonographic features increased significantly between test 4 and test 8.
One-way repeated-measures analysis of variance for correctly identified ultrasonographic features
Results of the 1-way repeated-measures ANOVA assessing the participants’ self-rated confidence in identifying ultrasonographic features are reported in Table 3. Participants rated their confidence on a simple scale from 0 to 100, where 0 indicated a complete lack of confidence and 100 indicated complete confidence. As indicated in Table 3, there was an overall significant effect for self-rated confidence during the module (F2401.11,37.48 = 8.872, p = 0.01). A Bonferroni post hoc analysis via pairwise comparisons indicated that self-rated confidence increased significantly between tests 3, 5 and 7.
One-way repeated-measures analysis of variance for self-rated confidence during the educational module
Assessment of the learning curve for the educational program
A learning curve that visually represents the association between learning (vertical axis) and experience (horizontal axis) was generated (Fig. 2). Analysis of the collected data indicated that as clinicians continued to repeatedly identify the ultrasonographic features and complete the interactive educational module, the percentage of the features that they correctly identified increased. This is also illustrated by the slope of the learning curve, which was 1.35. The learning curve complements the results of the 1-way repeated-measures ANOVA.
Learning curve for clinicians who participated in the educational module. Values are based on participants’ mean scores for each assessment they completed.
Difficulties identifying ultrasonographic features
During the educational module, participants were asked on 5 separate occasions to indicate which ultrasonographic features they had difficulty identifying. The features that they were least likely to report as being difficult to identify were LN shape and small axis length. On average, 2% of the clinicians expressed difficulty identifying these features. LN echogenicity, margin status, central hilar structure status and central necrosis status were more difficult for participants to correctly identify overall with 54%, 31%, 16% and 33% expressing identification difficulty on average, respectively. Although participants initially had difficulty correctly identifying the ultrasonographic LN features, their skill improved as they completed the module. The negative slopes, calculated on the basis of clinician self-identified difficulty, for echogenicity (−8.26), margin status (−2.72) and central hilar structure (−3.52) indicate that difficulty identifying the previously mentioned features decreased over time.
Participants’ experiences using the educational module
After completing the educational module, participants were asked about their overall experience with the program (Table 1). Overall, participants who completed the educational program found it helpful (86%), preferred the online nature of the program over traditional in-class approaches (74%) and would recommend the module to other clinicians (90%).
Discussion
Interactive online educational modules combine traditional didactic learning techniques with new learning theories to provide learners with the optimal learning experience. The interactive components of this online module provided participants with immediate corrective feedback and the opportunity to practise and develop their newly learned skills. To the best of our knowledge, this is the first educational module focused on teaching clinicians how to interpret ultrasonographic features of mediastinal LNs visualized during EBUS procedures.
Overall, the statistical analysis indicated that participating clinicians’ knowledge of how to correctly and confidently identify the ultrasonographic features increased significantly throughout the module. The learning curve had a positive slope and included a forecasted trend line predicting that inclusion of 2 additional LN sonographic scores in the educational module could result in participants achieving maximal scores and reaching a learning curve plateau. Analyses also revealed that self-rated difficulty with rating specific ultrasonographic features decreased as participants progressed through the module. Improvements in knowledge and confidence over the course of completing the module were expected results. These results suggest that while ultrasonographic feature detection and application are relatively niche skills in the field of EBUS, with the correct educational tools these skills are learnable.
Participants indicated which ultrasonographic features they had difficulty recognizing. The features they identified as difficult to recognize were ones that required subjective evaluation and could not be quantified using an objective score: echogenicity, margin status, central hilar structure and central necrosis. The present score does not use an objective textural analysis to differentiate between homogeneous and heterogeneous echotexture, meaning that even after clinicians complete the educational module there is a subjective nature to identifying this feature. However, trends indicated that as clinicians gained experience with identifying the features they found it less difficult to identify echogenicity. As with echogenicity, there is no true objective measurement for margin status, central necrosis or central hilar structure. Additionally, margin status, central necrosis and central hilar structure may be difficult to interpret, particularly when other features (echogenicity and central necrosis) are present in a single LN. Although participants initially expressed difficulty recognizing these subjective features, trends indicated that after they practised they had less difficulty identifying them correctly.
The results of this study have demonstrated that the educational module was well received by participants and capable of improving their knowledge and confidence related to interpreting ultrasonographic features. The use of ultrasonographic features to identify malignant or benign LNs is clinically relevant given the current landscape of mediastinal staging for non–small cell lung cancer. Guidelines mandate that EBUS transbronchial needle aspiration is the first-line staging modality and requires a minimum of 3 LNs to be systematically sampled.5,17 However, recommended assessment occurs in fewer than 50% of cases because of inadequate or inconclusive samples, small LNs and patient-specific characteristics.18 Additionally, approximately 40% of LNs biopsied during EBUS transbronchial needle aspiration are negative, which suggests that a significant proportion of LNs are undergoing tissue sampling procedures that may not be necessary. Ultrasonographic features have been shown to be accurate and valid in predicting LN malignancy;6–9 thus, the application of feature interpretation during mediastinal staging has the potential to reduce the number of biopsies required and to shorten the diagnostic pathway length in the case of inconclusive biopsy results. An educational module capable of improving knowledge retention and confidence in clinicians is necessary for the identification of ultrasonographic features of LNs to be used in clinical settings.
Limitations
We acknowledge several limitations of our study. Although our study population included clinicians specializing in thoracic surgery and respirology across different Canadian institutions, our sample size was small. Power analysis indicated that a sample size of 12 would allow for significant differences to be detected. However, a larger sample of clinicians could provide more generalizable results. Additionally, the educational module did not include an adaptive learning platform. The adaptive nature of the proposed platform would ensure that each clinician experiences a module tailored to the topics with which they have the most difficulty. Future research should consider the learning acquisition benefits of these platforms. This study did not include a comparator or control group. The addition of a comparator group would have enabled more robust analyses and enabled more conclusive results to be determined. However, the objective of this research was to see whether an online educational module based on ultrasonographic features could be created and used by clinicians. The results generated from this research provide a strong base to conduct future research including comparator groups to assess the true effect size related to the educational module and follow-ups to evaluate long-term retention and clinical adoption rates.
Conclusion
The incorporation of ultrasonographic features with EBUS transbronchial needle aspiration may enable clinical experiences for patients to be improved by reducing the time to diagnosis and potentially decreasing the requirement for staging of LNs that are benign on the basis of feature identification and diagnostic imaging. Given these possible benefits, a valid educational module is a necessity.
Acknowledgement
This research was funded by a McMaster Surgical Associates 2017 education grant.
Footnotes
This research was presented at the International Conference on Residency Education, Oct. 18–20, 2018, Halifax, N.S., and at the Annual Meeting of the Association for Surgical Education, May 1–3, 2018, Austin, Tex.
Competing interests: None declared.
Contributors: D. Hylton, Y. Shargall, C. Finley, J. Agzarian and W. Hanna designed the study. D. Hylton acquired and analyzed the data, which C. Fahim and W. Hanna also analyzed. D. Hylton wrote the article, which all authors reviewed. All authors approved the final version to be published.
- Accepted June 27, 2019.
In this issue
Article tools
Related Articles
Cited By...
- No citing articles found.