Using learning-by-concordance to develop reasoning in epistaxis management with online feedback: A pilot study

Abstract

Background

Epistaxis is a recurring cause for referral to emergency departments. Its management can be complex; hence, it is critical to provide appropriate support to Otolaryngology-Head and Neck Surgery (OHNS) residents to develop clinical reasoning skills to manage such cases. Learning-by-Concordance (LbC) is a recently developed educational tool that encourages learners to think through simulated clinical scenarios. A panel of ENTs provides insightful feedback to residents, reflecting a diversity of opinions about practice. Our study aimed to assess LbC's feasibility and perceived value for training OHNS residents in epistaxis management.

Methods

In this qualitative study, three OHNS surgeons, including two faculty members and one resident, wrote the LbC scenarios. The LbC tool was made available to participants through an online platform. A panel of four OHNS faculty provided feedback on answers to LbC questions. Otolaryngology-Head and Neck Surgery residents participated and provided their opinion on the value of this educational tool through an online questionnaire.

Results

A total of 10 one-hour sessions were required to create and upload the training tool. To provide insightful feedback embedded in the learning tool, the four panelists needed 60 min each. Of the 37 participating residents, 25 (68%) completed the training. Overall satisfaction was high: 88% appreciated the training method, and 92% wanted to use this type of training again. Most residents felt the training enabled them to improve their clinical reasoning when encountering a patient with epistaxis (92%) and their knowledge about epistaxis (96%).

Conclusion

Findings suggest that OHNS residents could benefit from clinical reasoning exercises with panelist feedback using the LbC approach for clinical presentations that require complex approaches to manage conditions such as epistaxis.

Keywords

Epistaxis concordance training expert panel reasoning feedback

Introduction

Epistaxis is a frequent cause of ENT consultation and emergency department visits. In a study of more than 20,000 patients admitted to an adult ENT emergency department, 11.5% of patients consulted for epistaxis, 13% of whom had to be hospitalized. One in four (24.57%) adult patients hospitalized via the ENT emergency department was admitted for epistaxis.¹ The management of epistaxis can be complex, especially when aggravating factors such as antiaggregant or anticoagulant therapy, hypertension, or an underlying pathology such as hereditary hemorrhagic telangiectasia (HHT) are present.^2,3 Thus, Otolaryngology-Head and Neck Surgery (OHNS) trainees should think critically about epistaxis management and receive feedback on their thinking.

The script concordance method may provide such opportunities. According to script theory,⁴ medical knowledge is organized into networks called “illness scripts” that enable physicians to integrate new information with existing knowledge, recognize patterns of clinical presentation, identify similarities and differences between disease states, and make judgments about the appropriateness and feasibility of diagnostic or treatment options. These illness scripts are updated and refined through experience and learning.^5–7

The implications of script theory for medical education are numerous. Because it is impossible for clinician-educators to simply transfer their clinical reasoning skills—based on “illness scripts” acquired through experience—to their trainees, they must create opportunities to help trainees develop and refine their own sets of scripts. These concerns led initially to the creation of the Script Concordance Test (SCT),⁸ used to assess the quality of micro-judgments trainees make continuously in problem-solving situations. Its principle involves exposing trainees to authentic clinical practice situations through clinical vignettes. Subsequently, hypotheses for diagnosis or action are proposed, followed by the presentation of new data. The trainee's task is to make a judgment about the effect the new data have on the proposed hypothesis, using a 5-point qualitative scale.

Trainee's responses are contrasted with those given by a panel of experienced professionals, which may vary due to uncertainty inherent in the selected clinical cases. Trainees receive a score based on the number of panel members who responded as they did. This online assessment method has been used to assess reasoning in medicine and other fields.^8–10 In Otolaryngology, Kania et al. reported on creating and administering an online SCT.¹¹ They found that such tools are feasible, helpful, and accurate for assessing clinical reasoning in this field. They suggested using this mode of questioning in residency programs and continuing professional development.¹²

Developments in information technology now make it possible to provide automated feedback to learners and scores. Indeed, after submitting their answer, learners see (1) how their answer compares to those given by the panelists, (2) the justifications provided by each panelist for their answers, and (3) an educational summary with hyperlinks to external resources. These features transform the SCT evaluation system into a powerful training system (i.e. Learning-by-Concordance, LbC), which leads participants (students or practicing professionals) to reflect on cases similar to those encountered in practice.^13–15 The possibility of introducing multimedia content, including audio and video materials, enriches the relevance of the learning tool.

To meet the training needs of residents in rhinology training, we have developed an LbC to support epistaxis management skills. We sought to establish principles for creating this learning tool for other ENT-related fields. The objectives of this work were, therefore: (i) to describe the development of an LbC tool in ENT on epistaxis, detailing the different stages of development; (ii) to assess the feasibility of using the LbC approach with ENT residents; (iii) to report the results of their evaluation of this tool.

Design, setting, and participants

Study type

Our study uses a qualitative design.

The team, the electronic platform, and the target population:

The LbC scenarios were developed by a team of three ENTs, including two faculty experts and a 4th-year resident, according to stages described by Charlin et al.¹⁵ The LbC was disseminated online using the Wooflash tool, provided by Wooclap. Wooclap offered technical support during implementation but was not involved in training dissemination. Table 1 describes the structure of the LbC: situation/options/data/judgment/expert comments/educational synthesis. ENT residents in the rhinology rotation were the target population. These residents had the basic theoretical knowledge and practical experience with epistaxis. The training aimed to develop their understanding and reasoning in the management of epistaxis. The entire course is available online: https://app.wooflash.com/join/BPII5Q4H.

Table 1.

Example of the learning-by-concordance approach to epistaxis.

Step 1	Step 2	Step 3	Step 4	Step 5	Step 6
A clinical case described in the vignette, supplemented or not by additional multimedia material	An option (diagnostic, investigative or treatment) related to the case	A new piece of information	5-point qualitative judgment scale	Expert panel answers with justifications	Educational synthesis of key clinical takeaways
A 56-year-old patient presents with severe epistaxis. His blood pressure on arrival was 157/99.	If you are thinking of: Administering an anti-hypertensive medication	And you subsequently learn that: The patient is already treated for hypertension and that he took his anti-hypertensive medication one hour ago	The impact of this new piece of information on your intention is: Strongly negative Negative Neutral Positive Strongly positive	Negative Justification: The reality of the hypertensive crisis must first be verified by monitoring blood pressure levels after the epistaxis has been managed. If high blood pressure persists, an anti-hypertensive drug may be used, but only after advice from cardiology colleagues.	Blood pressure levels are frequently higher than normal in patients with epistaxis, although the causal role of hypertension on the occurrence of epistaxis cannot be established

Creation of the training material

The LbC tool consisted of six clinical situations: epistaxis in a patient treated with antiplatelet therapy; epistaxis and anticoagulants; epistaxis and hypertension; epistaxis and HHT; first-line management (anterior packing); second-line management (after the failure of anterior packing). A “clinical vignette” briefly described each clinical situation, followed by 3–8 questions about issues likely to be encountered. These questions resembled those asked by professionals in such situations.

The panel

The panel comprised four ENT faculty members who specialized in rhinology and practiced in four French academic hospitals. These faculty members did not participate in the development of the training material. Each panelist was asked to indicate what they would do in each clinical situation by selecting one of the options on the 5-point Likert scale. They were also asked to provide brief (one or two sentences) justifications of their choice, which served as formative feedback for residents. Subsequently, a short educational synthesis summarizing the key takeaways was provided. This summary included hyperlinks to access additional online resources (consensus conferences, guidelines).

Data collection and analysis

The LbC was submitted to all ENT residents from the Ile-de-France region of France in the rhinology rotation of their training (37 residents). There were no exclusion criteria. Residents received the link to the LbC by e-mail, which was accessible from October 2021 to January 2022. The training was optional. The number of students who completed the training was retrieved directly from Wooflash. A link to an online evaluation questionnaire was provided within the LbC platform to collect trainees’ opinions. Survey results were anonymous. All students gave their written consent for the study. The questionnaire consisted of (1) the year of residency training and gender; (2) whether the trainee had completed the training in one session or more, and how much time they use to complete the training; (3) whether the format of the training was intuitive and easy to use, whether the student had adapted quickly to the format; (4) whether the training was engaging, and whether the situations addressed were realistic; (5) whether the training allowed the student to enrich their knowledge and advance their clinical reasoning when confronted with a patient with epistaxis; (6) whether the training and its format were appreciated or not and whether the student would like to use this type of training again in the future. Finally, students were asked to identify two strengths and two areas for improvement for the LbC exercise. The project was granted ethical approval (Société Internationale Francophone d'Education Médicale—10 January 2021).

Results

Creation and implementation of the training

Creating the LbC tool (writing the vignettes and uploading the content online) required approximately 10 working sessions of 1 h each. The clinical vignettes and the options proposed (therapeutic and diagnostic) were discussed among the faculty to select situations that would resemble those encountered in clinical practice. No significant technical difficulties were encountered, but some adaptations of the Wooflash tool were necessary to adjust to the fact that, in LbC, more than one answer may be considered correct. Wooclap provided the technical support to make the required adaptations.

In this study, the four panelists we invited agreed to participate. The average time for panelists to respond with justifications for all questions was approximately 60 min. It is common and expected in LbC tools that panelists’ answers show some variability. For this reason, each panelist is asked to provide a short justification of their reasoning. In the Epistaxis LbC, panelists’ responses were in complete agreement in five cases (15%), partial agreement in 26 cases (79%), and complete disagreement in two cases (6%). An example of discordant panelist responses is presented in Figure 1. The training was sent to the residents by e-mail, using a mailing list of ENT residents in the Ile-de-France region. The training was submitted to 37 residents enrolled in the rhinology teaching module as an optional learning activity. Of the 37, 25 (68%) residents completed the training (Figure 2). Year in residency training ranged from 1 to 3. The duration of the training was less than 20 min in 9% of cases, 20–40 min in 64% of cases, and more than 40 min in 27% of cases. In 91% of cases, the training was completed in one sitting.

Figure 1.

Example of discordant panelist responses.

Figure 2.

COREQ flow diagram of the study.

Subjective evaluation by the ENT residents

Trainees were residents in the first (32%), second (18%), third (23%), fourth (23%), and fifth (4%) year of residency training, the majority were men (55%) and they all completed the online evaluation questionnaire (100% response rate).

Table 2 presents the summarized results of the evaluation. The overall satisfaction was high: 88% appreciated the training method, 84% found the format enjoyable, 100% found the training interesting, and 92% wanted to use this type of training again in the future. Most trainees found the LbC format intuitive and easy to use (64%) and easily adapted to the training format (80%).

Table 2.

Results of the evaluation questionnaire completed by residents. Results are given as a percentage of responses collected.

	Slightly	Moderately	Very	Extremely
Did you appreciate this training method?	4	12	52	32
Did you find the format of this training method pleasant?	0	16	56	48
Did you find this training interesting?	0	0	44	56
Do you wish to use this type of training method again in the future?	0	8	60	32
Did you find the format of this training method intuitive and easy to use?	0	36	40	24
Did you adapt easily to the format of this training method?	4	20	56	20
Did this training allow you to advance your approach to reasoning clinically when faced with a patient presenting with epistaxis?	4	8	64	24
Did this training allow you to enrich your knowledge on epistaxis?	0	4	64	32
Did you find that the situations encountered during this training were realistic?	0	0	52	48

In general, the residents emphasized the effectiveness of the training from a pedagogical point of view: they felt that the training enabled them to improve their clinical reasoning when faced with a case of epistaxis (92%) and to enrich their knowledge of epistaxis (96%). All residents felt that the situations covered in the training were realistic.

Twenty residents (80%) commented positively about their experience using this training tool. In general, they appreciated the active and user-friendly aspects of the exercises, the interactivity and ergonomic nature of the electronic platform, and the conciseness of the clinical cases, which allowed for rapid review. They also appreciated the realism and variety of the situations presented, the adequate reflection of complexity in “real life” situations, and the fact that there was often more than one correct answer. Finally, they appreciated the access to the diversity of the panelists’ points of view, the insights in their justifications, and the educational summaries with key takeaways.

Eighteen residents (72%) described elements that could be improved or that bothered them: the diversity of the panelists’ opinions (at times, entirely different answers from each) was sometimes confusing—the lack of consensus forced the students to make up their mind; the brevity of the clinical vignettes could occasionally simplify the clinical situations a little too much; finally, the use of a 5-point Likert scale for the answer was sometimes felt to be too complex to use and to interpret results.

Discussion

To our knowledge, this work is the first description of the development and use of the LbC approach in OHNS. The Epistaxis LbC tool was straightforward to use. The positive feedback from residents suggests that the LbC format is adapted to residents’ learning, allowing them to practice complex clinical situations in a structured manner.

One of the main interests of LbC is to place learners in practical situations and allow them to make decisions in simulated complex situations, where experience plays an important role.¹⁴ For example, in clinical vignette six, a patient presents with active epistaxis, resistant to anteroposterior packing with a double balloon catheter. Embolization was considered. The trainees are asked to assess the impact of the new data on this decision: the patient has already been embolized, and she has already had sphenopalatine and ethmoidal artery ligation, which is a rare but actual situation for which there is no clearly defined strategy. Concordance training appears to be a powerful tool in this case, offering access to the reasoning of four experienced clinicians. Few other teaching methods can enrich a student's clinical reasoning when learning to intervene in such complex clinical situations.

In SCT, participants receive a score based on the number of panel members who responded as they did: it is recommended to have 15 or more panelists to obtain an accurate score.¹⁶ In LbC, the aim is not to provide a score but to provide formative feedback. In this respect, it is suggested that the number of panelists be sufficient to reveal nuances in reasoning in a particular domain. However, this number should not be excessive to avoid redundancy of responses and justifications. In our study, four panel members appeared adequate.

It should be noted that the LbC presented here already requires that trainees have a certain amount of experience: it is suitable for residency students but not, at least in its current form, for undergraduate medical students. Other LbC formats are ideal for the latter.¹⁷ Regarding in-service clinicians, it is possible to consider the development of LbC for continuing professional development—a format that has already been used successfully in other disciplines.¹⁸ Specifically, exposure to complex situations is of immediate interest to many ENT practitioners.

Although the primary objective of the LbC is to train students, the tool is also interesting in other respects. Collecting the opinions of a panel of experienced practitioners allows for a survey of current practices and, at times, highlights interesting discordance points. For example, in this LbC, several areas of disagreement between the panelists were revealed: they gave differing answers for 85% of questions, and for two vignettes, each panelist provided entirely different answers. These two situations were: (1) management of embolization in a patient on anticoagulant therapy and (2) management of vitamin K antagonist in a patient with epistaxis but with an INR in the therapeutic range. The divergence may partly be explained by the intuitive nature of panelists’ reasoning, reflecting insufficient or scantly known guidelines in OHNS. Learning-by-Concordance could thus help identify such gaps that could be addressed by academic societies in developing and disseminating best practice recommendations.

Finally, the development and use of LbC relies heavily on using digital tools. The tool must allow (i) for simple and rapid uploading of the clinical situations and the educational takeaways; (ii) the solicitation and collection of expert feedback; (iii) an intuitive and easy-to-use training for the residents, who are presented with clinical situations, questions, and feedback from panelists, as well as the educational summary. Although Wooclap was used in this study, plugins have recently been developed for the Moodle platform to support LbC educational tool development and administration.

The main limitation of this work lies in the fact that it is essentially a descriptive study based on self-reported data about the development and implementation of an LbC in rhinology. The data collected with a non-validated questionnaire reflect the subjective perceptions of students about this innovative format. Further work will be necessary to evaluate the benefit of LbC for students in terms of knowledge and reasoning skills.

Conclusion

The results of this first experiment using LbC in ENT demonstrate a great interest among residents and suggest applicability in other fields, including continued professional development. The conclusion of our study is that concordance learning tools can be developed and used efficiently for training future ENTs.

Supplemental Material

sj-pdf-1-sci-10.1177_00368504241274583 - Supplemental material for Using learning-by-concordance to develop reasoning in epistaxis management with online feedback: A pilot study

Supplemental material, sj-pdf-1-sci-10.1177_00368504241274583 for Using learning-by-concordance to develop reasoning in epistaxis management with online feedback: A pilot study by Benjamin Verillaud, Marine Veleur, Romain Kania, Ivry Zagury-Orly, Nicolas Fernandez and Bernard Charlin in Science Progress

Footnotes

Acknowledgements

The authors would like to thank Guillaume de Bonnecaze, Justin Michel, Geoffrey Mortuaire Cécile Rumeau, as well as Adeline Heurtel and the entire Wooclap team for participating in developing this training. The authors also thank all the students who attended the training and accepted to complete the evaluation questionnaire.

Author contributions

All authors made substantial contributions to the interpretation of data for the work; drafted the work and reviewed it critically for important intellectual content; approved the version to be submitted; agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Data availability

The data are available upon request (benjamin.verillaud@aphp.fr)

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The project was granted ethical approval (SIFEM #10012021-1).

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Informed consent

All participants gave their written consent for the study.

ORCID iDs

Benjamin Verillaud

Bernard Charlin

Supplemental material

Supplemental material for this article is available online.

References

Timsit

Bouchene

Olfatpour

, et al. Epidemiology and clinical findings in 20,563 patients attending the Lariboisière hospital ENT adult emergency clinic. Ann Otolaryngol Chir Cervicofac 2001; 118: 215–224.

Robard

Michel

Prulière Escabasse

, et al. Guidelines of the French Society of Otorhinolaryngology (SFORL) (short version). specific treatment of epistaxis in Rendu-Osler-Weber disease. Eur Ann Otorhinolaryngol Head Neck Dis 2017; 134: 37–41.

Kasle

Fujita

Manes

. Review of clinical practice guideline: nosebleed (epistaxis). JAMA Surg 2021; 156: 974–975.

Charlin

Tardif

Boshuizen

. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med 2000; 75: 182–190.

Charlin

Gagnon

Sauvé

, et al.

Composition of the panel of reference for concordance tests: do teaching functions have an impact on examinees’ ranks and absolute scores?

Med Teach 2007; 29: 49–53.

Lubarsky

Dory

Audétat

, et al. Using script theory to cultivate illness script formation and clinical reasoning in health professions education. Can Med Educ J 2015; 6: e61–e70.

Dory

Audétat

Rees

. Beliefs, identities and educational practice: a Q methodology study of general practice supervisors. Educ Prim Care 2015; 26: 66–78.

Charlin

Roy

Brailovsky

, et al. The script concordance test: a tool to assess the reflective clinician. Teach Learn Med 2000; 12: 189–195.

Ducos

Lejus

Sztark

, et al. The script concordance test in anesthesiology: validation of a new tool for assessing clinical reasoning. Anaesth Crit Care Pain Med 2015; 34: 11–15.

10.

Lubarsky

Charlin

Cook

, et al. Script concordance testing: a review of published validity evidence. Med Educ 2011; 45: 329–338.

11.

Kania

Verillaud

Tran

, et al. Online script concordance test for clinical reasoning assessment in otorhinolaryngology: the association between performance and clinical experience. Arch Otolaryngol Head Neck Surg 2011; 137: 751–755.

12.

Patel

. General practice trainees’ learning experiences of formative think-aloud script concordance testing. Educ Prim Care 2022; 33: 229–236.

13.

Deschênes

Goudreau

Fernandez

. Learning strategies used by undergraduate nursing students in the context of a digital educational strategy based on script concordance: a descriptive study. Nurse Educ Today 2020; 95: 104607.

14.

Fernandez

Foucault

Dubé

, et al. Learning-by-Concordance (LbC): introducing undergraduate students to the complexity and uncertainty of clinical practice. Can Med Educ J 2016; 7: e104–e113. Accessed December 16, 2022. https://pubmed.ncbi.nlm.nih.gov/28344697/.

15.

Charlin

Deschênes

Fernandez

. Learning by concordance (LbC) to develop professional reasoning skills: AMEE guide No. 141. Med Teach 2021; 43:614–621.

16.

Gagnon

Charlin

Coletti

, et al.

Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test?

Med Educ 2005; 39: 284–291.

17.

Chantal

Driss

Robert

, et al. Learning-by-Concordance of perception: a novel way to learn to read thoracic images. Acad Radiol 2023; 30: 132–137.

18.

Foucault

Dubé

Fernandez

, et al. Learning medical professionalism with the online concordance-of-judgment learning tool (CJLT): a pilot study. Med Teach 2015; 37: 955–960.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB