Abstract
Objective
Appropriate timing of subspecialty simulation is critical to maximize learner benefit and guide resource utilization. We aimed to determine optimal timing of a simulation-based curriculum designed to teach entry-level procedural skills for otolaryngology residency.
Study Design
Simulation curriculum intervention tested among 3 comparison groups of varying clinical levels.
Setting
Academic otolaryngology training program and medical school.
Subjects and Methods
We developed a simulation-based technical skills curriculum incorporating the following task trainers: flexible laryngoscopy, peritonsillar abscess drainage, and myringotomy and tube insertion. Preclinical medical students (n = 40), subintern rotators (n = 35), and midyear interns (n = 8) completed the simulation-based curriculum. Pre- and postintervention knowledge/confidence and “level appropriateness” were rated on a 5-point Likert scale, and effect size was calculated.
Results
Overall self-reported knowledge/confidence levels improved in all 3 groups preintervention (1.05, 2.15, 3.17) to postintervention (2.79, 3.45, 4.38, respectively; all P < .01). Preclinical medical students uniformly reported very little to no familiarity with the procedures prior to the curriculum, while interns approached independence following the intervention. Large effect sizes were seen in all tasks for preclinical students (d = 3.13), subinterns (d = 1.46), and interns (d = 2.14). Five-point Likert scale measures of level appropriateness (1 = too challenging, 5 = too easy) for preclinical students, subinterns, and interns were 2.70 (95% CI, 2.56-2.84), 3.11 (95% CI, 2.97-3.25), and 3.75 (95% CI, 3.35-4.15), respectively.
Conclusion
Subinternship may represent the optimal timing for entry-level skills simulation training. The proposed curriculum shows utility for clinical levels ranging from medical students to postgraduate year 1 resident levels, with large effect sizes for all tested groups.
In recent years, medical simulation has been rapidly gaining acceptance as a valuable resident education tool. 1 Simulation has been associated with the acquisition of competency in procedural and communication skills, increased learner confidence, improved patient safety, and potential health care system cost reduction.2-5 The Accreditation Council for Graduate Medical Education has recognized simulation as a means for demonstrating anatomic knowledge during procedural skills to fulfill an otolaryngology (ORL) program requirement. 6 As such, the majority of ORL residency programs have now incorporated some form of simulation into their scheduled curricula.6,7
While simulation is useful across all specialties, it may be particularly useful in surgical subspecialties such as ORL, where there is limited medical school exposure to the skills required early on in residency.7,8 Results showing utility of simulation for entry-level ORL resident procedures have been published, including nasolaryngoscopy, bronchoscopy, and myringotomy with tympanostomy tube placement.9-11 Additionally, the utility of a boot camp experience where multiple skills are simulated over the course of 1 to 2 days has been demonstrated.12-15
Most of this previous work has focused on graduate medical education, specifically ORL residents; however, with greater implementation and access to simulation training, preclinical and subinternship medical students may be feasibly included in training sessions. Notably, there are current efforts sponsored by the Society of University Otolaryngologists to develop a national simulation-based boot camp curriculum to be implemented regionally, with the aim to increase access to surgical simulation and provide a standard experience for trainees across the country; this model has been demonstrated successfully in the disciplines of neurosurgery and thoracic surgery, and Canada has deployed a national ORL boot camp curriculum since 2012.8,16-18 In this context, the question of optimal timing for boot camp experiences is especially pertinent. This study sought to determine the appropriate timing and learner level of experience for entry-level ORL simulation tasks to maximize educational benefit.
Materials and Methods
Boot Camp Design and Participants
This study was determined to be exempt by the University of Michigan Internal Review Board. A simulation-based technical skills curriculum was developed incorporating the following task trainers: flexible laryngoscopy, peritonsillar abscess drainage, and myringotomy and tube insertion. These tasks were selected because they are common intern-level procedures with preexisting task trainers. The flexible laryngoscopy trainer comprised performance of the task with a high-fidelity airway mannequin (TruCorp Airsim; TruCorp Ltd, Belfast, Northern Ireland). The peritonsillar abscess and myringotomy and tube task trainers were used as described by Bunting et al and Hong et al, respectively.11,19 The educational intervention consisted of 3 stations, each with 2 to 4 learners and each run by 1 or 2 senior residents and/or faculty who had experience with the task trainers and associated clinical tasks. Materials outlining learning objectives, background information, and basic procedural steps for each of the 3 simulation task trainers were provided to learners prior to the start of the intervention. Participants where included on the basis of voluntary participation. The intern group consisted of 2 consecutive classes of ORL interns at our institution (2016, 2017). The subintern group included all rotating fourth-year medical students on the ORL elective during the same period. The preclinical medical student pool included first- and second-year medical students who responded to an email request to participate in the proposed simulation course, and tested students were chosen at random from this volunteer pool to match the number of participants in the subintern group.
Assessment Tools and Outcome Measures
Prior to the simulation intervention, learners completed a self-assessment form for each proposed task, which included 5-point Likert scale measures of knowledge and confidence (anchors: 1 = no knowledge, unable to perform; 2 = some knowledge, need a lot of guidance; 3 = basic knowledge, guidance still needed; 4 = reasonably confident, some guidance needed; 5 = highly knowledgeable and confident, independent). An identical self-assessment form was completed following the intervention. On the postintervention questionnaire, learners were also asked to rate the “appropriateness” of the course overall on a 5-point Likert scale (1 = too challenging, 3 = appropriate for level, 5 = too easy). Additional free-response questions tailored to the level of clinical training were included on the assessment. Preclinical medical students were queried about prior exposure to ORL and simulation and about interest in ORL as a career. Subinterns and interns were queried on the elements of residency about which they are most anxious and which preparation/learning methods they find most useful. All participants were given an opportunity to give feedback about the experience. Participation in the study was not required for participation in the simulation session, and voluntary participation was indicated by completion of the assessment forms.
The primary outcome of this study was change in knowledge and confidence levels following the educational intervention. The secondary outcome was learner-assessed appropriateness of the boot camp for one’s level of education. Statistical assessment of curriculum efficacy included Wilcoxon sign rank test and effect size (Cohen’s d). Effect size is a standardized mean difference between 2 groups, and in this context, effect size complements metrics of statistical significance. 20 Effect sizes have the advantage of being readily comparable across interventions with different measures of effectiveness. With normally distributed data, effect sizes are equivalent to Z scores, meaning an effect size of 1, 2, or 3 would indicate that the mean postintervention score would exceed the preintervention score of about 84%, 87%, and 99.9% of the same group, respectively. Here, effect size reports the magnitude of difference between pre- and postintervention scores (ie, “effectiveness”); is not confounded by sample size; and is conventionally reported as small (0.2), moderate (0.5), or large (>0.8). 20 Responses to open-ended questions were compiled, and content analysis was performed to identify themes and the most common elements of feedback found in open-ended comments for each of the following categories: (1) timing of intervention, (2) prior clinical knowledge, (3) familiarity with tasks, (4) clinical experience with tasks, and (5) primary value of course.
Results
Efficacy, Effect Size, and Appropriate Timing of Intervention
Preclinical medical students (n = 40), subintern rotators (n = 35), and midyear interns (n = 8) completed the curriculum over a 2-year period. Overall mean self-reported knowledge/confidence levels improved for each of the 3 groups based on the 5-point Likert measure: preclinical medical students (preintervention = 1.05 to postintervention = 2.70, Wilcoxon sign rank test, P < .01), subinterns (2.15 to 3.45, P < .01), and interns (3.17 to 4.38, P < .01). These results are broken down by individual task and represented in Figure 1 (Wilcoxon sign rank test, all P < .01).
Pre- and postintervention knowledge and confidence scores. Improvements were seen for all tasks at all levels (P < .01, Wilcoxon sign rank). Note that postintervention scores equal or surpass preintervention scores of next-highest experience group.
Large effect sizes (Cohen’s d) were observed in all tasks for preclinical students (d = 3.54), subinterns (d = 1.69), and interns (d = 1.98). Effect sizes for individual tasks by group are shown in Table 1 , and notably all effect sizes are large.
Pre- and Postintervention Scores and Effect Size by Group. a
All tasks at all training levels shows a statistically significant improvement (P < .01, Wilcoxon sign rank).
Based on a 5-point Likert scale: 1 = no knowledge, unable to perform; 2 = some knowledge, need a lot of guidance; 3 = basic knowledge, guidance still needed; 4 = reasonably confident, some guidance needed; 5 = highly knowledgeable and confident, independent.
Effect size (Cohen’s d): small, 0.2; moderate, 0.5; large, >0.8. All effect sizes are large.
Five-point Likert scale measures of learner-reported level appropriateness (1 = too challenging, 5 = too easy) for preclinical students, subinterns, and interns were 2.70 (SD, 0.46; 95% CI, 2.56-3.84), 3.11 (SD, 0.42; 95% CI, 2.97-3.25), and 3.65 (SD, 0.58; 95% CI, 3.35-4.15), respectively ( Figure 2 ). Self-reported measures showed statistical difference, indicating distinction among groups (P < .01). In this context, scores <3.00 indicate that the course was perceived as too challenging; scores >3.00, too easy; and scores centered on 3.00, an ideal level for the learner’s level of training.
Learner perceptions of level appropriateness. Mean and 95% CIs are shown for each learner level. Subintern group shows reported measures indicating closest match of curriculum to “appropriateness for level.”
Participant Feedback
Common themes from learner feedback were identified within the following categories: (1) timing of intervention, (2) prior clinical knowledge, (3) familiarity with tasks, (4) clinical experience with tasks, and (5) primary value of course. Content analysis was performed to identify the most common responses and themes within each category, as reported in Table 2 . Preclinical students reported little prior ORL exposure, and many participants felt that it would be valuable to start with more basic tasks or additional clinical background prior to the intervention. A common theme of subintern feedback included the perception of sufficient clinical foundation yet limited live procedural experience. Finally, midyear interns thought that the simulation tasks were generally useful, although they reported having prior and continued clinical exposure to these tasks on a regular basis.
Assessment Tool Learner Responses: Content Analysis of Open-ended Questions. a
Abbreviations: ORL, otolaryngology; PGY-1, postgraduate year 1.
Responses to level-specific open-ended questions and identification of common themes based on learner feedback.
Open-ended response content analyzed for the following categories: (1) timing of intervention, (2) prior clinical knowledge, (3) familiarity with tasks, (4) clinical experience with tasks, and (5) primary value of course.
Medical student assessments included questions about prior experiences with ORL and simulation. Very little prior exposure to ORL was noted ( Table 2 ). Medical students were further asked about the influence that this intervention had with regard to ORL as a career choice (5-point Likert scale, 1 = less interested, 3 = no change, 5 = more interested), with a mean reported rating of 4.35 (SD, 0.48). Subinterns and interns were asked about anticipated sources of anxiety at the start of residency and effective learning methods to alleviate these concerns.
Discussion
Timing of Intervention
The practical question this study aims to answer is how and when entry-level skill simulation should be implemented. The most important finding from this study is that medical students identify benefit from ORL simulation training traditionally provided to current residents. In fact, medical students on a subinternship found the simulation training most appropriate for their skill level, indicating that they may be the ideal cohort with whom to work with on entry-level ORL simulation training. Our results suggest the task trainers utilized provide significant value to learners of varying levels of training, though perhaps for distinctly different reasons. Each group also had a large effect size, which complements statistical significance between pre- and postintervention.
Interestingly, the postintervention level of knowledge/confidence associated with each task surpassed the preintervention level of the next more experienced level of learner ( Table 1 ). For example, for each task, preclinical medical student postintervention knowledge/confidence scores surpassed subintern pre-intervention scores. Our group previously identified this trend, where junior residents self-rated their abilities as commensurate with those of a senior resident following a simulation curriculum teaching pediatric airway foreign body removal. 21 Such results provide a compelling argument to incorporate simulation early in training to enhance learner knowledge and skill development prior to performing these interventions on patients. Furthermore, as training requirements from the accrediting bodies continue to increase, 6 early proficiency will be necessary to continue to meet these Accreditation Council for Graduate Medical Education standards while providing safe and effective patient care. Implementation of level-appropriate simulation curricula may maximize trainee learning at the bedside to achieve these goals.
Preclinical medical students often have limited exposure to subspecialty fields such as ORL, which was corroborated by our data. Learners in this group reported a significant improvement in knowledge/confidence related to the tasks with a large effect size; however, the tasks were reported as trending toward too challenging for their level of training. Low familiarity with the chosen clinical tasks was reported, with a preference toward incorporating more basic tasks (eg, head and neck physical examination) and background clinical information into the curriculum. Increased interest in ORL was reported following the intervention, suggesting that exposure to ORL simulation early in medical school may influence students to consider ORL as a career choice. This benefit of an ORL-specific simulation experience should not be underestimated. Garnering early medical student interest in our specialty is of particular interest, as the applicant pool in ORL has steadily decreased since 2014. 22 We suggest the future simulation curricula targeting preclinical medical students would benefit from incorporating more basic task trainers with an added focus on providing robust clinical background information.
The subinternship group of learners reported significant improvement in knowledge/confidence related to the tasks, with a large effect size for each task. The mean 5-point Likert scale measure of learner-reported level appropriateness for this group was 2.89 (95% CI, 2.75-3.03), spanning the middle anchor of the scale (3, “appropriate for level”). This suggests that subinterns represent an ideal level of training to participate in this simulation-based ORL curriculum ( Figure 2 ). This group self-reported an adequate level of prior clinical knowledge and familiarity with the specific tasks performed, allowing for optimized utilization of time focused on learning the specific techniques associated with completing each task. As a secondary benefit to this study, we feel that this curriculum added a valuable standardized experience during the rotation month, which can otherwise show great variation in type and volume of clinical exposure.
The intern group of learners likewise reported significant pre- to postintervention improvements with large effect sizes; however, many learners reported having already gained significant clinical experience performing these entry-level ORL procedural skills and preference for exposure to the chosen simulation tasks earlier in training. The learner-reported level appropriateness for this group trended toward these tasks being too easy, likely given the high volume and routine performance of these tasks in the true clinical setting. Importantly, learners still felt that the intervention was useful for honing procedural skills and may be a valued supplement to their training on an individual basis. Indeed, simulation has been shown to aid in retention of surgical skills when repeated at spaced intervals, although the optimal spacing and frequency through training remains unclear. 23
Midyear interns were chosen as a study group to provide stratification of clinical experience, as the learning curve in residency is steep, and a starting intern likely has similar ORL experience as a rotating subintern.
Simulation Curricula
The 3 tasks selected for the study curriculum—flexible laryngoscopy, peritonsillar abscess drainage, and myringotomy and tube insertion—are commonplace among ORL training programs. Initial exposure to these tasks often occurs during the intern year. Within the inpatient setting, junior residents are often the first physicians to evaluate patients who require these diagnostic and therapeutic interventions.24,25 Therefore, it is prudent that competency of these entry-level tasks be achieved early within the residency experience. However, prior studies demonstrated that a learning curve exists among residents learning flexible laryngoscopy.24,26 The current reality of residency time constraints has sparked the development of multiple feasible task trainers for common procedures.9,11,19,25,27,28 Use of low-cost, well-described, and readily accessible task trainers in this study makes dissemination and implementation of the described curriculum feasible at other institutions.
There are currently at least 9 regional simulation-based ORL boot camps in the United States and a national ORL boot camp in Canada, and there are ongoing efforts to create a national boot camp curriculum in the United States. Discussions have revolved around which tasks to include and when to provide this experience, further highlighting a need to determine appropriate timing of simulation intervention and efficient use of resources. Such a model may theoretically include a common experience for all applicants matching into ORL. Our study suggests that the most appropriate timing for such an intervention would be just prior to or shortly after starting the intern year. Our institution currently holds a more intensive 1-day boot camp focusing on ORL emergencies and targeting preparation for postgraduate years 1 and 2. With exceedingly more advanced simulation trainers available,29-31 consideration should also be given to separate boot camps tailored toward higher-level trainees, such as that provided at the University of Pennsylvania targeted for late postgraduate year 3. We posit that the same principles will hold true across levels of training, with determination of level-appropriate simulation tasks to optimize learner value and pursuit of transferrable clinical competence being paramount.
Further Study and Limitations
Limitations of this study include a limited sample size collected from a single institution, a limited number of simulation tasks, and the use of learner-reported data. Learner recruitment for the intern group was limited by the available class size. Despite these limitations, statistically significant pre- to postintervention improvements and large effect sizes were identified in all learner groups. An effort to avoid biased response or coercion was made through voluntary participation and deidentified responses.
This study was designed to determine the appropriate timing of a simulation-based curriculum designed to teach critical entry-level procedural skills for ORL residency. Further studies should be performed to determine which procedural skills are most appropriate and of most benefit to learners at various levels of training, including additional tasks to complement those tested at the subinternship level.
Outcome measures in this study were defined by self-evaluation with a Likert scale, without implementation of objective measures to assess true acquisition of procedural skill. This limitation can be overcome in future iterations of the curriculum by using objective measures such as task-specific checklists, global rating scales, or Objective Structured Assessment of Technical Skills, similar to those already used at our institution and others.32,33 A deliberate decision was made to avoid inducing undue stress on our subintern group through structured evaluation, as this group represented the current year’s applicant group. The aim of this study was educational rather than a summative assessment, and use of performance to predict surgical aptitude remains controversial. 34 The use of objective assessment tools will help to further elucidate the difference between perceived benefit and actual benefit and may ultimately be used to show improvements that translate to clinical practice.
Conclusion
In an era of efforts to nationalize boot camp curricula, we aimed to determine efficacy and appropriate timing of a simulation-based curriculum designed to teach critical entry-level ORL procedural skills. The described curriculum showed positive effect across all groups tested, with differing benefits suggested by themes of open-ended learner feedback. Learner-reported measures indicate that subinternship may be the most appropriate timing for entry-level simulation training. Future studies should be aimed at incorporating objective measures of skill acquisition and determination of simulation tasks tailored to learners’ levels of experience.
Author Contributions
Disclosures
Footnotes
Acknowledgements
We acknowledge Patricia Mullan, PhD, Larry Gruppen, PhD, and Gurjit Sandhu, PhD, for input on study design and Cher Zhao, MD, for aiding with the subinternship course.
Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.
This article was presented at the AAO-HNSF 2018 Annual Meeting & OTO Experience; October 7-10, 2018; Atlanta, Georgia. This work was also presented at the Central Group on Educational Affairs Meeting; March 30, 2017; Chicago, Illinois.