Evaluating Simulator-Based Teaching Methods for Endoscopic Sinus Surgery

Abstract

A multitude of simulator systems for endoscopic sinus surgery (ESS) are available as training tools for residents preparing to enter the operating room. These include human cadavers, virtual reality, realistic anatomic models, and low-fidelity gelatin molds. While these models have been validated and evaluated as independent tools for surgical trainees, no study has performed direct comparison of their outcomes. To address this deficiency, we aimed to evaluate the utility of high-fidelity and low-fidelity trainers as compared to a traditional control (no simulator exposure) for novice trainees acquiring basic ESS skills. Thirty-four first-year medical students were randomized to 3 groups and taught basic sinus anatomy and instrumentation. Two groups received training with either the high-fidelity or low-fidelity trainer, while 1 group served as control. These groups were then tested with cadaveric specimens. These sessions were recorded and graded by an expert. There was no statistical difference in performance between the 3 study groups with regard to identification of anatomy, endoscopic competency, or completion of basic tasks. When the high-fidelity and low-fidelity arms were grouped into a single “trained” cohort, they demonstrated significantly improved time to completion for basic anatomy (P = .043) and total time (P = .041). This is the first study to perform a direct comparison of performance between high-fidelity and low-fidelity ESS simulators and controls. Although we found no difference in performance of novice trainees with regard to basic anatomical identification or procedural tasks associated with ESS, the use of ESS simulators may improve time to completion.

Introduction

Historically, resident training in endoscopic sinus surgery (ESS) has relied on the apprenticeship model to develop the technical skills, anatomic familiarity, and coordination of hand movements with endoscopic visualization necessary to perform this procedure in a compact, critical 3-dimensional space. ¹ In this system, the resident surgeon operates on live patients under the guidance of an attending surgeon through observation, supervision, and graduated autonomy to achieve proficiency. ² As the result of changing work hour restrictions, diminished experience, and the increased emphasis on quality improvement and patient safety, the operative skills of trainees graduating from surgical residency programs are diminishing. ³ These concerns have been supported by work demonstrating increased operative time ^4,5 and complication rates ⁶ for trainees as compared to experienced surgeons.

In light of these ethical and safety concerns, the implementation of surgical simulation has augmented anatomic laboratory dissection to improve trainee skills prior to surgery on live patients. ⁷ Simulators are touted as a promising, cost-effective modality to reconcile differences across residency training programs and objectively evaluate technical skills. ⁸ Of course, driving factors include improving patient safety, meeting Accreditation Council for Graduate Medical Education (ACGME) requirements in the setting of evolving work-hour restrictions, shortening the learning curve of complex procedures, and providing a low-stress environment more conducive to novices.

In response to the aforementioned limitations of intraoperative training, a multitude of simulator systems have been developed for ESS. These systems augment live patient and cadaver encounters and include virtual reality systems, ^9,10 realistic “high-fidelity” anatomic models, ^1,2,11 “low-fidelity” gelatin or 3D printed models, ^{12 –14} and ovine models. ¹⁵ Importantly, the fidelity of such constructs does not necessarily correspond to the degree of technological complexity, but rather the extent to which the simulator models the tissue properties and anatomical “correctness” of its intended structures. A recent review cites more than a dozen validation studies of simulators for functional ESS with varying levels of validation and evidence. ¹⁶ The virtual reality ES3 Endoscopic Sinus Surgery Simulator was developed by Lockheed Martin in 1998 and demonstrated construct, transfer, and face validity. ^9,10,17,18 Although no longer in production, this trainer demonstrated improved operative technique and performance during real surgery, leading to the development of next generation devices. ^10,19,20

However, barriers regarding access and cost of such high-fidelity virtual reality systems are important factors for many training programs. As a result, the introduction of low-fidelity gelatin trainers is associated with improved camera skills, instrumentation, and hand-eye coordination for trainees at a cost of < US$5 per mold. ¹⁴ To address deficiencies in anatomic realism, higher-fidelity systems modelled on computed tomography data of the nasal cavity and paranasal sinuses have been developed. ¹ While the face, content, and construct validity of low-fidelity and high-fidelity physical models is well-established for surgical trainees, no study has aimed to directly compare performance in low-fidelity gelatin models against those intended for a more anatomically accurate representation of ESS. In this study, we aimed to directly compare low-fidelity and high-fidelity simulator systems in developing skills in ESS for student learners. To further delineate the utility of surgical simulation experience for students, we constructed an untrained control group that also completed timed anatomic identification and basic surgical ESS tasks.

Methods and Materials

Approval was obtained from the Institutional Review Board at Saint Louis University School of Medicine. Informed consent was obtained for all participants. This randomized controlled study recruited 34 first-year medical students, with study design illustrated in Figure 1. All subjects received review on basic sinus anatomy and endoscopic techniques including camera and instrumentation. Sinus anatomy was reviewed through lecture format with handouts available throughout subjects participation. All subjects also studied prerecorded video demonstrating the major sinus anatomical structures and endoscopic techniques, including endoscopic techniques and instrumentation. A “hands-on” demonstration on endoscopic principles including endoscope handling, 2-handed surgical techniques, and a demonstration of available sinonasal instrumentation was performed without access to either simulator model with each subjects prior to participation.

Figure 1.

First-year medical students were all given a lecture on sinus anatomy, shown video demonstrations of ESS techniques, and familiarized with instrumentation and basic endoscopic techniques. Subsequently, participants either utilized a low-fidelity, high-fidelity, or no ESS trainer. After this session, patients were recorded and graded by a blind observer on performance during a cadaveric task session. ESS indicates endoscopic sinus surgery.

Subsequently, 2 groups of subjects received additional training on either a low-fidelity (11 subjects) or high-fidelity ESS trainer (12 subjects) with sinus instrumentation and the 0-degree endoscope. The remaining subjects (11 subjects) received no further training or experience. The low-fidelity trainer was a modified bovine gelatin construct utilizing a cylindrical cavity with embedded objects and obstacles validated by Malekzadeh et al. ¹² The high-fidelity trainer was a previously validated physical model that utilized a silicone-based matrix over a bone-like plaster construct. ¹ This model was originally designed for moderately detailed anatomy and realistic haptic feedback utilizing embedded electric probes to measure task completion and skill development. This aspect of the simulator was not employed in this study. All participants were given written instructions for the simulated cadaveric ESS and familiarized themselves prior to proceeding. The spectrum of tasks included identification of basic anatomical structures (septum, nasal floor, inferior turbinate, middle turbinate, nasopharynx, and eustachian tube opening), identification of structures of the middle meatus (uncinate process, ethmoid bulla, and maxillary sinus ostium), feigned syringe injections (middle turbinate, septum, uncinate), and foreign body removal. Subjects performed this simulated, timed cadaveric ESS without coaching or proctor assistance.

For each participant, the video-recording of cadaveric surgery was evaluated on anatomic knowledge, surgical skills including basic endoscopy, exposure of the middle meatus and probing of the maxillary sinus ostium, simulated injections, and foreign body removal of a small pipe cleaner. Notably, this was based solely on endoscopic views and did not include handling of the endoscope and instruments by the subjects. The blinded expert (M.L.) graded anonymized video recordings on a standardized global rating scale (GRS) utilizing a 5-point Likert-like scale (Table 1). ^15,22,23 Analysis of variance tested differences among the mean graded scores and times to completion for the 3 different groups. Wilcoxon signed-rank tests compared a combined “trained” group from both the high-fidelity and low-fidelity groups against the untrained group. Statistics were calculated using SAS version 9.4 (SAS Institute, Cary, North Carolina). Tests were 2-tailed, and significance was set at P < .05.

Table 1.

Global Rating Scale.^a

Use of endoscope	Inappropriate endoscopy and/or repeatedly awkward with endoscope		Appropriate endoscopy with occasional awkwardness	Appropriate, effortless endoscopy
	1          2		3	4          5
Identification of Anatomy	Unable to perform		Performs with	Performs easily
	1          2		3	4          5
Instrumentation	Repeated tentative or awkward movements		Competent use of instruments but occasionally awkward	Seamless instrumentation with no awkwardness
	1          2		3	4          5
Respect for tissue	Uses unnecessary force on tissue or caused damage by inappropriate instruments		Careful handling of tissue but occasional inadvertent damage to tissue	Consistently handled tissue appropriately
	1          2		3	4          5
Efficiency	Many unnecessary moves		Some unnecessary moves	No unnecessary moves with maximum efficiency
	1          2		3	4          5
Flow of operation	Frequent stops		Some forward planning	Effortless flow of steps
	1          2		3	4          5
Overall performance	Poor		Performs majority of surgery acceptably	Outstanding
	1          2		3	4          5

^aAdapted from Awad et al, ²¹ , Laeeq et al, ²² and Lin et al. ²³

Results

Grouped scores for the individual tasks including anatomic knowledge and nasal endoscopy, exposure and anatomic identification in the middle meatus, simulated injections, foreign body removal, and global rating for the high-fidelity, low-fidelity, and untrained groups with standard deviations are illustrated in Figure 2. There was no significant difference between the 3 groups in identification of basic anatomy (P = .230) or the middle meatus including probing the maxillary sinus ostium (P = .415). In addition, there was no difference observed regarding simulated injections (P = .469), foreign body removal (P = .200), or the overall GRS scale (P = .435).

Figure 2.

Global rating scale scores for tasks in the high-fidelity, low-fidelity, and untrained groups.

Mean time to completion in minutes for each of the tasks as well as the total time to completion for high-fidelity, low-fidelity, and untrained groups is shown in Figure 3. Notably, there was no difference in time to completion of any of the tasks or the total time to completion (range, P = .169-.647). However, when the high-fidelity and low-fidelity groups were combined into a “trained” group, this cohort had significantly shorter time to task completion for basic anatomy (P = .043) and total time (P = .041).

Figure 3.

Time to completion (minutes) for the high-fidelity, low-fidelity, and untrained groups.

Discussion

Although a wide variety of simulator systems are available as validated ESS training tools for medical students and residents, this is the first study to directly compare groups receiving high-fidelity training, low-fidelity training, and an untrained group receiving no “hands-on” instructional experience or simulation. Previous research to validate such sinus trainers focuses mainly on pre- and post-training performance measures at a wide range of training levels. Few studies have examined the time to completion for endoscopic sinus tasks. ^1,24 To date, most of the literature on simulators in ESS focuses on the development of new, moderately novel trainers and their subsequent validation. The absence of standardized recruitment, outcomes, and statistical measures for these relatively low-powered studies means that forming meaningful comparison and conclusions is difficult. ¹⁶ Consequently, choosing one simulator over another is based on opinion, accessibility, and cost.

Direct comparison between the 3 groups in this study yielded no statistical difference in performance or time to completion. The high-fidelity group had the highest mean GRS (P = .58) and trended to have higher scores on tasks that relied more on anatomical knowledge, although they did not perform significantly better. This finding may reflect the possibility of improved familiarity with sinus anatomy for this group, which would be expected with the use of a trainer that heeds anatomic realism. Although not significant, the low-fidelity group demonstrated the highest score on foreign body removal, which in theory could reflect improved training, tactile feedback, and manipulation based on experience navigating the embedded objects and malleable obstacles of the low-fidelity model. No group alone in this study had a significantly longer time to completion. However, when the low-fidelity and high-fidelity groups were grouped into a single “trained” group and compared against the control, an improvement in time to completion was noted in 2 categories (P = .041-0.043) indicating that either type of sinus trainer may yield faster times for certain tasks.

This study involved the participation of 34 first-year medical students, which is a relatively large, homogeneous group for this subject area. Nonetheless, the possibility of type II error implies that a significant difference between ESS trainers may not be identified in this study due to limited power. While the standardized GRS used herein has previously shown utility in the evaluation of training simulators, it must be recognized that such a scale has an ordinal, rather than interval, importance, and analysis of such results should be viewed with appropriate caution. Other limitations include the absence of more experienced participants such as residents or attending surgeons. It is possible that the relative unfamiliarity of first-year medical students with sinonasal anatomy and endoscopic techniques means that this population is not appropriate for performing certain tasks related to ESS, and the tasks being tested were too complex and demanding for first-year medical students, regardless of the exposure to simulators. This study is unable to comment on construct validity, as we did not include more experienced students, such as residents and attending surgeons. In addition, due to the planned absence of a pretest simulation scenario, conclusions regarding transfer validity are unable to be drawn.

Conclusions

The role of simulation in otolaryngology is well-established. Although a multitude of simulators have been developed and validate for ESS, the work herein makes the first direct comparison of students undergoing high-fidelity, low-fidelity, and no further training on ESS simulators. For the novice surgical learner, experience during a single ESS trainer session did not result in improved performance on simulated cadaveric endoscopic tasks, although it may improve time to completion in 2 areas. Despite recent innovation and validation in simulation-based training, it is important that such modalities are utilized as adjuvant, not replacement, tools in surgical resident education. ²⁵ Currently, this subject area lacks standardized recruiting and reporting measures which could potentially aid in further research and comparisons. Undoubtedly, additional investigations and larger studies are needed to further elucidate the role of simulation in ESS training.