Septoplasty Training During the COVID-19 Era: Development and Validation of a Novel Low-Cost Simulation Model

Abstract

Objective

In a context of increasingly limited surgical exposition, enhanced by the coronavirus disease 2019 (COVID-19) pandemic context, the objective of this article is to explain the development of a novel low-cost and simple replication animal-based septoplasty training model for otolaryngology residents, to assess its face and construct validity, and to validate a specific rating scale for each task.

Study Design

Experimental study.

Setting

Surgical simulation laboratory.

Methods

Septoplasty experts divided the procedure into key tasks. A simulator model to perform tasks was developed using pig ears to imitate human nasal septum cartilage, and a Specific Rating Scale was constructed. Trainees and faculty performed all tasks in the model. The participants were videotaped, and operative time, hand movements, and path length were recorded using a motion sensor device. Two blinded experts evaluated the videos with Global and Specific Rating Scales. All participants answered a satisfaction survey.

Results

Fifteen subjects were recruited (7 trainees and 8 faculty). Significantly higher Global Rating Scale score, shorter operative time and path length, and fewer hand movements were observed in the faculty group. The satisfaction survey showed high applicability to a real scenario (mean score of 4.6 out of 5). Specific Rating Scale showed construct and concurrent validity and high reliability.

Conclusion

This simulation model and its specific rating scale can be accurately used as a validated surgical assessment tool for endonasal septoplasty skills. Its low cost and simple replicability make it a potentially useful tool in any otolaryngology surgical training program.

Keywords

rhinology septoplasty simulation training validation medical education

Deviation or deformity of the nasal septum is a frequent cause of nasal obstruction and one of the most common complaints in the average rhinology practice.^1,2 Septoplasty is a surgical intervention to correct a septal deformity, often indicated when medical management fails. It is a core procedure in otolaryngology and one of the first learned by residents during their training, for it provides an entry to more complex procedures and endoscopic approaches.³

Regardless of the technique, septoplasty comprises mucosal preservation, flap elevation, and preservation of a cartilage frame to ensure nose stability.² There are 3 surgical approaches to septoplasty described in the literature: traditional endonasal, open rhinoseptoplasty, and endoscopic.

The traditional endonasal approach, most commonly used in our experience, involves direct visualization using a headlight and a nasal speculum, looking down into the nasal cavity from above, allowing little space for an assistant to observe.⁴ Therefore, this procedure can be challenging to teach in the operating room, encouraging trainers and trainees to find alternative ways to understand anatomy and surgical steps. It has been proposed that junior residents require at least 20 septoplasties to approximate a more experienced surgeon.⁵ However, a UK study reported that only 73.3% of senior residents consider themselves prepared to perform a septoplasty independently.³ According to a study performed in our institution, residents were exposed to a mean of 18 endonasal septoplasties during their training in otolaryngology.⁶ Other Chilean institutions report a median of only 12 septoplasties per trainee.⁷ This situation has significantly worsened during the past year due to the coronavirus disease 2019 (COVID-19) pandemic, which has led to a restriction in elective procedures, reducing residents’ exposure to this procedure.

In this scenario, surgical simulation has been demonstrated to be an efficient and effective tool in surgical skills acquisition.⁸ It offers a safe and realistic environment in which trainees can practice surgical skills, reducing patient risk and improving training residents’ confidence and performance.^9-11 Simulation models allow for deliberate practice, an objective structured form of training, with real-time feedback and unbiased evaluation.^11,12

Septoplasty training models described in the literature are scarce. A 3-dimensional printing model for endoscopic septoplasty training was created and validated in 2016 in Canada.¹³ Likewise, simulation models for training rhinoseptoplasty have been developed using sheep,¹⁴ chicken,¹⁵ rabbit,¹⁶ porcine,¹⁷ silicone,¹⁸ and human cadaver.¹⁹ However, to our knowledge, there are no simulation training models for endonasal septoplasty described to date.

The objectives of this study are to explain the development of a novel and low-cost animal-based endonasal septoplasty training model for otolaryngology trainees, to assess its face and construct validity, and to validate a specific rating scale for each task.

Materials and Methods

Simulator Fabrication

The first step in constructing the simulation model was to separate endonasal septoplasty surgery into key tasks that trainees should be competent in before facing the entire procedure. Two experts in septoplasty from the Department of Otolaryngology at the Pontifical Catholic University of Chile identified the main steps in septoplasty. After 3 meetings, 4 key tasks were chosen: superior tunnel, inferior tunnel, septum deviation resection, and suturing.

The first task (superior tunnel) in septoplasty consists of dissecting within the subperichondrial plane and elevating the perichondrium and mucosa from the cartilage intact using a Killian nasal speculum, a dissector, and a headlight.² Second (inferior tunnel), the caudal end of the inferior border of the cartilaginous septum is separated from the premaxilla, which requires tunnel-view skill using the headlight.⁵ Third (deviation resection), after both side flaps are elevated, a No. 15 blade is used to gently incise the septal deviation, preserving an adequate L-strut for nasal support, while elevating a contralateral mucosa flap to prevent septal perforation. Then, the deformity is separated from its cephalic attachment, using a rotational movement, and removed.² In the fourth and final task (suturing), once the septal deformity is resected, flaps are reapproximated using an absorbable suture, and the initial mucosal incision is closed.^2,5

Constructing the Simulation Model

Once these 4 key tasks were identified, a low-cost septoplasty model was designed with experts’ guidance. To simulate human nasal septum, diverse materials derived from silicone and ex vivo animal tissues were tested, with the porcine ear found to be the most successful due to its similarity to the human septum. A wooden frame with a nasal septum shape was constructed to support a partially dissected porcine ear ( Figure 1A ). A wooden base to support the frame was then developed ( Figure 1B ). This base could be placed on another 45° base ( Figure 1C ), which supported a synthetic human head consisting of a mask stuffed with a sponge ( Figure 1D ). Pig ears were purchased at a slaughterhouse in Santiago, Chile. Ears were preserved under refrigeration for no more than 2 days. Half an hour before the participant’s training, ears were predissected, separating the skin from cartilage excepting the frame zone. Then, the skin of the ear was fixed to the wooden frame on both sides with square brackets to support it ( Figure 1C ).

Figure 1.

(A) Wooden frame support, (B) wooden base supporting frame, and (C) 45° wooden base. The skin was bracketed to frame. Nondissected cartilage inside the frame and (D) human mask in position.

Simulation Assessment

A Specific Rating Scale (SRS) was constructed considering a list of fundamental objectives for each 1 of the 4 key tasks. Ratings were collected for 3 variables within each of the 4 key tasks and overall technique. A Likert scale from 1 to 5 was used for each rating, with a maximum possible score of 75 points ( Table 1 ). Instructive videos explaining each task, based on the SRS, were recorded and shown to the participants before the training. Participants received the simulator model prepared for each task. Also, they had available a headlight, a Killian nasal speculum, a No. 15 scalpel blade, a Cottle dissector, surgical forceps, a Mayo-Hegar, and a 3-0 Vicryl suture.

Table 1.

Specific Rating Scale.

Task 1: Superior tunnel making	Not achieved		Partially achieved		Completely achieved
Correct place of mucosa incision using a scalpel	1	2	3	4	5
Correct plane flap elevation (mucoperichondro-periosteal)	1	2	3	4	5
Dissected superior tunnel percentage	0%		50%		100%
Dissected superior tunnel percentage	1	2	3	4	5

Task 2: Inferior tunnel making
Nasal spine identification (correct color)	NO (1)	—	—	—	YES (5)
Inferior tunnel release and display (correct symbol sequence identification on nostril’s floor)	None	1 to 2	3 to 4	5	6
	1	2	3	4	5
Ethmoid’s perpendicular lamina identification (correct color)	NO (1)	—	—	—	YES (5)

Task 3: Cartilage deviation resection	Not achieved		Partially achieved		Completely achieved

Septal cartilage incision	1	2	3	4	5
L-strut preservation (1 × 1 cm)	1	2	3	4	5
Graft obtention	1	2	3	4	5

Task 4: Suturing	Not achieved		Partially achieved		Completely achieved

Preforms 4 stitches from 1 septum side to the other	1	2	3	4	5
Functional and firm suture	1	2	3	4	5
Needle preservation	1	2	3	4	5

Transversal skills	Not achieved		Partially achieved		Completely achieved

Headlight adequate illumination	1	2	3	4	5
Nasal speculum usage	1	2	3	4	5
Mucosa preservation	1	2	3	4	5

Simulator Validation

Before training, informed consent was obtained, and an initial demographic survey was completed by all participants. Two groups were identified based on their training experience: trainees and faculty. Each participant was recorded performing all 4 tasks. During every procedure, certain parameters were registered: the operative time, the number of hand movements, and path length traveled by each hand. The last 2 were measured using the Imperial College Surgical Assessment Device (ICSAD), a motion-tracking device that involves tracking sensors on the dorsum of each hand.²⁰ Once all tasks were completed, the final surgical product on the pig ear was photographed. Subsequently, 2 blinded experts then assessed the videos and the final product photographs using the validated Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills (OSATS), which is widely used and applied for objectifying skills acquisition ( Table 2 ),²¹ and the SRS created by experts in this study. Once training was finished, a satisfaction survey (4 areas, each with a score of 1 to 5) was completed by participants ( Table 3 ). All groups’ performances were compared to assess the construct validity of the simulation model. The satisfaction survey data were collected to assess face validity. Correlation between the construct obtained with SRS and OSATS was assessed to validate the newly developed rating scale.

Table 2.

Global Rating Scale (Objective Structured Assessment of Technical Skills).

Respect for Tissue
1	2	3	4	5
Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments		Careful handling of tissue but occasionally caused unintended damage		Consistently handled tissue appropriately with minimal damage

Time and Motion
1	2	3	4	5
Several unnecessary moves		Efficient time/motion but some unnecessary moves		Clear economy of movement and maximum efficiency

Knowledge and Handling of Instrument
1	2	3	4	5

Lack of knowledge of instruments		Competent use of instruments but occasionally appeared stiff or awkward		Obvious familiarity with instruments

Flow of Operation
1	2	3	4	5
Frequently stopped the procedure and seemed unsure of next move		Demonstrated some planning with the reasonable progression of the procedure		Obviously planned course of procedure with effortless slow from one movement to the next

Use of Assistants
1	2	3	4	5
Consistently placed assistants poorly or failed to use assistants		Appropriate use of assistants most of the time		Strategically used assistants to the best advantage at all times

Knowledge of Specific Procedure
1	2	3	4	5
Deficient knowledge. Needed specific instructions at most steps		Knew all important steps of the procedure		Demonstrated familiarity with all aspects of the procedure

Table 3.

Satisfaction Survey Administered to Participants.

Item	Strongly disagree				Strongly agree
1. I believe that the simulation model is useful for learning septoplasty skills	1	2	3	4	5
2. I believe that the simulation model applies to procedures in a real clinical setting	1	2	3	4	5
3. I would recommend training with the simulation model to a colleague	1	2	3	4	5
4. I would like to have this training as part of my formal instruction in septoplasty	1	2	3	4	5

Statistical Analysis

Data were analyzed with STATA version 13.0 (StataCorp). Descriptive statistics are shown, including the median and range. Given the small sample size, construct validity was assessed using the Mann-Whitney-Wilcoxon test (nonparametric), searching for statistically significant differences among groups’ performances for each parameter. Face validity was measured with mean and standard deviation for each item of the satisfaction survey. Correlation between SRS and OSATS was assessed with Spearman correlation. Internal consistency was assessed using Cronbach’s α for SRS and GRS. P < .05 was considered statistically significant. The sample size was calculated based on parameters from similar previous publications (standard deviation 1.25 and a mean difference of 2.3),²² obtaining a minimum of 5 trainees and another 5 nontrainees.

Ethical Concerns

This project was formally reviewed and approved by the Scientific Ethics Committee at the Faculty of Medicine of the Pontifical Catholic University of Chile, Santiago, Chile (ID 181129010).

Results

Fifteen subjects were recruited (7 trainees and 8 faculty). Participants’ baseline characteristics are shown in Table 4 .

Table 4.

Participant Characteristics.

Characteristic	n	Sex, M/F, No.	Age, median (range), y	Years in practice, median (range)	No. of septoplasties, median (range)	Septoplasty instruction, % (No./total No.)
Trainees	7	5/2	30 (25-34)	2 (1-3)	3 (0-15)	28 (2/7)
Faculty	8	5/3	41 (29-58)	12.5 (3-30)	67.5 (20-6000)	100 (8/8)

Construct Validity

The quality of performance analysis showed a significantly better score for faculty in the GRS using OSATS, with a median score of 23.25 out of 30 (range, 17.5-30) compared to 17.5 points obtained by trainees (range, 13-22.5) (mean score difference, 6.68 points; 95% CI, 1.75-11.61; P = .0145) ( Figure 2A ).

Figure 2.

Boxplot showing Objective Structured Assessment of Technical Skills (A, points), operative time (B, minutes), hand movements (C, number), and path length (D, meters). Significant differences among groups were found (Mann-Whitney-Wilcoxon test). Brackets: Participants per group.

In terms of operative time, considering total time to complete all tasks, the analysis showed shorter median operative time for faculty (11.7 minutes; range, 6.68-26.61) compared to trainees (25.1 minutes; range, 16.43-29.52) (mean time difference, 9.84 minutes; 95% CI, 2.44-17.24; P = .022). Two participants, one in each group, could not complete the fourth task due to the sutures’ needle breaking. Only participants who completed the 4 tasks were considered in this analysis (13 of 15) ( Figure 2B ).

To register the participants’ hand movements and path length assessment, ICSAD was available for only 12 participants, excluding 1 trainee and 2 faculty. The analysis was made considering participants who completed all tasks. Therefore, the same 2 participants excluded from the operative time analysis for incomplete tasks were also excluded from this analysis. Significantly fewer hand movements were observed in the faculty group with a median number of 247 (range, 212-408) throughout the 4 tasks, compared to 465 (range, 205-465) in the trainee group (mean difference, 169.6; 95% CI, 60.58-278.61; P = .028) ( Figure 2C ). Similarly, significantly shorter path length was observed in faculty members (64.93 m; range, 51.14-97.88 m) as compared to trainees (121.27 m; range, 111.37-145.059 m) (mean difference, 53.23 m; 95% CI, 30.55-75.91; P = .009) ( Figure 2D ).

Specific Rating Scale Validation

The SRS created for the assessment of the simulation model tasks also showed significantly better scores according to a higher level of expertise, with a median of 56.5 (range, 50-66) and 64.75 (range, 56.5-68) points, for trainees and faculty, respectively (mean score difference, 5.66 points; 95% CI, 0.56-10.78; P = .031) ( Figure 3A ). Furthermore, a significant correlation (ρ = 0.9276, P < .001) between SRS and OSATS was found ( Figure 3B ). Internal consistency, assessed with Cronbach’s α, was statistically significant at 0.8288 for SRS.

Figure 3.

(A) Boxplot showing Specific Rating Scale (SRS) median scores. Significant differences among groups were found (Mann-Whitney-Wilcoxon test). Brackets: Participants. (B) A significant Spearman correlation between Global Rating Scale (GRS) and SRS was found.

Face Validity

In the satisfaction survey, 100% of participants found the simulation model to be useful in learning septoplasty skills. In terms of applicability to a real scenario, participants valued the model with a mean (SD) score of 4.6 (0.51) out of 5. When participants were asked if they would recommend using the model to a peer, the mean (SD) score was 4.9 (0.26) out of 5. Finally, all participants would have liked to use the model during their otolaryngology training.

Discussion

In this study, we created a low-cost and straightforward replication of an endonasal septoplasty simulation model as a tool for training and objectively assessing otolaryngology trainees. The model was achieved using pig ears to simulate the human nasal septum, wood planks, and a human latex mask, with a total estimated cost of USD 30. Only the pig ears need to be replaced after the procedure (USD 2). The model assembly requires a previous subperichondrial ear dissection that can be performed by any trainee in a short time (30 minutes). These features make this model accessible to all otolaryngology training programs.

In a context of increasingly limited surgical exposition, enhanced by the COVID-19 pandemic context where elective procedures and the number of people remaining in the operation room should be restricted,²³ our low-cost model may be especially useful for junior residents who may achieve basic competence faster and thus take more advantage of surgical opportunities. Furthermore, it has been described that resident involvement in septoplasty correlates with increased time and cost, especially in junior resident–involved cases.^24,25 In our context, the median of septoplasties performed by trainees is below the expected learning curve described in the literature (20 procedures)⁵ and the Accreditation Council for Graduate Medical Education (ACGME) case log data (mean 52.9 procedures; range, 6-20).²⁶ Hence, the development of this simulation training model can potentially facilitate the practice of essential steps of this complex procedure for novice learners and thus shorten the learning curve.²⁷ To our knowledge, there is no other endonasal septoplasty simulation tool available.

The results of this study demonstrate that simulation training with this model has construct validity since it differentiates between levels of surgical expertise.²⁸ OSATS, operative time, hand movements, and path length analysis showed statistically significant differences between the 2 groups. These results make this model a useful tool for objective assessment of endonasal septoplasty skills.

Face validity, which reflects reality, was assessed with a 4-question satisfaction survey, in which all the participants were asked about the model’s usefulness in learning septoplasty skills, the applicability to a real scenario, if they would recommend using the model to a partner, and if it should be included in a surgical training program. All questions obtained a mean score of over 4.6 out of 5. In addition, septoplasty experts actively participated in the model design, giving feedback through formal meetings during the development process.

Likewise, an SRS was developed following experts’ fundamental concerns about each of the tasks to finally identify 4 key tasks and 5 areas of marking, with a total score of 75 points ( Table 1 ). Concurrent validity was demonstrated by a positive correlation between the assessments’ results when using this scale and OSATS. For construct validity, significant difference between the 2 expertise level groups was found. Internal consistency measures whether different items of a composite score show significance to the total score. In this study, SRS and GRS obtained high reliabilities, indicating that all items should be considered for final scoring. A similar septoplasty SRS to assess trainees in an operating room setting was published in the literature by Obeid et al,²⁹ matching the majority of the key tasks assessed in our specific assessment tool and simulation model.

A possible concern about the model is pig ear management. They were purchased at a slaughterhouse and maintained in refrigeration for no more than 2 days and predissected 30 minutes before model assembly. Initially, during the model development, ears were frozen to preserve them for a longer time, but after defrosting, the subperichondrial plane adhesion was higher than those kept in the refrigerator. On the other hand, ears kept for more than 2 days in refrigeration were too soft for the flap elevation.

A potential weakness of this study is variability of the pig ears during the validation process, since it could cause differences in the complexity of the model among participants and an imbalance between groups. This effect was intended to be neutralized by a matching, in which each of the 2 ears of a batch was used in different groups. However, the lack of standardization is present in the real world, where surgical cases are variable.

Another issue that could affect the fidelity/authenticity of the model in regards to the real procedure was the inferior tunnel. In this model, the cartilage is naturally attached to the mucosa in a specific region that simulates the cartilaginous septum (superior tunnel). In the case of the inferior tunnel, where the mucosa adheres to the bone in a real human septum, the skin is separated from the wooden frame. Nevertheless, the development of this task, in this model, does not allow for the dissection of the inferior tunnel, and the main objective of the exercise is to visualize the tunnel. Thus, after separating the pigskin adhered to the floor of the nostril, participants were requested to identify different colors located in the wooden frame, simulating the nasal spine and the perpendicular lamina of the ethmoid. In addition, in the premaxilla region—with caudo-cephalic orientation—a sequence of 6 symbols of the Spanish alphabet had to be identified by participants in order ( Table 1 and Figure 4 ).

Figure 4.

Inferior tunnel task. (A) Nasal spine color identification (green arrow); nostril floor sequence identification (blue arrow); perpendicular lamina color identification (red arrow). (B) Skin in situ before tunnel making (yellow).

Two of 15 participants could not complete the last task, consisting of suturing both flaps after deviated cartilage is removed by performing 4 stitches from 1 septum side to the other ( Table 1 ) due to breaking of the suture’s needle. This result led to the exclusion of these participants from the analysis of operative time, hand movements, and path length, given that the values of these parameters would be infinite. Also, ICSAD was not available for all participants’ assessments, causing another 3 participants—1 in the trainee group and 2 in the faculty group—to be excluded from hand movements and path length analysis. Further stages of this or other similar studies should include more participants.

Conclusions

The simulation model and the SRS assessed in this study can be accurately used as a validated surgical assessment tool for skills that are required for endonasal septoplasty. This low-cost and simple replication simulation model is accessible and can potentially be used in any otolaryngology surgical training program for trainees. New studies will need to address learning curves and predictive validity, to better understand skill acquisition and transfer to the operating room setting.

Author Contributions

Andrés Rosenbaum, designed the study, contributed to model design, performed experiments, analyzed the data, wrote the manuscript; Gabriel Faba, devised the project, designed the model and worked the technical details, approved the manuscript; Julián Varas, supervised the findings of this work, contributed to the interpretation of the results and to the manuscript; Tomás Andrade, directed the project, contributed to the implementation of the research and to the writing of the manuscript.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.

References

Konstantinidis

Triaridis

Karagiannidis

Kontzoglou

Long term results following nasal septal surgery. Focus on patients’ satisfaction. Auris Nasus Larynx. 2005;32(4):369-374.

Shah

Roxbury

Sindwani

Techniques in septoplasty: traditional versus endoscopic approaches. Otolaryngol Clin North Am. 2018;51(5):909-917.

Gupta

Phillips

Salim

Hopkins

Oakley

. Septoplasty training revisited. Bull R Coll Surg Engl. 2014;96:e10-e13.

Killian

The submucous window resection of the nasal septum. Ann Otol Rhinol Laryngol. 1905;14:363-393.

D’Ascanio

Manzini

Quick septoplasty: surgical technique and learning curve. Aesthetic Plast Surg. 2009;33(6):814-818.

Lagos

Waissbluth

Heider

Andrade

Faba

Rhinoseptoplasty: three-year experience at the Pontificia Universidad Católica de Chile. Rev Otorrinolaringol Cir Cabeza Cuello. 2018;78(3):245-250.

Gauna

Goldman

Torrente

Surgical skills evaluation of the graduates from the training program in Otorhinolaryngology of the University of Chile. Rev Otorrinolaringol Cir Cabeza Cuello. 2018;78:133-140.

McGaghie

Issenberg

Cohen

Barsuk

Wayne

DB.

Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706-711.

Abou-Elhamd

K-EA

Al-Sultan

Rashad

. Simulation in ENT medical education. J Laryngol Otol. 2010;124(3):237-241.

10.

Musbahi

Aydin

Al Omran

Skilbeck

Ahmed

Current status of simulation in otolaryngology: a systematic review. J Surg Educ. 2017;74(2):203-215.

11.

Chang

Lin

Bowe

, et al. Fabrication and validation of a low-cost, medium-fidelity silicone injection molded endoscopic sinus surgery simulation model. Laryngoscope. 2017;127(4):781-786.

12.

Bhatti

Ahmed

Improving skills development in residency using a deliberate-practice and learner-centered model. Laryngoscope. 2015;125(suppl):S1-S14.

13.

Doorly

Taylor

Schroter

RC.

Mechanics of airflow in the human nasal airways. Respir Physiol Neurobiol. 2008;163(1-3):100-110.

14.

Dini

Gonella

Fregadolli

Nunes

Gozzano

A new animal model for training rhinoplasty. Rev Bras Cir Plást. 2012;27(2):201-205.

15.

Weinfeld

AB.

Chicken sternal cartilage for simulated septal cartilage graft carving: a rhinoplasty educational model. Aesthetic Surg J. 2010;30(6):810-813.

16.

Cologlu

Uysal

Kocer

Kankaya

Oruc

Uysal

Rhinoplasty model in rabbit. Plast Reconstr Surg. 2006;117(6):1851-1859.

17.

Chark

Oliaei

Manuel

Wong

BJ.

Porcine cartilage model for simulation of nasal tip aesthetics and mechanics. Aesthetic Surg J. 2011;31(5):501-505.

18.

Zabaneh

Lederer

Grosvenor

Wilkes

Rhinoplasty: a hands-on training module. Plast Reconstr Surg. 2009;124(3):952-954.

19.

Coan

Neff

Mukundan

Marcus

JR.

Validation of a cadaveric model for comprehensive physiologic and anatomic evaluation of rhinoplastic techniques. Plast Reconstr Surg. 2009;124(6):2107-2117.

20.

Datta

Mackay

Mandalia

Darzi

The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg. 2001;193(5):479-485.

21.

Hiemstra

Kolkman

Wolterbeek

Trimbos

Jansen

FW.

Value of an objective assessment tool in the operating room. Can J Surg. 2011;54(2):116-122.

22.

Kazan

Viezel-Mathieu

Cyr

Hemmerling

Gilardino

MS.

The Montreal Augmentation Mammaplasty Operation (MAMO) simulator: an alternative method to train and assess competence in breast augmentation procedures. Aesthet Surg J. 2018;38(8):835-849.

23.

Boccalatte

Larrañaga

Perez Raffo

, et al. Brief guideline for the prevention of COVID-19 infection in head and neck and otolaryngology surgeons. Am J Otolaryngol. 2020;41(3):102484.

24.

Pollei

Barrs

Hinni

Bansberg

Walter

LC.

Operative time and cost of resident surgical experience: effect of instituting an otolaryngology residency program. Otolaryngol Head Neck Surg. 2013;148(6):912-918.

25.

Quinn

Alt

Ashby

Orlandi

Time, resident involvement, and supply drive cost variability in septoplasty with turbinate reduction. Otolaryngol Head Neck Surg. 2018;159(2):310-314.

26.

Baugh

Franzese

Extremes in otolaryngology resident surgical case numbers: an update. Otolaryngol Head Neck Surg. 2017;156(6):1060-1066.

27.

Ryu

Dharampal

Mostafa

Sharlin

Systematic review of patient-specific surgical simulation: toward advancing medical education. J Surg Educ. 2017;74(6):1028-1038.

28.

Van Nortwick

Lendvay

Jensen

Wright

Horvath

Kim

Methodologies for establishing validity in surgical simulation studies. Surgery. 2010;147(5):622-630.

29.

Obeid

Al-Qahtani

Ashraf

Alghamdi

Marglani

Alherabi

Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy. 2014;28(4):e163-e167.