Abstract
Importance
Surgical difficulty quantification is crucial to develop educational curricula for progressive training. This need remains unmet in endoscopic endonasal surgery (EES).
Objective
The objective was to classify the basic skills and procedures of EES according to the perceived difficulty by experts to build a progressive curriculum.
Design
Three-round modified Delphi study.
Setting
E-survey with an international panel of EES experts.
Participants
Fifty-nine international experts were selected on their publications in the field of EES and medical education and invited to participate. Fifteen experts from 5 countries completed the survey.
Intervention
A 3-round modified Delphi study was conducted.
Main Outcome Measures
In round#1, basic skills and EES procedures were identified and the basic skills required for these procedures were listed. In round#2, the expert panel ranked the difficulty of basic skills, and in round#3, the difficulty of the EES procedures. The basic skills were grouped into 3 categories of increasing difficulty, and the subsequently-calculated difficulty of each EES procedure was compared with the experts’ score. Consensus was defined at ≥80% agreement.
Results
Twenty-three basic skills and 26 EES procedures were identified. Basic skills were ranked in 3 groups of increasing difficulty, from easiest (navigation with a 0° telescope) to most difficult (suturing in the nasal cavity), with ≥80% agreement. The least difficult procedure ranked was polypectomy, whereas the most difficult was Vidian neurectomy. The correlation between the expert score and the calculated score reached an R2 of .75, reflecting the cumulative effect of these required competencies on procedure difficulty.
Conclusions
Consensus was achieved to rank basic skills and EES procedure according to the perceived difficulty by experts.
Relevance
This study provides a basis for quantifying surgical difficulty in EES and pave the way for developing a progressive educational curriculum.
Key Messages
Quantifying the difficulty of surgical skills in endoscopic endonasal surgery (EES) is essential for developing a step-by-step progressive training approach.
EES procedures can be broken down into several basic skills that can be practiced independently before performing the full procedures.
There is a high correlation between the difficulty of basic skills and the difficulty of EES procedures as rated by experts.
Introduction
Quantifying the surgical difficulty is of high importance in designing educational strategies and building surgical skills in a stepwise manner. For efficient learning, it is indeed necessary to gradually increase the difficulty of exercises according to the skills or the level of experience.1 -3 This proficiency-based approach increases the efficiency of educational strategies. 4
The construction of any educational curricula ideally follows a 6-phase process described by Thomas et al 5 The first steps require (1) the identification of priority objectives and (2) the ranking of difficulties in order to enable relevant, progressive, and personalized training. Defining the level of training or assessment exercises helps in referring learners to the appropriate learning path and in avoiding 2 major pitfalls: annoying or, more importantly, discouraging the learner.6 -8 The lack of a clear definition of the level of difficulty also leads to great heterogeneity in training.
However, quantifying surgical difficulty is challenging because perceived difficulty is highly subjective and influenced by not only surgical or procedural factors, but also by patient, disease and working environment-related factors.
The example of laparoscopic surgery is of high interest because the educational process has been particularly studied and scientifically built on a stepwise curriculum where modular training has been progressively implemented and validated. Skills and procedures were “deconstructed” into component tasks (task analysis and task deconstruction), and the difficulty of each task has been evaluated to build a progressive curriculum.9 -16 In this modular training approach, trainees are allowed to perform continuously more complex parts of a procedure after achieving proficiency in the preceding steps. This educational strategy has then been successfully transferred to education in robotic surgery.17,18
To date, no study has attempted to describe the degree of difficulty of endoscopic endonasal surgeries (EES) objectively or subjectively. Therefore, the present study design is original.
The objective of this study was to quantify the difficulty degree of basic steps and overall procedures in EES. These data could be useful as a basis for education or as a guide in the early stages of experience in EES.
Methods
A three-round modified Delphi method (graphical abstract) was performed to quantify the difficulty degree of basic steps and EES procedure. As there are no data from patients, the Institutional Review Board of Montpellier University Hospital was waived.
Participants
Academic experts in EES were selected based on their clinical experience in endoscopic endonasal surgery and their academic work in medical education. Inclusion criteria for experts’ selection were (1) having performed all the listed EES procedures, (2) having published in the last 5 years a minimum of 10 articles pertaining to endoscopic endonasal/skull base surgery, and (3) having participated in the design of an EES simulator or having published at least 1 article on medical education in this field. A Pubmed Central database query was performed for the 2014 to 2019 period to identify experts according to criteria (2) and (3) using the following strategy:
((“education”[Subheading] OR “education”[All Fields] OR “educational status”[MeSH Terms] OR (“educational”[All Fields] AND “status”[All Fields]) OR “educational status”[All Fields] OR “education”[All Fields] OR “education”[MeSH Terms]) AND (“surgery”[Subheading] OR “surgery”[All Fields] OR “surgical procedures, operative”[MeSH Terms] OR (“surgical”[All Fields] AND “procedures”[All Fields] AND “operative”[All Fields]) OR “operative surgical procedures”[All Fields] OR “surgery”[All Fields] OR “general surgery”[MeSH Terms] OR (“general”[All Fields] AND “surgery”[All Fields]) OR “general surgery”[All Fields])) AND ((“otolaryngology”[MeSH Terms] OR “otolaryngology”[All Fields]) OR (“neurosurgical procedures”[MeSH Terms] OR (“neurosurgical”[All Fields] AND “procedures”[All Fields]) OR “neurosurgical procedures”[All Fields] OR “neurosurgery”[All Fields] OR “neurosurgery”[MeSH Terms])). This search led to a total of 5623 articles.
Delphi Round#1
The first round of the survey consisted of the creation of an initial list of basic steps and EES procedures by a working group of 5 physicians performing EES and medical education (V.F., L.H.P.N., M.A.T., C.R., P.G.) from 3 different academic centers. In this brainstorming activity, each member listed in free text all the possible basic skills and EES procedures to be acquired during an ENT curriculum. Each item was then discussed and modified according to the objectives of the study. Items related exclusively to skull base procedures were excluded from the final list. An effort was made to avoid anatomical or clinical considerations when defining basic skills, to focus on the skills themselves rather than on the context in which they are performed. As an example, the proposed item “neurovascular structures peeling” was broken down into a “peeling” and a “dissection” basic skill. Similarly, each EES procedure (Table 1) was analyzed and deconstructed into several basic skills required (Table 2). At the end of this four month process, 23 basic skills and 26 EES procedures were listed with a 100% agreement between members.
Lists of Relevant Basic Skills and EES Procedures From the Round#1 (Brainstorming Phase) Listed in Alphabetical Order.
Abbreviation: EES, endoscopic endonasal surgery.
Glue, hemostatic matrix.
Evaluation of Basic Skills Required to Perform EES Procedures.
Abbreviation: EES, endoscopic endonasal surgery.
Subsequently, a questionnaire consisting of the all the agreed-upon items was elaborated (Supplemental Material 1) and sent to a panel of experts. Items were randomly distributed within the questionnaire to minimize bias.
Delphi Round#2: Basic Skills Ranking Survey
Expert panelists meeting inclusion criteria were invited by e-mail to review the items summarized in round#1, and to rank the basic skills from least (1/23) to most difficult (23/23) skill to learn, regardless of the context (eg, indication or surgical anatomy). The mean ranking among experts was calculated for each basic skill. At the closure of the round, the principal investigator (V.F.) consolidated the 23 basic skills into 3 basic skill groups of increasing difficulty. The first group was composed of basic skills with an average rank of 1-8 ([1-8[), the second with an average rank of 8-15 ([8-15[), and the third, of 15-23 ([15-23]).
Delphi Round#3: Review of Round#2 and EES Procedures Ranking Survey
In the final round, panelists were asked to review the basic skills summary ranking established in round#2, and to provide explanations in cases of disagreement (Supplemental Material 2). Consensus was obtained if >75% of panelists approved this ranking. In addition, panelists were asked to rate the overall degree of difficulty of EES procedures using a 6-point Likert scale from least to most difficult.
Each Delphi panelist received a questionnaire that included the items and ratings summarized in the previous round. In cases of disagreement, experts were asked to revise his/her judgments or “to specify the reasons for remaining outside the consensus,” thus providing an opportunity for further clarification of both the information and their judgments of the relative item difficulty.
Questionnaires Management
All questionnaires were sent electronically, and results were retrieved by the same investigator (V.F.) who did not participate as panelist. Thus, respondents remain unknown to each other, providing the required anonymity that is characteristic of the Delphi process.
Statistical Analysis
Statistical analysis was performed with SPSS 24.0 (IBM, Armonk, NY) for Windows 10. Basic skills items were ranked according to mean score (ranging from 1 to 23), while EES procedures were ranked according to a 6-point Likert Scale.
We then sought to determine whether the combination of basic skills required for a procedure could be related with the procedure difficulty score established by the experts. For each procedure, a score was established by combining the difficulties of the required basic skills (Basic Skills Score, BSS). For this purpose, we used the results of Delphi round#2: each basic skill in group 1 was assigned the value of 1; each skill in group 2 was assigned the value of 2; each skill in group 3 was assigned the value of 3. If a basic skill was not systematically useful for the procedure, it was not computed. The procedure score was the result of the addition of the scores assigned to the required basic skills. We verified the correlation of this score with the experts’ difficulty score by calculating the R 2 coefficient of determination.
Results
A total of 59 experts meeting the 3 inclusion criteria were identified. Of these, 15 (25%) experts from 5 different countries participated in the round#2 of the Delphi method. Nine participants (60%) had over 20 years of experience in EES surgery, 2 (13%) had 15 to 19 years of experience, 3 (20%) had 10 to 14 years of experience, and 1 (7%) had 5 to 9 years of experience. All panelists were ENT surgeons (6 from the USA, 4 from France, 2 from Canada, 2 from Spain, and 1 from Australia).
The 3 groups of basic skills issued from Delphi round#2 (1: low difficulty; 2: intermediate difficulty, 3: high difficulty) are summarized in Table 3. Group 1 is comprised of 5 items, Group 2 of 12 items, and Group 3 of 6 items.
Group of Basics Skills Built From Round#2. Group 1: Low-Difficulty Items; Group 2: Intermediate Difficulty Items; Group 3; High-Difficulty Items.
In Delphi round#3, all items ranked in group 1 reached 100% agreement. Nine of the 12 items of group 2 reached 100% agreement, and all items reached >80% agreement. Shaving skill (82% agreement) required to be ranked in group 3 for 2 respondents, due to its potential danger for anatomical structures according to these respondents. Application of liquid (91% agreement) needed to be ranked in the high-difficulty group for 1 respondent, due to the main indication of the skill: closure of cerebrospinal fluid (CSF) leak. Out-fracture of thin bone structure (82% agreement) needed to be ranked in group 3 for 2 respondents because it required mastery of the force applied on bone structures. Five of the 6 items from group 3 reached 100% agreement. Coagulation skill (91% agreement) needed to be ranked in group 2 for 1 respondent, without providing any justification.
Relation Between Basic Skills Difficulties and Procedures Difficulties
Table 4 summarizes the ranking of EES procedures difficulty according to experts (round#3 results). The obtained scores are well correlated to the scores based on basic skills (Figure 1) with a coefficient of determination R2 = .75.
Ranking of EES Procedures Difficulty by Panelists From 1 (Less Difficult) to 6 (Most Difficult) in Delphi Round#3 and Scores From the Basic Skills Study (From 1 to 26).
Abbreviation: EES, endoscopic endonasal surgery.
Correlation between “Expert score” and “Basic skills score” for EES procedures. EES, endoscopic endonasal surgery.
Discussion
The segmentation of the learning process into stages of increasing difficulty is crucial when developing a training curriculum. As an example, it is important to train to anterior and posterior ethmoidectomy before attempting any endonasal orbital decompression, or to train to anterior ethmoidectomy before any frontal recess dissection. However, designing this breakdown and determining the level of difficulty can be challenging. The Delphi method has been found to be effective in identifying and prioritizing technical procedures in otorhinolaryngology for simulation-based training. 19 Fifty-nine international experts were identified based on their scientific publications and contacted via e-mail by the authors. While this recruitment method may explain the relatively-low response rate in the first round—only 15 experts participated—it ensured an objective and transparent selection process, strengthening the validity of the resulting consensus. Using this method, we were able to subdivide procedures into basic tasks and skills, which are then combined to perform more complex procedures.
Breaking down procedures into basic skills may be an unusual and challenging task for surgical educators, as their judgment may be influenced by anatomical considerations—some basic skills may seem more complex or risky depending on the anatomical region. To minimize bias and avoid classification based on perceived procedural risk, experts were explicitly instructed to evaluate each task solely on its technical execution, without taking its anatomical and procedural context into account. This approach facilitated a more objective assessment of procedural difficulty by focusing on the level of technical mastery required for each core skill. Ultimately, it allowed for the creation of 3 procedure groups with increasing levels of difficulty.
Thus, we have shown that it is possible to reach consensus on the important skills and level of difficulty of tasks and procedures in endonasal surgery. This breakdown into subskills allows to (1) build dedicated exercises to specifically train these subskills, (2) more accurately evaluate the level of learners, the difficulties to be overcome, and the points for improvement, (3) more precisely adapt training to this evaluation by selecting relevant activities. Thus, the knowledge of difficulty levels allows for building a training program, in a focused, progressive, and learner-adapted approach. We proposed an overview of training possibilities on box trainers,20 -22 synthetic static,22 -28 synthetic dynamic,22,29 -31 virtual reality,32 -38animal,21,39 -41 human cadaver,42 -47 or hybrid48 -50 models for each basic skills in Supplemental Material 1 and procedures in Supplemental Material 2, to train students before practicing on patients.
The relative correlation between scores based on the ranking of basic skills (“subskills score”) and the “expert score” (which was independently rated) for evaluating the difficulty of EES procedures suggests that the difficulty of basic skills has a cumulative effect on the difficulty of EES procedures in which they are required. It is quite intuitive: A novice should not perform a neurectomy of the Vidian nerve, because he or she will have too many technical obstacles to overcome simultaneously. This does not mean that only one subskill should be trained and then moved on to another only once that subskill is mastered, as it has been shown in the field of sports that nonlinear learning, varying activities, is more efficient than repeating the same activities. 51 However, it appears, based on our results, that it is preferable to avoid activities that combine several skills that have not yet been mastered, and to work separately on these subskills to avoid overwhelming the learner’s attention capacity and to avoid the frustration of failure. Indeed, the learner’s ability to succeed in a procedure is limited by the subskill(s) in which he/she has difficulty, as well as by the addition of these difficulties that monopolize all his/her attention capacities. This implies that a focused and targeted approach to learning, where fundamental skills are prioritized and excessive difficulties are avoided, may lead to better outcomes. Such a segmentation, although somewhat artificial, can nonetheless support more accurate educational diagnostics and help tailor feedback and guidance to the learner’s specific needs. It also allows educators to direct the learner back to targeted practice of a specific core skill when difficulties arise. Following this principle, a framework for a training curriculum in EES is proposed in Figure 2.
Proposed framework for a personalized skill-based EES curriculum. EES, endoscopic endonasal surgery.
While most of the procedures listed appear to adhere to the principle of difficulty increasing in correlation with the complexity of the required subskills, a small number of exceptions have been observed (Figure 1). We identified 2 types of situations: (1) experts assigned to a procedure a higher score (more difficult) than “subskills score” did, and (2) experts assigned a lower score (less difficult) than “subskills score” did. The first situation includes Draf 3 procedures and frontal sinus stent placement. The second situation includes polypectomy, canine fossa trephination, anterior ethmoidectomy, and endoscopic septoplasty. For all these procedures, we hypothesize that experts took into consideration the anatomy and the potential complications. A perfect mastering of anterior skull base anatomy is crucial for successful Draf 3 procedures to prevent CSF leaks. In contrast, endoscopic septoplasty is considered a relatively-low-risk procedure that can be performed by novices, even if they encounter some technical difficulties. Another example of this is the anterior ethmoidectomy, which is considered easier to learn than the posterior ethmoidectomy despite the more complex anatomy due to the numerous variations of the fronto-ethmoidal cells and the frequent use of angled endoscopes.
Other methodologies than expert consensus have been proposed to establish the difficulty of surgical procedures. For example, it would be possible to rely on the study of learning curves for all procedures (ie, “The time taken and/or the number of procedures an average surgeon needs to be able to perform a procedure independently with a reasonable outcome” 52 ). However, during the surgical curriculum, procedures are encountered and performed in a more or less random order, which can influence the learning curves. Furthermore, it can be difficult to define the criteria to be analyzed to consider that the result is “reasonable.” Too often, surgical time is the only criterion used, which is not sufficient. Moreover, learning a specific procedure (for example, “orbital decompression”) often relies on skills acquired during other procedures (here, ethmoidectomy). The “number of procedures required” raw is itself a questionable criterion, because, mainly, (1) it depends on the starting level of the learner and (2) learning is not a linear process. It is therefore difficult to know the learning curves for each type of procedure and to organize the teaching program based on these values that are not yet well defined in the literature.
Another way of analyzing the surgical difficulty of procedures would be to analyze complications, based on the assumption that difficult procedures are associated with higher complication rates. In a study focused on extended endoscopic endonasal and base of skull surgeries, Snyderman et al 53 validated training levels using their patients’ database: they ranged EES procedures based on the rate of complications. Thus, 5 groups were identified: (1) endoscopic sinus surgery and sphenopalatine artery ligation; (2) advanced sinus surgery comprising CSF leak and intrasellar surgery; (3) procedures performed in the extradural sagittal plane, orbital surgery, optic nerve decompression, and extrasellar surgery; (4) intradural skull base surgery and procedures performed in the extradural coronal plane; and (5) vascular surgery and intradural procedures in the coronal plane. However, this classification did not rank common EES procedures themselves (all included in group 1). Furthermore, it is important to note that there may be other factors that contribute to complications, such as patient factors or team experience, that should also be considered when interpreting complication rates. Thus, if one were to base the procedure on the complication rate alone, tonsillectomy could be considered more difficult than tympanoplasty. Nevertheless, the complication rate is an important consideration, as a young apprentice should not be entrusted with surgical procedures with a high risk of complications, thus ensuring the best possible outcome for patients.
Recently, new approaches have been proposed to assess procedural difficulty in surgical training. Mental workload, for instance, has shown potential, having been used in laparoscopic studies to evaluate simulator validity. 54 It may correlate with task difficulty, although this relationship requires further investigation. Building on this idea, Lim et al 55 demonstrated that physiological metrics—such as eye-tracking, electroencephalography, and heart rate variability—were associated with task difficulty and multitasking demands in robotic surgery, using multimodal neural network models. To our knowledge, no comparable approach has yet been applied in the field of otolaryngology.
Conclusion
In this study, we conducted an international expert survey to classify difficulties encountered by trainees in EES, aiming to help teachers construct a dedicated surgical curriculum in EES. First, basic skills were classified in 3 groups of increasing difficulty according to experts’ ranking. According to this result, it is suggested that trainees should first master the basic skills in the first group, then proceed to learn those in the second group, and finally move on to the third group. Experts were also asked to rate the difficulty of each EES procedure to help designing a step-by-step learning of procedures. Each of them has been deconstructed in basic skills, and we studied the correlation between basic skills difficulty and EES procedure difficulty. This relation was not linear, indicating that other factors also influence the difficulty of the surgical procedure (eg, surgical indication such as benign or malignant tumor, patient’s clinical context, and comorbidities). However, the knowledge of the global difficulty level allows us to better stratify the learning process and the curriculum in EES. All additional difficulty factors must also be considered during training (bleeding simulation, stress management, etc.), but our discussion focused on the technical difficulty of the gestures (psychomotricity).
Supplemental Material
sj-docx-1-ohn-10.1177_19160216251390285 – Supplemental material for Quantifying Difficulty in Endoscopic Endonasal Surgery—A Modified Delphi Method Approach
Supplemental material, sj-docx-1-ohn-10.1177_19160216251390285 for Quantifying Difficulty in Endoscopic Endonasal Surgery—A Modified Delphi Method Approach by Valentin Favier, Marc A. Tewfik, Lily H. P. Nguyen, Cécile Rumeau and Patrice Gallet in Journal of Otolaryngology - Head & Neck Surgery
Footnotes
Acknowledgements
The authors would like to warmly acknowledge all the experts who contributed to the panel and the Direction of Research and Innovation of the Montpellier University Hospital, Montpellier, France for its support.
Author Contributions
Valentin Favier: Design, acquisition, analysis, interpretation of data, draft, revision, final approval, agree to be accountable for all aspects of the work.
Marc A. Tewfik: Analysis, interpretation of data, revision, final approval, agree to be accountable for all aspects of the work.
Lily H. P. Nguyen: Analysis, interpretation of data, revision, final approval, agree to be accountable for all aspects of the work.
Cécile Rumeau: Design, analysis, interpretation of data, revision, final approval, agree to be accountable for all aspects of the work.
Patrice Gallet: Design, analysis, interpretation of data, draft, revision, final approval, agree to be accountable for all aspects of the work.
Data Availability Statement
All the data used in this article are available in the manuscript or Supplemental Material.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Ethical Approval
This study was approved by the Institutional Review Board of Montpellier University Hospital (ref no. IRB-MTP_2023_05_202301415).
Supplemental Material
Additional supporting information is available in the online version of the article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
