Metrics Development for Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis With and Without Spondylolisthesis by an International Expert Panel

Abstract

Study Design:

Prospective study.

Objectives:

To develop, operationally define, and seek consensus from procedure experts on the metrics that best characterize a reference approach to the performance of a minimally invasive unilateral laminotomy for bilateral decompression (ULBD) for lumbar spinal stenosis.

Methods:

A Metrics Group consisting of 3 experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a “reference” procedure. In a modified Delphi panel, 26 spine surgeons from 14 countries critiqued these metrics and their operational definitions before reaching consensus.

Results:

Performance metrics consisting of 6 phases with 42 steps, 21 errors, and 17 sentinel errors were identified that characterize the procedure. During the peer review, these were evaluated, modified, and agreed.

Conclusions:

Surgical procedures can be broken down into elemental tasks necessary for the safe and effective completion of a reference approach to a specified surgical procedure. Spinal experts from 16 countries reached consensus on performance metrics for the procedure. This metric-based characterization can be used in a training curriculum and also for assessment of training and performance in clinical practice.

Keywords

spinal stenosis objective assessment metrics proficiency-based progression unilateral decompression Delphi

Introduction

Neurosurgical and orthopedic spine surgery programs rely on cadaveric and operating room training as their standard teaching methods. As these training forms can be limited by either institutional infrastructure and/or biohazard regulations with operating room time even more limited by work time regulations, legal issues, and high costs.^1,2 Furthermore, the surgical learning curve is associated with an increased risk of complication and higher morbidity and mortality for the patient.^3,4 Given that cadavers and animals tend to have different characteristics when compared to a living patient training options of specific procedure steps on cadavers or animal models have limitations and new options would be valuable.

Surgical training, at both trainee and master levels, intends to enable a surgeon to acquire the theoretical and practical knowledge necessary to perform a designated surgical procedure skillfully, reliably, and safely. Many experienced surgeons who expertly perform a specific procedure are able to identify and agree on the essential “steps” to be completed and “errors” to be avoided, whereas they rarely think about the procedures they perform with the level of detail needed to identify those key features.⁵

By psychologist definition those key features have to be subjected to a detailed task analysis specifying the order, duration, and result of a specific action and determine them to be unambiguously measurable to be called metrics.⁶ The metrics do not have to capture every aspect of performance, but should allow differentiation in the way developed by Dreyfus and Dreyfus⁷ to serve as a tool to evaluate effectiveness with a quantitative score.

The systematic implementation of a simulation curriculum in neurosurgery or orthopedic training programs has been proven to be feasible, is favorably regarded, and has a positive impact on trainees of all levels.⁸ Therefore, a few academic centers around the world have focused on the role of highly realistic spine surgery simulation models to train spine surgeons^9

-14 and even though simulation training is not a substitute for operating room experience, it may decrease the time teaching intraoperatively and reduce the morbidity secondary to resident surgical errors.

Simulation-based medical education with deliberate practice was proven to work even better in technical skill acquisition and maintenance for several clinical skills than traditional apprenticeship-styled clinical education.¹⁵

The aim of this study was (1) to develop and objectively define performance metrics that comprehensively characterize a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis with and without spondylolisthesis (Figures 1 and 2), (2) to see if the metrics are assessible on a simulator, and (3) to seek consensus on the completeness and appropriateness of the metrics from a panel of experienced orthopedic and neurological surgeons.

Figure 1.

Illustration of the incision site on the right side for a L4/5 minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis.

Figure 2.

Illustration demonstrating the removal of ligamentum flavum through a tubular retractor as part of a unilateral laminotomy for bilateral decompression of lumbar spinal stenosis.

Material and Methods

Metric Definitions

As described before metrics are composed of different parts, such as phases, steps, errors and sentinel errors.¹⁶

Metric: A standard of measurement of quantitative assessments used for objective evaluations to make comparisons or to track performance.

Procedure phase: A group or series of integrally related events or actions that, when combined with other phases, make up or constitute a complete operative procedure.

Step: A component task, the series aggregate of which constitutes the completion of a specific procedure.

Error: A deviation from optimal performance.

Sentinel error: An occurrence involving a serious deviation from optimal performance of a procedure that exposes the patient to serious risk but falls short of being a critical error (ie, life threatening)

Metrics Development

Three experienced spine surgeons (2 neurosurgeons, 1 orthopedic surgeon), each with over 25 years of clinical practice, and an educational expert formed the Metrics Group that characterized a lumbar decompression surgery for spinal stenosis as a “reference” procedure. In the first step, all 3 experts individually described their classic lumbar decompression procedure. As it became evident that this resulted in a nondetailed description with just few steps identified, the next task was to explain it as they would explain it to a resident or fellow. Besides seeking consensus in the description and reviewing the current literature,^17

-20 a series of 9 videos of lumbar decompressions available online (Google, YouTube, VuMedi, PubMed) were reviewed in detail. Based on a detailed task analysis and deconstruction process⁶ as well as decades of experience practicing and teaching spinal surgery the units of performance that are integral to skilled task performance were identified. During individual face-to-face meetings, phone calls, and four 1.5- to 2-hour online conferences along with countless email exchanges metric units were identified and then unambiguously defined so that they could be subsequently scored as either occurring or not occurring with a high degree of reliability. For the online sessions, the use of Skype videoconferencing (Microsoft, Redmond, WA; available at www.skype.com) enabled the investigators who reside in different geographic locations to look at videos simultaneously and discuss results facing the same screens. The metrics included the phases in general order as well as the specific steps and the instruments used. The metrics also described for each procedural step what should not be done (errors) thus characterizing behavior that deviated from optimal performance.²¹ The intent again, was to create unambiguous operational definitions (rather than descriptions) for each performance unit namely procedure steps, errors, and sentinel errors. The goal was to characterize a “reference” minimal invasive decompression procedure without unusual or complex pathology and subsequent unexpected obstacles.

Metric Stress Testing and Reliability of Identification

With the Metrics Group content with characterization of the procedure, the metrics were “stress tested” by subjecting them to an analysis of reliability assessing how independently scored in blinded fashion. Eight video recordings of complete minimal invasive decompression procedures that were performed by surgeons using either tubular or spatula retractors were independently reviewed and scored in binary fashion as either occurring or not. Differences in the scoring of each metric by the reviewers were discussed and if necessary, definitions of steps and errors were clarified, with the items modified, or dropped and new ones added until the Metrics Group was satisfied that the metrics accurately and unambiguously characterized the specifics of a minimal invasive decompression procedure. The extent of agreement between 2 raters ranged between 0 = no agreement and 1 = complete agreement.

Experts Perform the Procedure to Evaluate the Metrics

To test the metrics further each expert performed 2 procedures on a simulator while being scored individually by the other 2 using the metrics. The procedures were recorded, and the records reviewed by all experts conjointly. Differences in the results and scores once again were discussed with steps missing or unnecessary deleted or modified in 3 phases (Table 1).

Table 1.

The Steps of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed on After Expert’s Procedure.

Procedure Phase	Name	Steps Before Experts’ Procedures	Steps After Experts’ Procedures	Deleted	Modified
I	Starting point, skin incision, and access	8	8	0	1
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	9	9	0	1
III	Flavectomy	4	4	0	0
IV	Complete ipsilateral decompression	9	8	1	0
V	Contralateral hemilaminotomy	11	11	0	0
VI	Hemostasis and closure	5	5	0	1

Modified Delphi Panel to Evaluate the Metrics

The metrics were then sent for appraisal to a group of surgeons who were also expert in the performance of the procedure. A questionnaire was developed by the Metrics Group seeking consensus for each of the procedural steps and potential errors that needed to be evaluated individually. The questionnaire was then sent by the educational specialist via SurveyMonkey (Dublin, Ireland; available at http://surverymonkey.com) to every surgeon who agreed to participate.

While the metrics developed by the Metrics Group for a reference procedure for unilateral approach to a bilateral decompression for lumbar stenosis might not depict the exact techniques used by the individual panelists, it was agreed that the operative steps accurately reflect the essential components of the procedure.

An affirmative vote by a participant indicated that the metric definition presented was accurate and acceptable as written but not necessarily that it was the manner in which that particular panelist might have chosen to complete the step. Therefore, panel members were encouraged to comment on the metrics even though they might not perform the steps in the exact same way. “Consensus” meant that panel members voted unanimity and that a particular metric definition was “not wrong or inappropriate.”

Subjects

Twenty-six experienced spine surgeons from 14 countries (Table 2) were asked to take part in an expert panel. Participants were selected (to ensure regional representation) from the international and regional faculty of AOSpine as well as Society for Minimally Invasive Spine Surgery (SMISS) and by their experience in minimally invasive spine surgery providing an equal distribution of orthopedic surgeons and neurosurgeons.

Table 2.

Number of Surgeons From Each Country Represented in the Delphi Panel.

Country	No. of Surgeons
Australia	1
Austria	1
Brazil	2
Canada	1
Colombia	1
Egypt	1
France	1
Germany	3
Italy	1
Jordan	1
South Korea	1
Spain	1
United Kingdom	1
United States	10

Results

Procedure Phases

The procedural metrics resulting from task deconstruction were grouped into 6 phases. Each phase (eg, “Identification of ipsilateral hemilamina ipsilateral hemilaminotomy, and identification of ligamentum flavum” or “Contralateral decompression”) contains a series of related, unambiguously defined, observable procedure steps with specific beginning and ending points (Table 3).

Table 3.

The Operational Definitions of the Beginning and End of the Different Phases of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis.

Procedure Phase	Name	Begins and Ends
I	Starting point, skin incision, and access	Begins	Preparation phase above has been completed
		Ends	Retractor positioned on bone in correct level
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	Begins	Surgeon ready to identify lamina
		Ends	The ligamentum flavum has been identified
III	Flavectomy	Begins	Phase II completed
		Ends	Central portion of ipsilateral ligamentum flavum has been removed
IV	Complete ipsilateral decompression	Begins	Ipsilateral medial flavum has been removed
		Ends	Lateral recess is opened
V	Contralateral hemilaminotomy	Begins	Ready to start contralateral decompression
		Ends	Contralateral decompression and flavectomy performed
VI	Hemostasis and closure	Begins	Hemilaminotomies have been performed
		Ends	Hemostasis applied and wound closed

By processing the information from the expert operations and the Delphi panel survey all phases were accepted with changes, but no extra phase was proposed or was a phase deleted (Table 4) while phase V was renamed. Phases I and VI were agreed on with just linguistic changes, while wording and content was altered within the other phases (II-V)

Table 4.

The Procedural Phases of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis.

Procedure Phase	Name
I	Starting point, skin incision, and access
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum
III	Flavectomy
IV	Complete ipsilateral decompression
V	Contralateral decompression
VI	Hemostasis and closure

Procedure Steps

Within the reference procedure, 44 steps were identified by the Metrics Group. After the expert operations and reevaluation of the steps, 1 was added, 1 was deleted, and 3 were modified. These were distributed unequally within the phases with the most occurring in the fifth phase (“contralateral decompression”). After discussion of the feedback from the expert panel, 2 steps were deleted and 11 were modified. (Table 5).

Table 5.

The Steps of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Steps Before Delphi	Steps After Delphi	Deleted	Modified
I	Starting point, skin incision, and access	8	8	0	3
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	9	9	0	0
III	Flavectomy	4	4	0	3
IV	Complete ipsilateral decompression	7	6	1	1
V	Contralateral hemilaminotomy	11	10	11	2
VI	Hemostasis and closure	5	5	0	2

Procedure Errors and Sentinel Errors

All potential errors identified had been noted to occur during the stress testing of the metrics. Identical errors and sentinel errors could occur during different phases of the procedure (eg, “dural tear with CSF leak”), but were named within the phases individually. Overall, 18 errors and 20 sentinel errors were defined by the Metrics Group and affirmed. No errors and sentinel errors were added, but 1 sentinel error was deleted and 1 sentinel error occurring in 2 phases was changed to an error instead of a sentinel error. Furthermore, changes were made in 3 errors adding information or changing wording. In the end, 21 errors and 17 sentinel errors were defined (Tables 6 and 7).

Table 6.

The Errors of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Errors Before Delphi	Errors After Delphi	Added
I	Starting point, skin incision, and access	2	2	0
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	6	7	1
III	Flavectomy	3	3	0
IV	Complete ipsilateral decompression	2	3	1
V	Contralateral hemilaminotomy	4	5	1
VI	Hemostasis and closure	1	1	0

Table 7.

The Sentinel Errors of the Reference Procedure of Minimal Invasive Unilateral Laminotomy for Bilateral Decompression of Lumbar Spinal Stenosis and the Changes Agreed and Voted on by the Delphi Panel.

Procedure Phase	Name	Sentinel Errors Before Delphi	Sentinel Errors After Delphi	Deleted
I	Starting point, skin incision, and access	2	1	1
II	Identification of ipsilateral hemilamina, ipsilateral hemilaminotomy, and identification of ligamentum flavum	5	4	1
III	Flavectomy	2	2	0
IV	Complete ipsilateral decompression	5	4	1
V	Contralateral hemilaminotomy	5	4	1
VI	Hemostasis and closure	1	1	0

After affirmation of the panel suggestions the metrics were agreed as an accurate characterization of a minimal invasive unilateral laminotomy for bilateral decompression for lumbar spinal stenosis. An example of the complete metrics and steps for a phase is shown in Figure 3.

Figure 3.

The agreed metrics for Phase IV of the reference procedure (starting point and entrance angle). Fields in white show the metrics that can be scored by an assessor.

Discussion

Assessment of medical education and competence is always difficult and becomes even more challenging within postgraduate training. Ideally, the assessment of competence should provide insight into actual performance as well as the capacity to find and generate new knowledge and improve overall performance. Competence is contextual, reflecting the relationship between a person’s abilities and the tasks he or she is required to perform in a particular situation in the real world.

Medical training and its assessment have to meet certain criteria. The first criterion, reliability, refers to the reproducibility, consistency, and generality of the results over time. The second criterion, feasibility, reasons that, in order to be accepted, a new tool should be cost-effective, easy to execute, and not time-consuming. The third criterion, validity, asks if the tool assesses what it is supposed to assess. Validity is divided into 5 subcategories: predictive validity, which asks if the tool can predict future performance; face validity, which asks if the tool reflects real life; construct validity, which asks if the tool measures what it is designed to measure; content validity, which asks if the domain that we are assessing is assessed by the tool; and criterion-related validity, which asks if the result assessed by a tool correlates with that measured by the current gold standard tool.

Metric development is therefore crucial as the metrics build the fundament of good medical training. Metrics define how the training should be characterized and the procedure performed by the trainee as well as grant the opportunity for assessment of the trainee’s performance and progress.

In contrast to other, mostly quantitative approaches, metrics provide for an individual not associated with the initial development to use them to score performance reliably as they provide precise definitions of performance and a binary scoring system.^6,22,23

Especially training in spine surgery varies a lot due to regional and national differences as well as differences between neurosurgical and orthopedic approaches to surgical procedures. Furthermore, with duty hour limitations and regulatory pressure for enhanced quality and outcomes limitations occur in trainees’ access to fundamental skills training. In addition, a cultural shift is seen from competence to mastery learning.²⁴ Fundamental skills training as part of a curriculum might help mitigate this challenge to surgical education.^12,25 This is what we aim to achieve.

Conclusion

With a total of 29 surgical experts from 14 countries achieving consensus on the metric-based definition of a reference procedure, this study validates that definition and exact characterization of the phases and steps depicting a reference procedure for a minimal invasive unilateral laminotomy for bilateral decompression of lumbar spinal stenosis is possible. Furthermore, face and content validity for the resulting steps and error metrics can be obtained by simulator-based procedures by experts and through review by a panel of experts in a modified Delphi panel.

Footnotes

Authors’ Note

Following is the list of names of the surgeons in the expert panel (in alphabetical order):

Neel Anand, Department of Surgery, Spine Trauma, Spine Center, Cedars Sinai Medical Center, Los Angeles, USA

Richard Assaker, Professor in Neurosurgery, CHRU Lille, Hôpital Salengro, Pôle de Neurochirurgie, Lille, France

Muhammed Assous, FRCS, FRCS Orth, CCST, Consultant Spinal Surgeon, Razi Spine Clinic-Minimally Invasive Spine Surgery, Amman, Jordan

Dott Prof Massimo Balsano, Chirurgo ortopedico specialista della colonna vertebrale, Direttore Ortopedia e Traumatologia A, Centro Regionale Specializzato in Chirurgia Vertebrale, Azienda Ospedaliera Universitaria Integrata Verona (AOUI), Ospedale Borgo Trento, Italy

Christof Birkenmaier, MD, Associate Professor, Head of Spine, Department of Orthopedics, Physical Medicine & Rehabilitation, University of Munich, Grosshadern Medical Center, Germany

Tarek ElHewala, MD, Lecturer of Orthopedics and Spine Surgery, Faculty of Medicine, Zagazig University, Zagazig, Egypt

Asdrubal Falavigna, MD, PhD, Department of Neurosurgery, Director of Postgraduate Program in Health Science, Caxias do Sul University, Caxias do Sul, Brazil

Alberto Gotfryd, MD, PhD, Santa Casa of São Paulo Medical School and Hospitals, Brazil

Jeff Golan, MD, FRCS(c), Chief of Neurosurgery, Jewish General Hospital and Assistant Professor, McGill University, Montreal, Canada

Michael Grevitt, BSc FRCS, FRCS(Orth), Consultant Spinal Surgeon, Nottingham University Hospitals NHS Trust, UK

Patrick C. Hsieh, MD, Professor of Neurological Surgery, Edwin M. Todd/Trent H. Wells, Jr Professor of Neurosurgery, Director of Minimally Invasive Spine Surgery, Department of Neurological Surgery, Keck School of Medicine, University of Southern California, USA

Jin-Sung Luke Kim, MD, PhD, Professor, Spine Center, Department of Neurosurgery, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea

Isador H. Lieberman, MD, Orthopaedic & Spinal Surgeon, President Texas Back Institute, Plano, Texas, USA

Prof Dr Bernhard Meyer, Direktor der Neurochirurgischen Klinik, Technische Universität München, Munich, Germany

Jean-Pierre Mobasser, Neurosurgeon and President Goodman Campbell Brain and Spine, Indianapolis, IN, USA

Isaac L. Moss, MD, MASc, FRCSC, Assistant Professor of Orthopedic Surgery, Comprehensive Spine Center, UConn Musculoskeletal Institute, Farmington, Connecticut, USA

Avelino Parajón, Jefe de Sección de Neurocirugía, Hospital Universitario Ramón y Cajal, Madrid, Spain

Frank Philipps, MD, Professor, Rush University Medical Center, Chicago, USA

Kris Radcliff, MD, Associate Professor, Department of Orthopedic Surgery, Thomas Jefferson University, Philadelphia, PA, USA

Néstor Taboada, Clinica PortoAzul, Barranquilla, Colombia

Paul Taylor, CSC, MBChB, LLM, FRCSEd, FACLM, FRACS, DMCC, Mount Medical Centre, Perth, Australia

Univ-Prof Dr Claudius Thomé, Universitätsklinik für Neurochirurgie, Innsbruck, Austria

Juan Uribe, MD, FAANS, Professor and Vice-Chairman Department of Neurological Surgery, Chief Division of Spinal Disorders, Volker K. H. Sonntag Chair of Spine Research, Barrow Neurological Institute, Phoenix, AZ, USA

Jeffrey C. Wang, MD, Chief, Orthopaedic Spine Service, Co-Director USC Spine Center, Professor of Orthopaedic Surgery and Neurosurgery USC Spine Center, Los Angeles, CA, USA

Acknowledgments

Thanks to Realist (RealSpine) Leipzig, Germany for supporting the study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This supplement was supported by funding from the Carl Zeiss Meditec Group. The study was funded by the AO Spine TK.

ORCID iD

Carolin Melcher

References

Iglehart

. Revisiting duty-hour limits—IOM recommendations for patient safety and resident education. N Engl J Med. 2008;359:2633–2635. doi:10.1056/NEJMp0808736

Lambert

Smith

Goldacre

. The impact of the European Working Time Directive 10 years on: views of the UK medical graduates of 2002 surveyed in 2013-2014. JRSM Open. 2016;7:2054270416632703. doi:10.1177/2054270416632703

Ahn

Iqbal

Manning

, et al. Minimally invasive lumbar decompression—the surgical learning curve. Spine J. 2016;16:909–916. doi:10.1016/j.spinee.2015.07.455

Birkmeyer

Finks

O’Reilly

, et al.; Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369:1434–1442. doi:10.1056/NEJMsa1300625

Adams

. The changing face of motor learning. Hum Mov Sci. 1990;9:209–220. doi:10.1016/0167-9457(90)90003-V

Gallagher

. Metric-based simulation training to proficiency in medical education: what it is and how to do it. Ulster Med J. 2012;81:107–113.

Dreyfus

. Mind over Machine: the Power of Human Intuitive Expertise in the Era of the Computer. New York, NY: Free Press; 1986.

Kirkman

Ahmed

Albert

Wilson

Nandi

Sevdalis

. The use of simulation in neurosurgical education and training. J Neurosurg. 2014;121:228–246. doi:10.3171/2014.5.Jns131766

Gasco

Holbrook

Patel

, et al. Neurosurgery simulation in residency training: feasibility, cost, and educational benefit. Neurosurgery. 2013;73(suppl 1):39–45. doi:10.1227/NEU.0000000000000102

10.

Ghobrial

Anderson

Chitale

Campbell

Lobel

Harrop

. Simulated spinal cerebrospinal fluid leak repair: an educational model with didactic and technical components. Neurosurgery. 2013;73(suppl 1):111–115. doi:10.1227/NEU.0000000000000100

11.

Gragnaniello

Abou-Hamden

Mortini

, et al. Complex spine pathology simulator: an innovative tool for advanced spine surgery training. J Neurol Surg A Cent Eur Neurosurg. 2016;77:515–522. doi:10.1055/s-0036-1583940

12.

Pereira

Aziz

. Simulation in spinal surgery and the transition from novice to expert. World Neurosurg. 2015;84:1511–1512. doi:10.1016/j.wneu.2015.06.066

13.

Rehder

Abd-El-Barr

Hooten

Weinstock

Madsen

Cohen

. The role of simulation in neurosurgery. Childs Nerv Syst. 2016;32:43–54. doi:10.1007/s00381-015-2923-z

14.

Youngerman

Zacharia

Hickman

Bruce

Solomon

Benzil

. Making milestones: development and implementation of a formal socioeconomic curriculum in a neurosurgical residency training program. Neurosurgery. 2016;79:492–498. doi:10.1227/NEU.0000000000001126

15.

McGaghie

Issenberg

Cohen

Barsuk

Wayne

. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86:706–711. doi:10.1097/ACM.0b013e318217e119

16.

Angelo

Ryu

Pedowitz

Gallagher

. The Bankart performance metrics combined with a cadaveric shoulder create a precise and accurate assessment tool for measuring surgeon skill. Arthroscopy. 2015;31:1655–1670. doi:10.1016/j.arthro.2015.05.006

17.

Alimi

Hofstetter

Pyo

Paulo

Härtl

. Minimally invasive laminectomy for lumbar spinal stenosis in patients with and without preoperative spondylolisthesis: clinical outcome and reoperation rates. J Neurosurg Spine. 2015;22:339–352. doi:10.3171/2014.11.spine13597

18.

Thomé

Zevgaridis

Leheta

, et al. Outcome after less-invasive decompression of lumbar spinal stenosis: a randomized comparison of unilateral laminotomy, bilateral laminotomy, and laminectomy. J Neurosurg Spine. 2005;3:129–141. doi:10.3171/spi.2005.3.2.0129

19.

Overdevest

Jacobs

Vleggeert-Lankamp

Thomé

Gunzburg

Peul

. Effectiveness of posterior decompression techniques compared with conventional laminectomy for lumbar stenosis. Cochrane Database Syst Rev. 2015;(3):CD010036. doi:10.1002/14651858.CD010036.pub2

20.

Papavero

Thiel

Fritzsche

Kunze

Westphal

Kothe

. Lumbar spinal stenosis: prognostic factors for bilateral microsurgical decompression using a unilateral approach. Neurosurgery. 2009;65(6 suppl):182–187. doi:10.1227/01.NEU.0000341906.65696.08

21.

Seymour

Gallagher

Roman

O’Brien

Andersen

Satava

. Analysis of errors in laparoscopic surgical procedures. Surg Endosc. 2004;18:592–595. doi:10.1007/s00464-002-8927-2

22.

Gallagher

O’Sullivan

Leonard

Bunting

McGlade

. Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments. ANZ J Surg. 2014;84:568–573. doi:10.1111/j.1445-2197.2012.06236.x

23.

Liebig

Holtmannspötter

Crossley

, et al. Metric-based virtual reality simulation: a paradigm shift in training for mechanical thrombectomy in acute stroke. Stroke. 2018;49:e239–e242. doi:10.1161/strokeaha.118.021089

24.

Barsuk

Cohen

Vozenilek

O’Connor

McGaghie

Wayne

. Simulation-based education with mastery learning improves paracentesis skills. J Grad Med Educ. 2012;4:23–27. doi:10.4300/jgme-d-11-00161.1

25.

Adermann

Geißler

Bernal

Kotzsch

Korb

. Development and validation of an artificial wetlab training system for the lumbar discectomy. Eur Spine J. 2014;23:1978–1983. doi:10.1007/s00586-014-3257-3