Sage Journals: Discover world-class research

Abstract

Study Design

Modified Delphi study.

Objectives

Telementoring, the practice of providing remote guidance and teaching from a distance using telecommunication technology, has demonstrated feasibility and value in assisting surgeons in remote locations. However, limited evidence exists regarding its effectiveness in improving surgeon performance. This study aimed to develop metrics to assess spine surgeon performance in scoliosis surgery, potentially within the context of a telementoring training program.

Methods

Ten expert spine surgeons participated in a four-round modified Delphi process including both online and in-person meetings. The resulting metrics were validated for objective assessment by eleven surgeons, who reviewed and rated a video recording using the established rubric.

Results

The final set of metrics, comprising 50 procedural steps and 28 error categories, was unanimously approved by the panel. Additionally, the panel agreed that utilizing cameras, smart lenses, and output from intraoperative imaging monitors would provide sufficient visibility for assessing both steps and errors.

Conclusions

A set of assessment metrics for adolescent idiopathic scoliosis surgery was successfully defined and validated by reviewing example videos. The longer-term research objective is to employ this rubric to assess surgeon’s performance throughout a telementoring program, thus assessing its educational impact. The rubric could also be used in other contexts, such as live surgical observation.

Keywords

telementoring teleproctoring performance assessment spine surgeon education surgical treatment of AIS

Introduction

Telementoring is defined as the remote assistance and evaluation of surgical procedures by a mentor or proctor, who is located at least one kilometer from the operating surgeon and the patient.¹ The literature often uses this term interchangeably with ‘teleproctoring’, however only ‘telementoring’ will be used here, covering both terms. These are different from the term ‘telesurgery’, which describes remote surgery performed by a surgeon through a surgical robot. Compared to telesurgery, telementoring is considered easier to establish, with lower associated costs and reduced sensitivity to network latency issues.² Telementoring has been demonstrated to be both feasible and valuable, particularly in assisting surgeons in remote or under-resourced areas of the world^3,4 and in cases where proctor mobility was constrained, such as during the COVID-19 Pandemic.^5,6 Numerous studies have demonstrated its safety and effectiveness as a surgical adjunct.^3-7

From an educational perspective, the integration of emerging technologies such as video conferencing, instant messaging, and cloud-based applications has significantly transformed the landscape of distance education, making it more interactive, engaging, and accessible. Crucially, telementoring has proven particularly effective in supporting the professional development of in-service teachers, enhancing their technology integration skills and enabling them to adapt teaching practices to the evolving digital landscape. Moreover, telementoring has a positive impact on student outcomes. By providing personalized guidance and support, telementoring fosters deeper student engagement, improves academic achievement, and cultivates a stronger sense of motivation and self-efficacy.⁸ In regard to its applications in the field of health, telementoring has been shown to be effective in supporting the development of healthcare professionals.^9-12

The potential of telementoring in spine surgery is substantial, as it offers a pragmatic and cost-effective solution to address the challenges of a shortage of specialized spine surgeons, particularly in remote or underserved areas. By enabling experienced surgeons to virtually guide and mentor their colleagues in real-time, telementoring has the potential to enhance the quality of care, improve patient outcomes, and expand access to advanced surgical techniques.¹³ A key advantage of telementoring in spine surgery is its ability to provide robust educational opportunities for the next generation of surgeons. Through virtual observation and interaction with seasoned experts, medical students, residents, and early-career surgeons can accelerate the acquisition of crucial skills and knowledge, ultimately fostering the development of a more proficient surgical workforce.¹⁴ Furthermore, the integration of virtual reality and augmented reality technologies into the telementoring process has the potential to create even more immersive and effective learning environments for aspiring spine surgeons.¹⁵

Although there is evidence that telementoring is perceived as user friendly, there is limited evidence on its impact on improving surgeon performance. To address this gap, it is first necessary to create assessment tools to measure proficiency in the skill sets required for performing surgical procedures. Therefore, this study aimed to take the necessary step to develop a set of metrics (a rubric) to assess improvements in surgeon performance in the surgical treatment of Adolescent Idiopathic Scoliosis (AIS) and to validate the rubric.

Materials and methods

A modified Delphi process in accordance with Zeliff and Heldenbrand,^16,17 consisting of four rounds, was used to develop an assessment rubric for posterior adolescent idiopathic scoliosis (AIS) surgery.

Ethical Approval

The study has no direct involvement of patients and was deemed exempt from ethical approval by the Palestine Polytechnic University Institutional Board Review (approval ID KA/41/2023). Informed consent for the use of the surgical videos was obtained.

Panelists

For the Delphi process 10 international experts in the field of idiopathic scoliosis surgery were selected as panelists from nine countries Argentina, Brazil, Egypt, Germany, India, Palestine, Turkey, UK, USA.

For the rubric validation, 11 surgeons were involved (7 from the Delphi panel and 4 not involved in the development of the rubric).

Modified Delphi Process

Its steps included:

1. Establishment of the Delphi panel. Selection of expert panelists to participate in the iterative feedback process.

2. Development of a list of surgical competencies. Panelists were asked to identify and list the competencies (and associated potential errors) that may be evaluated in each step of posterior scoliosis surgery. Using the Delphi tool (Survey Monkey), panelists individually developed lists of competencies for each surgical step. These lists of competencies were then analyzed for accuracy and redundancy, grouped into stages of surgery (eg, exposure of the spine, placement of pedicle screws or other anchors, performing fusion, wound closure) and compiled into a final list.

3. Evaluation of competencies. Panelists rated the importance and necessity of each competency (and associated errors) in the compiled list on a Likert scale (1 to 9, from totally incompetent to excellent competence) on the Delphi tool. Results were analyzed using the median, 1st quartile, 3rd quartile, and interquartile range (IQR) calculations to assess consensus.

4. Re-evaluation of competencies. The results of the analysis of the previous step were sent to the panelists to ask for any changes in opinion after seeing these initial results to achieve a higher level of agreement. Panelists were given the ability to change their answers following the overall results if they deemed necessary.

In addition to the Delphi process, one online and two face-to-face consensus meetings were conducted with the panelist to discuss and finalize the marking rubric.

The final results were analyzed by calculating the median, 1st quartile, 3rd quartile, and the interquartile range (IQR). An IQR equal to or less than 1.2 for any item was accepted as a “consensus (C)”, whereas any IQR greater than 1.2 was classified as “no consensus” (NC). All competencies (and errors) that have reached consensus (C) with a median score equal to or greater than 5 were included in the final evaluation rubric.

The final step in the development of the rubric was a grammatical check performed by two native English-speaking surgeons.

Rubric Validation, Stress Testing, and Reliability of Identification

This step assessed how reliably the rubric could be independently scored by 11 surgeons. This required independent reviewing and scoring of video recordings of the procedure. Panelists were asked to assess the competencies of a surgeon (AA, the designated telementor) performing scoliosis surgery from a recorded surgical video.

Adolescent Idiopathic Scoliosis Video Acquisition.

A video recording of scoliosis surgery performed in the Kenyata National Hospital, Nairobi, Kenya was recorded using technology developed by Proximie Inc (Proximie SAL. Bachoura, Beirut, Lebanon) with multiple cameras installed in the operating room and video feeds streamed into a central software program with HIPAA compliance. Although this tool is specifically designed for remote telementoring, it was used only for the recording of the case for the purposes of this study. The output included recordings from 3 different cameras in the operating room (one overhead at the center of the pilot surgical light, one from the cranial side of the operating table and another covering the entire operating room including the fluoroscopy screen) compiled as a tile to be viewed simultaneously (Figure 1).

Figure 1.

A sample video instant from the output of three cameras compiled as a tile (a) pilot lamp, (b) cranial side of the table, (c) room view.

Rubric Stress Testing and Reliability of Identification.

The video was marked by 11 surgeons (5 expert surgeons from the Delphi panel, 4 other spine surgeons who were not involved in the development of the rubric, and 2 surgeons who were involved in the development of the rubric). Each surgeon rated the performance level for completing the specified task (competency) on a Likert scale from 1 to 9, along with the presence and number of errors at each step. Additionally, this panel evaluated the video’s quality specifically considering whether performance level and errors could be accurately assessed from the video. Any difficulties encountered in viewing or assessing the video were documented into the evaluation sheet. The panel was given the marking choice of ‘cannot assess’ for each step and error and was asked to specify their reasons in the open text comments. A group of five surgeons (2 Delphi panel members, 2 other surgeons and 1 developer) repeated the same viewing and marking procedure after a time lag of 3 to 5 weeks for the calculation of the intraobserver and interobserver reliabilities. Only these evaluations were used for the purposes of Kappa statistics whereas the results from the entire panel were used for the evaluation of the capability to assess.

Statistical methods for the Delphi step included the descriptive statistics of the scores from the Delphi panel. In the validation step, in addition to the descriptive statistics, reliability of the rubric was evaluated by Cronbach-Alpha and Kappa statistics were used to evaluate the intra-and inter-observer reliability of the marking system. For the purpose of this analysis, the Likert scale used in this step was simplified into a scale of 5 to rate how well the surgeon performed the step in which the marks were grouped as 1 (very poor), 2 (marks of 2 and 3), 3 (marks of 4 and 5), 4 (marks of 6 and 7) and 5 (marks of 8 and 9, excellent).

Results

Development of the Rubric

A Delphi panel consisting of nine experienced idiopathic scoliosis surgeons attended the initial consensus meeting on the general objectives of the study. In the 2nd round, this panel identified nine phases of idiopathic scoliosis surgery (Table 1) resulting in an initial list of 50 competencies/tasks and 29 potential errors. In the 3^rd round, 10 items (competencies or errors) were eliminated due to lack of consensus on their necessity and/or importance (IQR>1.2). In the 4th round, 10 new items were added, and the list was finalized at another consensus meeting.

Table 1.

List of Surgical Phases.

Surgical Phase
1. Patient selection/indications for surgery
• 3 competencies, 3 errors
2. Measurement and classification/selection of fusion levels
• 5 competencies, 3 errors
3. Positioning of the patient
• 11 competencies, 1 error
4. Surgical exposure
• 6 competencies, 2 errors
5. Placement of the anchors (i.e., screws or hooks) and rods
• 8 competencies, 6 errors
6. Correction maneuvers including evaluation and correction of balance
• 8 competencies, 4 errors
7. Performance of fusion
• 4 competencies, 3 errors
8. Wound closure
• 1 competency, 1 error
9. Transfer to bed
• 4 competencies, 5 errors

A final rubric consisting of 9 surgical phases, 50 tasks/competencies and 28 potential errors was developed (Supplemental Material) and analyzed for internal consistency. The panel also decided that phases 1, 2, and 9 (ie, Patient selection/indications for surgery, Measurement and classification/selection of fusion levels, and Transfer to bed) may not be possible to evaluate on a video recording. Consequently, although these phases were included in the overall rubric, they were excluded from the following validation step.

Rubric Stress Testing and Reliability of Identification

The video recording used for this stage started at the time of the induction of anesthesia and ended when the patient was transferred to the bed, a total time of 4 h, 35 min and 38 sec with all three cameras recording the entire procedure simultaneously.

Evaluation data gathered from the 1st evaluation round were analyzed for internal consistency reliability for each surgical phase as well as a total value (Table 2). The overall (Total) reliability was very high: three of the phases were reliable (.4-.6), one was quite reliable (.6-.8) and two were very reliable (>.8).

Table 2.

The Internal Consistency Reliability Analysis of the 1st Evaluation Round.

	Internal Consistency Reliability (Cronbach-Alpha)
Phase 3	.51
Phase 4	.83
Phase 5	.67
Phase 6	.87
Phase 7	.59
Phase 8	.59
Total	.87

The inter and intraobserver variances of the five assessors as denoted as Kappa values¹⁸ can be seen in Tables 3 and 4 respectively. In regard to the interobserver agreement, out of 20 pairings (10 for the 1st evaluation and 10 for the 2nd), 17 demonstrated substantial agreement and 3 demonstrated excellent interobserver agreement. Regarding intraobserver agreement, three of the five surgeons demonstrated substantial agreement whereas the other two demonstrated excellent agreement.

Table 3.

The Interobserver Agreement Results of the Five Surgeons Participating in the Assessment for the 1st and the 2nd Evaluations.

	Weighted Kappa-Se 1^st Evaluation	Weighted Kappa-Se 2^nd Evaluation
Surg 1- Surg 2	.734-.050	.786-.040
Surg 1- Surg 3	.807-.053	.749-.046
Surg 1- Surg 4	.700-.056	.733-.045
Surg 1- Surg 4	.672-.051	.660-.043
Surg 2- Surg 3	.809-.037	.834-.036
Surg 2- Surg 4	.714-.044	.741-.044
Surg 2- Surg 5	.687-.034	.715-.043
Surg 3- Surg 4	.764-.050	.753-.039
Surg 3- Surg 5	.699-.045	.742-.043
Surg 4- Surg 5	.716-.052	.696-.044

Table 4.

The Intraobserver Agreement Results of the Five Surgeons Participating in the Assessment.

	Weighted Kappa-Se
Surgeon 1	.761-.043
Surgeon 2	.833-.035
Surgeon 3	.796-.053
Surgeon 4	.850-.034
Surgeon 5	.679-.051

The number of ‘cannot assess’ answers varied widely by individual competencies/errors and the surgical phases (Table 5). Difficulties were encountered most frequently during the phases of positioning, placement of the anchors, and correction maneuvers, and were almost exclusively related to the inability of the viewer to see whether the task was performed appropriately due to the positioning of the cameras or obstacles to them (eg, surgeon’s head in the field of view). A sample for such an answer sheet may be seen in Figure 2.

Table 5.

The Range and Average Values for the Rate of ‘Cannot Assess’ Answer for the Surgical Phases.

	Average (%)	Range (%)
Positioning of the patient	18.1	0-82
Surgical exposure	0	0-0
Placement of the anchors (ie, screws or hooks) and rods	12.0	0-50
Correction maneuvers	22.5	0-40
Performance of fusion	0	0-0
Wound closure	0	0-0

Figure 2.

A sample sheet from the evaluation form demonstrating the reasons for the answer of ‘cannot assess’ during the positioning phase.

Discussion

The main result from this study was the development of a rubric to assess surgeon performance and competence in posterior surgery for adolescent idiopathic scoliosis (AIS). The created rubric showed good internal consistency and a good inter- and intraobserver reliability.

The need for developing this rubric was originally identified to assess learner improvement within a telementoring program, as remote methods are gaining popularity in surgical training. However, it would be applicable for the assessment of AIS surgery performance in other context such as live observation of surgery in the operating room. This live setting would even allow to assess additional steps and errors that were removed from the current rubric because could not be assessed when relying solely on videos.

Surgical education is undergoing a significant paradigm shift, moving from time-based training models to competency-based medical education (CBME).¹⁹ This evolution requires well-defined objectives and reliable methods for assessing learners’ skills and competences at various stages of their training. In this framework, valid assessment tools are essential not only for achieving surgical competencies but also for promoting effective and meaningful learning.^20,21 The rubric created in this study serves the purpose of filling the gap in assessing the specific skills of posterior surgery for AIS that can well complement more general assessment methos such as the Objective Structured Assessment of Technical skills (OSATS).²²

The use of the Delphi method or similar consensus-building approaches for developing assessment rubrics is well established in the literature. St-Louis and colleagues, for instance, reported the development of the Open Surgery Competency Assessment for Residents (OSCAR) tool. Following a comprehensive literature review to identify key competencies, they applied a Delphi process to determine the 12 most relevant surgical competencies for resident assessment.²³ Comparable methodologies have been employed to define entrustable professional activities (EPAs) and core competencies in other health professions, including dentistry^24,25 and nursing.^26,27 Burke et al. adopted an accelerated Delphi process to develop a core curriculum for robotic surgery.²⁸

To date, however, the results of these rubric based assessment in actual assessment scenarios over consecutive recorded procedures has been reported in a limited number of studies. Cheon and colleagues for example, using the Council of Ophthalmology’s Ophthalmology Competency Assessment Rubric, conducted evaluations of 10 simulated cataract surgery videos generated by residents. Two faculty experts, along with 10 residents (through peer and self-assessment), participated in the evaluation process. While expert and peer assessments demonstrated high inter-rater reliability, a systematic discrepancy was observed between these and the residents’ self-evaluations.²⁹ It remains unclear whether this divergence is specific to the rubric employed or reflects a more generalizable phenomenon across self-assessment practices. The present study cannot address this question due to the absence of self-assessment data.

Finally, beyond assessment purposes, the Delphi methodology has also been widely applied in the development of guidelines and curricular frameworks, particularly for competencies related to emergency procedures (eg, Refs. 30,31). In summary, the Delphi method is a validated and reliable approach for the development of assessment rubrics and educational content. Our study further supports this statement by demonstrating the feasibility of the process and that it may be used for distant education and assessment, and also, by providing data on its validation.

This study has several potential limitations related to the methodology used and the quality of the video used for the validation step. First, the quality of the recording, although using three different cameras from three different angles, was less than ideal, resulting in several ‘cannot assess’ answers in the rubric. This problem might be less relevant in the assessment of live surgeries however a better recording system for purposes of telementoring may be needed. A possible improvement of the quality of the recording is the use of smart glasses which provide a view of the surgery from the principal surgeon’s perspective and enhance the view of the operating field.³² Another limitation is that the surgeon who performed the surgery for the purpose of the recording was the originator of the telementoring project and a member of the Delphi panel. It may therefore be introducing a bias in the assessment process since he performed the surgery according to the rubric that may have increased the number of steps visible on the recording (decreasing the number of ‘cannot assess’ answers from the assessors). This surgeon was excluded from the validation process to eliminate a potential bias in the scoring of his own surgery.

In conclusion, this study developed and validated an assessment rubric for posterior surgery for AIS. Future work would focus on validating further this rubric to monitor the progression of surgical skills during a telementoring training program.

Supplemental Material

Supplemental Material - Defining Metrics for Assessing Surgeon Performance During a Telementoring Program for Adolescent Idiopathic Scoliosis Surgery

Supplemental Material for Defining Metrics for Assessing Surgeon Performance During a Telementoring Program for Adolescent Idiopathic Scoliosis Surgery by Alaaeldin Ahmad, Engin Çetin, Steven Theiss, Selcen Pehlivan, Michael Cunningham, Monica Ghidinelli, and Emre Acaroğlu in Global Spine Journal.

Footnotes

Acknowledgments

The authors would like to extend their gratitude to our other Delphi panelists: Colin Nnadi (Oxford, UK), Ruben Alejandro Morales Ciancio (Mendoza, Argentina), Alpaslan Şenköylü (Ankara, Turkey), Youssry Elhawary (Cairo, Egypt), Bhavuk Garg (New Delhi, India), Cristiano Menezes (Nova Lima, Brazil), and Carolin Melcher (Neustadt in Holstein, Germany); our independent video evaluators: Luca Erdmann Bini Cordeiro (Belo Horizonte, Brazil), Francisco Moura Neto (Belo Horizonte, Brazil), and Mehmet Çetinkaya (Istanbul, Turkey); Moyo Kurt (Utrecht, Holland) for the invaluable contributions to this project, and George Museivi head of spine department in Kenyatta hospital. We would also like to thank Proximie Inc who donated the hardware and software used for the videos.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Proximie Inc provided cameras with associated hardware and software to anonymize the recordings and provide short-term access to the Delphi panelists. Support for data collection and manuscript preparation was provided by the AO Education Institute through AO Spine curriculum design and educational research projects.

Ethical Statement

ORCID iDs

Alaaeldin Ahmad

Monica Ghidinelli

Michael Cunningham

Emre Acaroğlu

Supplemental Material

Supplemental material for this article is available online.

References

Barba

Stramiello

Funk

Richter

Yip

Orosco

. Remote telesurgery in humans: a systematic review. Surg Endosc. 2022;36(5):2771-2777. doi:10.1007/s00464-022-09074-4

Reppucci

Houston

SKS

3rd Kras

Miller

. Three-dimensional telesurgery and remote proctoring over a 5G network. Digit J Ophthalmol. 2021;27(3):38-43. doi:10.5693/djo.01.2021.06.003

Greenfield

Luck

Billingsley

, et al. Demonstration of the effectiveness of augmented reality telesurgery in complex hand reconstruction in gaza. Plast Reconstr Surg Glob Open. 2018;6(3):e1708. doi:10.1097/GOX.0000000000001708

McCullough

Kulber

Sammons

Santos

Kulber

. Google glass for remote surgical tele-proctoring in low- and middle-income countries: a feasibility study from Mozambique. Plast Reconstr Surg Glob Open. 2018;6(12):e1999. doi:10.1097/GOX.0000000000001999

Beyhoff

Zhu

Zanders

, et al. Teleproctoring for training in structural heart interventions: initial real-world experience during the COVID-19 pandemic. J Am Heart Assoc. 2022;11(4):e023757. doi:10.1161/JAHA.121.023757

Rai

Deib

Smith

Boo

. Teleproctoring for neurovascular procedures: demonstration of concept using optical see-through head-mounted display, interactive mixed reality, and virtual space sharing-A critical need highlighted by the COVID-19 pandemic. AJNR Am J Neuroradiol. 2021;42(6):1109-1115. doi:10.3174/ajnr.A7066

Datta

MacQueen

Schroeder

, et al. Wearable technology for global surgical teleproctoring. J Surg Educ. 2015;72(6):1290-1295. doi:10.1016/j.jsurg.2015.07.004

Garlinska

Osial

Proniewska

Pregowska

. The influence of emerging technologies on distance education. Electronics (Switzerland). 2023;12(7):1550. doi:10.3390/electronics12071550

Jin

Brown

Patwa

Nirmalan

Edwards

. Telemedicine, telementoring, and telesurgery for surgical practices. Curr Probl Surg. 2021;58(12):100986. doi:10.1016/J.CPSURG.2021.100986

10.

Huang

Knight

Guetter

, et al. Telemedicine and telementoring in the surgical specialties: a narrative review. Am J Surg. 2019;218(4):760-766. doi:10.1016/J.AMJSURG.2019.07.018

11.

Dawe

Kirkpatrick

Talbot

, et al. Tele-mentored damage-control and emergency trauma surgery: a feasibility study using live-tissue models. Am J Surg. 2018;215(5):927-929. doi:10.1016/j.amjsurg.2018.01.016

12.

Hickman

Dean

Puri

Singh

Siegel

Patel

. Ophthalmic telesurgery with a low-cost smartphone video system for surgeon self-reflection and remote synchronous consultation: a qualitative and quantitative study. Telemed Rep. 2022;3(1):30-37. doi:10.1089/tmr.2021.0037

13.

Butt K

Augestad

. Educational value of surgical telementoring. J Surg Oncol. 2021;124(2):231-240. doi:10.1002/JSO.26524

14.

Gerardo

Lele

Sundaram

Ponsky

. Surgical telementoring: feasibility, applicability, and how to. J Surg Oncol. 2021;124(2):241-245. doi:10.1002/jso.26511

15.

Yoo

Patel

Hrynewycz

Brundage

Singh

. The utility of virtual reality and augmented reality in spine surgery. Ann Transl Med. 2019;7(Suppl 5):S171. doi:10.21037/ATM.2019.06.38

16.

Zeliff

Heldenbrand

. What’s being done in the international business curriculum? Bus Educ Forum. 1993;48:23-25.

17.

Acaroğlu

Mmopelwa

Yüksel

, et al. The Global Spine Care Initiative: a consensus process to develop and validate a stratification scheme for surgical care of spinal disorders as a guide for improved resource utilization in low- and middle-income communities. Eur Spine J. 2018;27(Suppl 6):879-888. doi:10.1007/s00586-017-5332-z

18.

Landis

Koch

. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.

19.

Nousiainen

McQueen

Hall

, et al. Resident education in orthopaedic trauma: the future role of competency-based medical education. Bone Joint Lett J. 2016;98-B(10):1320-1325. https://pubmed.ncbi.nlm.nih.gov/27694584/. Accessed November 14, 2022.

20.

Holmboe

Sherbino

Long

Swing

Frank

. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682.

21.

Lockyer

Carraccio

Chan

, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017;39(6):609-616.

22.

Martin

Regehr

Reznick

, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273-278.

23.

St-Louis

Shaheen

Mukhtar

Adessky

Meterissian

Boutros

. Towards development of an open surgery competency assessment for residents (OSCAR) tool - A systematic review of the literature and Delphi consensus. J Surg Educ. 2020;77(2):438-453. doi:10.1016/j.jsurg.2019.10.006

24.

Sethi

Haq

Zaidi

SJA

Baig

. Developing entrustable professional activities for undergraduate operative dentistry clerkship. BMC Med Educ. 2024;24(1):1505. doi:10.1186/s12909-024-06525-5

25.

Chen

Cheng

, et al. Developing competency-based medical education for dental education in Taiwan: a pilot study of tooth extraction entrustable professional activities. J Dent Sci. 2024;19(Suppl 2):S128-S135. doi:10.1016/j.jds.2024.09.032

26.

Meng

Haidan

Junye

Qian

Xin

Cheng

. Development of entrustable professional activities for novice nurses: a modified Delphi study. Nurse Educ Pract. 2024;81:104181. doi:10.1016/j.nepr.2024.104181

27.

Almalkawi

Jester

Terry

. Developing a consensus-based scoring rubric to enhance practice-based assessment of student nurses' clinical competence: a Delphi study. Nurse Educ Today. 2021;100:104859. doi:10.1016/j.nedt.2021.104859

28.

Burke

Fleming

King

, et al. Utilising an accelerated Delphi process to develop consensus on the requirement and components of a pre-procedural core robotic surgery curriculum. J Robot Surg. 2023;17(4):1443-1455. doi:10.1007/s11701-022-01518-2

29.

Cheon

de Jager

Egan

Bona

Law

. Evaluation of ophthalmology residents' self-assessments and peer assessments in simulated surgery. Can J Ophthalmol. 2020;55(5):382-390. doi:10.1016/j.jcjo.2020.05.005

30.

Michelson

Williams

Dart

, et al. Development of a rubric for assessing delayed diagnosis of appendicitis, diabetic ketoacidosis and sepsis. Diagnosis (Berl). 2020;8(2):219-225. doi:10.1515/dx-2020-0035

31.

Balki

Hoppe

Monks

Cooke

Sharples

Windrim

. Multidisciplinary Delphi development of a scale to evaluate team function in obstetric emergencies: the PETRA scale. J Obstet Gynaecol Can. 2017;39(6):434-442.e2. doi:10.1016/j.jogc.2017.01.030

32.

Baldwin

Mallidi

Baldwin

, et al. Through the looking glass: real-time video using 'smart' technology provides enhanced intraoperative logistics. World J Surg. 2016;40(1):242-244. doi:10.1007/s00268-015-3235-x

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB