Opportunity to Perform – Should Simulation-Based Surgical Training be Provided on Demand or on Supply?

Abstract

Introduction

The aim of the study was to assess the surgical case volume of residents before and after simulation-based training in hip fracture surgery provided on demand versus knee and shoulder arthroscopy provided on supply.

Materials and Methods

A retrospective analysis of surgical case volume in hip fracture surgery and arthroscopic shoulder and knee procedures 90 days before and after simulation-based training of either procedure. Sixty-nine orthopedic residents voluntarily participating in either simulation-based training. Hip fracture surgery simulation was provided on supply, ie, whenever 1–2 residents applied for the course, while the arthroscopic simulation course was supplied twice yearly.

Results

Thirty-four residents participated in hip fracture simulation on demand and 35 residents participated in arthroscopic simulation on supply. The surgical case volume of hip fracture osteosynthesis increased from median 2.5 (range: 0–21) to median 11.5 (1–17) from 90 days before to the 90 days after the simulation-based training on demand. The median difference was 6.5 procedures (p < 0.0003). On the contrary, the surgical case volume in shoulder and knee arthroscopy was low both before and after the simulation on supply, ie, median 2 (0–22) before and median 1 (0–31) after. The median difference was 0 (p = 0.21).

Conclusions

Simulation on demand was associated with increased opportunities to perform in the clinical environment after the simulation-based training compared with simulation on supply. Simulation-based training should be aligned with the clinical rotation of the residents. Simulation on demand instead of supply on fixed dates may overcome this organizational issue of aligning training with the opportunity to perform.

Introduction

Simulation-based training has become mainstay in medical education, providing the advantage of training skills and procedures without compromising patient safety. ¹

In orthopedic surgery, simulation is widely used for practicing different surgical techniques - specifically concerning trauma injuries.^2–6 However, Kalun and colleagues report weak evidence of skills transfer from simulation-based training to the clinic. ⁷ As a consequence, the authors conclude that there is an immediate need for research that focuses on skills transfer, which will allow simulation-based training to be incorporated effectively into Orthopedic Surgery education programs. While Kalun et al.'s review emphasizes training design, the broader literature on transfer of training suggests three overall training inputs that influence conditions of transfer. According to the adapted model of the transfer process, these three training inputs are; resident characteristics, training design, and work environment. ⁸

Within the work environment, Grossman & Salas highlight that residents need relevant opportunities to apply their newly acquired competencies and skills in the workplace. ⁸

Opportunity to perform is highly emphasized in the literature on transfer of training, in which limited opportunities is seen as one of the strongest barriers to successful transfer of training. ⁹ However, a previous study found no correlation between participation in simulation-based laparoscopic training and volume of cases performed in the operating room (OR). ⁹ Orthopedic residents have different levels of experience and prerequisites when entering training, and therefore display different learning curves. The idea of simulation-based training is to pre-train novices and thus advance their clinical learning curve. Theoretically, offering simulation-based training on demand, ie, immediately before being exposed to opportunities to perform in the clinical setting should be advantageous compared with training on supply, ie, training irrespective of alignment with the clinical rotation and the opportunity to perform. ³ The existing literature on transfer of training highlights the importance that training should fit the demand of the clinical workplace. ¹¹

Little is known as to whether orthopedic residents perform relevant procedures after their participation in simulation-based training, ie, transfer their newly acquired simulation skills to the OR, matching their newly acquired skills with relevant opportunities to perform. We set out to investigate how two different simulation-based training designs correlate with residents’ opportunities to perform in the clinical setting. Based on surgical case volume before and after participation in simulation-based training, the present study seeks to answer the question: Should surgical simulation-based training be on demand or on supply?

Materials and methods

At our simulation center, arthroscopic simulation-based training was supplied on fixed course dates twice yearly. In contrast, simulation-based training in hip fracture surgery was conducted on demand, based on the preference of the residents and their clinical supervisors.

To answer the research question, we compared the resident's surgical activity in the OR before and after participation in simulation-based courses in arthroscopy or hip fracture, respectively.

The hip fracture course focuses on the dynamic hip screw (DHS) procedure and is designed as a time-dispersed mastery-learning program with two to three training days and with a maximum of four hours of training per training day. ¹² The construct validity of the virtual reality (VR) hip fracture simulator (TraumaVision, Swemac Simulation AB, Sweden) has been established, ie, the simulator can distinguish between varying levels of clinical expertise. ¹³ The simulator provides the opportunity to train the basic skills required for the clinical DHS procedure, eg, hand–eye coordination and the ability to work in three dimensions based on two-dimensional visual information provided by simulated radiographs and haptic feedback.

The arthroscopy course was supplied twice yearly as a 1 day 9-h course from 8 AM – 5 PM. After 75 min welcome lecture introducing the course and basic arthroscopy, the participants trained on simulators for the rest of the day. The curriculum included basic triangulation and navigation skills, identification of anatomic structures, loose body removal, partial meniscal resection, and arthroscopic examination of the shoulder and potentially Bankart repair.

Data collection and participants

We collected data on orthopedic residents who had participated in either of the two simulation-based training courses and compared their surgical case volume of hip fracture from 2017 to 2022, arthroscopic knee or shoulder procedures from 2014 to 2019, respectively. All residents participating in the simulation training were included in the data analysis and none were excluded.

Simulation training on demand: The surgical case volume of orthopedic residents who participated in the hip fracture simulation-based training was analyzed from 90 days before to 90 days after the training. All hip fracture osteosyntheses were analyzed, ie, cannulated screws, DHS, and intramedullary nailing. The training was held on demand for 1–2 residents, training on 2 simulators.

Simulation on supply: Likewise, the case volume of all arthroscopic shoulder and knee procedures was analyzed before and after the twice yearly simulation-based course.

In Denmark, all surgical cases are logged and available from a business intelligence portal (BI), which is a backend of the electronic patient records. Surgical case data were collected in a pseudomized fashion from the BI database. Data on types of procedures, dates of procedures, and the residents’ roles were collected and merged with employment periods and the dates for participation in one of the two courses. In order to investigate the residents’ opportunity to perform, a total of 90 days after participation in the course were analyzed and compared to their pre-course activities. All simulation data were thus prospectively collected and data regarding clinical procedures were retrospectively assessed from BI.

Statistical analysis: Data were checked for normal distribution with QQ plots and Shapiro Wilk test. The non-parametrical paired data were analyzed with Wilcoxon signed-rank test using Prism for Mac 9.0. The significance level was 0.05. We did not perform a sample size calculation, because the arthroscopy course was no longer running in the same format, when the study was initiated. The number of participants was thus fixed and could not be changed.

Ethical considerations: This study was performed in accordance with the ethical standards in the 1964 Declaration of Helsinki. The Ethical Committee, Central Denmark Region exempted from ethical approval in their letter no. 251/2016 according to Danish legislation. Informed consent: All participants gave their written consent to participate in the study and to use their data in an anonymized form for publication.

Results

A total of 69 residents were included in the present study, with 34 (15 female, 19 male) in the hip fracture dataset and 35 (14 female, 21 male) residents in the arthroscopy data set. There was no statistically significant difference in the female/male ratio, median age of 29 years (range 26–37), and hand dominance between the groups (p > 0.05). Consequently, no subgroup analysis was performed.

The surgical case volume of hip fracture osteosynthesis increased from median 2.5 (range: 0–21) to median 11.5 (1–17) procedures per resident from 90 days before to 90 days after the simulation on demand training. The median difference was 6.5 procedures (p < 0.0003). In comparison, the median surgical case volume in shoulder and knee arthroscopy was low both before and after the simulation on supply, ie, median 2 (0–22) before and median 1 (0–31) after. The median difference was 0 (p = 0.21).

As seen in Figure 1A, we observed a large inter-individual variation in number of procedures performed before and after participation in simulation-based hip fracture training. Similar to the hip fracture data, we found a large inter-individual variation in number of procedures performed before and after participation in simulation-based arthroscopy training (Figure 1). The resident with the highest volume of arthroscopy cases prior to simulation-based training had performed 22 arthroscopies and performed 0 arthroscopies in the following 90 days. Furthermore, 12 residents had not performed any arthroscopic procedure in the 90 days before the simulation. In total, the 35 residents performed a median of 2 arthroscopy procedures before participation in the simulation-based training and a median of 1 procedure after, with a median difference of 0 procedures (Figure 1B).

Figure 1.

Surgical procedures. 1A: Number of surgical procedures performed per resident in the 90 days before and after the simulation on demand (hip fracture osteosyntheses) and simulation on supply (arthroscopies). 1B: Difference after-before the simulation-based training with median and interquartile range as whiskers. Median (range) given above. *p < 0.0003, **p = 0.21.

Discussion

The aim of our study was to assess first-year orthopedic residents’ opportunities to perform in the OR in the post-course periods by investigating the case volume 90 days before and 90 days after simulation-based training. Simulation on demand (hip fracture training) was associated with a significant increase in performed procedures after the training, while simulation on supply (arthroscopy training) was associated with a low number of median procedures both before and after the simulation-based training.

Planning and executing simulation-based training on supply in arthroscopic skills did not align with the clinical opportunities for skill transfer. The current Danish curriculum for orthopedic residents requires more than 25 arthroscopic knee procedures and more than 20 shoulder/elbow surgeries including osteosyntheses and arthroscopies. ¹⁴ Arthroscopy is most often performed as elective surgery and it should thus be feasible to provide optimal learning and transfer opportunities for residents. Nonetheless, this was not the case in the current cohort. A reason for this finding could be, that arthroscopic knee and shoulder surgery primarily is performed at specialized units. Consequently, residents are exposed to this type of surgery in limited time slots of 2–3 months during rotation of surgical training. This may partially explain the low surgical case numbers both before and after participation in surgical simulation-based training in the present study. This highlights the need to structure simulation-based training as on demand instead of supply.

Both simulation-based training courses were free of charge for the participants and voluntary, as such the courses were not part of a mandatory curriculum. A selection bias regarding the attending participants may thus have occurred. This bias would however be directed at selecting the most ambitious and determined residents seeking learning opportunities. The lack of increased numbers of arthroscopic procedures after the simulation is therefore likely to be a structural problem of the timing of the course in relation to their clinical rotation in relevant subspecialties, rather than the participants’ willingness to engage in clinical arthroscopy after the course participation.

Both arthroscopy and hip fracture procedures are a part of the suggested national curriculum for simulation-based training in orthopedic surgery and The Core Competencies for General Orthopaedic Surgeons.^5,15 Hence, it makes sense that these procedures are learned in the simulated setting and made compulsory. However, in order to reap the benefit of training, there is a need for a greater structural and organizational alignment in order to increase the transfer of training. This approach should be focused on aligning simulation-based training with clinical opportunities to apply newly acquired skills, thus facilitating training by structural integration rather than by chance. ¹⁰ The benefits of personalized training, such as proficiency-based training and individualized feedback are well-established in simulation-based surgical training and the findings in our analysis highlight a gap in the alignment between the simulation-based training and the clinical work environment.^16,17 Combining non-technical skill training with the acquisition of technical skills may further contribute to skill development. ¹⁸

It is well-known that self-efficacy among residents is influenced by simulation-based training and that self-efficacy correlates with engagement and performance.^19,20 Teman and colleagues found that the residents’ confidence and decrease in ownership of responsibility for patients are two main factors for faculty decisions to granting autonomy to surgical residents. ²¹ This raises the importance of the supervisor's awareness of resident's participation in simulation-based training activities. As some learners require more training and repetitions than others it could have a negative influence on the self-efficacy and motivation of the resident and thereby affect their autonomy in the clinic. ³ In continuation, Huang et al report that self-efficacy influences learner's motivation and transfer attempts. Furthermore, the workplace should support the learner by providing learning opportunities. ²² As such, it is important that the timely correlation between simulation-based training and clinical activities is supported in the work environment. In this endeavor, it is important that we consider within-person variability and view the transfer of training as trajectories over time.^23,23

Limitations of the present study include the lack of testing the validity of the data from the BI portal. The BI portal contains data recorded in the electronic patient records by clinicians and secretaries. Hence, data could be subject to information bias, which might have led to non-differential inaccuracy in the estimates of the residents’ surgical activity. In addition, we only included data from procedures where the residents were listed as primary surgeons with or without supervision. Theoretically, the diagnostic part of arthroscopies could have been undertaken by the residents and the therapeutic part by a consultant, who is listed as primary surgeon. If this is the case, the opportunity to perform and apply the newly acquired skills may be underreported. In a previous publication based on BI data, we observed that the residents’ role as assistants may have been under-registered. This is not the case for primary surgeons, because case notes are legal documents and the primary surgeons carry the legal responsibility for the procedure. While the included data has been treated equal, we acknowledge that this could be nuanced by workplace-based approaches in surgical education as the Zwisch model for teaching and assessment in the OR. Within this model, supervision in the OR is divided into four stages; Show and Tell, Smart Help, Dump Help, and No Help. In this manner, we suggest that the current study can serve as a starting point for aligning simulation-based training and clinical learning and that future research on transfer of training and opportunity to perform after simulation-based surgical training. Furthermore, we only analyzed the opportunity to perform from a quantity perspective and not focusing on the quality of the procedures performed before and after simulation-based training. As such we did not investigate if simulation on demand impacts the level of competence compared with simulation on supply. This interesting research question was beyond the scope of this retrospective analysis paper, and we suggest that future research investigates how a timely correlation between simulation-based training and workplace-based training impacts the level of competence. Another limitation is that the study was initiated after the arthroscopy course was discontinued in the described form, the study was thus not based on a sample size calculation. However, the study results are still clear, and the study was therefore not underpowered.

Conclusion

Simulation-based training has to fit and align with clinical training. On demand simulation-based training may offer a potential solution to this challenge.

In a future study, we wish to investigate, whether individual (or small group) arthroscopic simulation-based training on demand, can be better timely coordinated with the surgical rotation and prove to be advantageous, by providing the opportunity to transfer and extend the acquired surgical skills in the clinical setting, ie, a higher case volume after participation in simulation-based training.