Efficacy of a Virtual Arthroscopic Simulator for Orthopaedic Surgery Residents by Year in Training

Abstract

Background:

Virtual reality arthroscopic simulators are an attractive option for resident training and are increasingly used across training programs. However, no study has analyzed the utility of simulators for trainees based on their level of training/postgraduate year (PGY).

Purpose/Hypothesis:

The primary aim of this study was to determine the utility of the ArthroS arthroscopic simulator for orthopaedic trainees based on their level of training. We hypothesized that residents at all levels would show similar improvements in performance after completion of the training modules.

Study Design:

Descriptive laboratory study.

Methods:

Eighteen orthopaedic surgery residents performed diagnostic knee and shoulder tasks on the ArthroS simulator. Participants completed a series of training modules and then repeated the diagnostic tasks. Correlation coefficients (r²) were calculated for improvements in the mean composite score (based on the Imperial Global Arthroscopy Rating Scale [IGARS]) as a function of PGY.

Results:

The mean improvement in the composite score for participants as a whole was 11.2 ± 10.0 points (P = .0003) for the knee simulator and 14.9 ± 10.9 points (P = .0352) for the shoulder simulator. When broken down by PGY, all groups showed improvement, with greater improvements seen for junior-level residents in the knee simulator and greater improvements seen for senior-level residents in the shoulder simulator. Analysis of variance for the score improvement variable among the different PGY groups yielded an f value of 1.640 (P = .2258) for the knee simulator data and an f value of 0.2292 (P = .917) for the shoulder simulator data. The correlation coefficient (r²) was –0.866 for the knee score improvement and 0.887 for the shoulder score improvement.

Conclusion:

Residents training on a virtual arthroscopic simulator made significant improvements in both knee and shoulder arthroscopic surgery skills.

Clinical Relevance:

The current study adds to mounting evidence supporting virtual arthroscopic simulator–based training for orthopaedic residents. Most significantly, this study also provides a baseline for evidence-based targeted use of arthroscopic simulators based on resident training level.

Knee and shoulder arthroscopic surgery are among the most commonly performed procedures worldwide. ^{19,24,26,32,37} There is a steep learning curve associated with arthroscopic surgery for orthopaedic surgery trainees. ^9,33 Extensive hands-on training is typically required to develop surgical competency. ^32,33 The current minimum number of cases may not be sufficient to develop competency in arthroscopic surgery. ³⁰ Price et al ³² found that it takes about 170 procedures before a surgeon develops consultant-level motor skills in knee arthroscopic surgery. With work-hour restrictions, patient safety concerns, and fellows often taking priority over residents in performing cases, it is challenging for residents to obtain high-level arthroscopic skills by the end of their residency training. ^{12,15,20,25,30,31} Senior orthopaedic residents have reported feeling that there is insufficient time dedicated to arthroscopic training and that they are less prepared in arthroscopic surgical procedures compared with open procedures. ²⁰

Virtual reality arthroscopic simulators are an attractive option for resident training. ^2,4 Simulators provide trainees with a risk-free environment in which to hone their arthroscopic skills. ^9,12,33,37 Arthroscopic surgery is highly dependent on technical ability, so virtual reality arthroscopic simulation holds great potential for having a positive impact on improving surgical skills. ^{1,4,10,11,21,37} Another benefit of virtual reality simulation is that it can be scored and assessed, serving as a quantitative measure of a trainee’s improvement. ^27,37,38 Previous work has shown that surgical simulator training can lead to improved arthroscopic skills in orthopaedic surgery residents. ^9,16,22,33 A study by Coughlin et al ¹² used an arthroscopic simulation model to successfully teach and evaluate basic arthroscopic skills with good construct validity. The benefits of virtual reality simulation have been shown in other fields, such as general surgery, ophthalmology, urology, and aviation. ^13,17,18,34 The Accreditation Council for Graduate Medical Education (ACGME) recently approved mandates to implement surgical simulation training in all orthopaedic residency programs. ^3,12,36 However, the role of virtual reality simulators in orthopaedic surgical training is evolving, with further evidence-based research needed to define best practices. ^9,33,37

The purpose of this study was to evaluate the efficacy of a virtual arthroscopic simulator in orthopaedic surgery education by assessing improvements in residents’ performance based on their level of training/postgraduate year (PGY). Our first aim was to determine if the ArthroS arthroscopic simulator (VirtaMed) can improve trainee performance and to measure that improvement. Our second aim for this study was to determine the utility of the simulator for orthopaedic trainees based on their level of training—in other words, to determine at what point in training the simulator offers the most benefit for trainees. We hypothesized that the simulator would enhance trainee performance in arthroscopic surgery. We further hypothesized that given the steep learning curve associated with arthroscopic surgery, all resident levels would benefit.

Methods

Participants

Eighteen orthopaedic surgery residents participated in our single-center study. The study was designed to coincide with the incorporation of simulator training in this 5-year residency program. Each participant was assigned a random unique identifier. Demographic information was recorded for each participant, including sex, age, and PGY. The simulator used in this study was the ArthroS, consisting of a shoulder simulator and a knee simulator. Each participant underwent a standard orientation to the simulator. The fidelity and potential for educational benefit with the ArthroS have been validated in prior studies. ^9,28,35 This study was funded by a grant from the American Board of Orthopaedic Surgery (ABOS) and was approved by our institutional review board.

Simulator Training and Testing

Each resident completed a diagnostic task on the knee simulator and the shoulder simulator, and composite scores for these tasks were recorded. The residents then completed knee and shoulder simulator training over the same 2-month period (regardless of which rotation they were on at the time). They performed the tasks outside of clinical duties but adhered to ACGME work-hour restrictions.

The knee simulator was subdivided into 3 groups of modules: “Basic Skills,” “Diagnostic Cases,” and “Surgical Cases.” The Basic Skills group consisted of 3 modules in triangulation, 2 modules in loose body removal, and 2 modules in guided meniscectomy. The Diagnostic Cases group consisted of an evaluation of different types of meniscal tears, an evaluation of grade 1-4 arthrosis in the knee with associated meniscal tears, and an evaluation of anterior cruciate ligament ruptures with associated meniscal tears. The Surgical Cases group consisted of modules requiring the trainee to identify 3 different types of meniscal tears and perform meniscectomy (with progressive levels of difficulty); modules requiring the trainee to identify grade 1-4 arthrosis in the knee, find meniscal lesions, and perform meniscectomy; a module requiring the trainee to identify synovitis and remove the inflamed synovium using a shaver; and finally, 2 modules requiring loose body removal from the knee joint. Figure 1 shows a resident using the ArthroS knee simulator and a close-up view of the monitor during one of the Basic Skills modules.

Figure 1.

(A) A resident using the ArthroS knee simulator. (B) The monitor of the knee simulator during a “Basic Skills” module.

The shoulder simulator was subdivided into 2 groups of modules: Basic Skills and Diagnostic Cases. These were performed in the beach-chair position. The Basic Skills group consisted of 2 modules designed to teach the trainee to navigate the camera in the glenohumeral joint and the subacromial space, 4 modules simulating triangulation (each with progressive levels of difficulty), 2 modules simulating the removal of loose bodies, and 2 modules simulating guided subacromial decompression (acromioplasty). The Diagnostic Cases group consisted of 10 modules simulating basic navigation skills of the glenohumeral joint and the subacromial space; an evaluation of the acromion and the biceps tendon; and a diagnostic evaluation of loose bodies, rotator cuff tears, Bankart lesions, labral tears, and superior labral from anterior to posterior (SLAP) tears. Residents were allowed free access to the simulation laboratory during the time of the study. They were required to complete training while adhering to work-hour restrictions. The mean time required to complete 1 knee module for all residents was 17 minutes (range, 5-56 minutes). The mean time required to complete 1 shoulder module for all residents was 46 minutes (range, 14-116 minutes).

After completion of the training modules on the simulator (immediately after the 2-month training period), each resident performed a final diagnostic knee task and a final diagnostic shoulder task. One final score was recorded for each resident on each task.

Data Collection and Statistical Analysis

The simulator produces a composite score (range, 0-50) that factors in correct anatomic identification, efficiency of movement and hook manipulation, respect for surrounding tissue, and time to completion of tasks. This composite score is based on the Imperial Global Arthroscopy Rating Scale (IGARS), which was developed by Bayona et al ⁷ as an assessment tool for both real and simulated arthroscopic tasks. ²⁹ We elected to utilize this composite score as our primary outcome measure to avoid overly complicated multivariate data analysis. Furthermore, using the composite score facilitates intraparticipant and interparticipant comparisons as well as a comparison of our data with other studies reporting outcomes as the IGARS or to other training or research applications of the ArthroS simulator.

Summary statistics were calculated in terms of means and standard deviations for the composite scores by PGY and for the participants as a whole. Group differences for discrete variables were evaluated using analysis of variance (ANOVA). Statistical significance was set at alpha (P value) equal to .05. The correlation coefficient (r²) was calculated for the score improvement variable as a function of PGY. All analyses were performed using Excel (Microsoft).

Results

We found significant improvements for both knee and shoulder skills at all levels of training. For the knee simulator, the mean score improvement was 11.2 ± 10.0 points (P = .0003) (Appendix Table A1). For the shoulder simulator, the mean score improvement was 14.9 ± 10.9 points (P = .0352) (Appendix Table A2). Of the 18 residents who participated in the study, all 18 completed knee simulation testing and training, and 17 completed shoulder simulation testing and training (the resident who did not complete the shoulder simulation was a PGY 5). Figure 2 and Appendix Tables A3 to A7 show mean improvements and composite scores by PGY for the knee simulator. Figure 3 and Appendix Tables A8 to A12 show mean improvements and composite scores by PGY for the shoulder simulator.

Figure 2.

Performance (mean composite score) on the knee simulator before and after training by postgraduate year (PGY). The change in the mean score for each group is also shown.

Figure 3.

Performance (mean composite score) on the shoulder simulator before and after training by postgraduate year (PGY). The change in the mean score for each group is also shown.

Individual score improvements for all participants are presented in Appendix Tables A1 to A12, with Tables A1 and A2 showing all participants’ knee and shoulder scores, Tables A3 to A7 showing knee scores broken down by PGY, and Tables A8 to A12 showing shoulder scores broken down by PGY.

ANOVA for the score improvement variable in the knee simulator and shoulder simulator data among the different PGY groups yielded f values of 1.640 (P = .2258) for the knee simulator and 0.2292 (P = .917) for the shoulder simulator. The correlation coefficient (r²) for the knee score improvement was –0.866, while the r² for the shoulder score improvement was 0.887 (Figure 4).

Figure 4.

Correlation coefficients (r²) for composite score improvement in relation to postgraduate year (PGY) for the (A) knee simulator and (B) shoulder simulator.

Discussion

The current study is the first to evaluate improvement in arthroscopic simulator training by PGY for an entire residency program. We report 2 important findings: First, the results confirmed our first hypothesis that the ArthroS simulator significantly improves trainee arthroscopic performance for both the knee simulator and the shoulder simulator. Second, our study indicates that the knee simulator is more beneficial for junior residents (PGY 1 and 2), while the shoulder simulator is more beneficial for senior residents (PGY 4 and 5).

These relationships between the residents’ year in training and improvement on the arthroscopic simulator are novel findings. The r² for the knee and shoulder score improvement (–0.866 and 0.887, respectively) were both strong. There was a strong negative correlation between PGY and score improvement in the knee group and a strong positive correlation between PGY and score improvement in the shoulder group. To elaborate on this point, analysis of the raw scores showed that seniors started at higher baseline levels for the knee tasks, likely because they had greater exposure to knee arthroscopic surgery at that point in their training. This may have led to them having less room to improve. Meanwhile, the junior residents, who had limited or no experience in knee arthroscopic surgery, showed more significant improvement after using the simulator because they started at a lower baseline skill level. Regarding the senior residents’ greater improvement on shoulder arthroscopic surgery, we believe that the seniors, having mastered basic arthroscopic skills, were more apt in attaining finer psychomotor skills, which we think are required to score higher for the shoulder tasks. It is important to note that only 2 PGY 5 residents participated in the shoulder arthroscopic tasks for this study. Nevertheless, we believe that these findings should guide further studies in the field and guide the development of resident arthroscopic simulator curricula.

Overall, we found that our residents as a group made significant improvements in their arthroscopic skills on the simulator for both the knee and the shoulder. These results make a strong case for incorporating simulator-based arthroscopic surgery training into orthopaedic residency programs. Orthopaedic surgery trainees face a steep learning curve during arthroscopic surgery, with little room for error around delicate articular surfaces. ^9,14,23,33 The challenges of acquiring arthroscopic skills during traditional residency training have been well described in prior studies. ^∥ The importance of how work-hour restrictions have negatively affected case volumes and the fact that the current number of cases performed during resident training may not be sufficient to develop arthroscopic competency cannot be overstated. ^{12,15,20,25,30,31} The improvement seen in the trainees in this study suggests that the simulator can help meet these challenges in the arthroscopic training of orthopaedic residents. Anecdotally, residents also felt that this training was helpful; they appreciated the opportunity to practice arthroscopic skills. In general, however, they felt that the simulator was not a true substitute for hands-on arthroscopic surgical training. The residents on sports medicine and arthroscopic surgery rotations did make great improvements in their skills during the rotations. We felt as faculty that the simulator was helpful for these residents. However, because arthroscopic skills vary somewhat from resident to resident, teasing out the contribution of the simulator to their actual operative skill development is beyond the scope of this study. We suggest that further investigation of these concepts is warranted.

We must note that when subgroups were analyzed based on PGY, improvements did not reach statistical significance (with the exception of the PGY 3 group on the knee simulator). A power calculation confirmed that we were underpowered to detect differences in the magnitude of improvement in each subgroup. Power analysis showed that 61 participants in each PGY group would be needed to achieve significance. Likewise, ANOVA for the score improvement variable in the knee simulator data and shoulder simulator data among PGYs did not yield statistically significant findings. Again, this is not surprising, given that the number of data points in each PGY group (2-4 participants) was small and thus subject to type II error. Nevertheless, the absolute numbers for the means show clear improvement in performance at all levels of training for both knee and shoulder simulator tasks. We believe that a larger-scale study would confirm that the magnitude of improvement was significant, but it was not feasible for this initial study.

Our results corroborate those of prior studies, which have also shown that simulator training benefits orthopaedic surgical residents. Rebolledo et al ³³ compared 14 junior orthopaedic residents at a single institution who were randomized to undergo knee and shoulder arthroscopic surgery training with either a surgical simulator or didactic lectures with arthroscopic models. After their respective training, performance in diagnostic knee and shoulder arthroscopic surgery was assessed using a cadaveric model. Residents who trained with a surgical simulator outperformed the residents with didactic training in that study. ³³ Camp et al ⁹ randomized 45 orthopaedic surgery residents to 1 of 3 groups: control, training on cadaveric specimens, and training with the use of a simulator. Before and after training, residents performed diagnostic knee arthroscopic surgery on cadaveric specimens, which were timed and video-recorded. In that study, residents practicing on cadaveric specimens improved twice as fast as those utilizing a high-fidelity simulator; however, based on a cost estimation specific to their institution, the authors concluded that simulator training may be more cost-effective if it is used at least 300 hours per year. ⁹

Middleton et al ²⁸ randomized 17 arthroscopic surgery novices to a training program on a benchtop or a virtual reality knee arthroscopic simulator. Participants performed diagnostic arthroscopic surgery on both simulators before and after the training program, and performance was assessed using wireless objective motion analysis and a global rating scale. In that study, both simulators delivered improvements in arthroscopic skills. Benchtop training led to skills that readily transferred to the virtual reality simulator. Yet, skills acquired after virtual reality training did not transfer as readily to the benchtop simulator. ²⁸ Banaszek et al ⁶ reported similar results but noted significantly greater improvement with high-fidelity virtual simulation compared with low-fidelity benchtop simulation. We echo the authors of each of these studies, who were all careful to note that additional studies on simulator training are still warranted.

We elected to utilize the composite score on the simulator as our primary outcome measure. After careful consideration, we felt that analysis of the subvariables would be a search for statistical significance but would not necessarily yield meaningful data. We felt that by choosing an important and easily measurable outcome, we would also make our findings more powerful. Furthermore, the composite score from the ArthroS is fundamentally similar to the IGARS. ⁷ Choosing to use the composite score allows easy comparisons among participants and facilitates comparing data from our study with other similar studies.

Middleton et al ²⁹ performed a prospective study using the ArthroS to determine which commonly used global rating scale (Arthroscopic Surgical Skill Evaluation Tool [ASSET], Basic Arthroscopic Knee Skill Scoring System [BAKSSS] or IGARS) demonstrated superior assessment of simulated arthroscopic skills. Sixty-three participants of varying surgical experience performed a number of arthroscopic tasks on the ArthroS simulator. Two blinded observers graded each participant’s performance using the 3 global rating scales. Performance was also assessed by validated objective motion analysis. The authors found that all of the global rating scales demonstrated construct validity and concluded that no single global rating scale demonstrated superiority as an assessment tool. ²⁹ Based on those results, we felt comfortable utilizing the ArthroS composite score. Our results further validate the use of the composite score, which we believe will be a useful outcome metric for further research with the ArthroS simulator and will allow easy comparisons among studies. In addition, our results suggest that the composite score can serve as a simple and easily comparable metric for trainees and educators to track progress, set milestones, and compare outcomes on the simulator.

To our knowledge, this study is the first to evaluate arthroscopic performance with regard to PGY. We anticipate that the findings may help set educational standards for future orthopaedic surgery trainees, particularly as simulators become more common and their specific optimal applications are better characterized. We are utilizing our data to develop an educational program for virtual arthroscopic training in our orthopaedic surgery residency program. At our institution, the completion of a minimum number of tasks and modules on the ArthroS simulator is now mandatory for PGY 1 and 2 residents before they begin their first sports medicine rotation. In addition, the simulator is accessible for trainees who have completed a simulation laboratory orientation, and trainees are encouraged to practice their arthroscopic skills in the laboratory.

We are aware of limitations in this study. The most significant limitation was the lack of measurement of actual surgical performance for the participants (performance was measured solely on the simulator instead). Another limitation of the study was the lack of subanalyses of various components that made up the composite score for each participant (ie, correct anatomic identification, efficiency of movement and hook manipulation, respect for surrounding tissue, and time to completion of tasks). As mentioned before, we elected not to look independently at all of the subvariables that go into the composite score to avoid overly complicated multivariate data analysis and to allow the facilitation of comparisons among participants and data from our study to other similar studies. We believe that such subanalyses can be helpful for individual participants to gauge their progress and identify areas they need to improve, but for the study as a whole, they are not necessary.

Additionally, as noted above, our subgroups were not sufficiently powered to detect that improvement in each individual PGY group was in fact significant. Finally, one confounding factor in this study was ongoing surgical training; PGY 3 residents on the sports medicine rotation would have been reinforcing their simulator training with hands-on arthroscopic surgery as compared with the majority of residents, who were on other services. We suspect this may be the reason why there was significant improvement on the knee simulator for the PGY 3s as a subgroup. Other factors may be variability in baseline interest in arthroscopic surgery/sports medicine or differences in intended career path. It is important to note that our results may not directly transfer to other training programs, given program-specific variation in exposure to arthroscopic surgery at different times, for different durations, at different volumes, and in different circumstances (eg, presence of fellows).

One significant limitation to implementing arthroscopic simulator training may be the expense associated with it. At our institution, the simulator (including both shoulder and knee components) cost US$97,000 plus a training and installation fee of $1200 and a 4-year extended warranty of $38,800. This cost was covered by a mission support grant from Baylor College of Medicine as part of building a comprehensive surgical simulation laboratory. Camp et al ⁹ noted that although pricing varies among models and institutions, virtual reality simulation could potentially be a more cost-effective training model for programs lacking funding for a cadaveric training center. The authors of that study reported that at their institution, the simulator would need to be used just over 300 hours annually to yield a higher rate of financial benefit than a cadaveric laboratory for the program. ⁹ It is important to note that there are less expensive options for hands-on arthroscopic skill development, such as Fundamentals of Arthroscopic Surgery Training (FAST). Similarly, Arealis et al ⁵ have made available a way to create a simple and cost-effective arthroscopic simulator using readily accessible materials and a low-cost web camera. Further study is needed to compare an arthroscopic simulator with low-cost alternatives such as FAST.

Conclusion

We found that residents training on a virtual arthroscopic simulator made significant improvements in both knee and shoulder arthroscopic skills. Furthermore, our results may indicate that the knee simulator may be more beneficial for junior residents, while the shoulder simulator appears to be more beneficial for senior residents. The current study adds to the mounting evidence supporting virtual arthroscopic simulator–based training for orthopaedic surgery residents. Our greater purpose is to foster data-driven educational standards for virtual arthroscopic training in orthopaedic surgery residency.