Abstract
Objective
Metric based virtual reality simulation training may enhance the capability of interventional neuroradiologists (INR) to perform endovascular thrombectomy. As pilot for a national simulation study we examined the feasibility and utility of simulated endovascular thrombectomy procedures on a virtual reality (VR) simulator.
Methods
Six INR and four residents participated in the thrombectomy skill training on a VR simulator (Mentice VIST 5G). Two different case-scenarios were defined as benchmark-cases, performed before and after VR simulator training. INR performing endovascular thrombectomy clinically were also asked to fill out a questionnaire analyzing their degree of expectation and general attitude towards VR simulator training.
Results
All participants improved in mean total procedure time for both benchmark-cases. Experts showed significant improvements in handling errors (case 2), a reduction in contrast volume used (case 1 and 2), and fluoroscopy time (case 1 and 2). Novices showed a significant improvement in steps finished (case 2), a reduction in fluoroscopy time (case 1), and radiation used (case 1). Both, before and after having performed simulation training the participating INR had a positive attitude towards VR simulation training.
Conclusion
VR simulation training enhances the capability of INR to perform endovascular thrombectomy on the VR simulator. INR have generally a positive attitude towards VR simulation training. Whether the VR simulation training translates to enhanced clinical performance will be evaluated in the ongoing Norwegian national simulation study.
Introduction
Acute ischemic stroke remains a major cause of morbidity and mortality if left untreated, particularly in the case of large vessel occlusions (LVO). 1 Rapid treatment with endovascular thrombectomy (EVT) is a highly effective treatment for LVO stroke with a number needed to treat as low as 2.6.2,3 Besides rapid triage and correct patient selection, complete and timey reperfusion are key prerequisites for good patient outcome.4,5 Thus, the technical skillset of the individual operator is crucial. That being said, in smaller centers that perform a limited number of procedures, it can be challenging to maintain a high level of technical expertize and excellence. Additionally, strict minimum volume requirements for certification may not allow smaller centers to offer EVT and thereby prevent establishing EVT in rural areas, remote from Comprehensive Stroke Centers. 6 We previously suggested that simulation training might act as a complimentary tool that can partially substitute experience gained through treating patients, especially in low volume centers, thereby accelerating learning curves of trainees and ensuring an adequate level of experience of interventional radiologists (INR) performing EVT procedures. 7 At our institution, we have adopted simulation based team training since 2017 to improve the interplay between the various members of the stroke team, which has reduced door to needle times and improved patient outcomes considerably. 8
In addition to the team-based simulation approach, we developed a pilot training program in which INR who perform EVT at our hospital attended a structured training program with metric based virtual simulation. We hypothesized that the simulator-based task training curriculum helps INR to develop their skill set, leading to better performance on the simulator.
Methods
INR and radiology residents were recruited to perform simulated EVT on a high-fidelity simulator. Participating in this study was voluntary. The study consisted of a didactic introduction, a baseline evaluation, a simulation training period and a post-simulation evaluation, as described below.
Endovascular simulator
For the virtual simulation training, we used a portable high-fidelity endovascular simulator (Mentice VIST® G5 simulator) that provides realistic scenarios for hands on procedural training. The integrated thrombectomy software program (Neurovascular Thrombectomy 1.0) used in the study was constructed by a Delphi panel of 21 experienced INR. This group of INR deconstructed the EVT procedure into 10 phases, 46 steps, and 57 possible errors. 9 Collected metric based data parameters used in the study were total procedure time (min), the number of accomplished steps that were technically executed correctly, the number of handling errors, applied intraarterial contrast volume (ml), fluoroscopy time (min) and patient radiation dose exposure (mGy).
Pre- and post-simulation survey
Participants were asked about their expectation and general attitude towards VR simulator training. Furthermore, participants were questioned about whether they think simulation training would improve their overall performance. Finally, they were asked to rate the perceived impact of simulator training on real patient treatment. The detailed questionnaire that was used for the pre- and post-simulation survey can be found in the Supplemental Material (Survey, supplement).
Didactic introduction
All participants were given a standardized theoretical introduction by an experienced simulation instructor who was familiar with both clinical EVT procedures and VR simulation training. The radiology residents, who generally had limited prior EVT experience were introduced to the relevant basic anatomy of the cervical and intracranial vasculature, to the properties and geometry of the catheters and other endovascular devices, including the basic techniques for the EVT procedure.
Pre- and post-simulation assessment of participants’ EVT skills
The Mentice simulation curriculum consisted of nine patient case scenarios, each with distinctive tasks and learning objectives (Cases overview, supplement). We chose two scenarios for the assessment of participants’ baseline skills.
The first case scenario (#1) was hereby chosen with the educational goal to perform safe access to the carotid artery through a Type I-II aortic arch using multi-purpose forward curve catheters and perform clot extraction in a right M1 segment middle cerebral artery occlusion.
For the second case scenario (#4), a more challenging procedure was chosen with the main goal to perform safe access to the carotid artery through a Type III aortic arch using a Simmons catheter and perform clot extraction in a left M1 segment middle cerebral artery occlusion.
Individual performance on these two case-scenarios was tested (initial pre-simulation assessment) and re-tested after 5 months of training (post-simulation assessment). Both case-scenarios were exclusively used for evaluation of participants’ baseline skills, and were thus not used for training purpose. In turn, participants were given free access to the remaining seven cases for training purposes.
Simulation training period
After completing the baseline assessment, the 5 months training period started, during which participants had free access to the Mentice simulator. We defined no upper training limit, but at least 30 cases without supervision had to be completed during the training period. As a stationary training environment, we established an office-room that was available for all participants 24 h, every day (Figure 1).
"Office room" (1).
Statistical analysis
All statistical analyzes were performed using SPSS Statistics version 26 (IBM Cooperation, Armonk, NY, USA). Participants who had performed more than 50 thrombectomies in total, were ranked as experts, those performing less than 50 thrombectomies (including those with no prior practical thrombectomy experience at all) were categorized as novices. Numerical results were described by providing means and standard deviations. Differences between before and after training were examined using paired samples t-tests in cases with positive correlation between before and after scores, and with independent samples t-tests (with possible Welch correction if SD in one sample was more than 50% larger than in the other sample) in cases with negative correlation between before and after scores. A significance level of 0.05 was used to determine statistical significance.
Results
The study was performed from October 2019 up to February 2020, and six of the seven INR performing EVT participated in the study. Additionally, four out of eleven radiology residents with no prior experience in interventional neuroradiology participated. For case 2 we lost one INR due to sabbatical leave and one resident due to change of residence. During the five months of training, the participants performed in total 436 training sessions. All the participants performed at least 30 cases (median 37). 71 (16.3%) of the training sessions were aborted due to a runtime error in the program. The participant had to either reopen the software or restart the computer to remove the error.
Because the questionnaire was based on the participants with clinical experience, we only included the six interventional radiologists, but not the four radiology residents in the pre- and post-questionnaire. The two most experienced INR had performed over 100 EVT procedures, two INR had performed approximately 50 procedures, and the remaining two INR had performed 23 and 10 procedures, respectively.
Pre- and post-simulation assessment - case scenario 1 and 2
When comparing the pre- and post-simulation performance of participants in the first and second case-scenario (Cases overview, supplement) total time used improved significantly. In the first case scenario the participants improved from 39.4 to 18.4 min (p < 0.001), while in the second case scenario from 31.5 to 17.2 min (p < 0.001). Both groups were able to successfully execute more procedure steps [(case 1: mean 25.8 pre-simulation vs. 32.6 post-simulation, p = 0.02) (case 2: mean 25.6 pre-simulation vs. 38.4 post-simulation, p = 0008)] and nominally made less handling errors, although this difference did not reach statistical significance (p = 0.67). Equally, there was a reduction in applied contrast volume (case 1: 109.6 ml pre-simulation vs. 75.1 ml post-simulation, p = 0.04.Case 2: 109.8 ml pre-simulation vs. 63.5 ml post-simulation, p = 0.05), fluoroscopy time (case 1: 16.6 s pre-simulation vs. 7.4 s post-simulation, p = 0.004.Case 2: 15.3 s pre-simulation vs. 7.6 s post-simulation, p = 0.007) and radiation dose (case 1: 54.7 mGy pre-simulation vs. 25.7 mGy post-simulation, p = 0.02.Case 2: 84.9 mGy pre-simulation vs. 20.6 post-simulation, p = 0.02) by assembling both groups. For detailed comparison of pre- and post-simulation metrics of case 1 and 2, see Tables 1 and 2.
Benchmark case 1 on the simulator.
All values are given in mean (standard deviation), n = numbers. ml and J/kg are fictive values calculated from the simulator.
Benchmark case 2 on the simulator.
All values are given in mean (standard deviation), n = numbers. ml and J/kg are fictive values calculated from the simulator.
Pre- and post-simulation survey
Prior to the simulation training, five of the six INR thought that VR simulator training would lead to large time savings on the simulator, only three thought it would translate into large time-savings in clinically treated patients. Four INR thought that VR training would have impact on recanalization results in clinically treated patients, only one that it would have large impact. In the survey after the simulation training, unchanged, five of the six INR thought that VR simulator training would lead to large time-savings on the simulator, and three INR thought the training would also translate to time savings in EVT in clinical practice, yet only with a small impact. Only two INR thought that VR training would have impact on recanalization results in clinical practice. Informal feedback from the INR training on their simulator experience highlighted that they sometimes perceived the training sessions as somewhat boring and unpleasurable, as the system provided little positive feedback.
Discussion
In this simulator-based task training study, VR simulation training enhanced the capability of INR to perform EVT in a VR simulation environment.
As simulation task training is still lacking implementation in the educational curriculum, there is limited information about its translation into clinical results. We could earlier show that EVT simulation team training contributes to reduce treatment times in stroke and that this training has positive effects on patient outcome. 8 EVT simulation task training in addition to team based simulation training has been shown to reduce treatment times in acute patients with large vessel occlusions stroke. 10
In our study, all participants, regardless of their level of experience, showed significant improvement over the 5 months training period in their total procedural times. Additionally, at the end of the simulation training, participants were able to successfully execute more procedure steps, and fluoroscopy time, radiation dose and contrast volumes were reduced (Tables 1 and 2): Novices showed significant improvement in the number of successfully executed procedure steps (case 2), a reduction in fluoroscopy time (case 1) and radiation dose (case 1). Experts showed significant improvements in handling errors (case 2), a reduction in contrast volume used (case 1 and 2), and fluoroscopy times (case 1 and 2). The results from the current study therefore disagree with results of previous studies, which suggested that simulation training might have less effect in experienced participants.11,12
Both prior to the simulation training and after having completed the training curriculum, the participating INR had a positive attitude towards VR simulation training. This generally positive attitude of INR towards simulation training has been shown in several studies, but this may be partially related to selection bias, since participation in our and most previous studies was on a voluntary basis.13–15 Indeed, simulator-based task training has only recently been introduced into stroke training and one of the main obstacles to overcome is the acceptance of this training method as supplement to the traditional supervised learning approach. 7 Simulation training is still often considered a potential training tool mainly for beginners in the absence of experienced supervisors and acceptance among senior physicians is limited, although it has the potential to serve as a useful and valuable lifelong training and preparation tool. 7 The limited acceptance may be partially related to the lack of positive feedback that is provided during the simulation itself, and the relative “boring” simulation environment, as it was described informally by some of the participants. There were technical issues with some of the training sessions, which had negative impact on the motivation for training, we believe updates of simulators and technical improvements will limit this obstacle in the future. A greater degree of gamification of the simulation training, for example through internal competitions among physicians with awards, could probably lead to wider acceptance and use of simulators, as playing games improves learning outcomes and has positive impact on behavioral, perceptual, cognitive and social skills.16–18 Gamification of task simulation training could thus improve intrinsic and extrinsic motivation and user engagement.
Our hospital has a long tradition and culture for simulation training.8,19,20 Other hospitals without a similar simulation culture might experience institutional challenges introducing simulation training.21,22 Thus, we are currently testing our simulation training curriculum in a nation-wide study in Norway. Experience from this large collaboration will provide valuable information about broader implementation of simulation training in a busy routine hospital setting. By analyzing data from the national Norwegian EVT registry, we will also be able to acquire data about impact of this training on patient outcomes in clinically treated LVO stroke patients.
Our study has limitations. First, although almost the complete department for interventional neuroradiology participated in the study, the number of participants was small, and our study was a single-center pilot study. As such, the current results are not generalizable to other hospitals. Second, participation was voluntary and simulation is a longstanding cornerstone of our hospital culture, which means that the attitude of the participants towards simulation training may have been more positive compared to the average physician. Because daily clinical routines had to be covered, not everyone in the department could participate in the simulation-training program. Third, we performed only one post-simulation assessment using two cases, immediately after the training period finished. Thus, we cannot analyze long-term effects of task training and effects on clinically treated patients. This study was a pilot study for a currently ongoing nation-wide simulation study in Norway, which will soon provide more evidence for some of the questions that could not be answered in the current study.
In conclusion, this simulation-based pilot study shows that VR simulation task training enhances the capability of INR to perform EVT in a VR simulation environment. INR generally have a positive attitude towards VR simulation training. Whether VR simulation-based task training for EVT also translates to enhanced clinical performance will be evaluated in the ongoing Norwegian simulation collaboration.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
