Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability

Abstract

Objective

In a previous randomized controlled trial, we found immersive virtual reality (VR) simulation to be effective for teaching procedural skills to medical students. We further investigated this interface’s usability and cognitive load.

Methods

This was a secondary analysis of data from a previous randomized controlled trial. Twenty-two medical students with no or limited experience with VR and chest tube insertion received training for chest tube insertion using a commercially available immersive VR simulation. Participants completed post-training surveys on usability (System Usability Scale, SUS, from 0-100) and cognitive load (Leppink’s scale, 11-point, 10 items). Three types of cognitive loads (intrinsic, extraneous, and germane) were evaluated. Modified Objective Structured Assessment of Technical Skills (OSATS, 5-point, 11 items) for technical skills in a mannequin simulation were assessed after VR training, and in knowledge scores before and after training were extracted to analyze their relationships with usability and cognitive load. Data are presented as median (interquartile range).

Results

Median scores (%) for the knowledge test were 46.7 (40.0-53.3) at baseline and 86.7 (80.0-90.3) after training. The OSATS score was 40.5 (35.5-49.3), and SUS was 82.5 (73.8-88.8, with significant correlation between these variables (r = 0.51, P = 0.04). The intrinsic, extrinsic, and germane cognitive loads were 3.7 (1.8-6.1), 0.15 (0-1.4), and 9.2 (6.0-10), respectively.

Conclusion

Cognitive load and usability of immersive VR simulation were reported to be excellent. Along with its effectiveness shown previously, VR simulation is a highly acceptable approach for teaching technical skills to medical students.

Background

Immersive virtual reality (VR) simulation is being used increasingly in medical and surgical education, offering interactive experiences in an immersive environment.^1-5 With recent advances in hardware and software, VR has become more accessible and affordable, providing flexible and autonomous learning opportunities with automated instructions and integrated real-time feedback.^1-5 Further, this new technology provides an environment for learners to safely explore procedural skills without compromising patient safety and is expected to enhance the acquisition and retention of technical skill and theoretical knowledge. Evidence for the effectiveness of VR simulation in medical and surgical education has been reported.^1-5 In a recently reported randomized controlled trial we found that VR was effective in teaching technical skills (chest tube insertion) to medical students. ³

Although recent research has demonstrated VR’s educational potential, user experience with this technology has not been sufficiently studied. A randomized controlled trial comparing immersive VR and screen-based VR demonstrated higher cognitive load and impaired performance among learners trained with immersive VR. ⁶ The cognitive load theory is an important consideration when developing an educational program delivered with new technology, given that learners need to learn how to use these new technologies in addition to the content for which they are being used.^7,8 The main premise of this theory is that the capacity of our working memory and information processing for long-term memory is limited.^7-11 Because of the additional mental burden required to learn how to use the technology, it is important to assess and manage the learner’s cognitive load in order to not exceed the individual learner’s cognitive capacities. The theory describes three different types of cognitive load: intrinsic, extraneous, and germane. The intrinsic load is based on the learning task (inherent to the complexity of the task). The extraneous load is determined by how the teaching is delivered, how the learning task is presented, or the learning situation (poor teaching and instructions, and distractions in learning environment lead to higher extraneous load). The germane load refers to the learning process and the retention (the effort required to process new information into long-term memory by developing mental schema). In the medical education context, intrinsic load should be managed by adjusting the curriculum to learners’ pre-existing knowledge and skills, while extraneous load should be reduced by incorporating user-friendly technologies and avoiding irrelevant elements and complex instructions to foster generative processing.^7,8

In addition to cognitive load, usability is an essential concept to evaluate the integration and ease of use of technology as a learning tool. ¹² Usability problems can be a barrier in successful implementation of VR in medical education. ¹³ It is postulated that the concepts of usability and cognitive load theory are interrelated, wherein poor usability can be a significant contributor to higher levels of extraneous cognitive load. ⁷

In a national survey of simulation centres, it was found that lack of evidence for its utility is one of the major barriers to implementation of VR in medical education. ¹⁴ Such evidence should be multifaceted, including the effectiveness and the efficiency in learning, and the user experience. The objective of this study is to address these gaps by investigating user experience in VR applications for technical skill learning, particularly with respect to usability and cognitive load.

Methods

Experimental Design

A previously reported randomized controlled study ³ was conducted in July, 2023 (IRB approval A03-E17-23B), involving 30 medical students with no or limited experience with VR and no clinical experience with chest tube insertion. In that study, knowledge and performance were compared between students trained with online learning alone vs those with online learning + VR training. Subjects trained with VR simulation + online learning showed greater technical skill when assessed in a mannequin-based simulation, compared to participants prepared only with the online module.^3,15 For the present study, we conducted further analysis of data based on students trained with VR simulation between July and September, 2023 (n = 22).

Participants completed a pre-training survey and multiple-choice knowledge assessment, then received access to the online learning module on chest tube insertion. They then received two training sessions with immersive VR simulation (Figure 2). The first session consisted of a tutorial to familiarize participants with use of the VR equipment and working in a virtual environment, followed by two repetitions of the immersive VR chest tube insertion simulation. The second session took place approximately a week after the first session with two additional trials of the VR simulation. After the fourth VR simulation, all participants were asked to perform chest tube insertion in a mannequin model (Figure 3), while being evaluated by a trained assessor. After this technical skill assessment, participants completed a post-training multiple-choice knowledge assessment and a survey on the user experience to evaluate the usability and cognitive load of using the VR simulator. In addition, participants were asked to provide suggestions for improvement in VR simulation through an open-ended question. Flow of the intervention and evaluation is shown in Figure 1.

Figure 1.

Flowchart of the study. VR, virtual reality.

VR Simulation

Immersive VR simulation was provided using the chest tube module of a commercially available product (Vantari VR, Sydney, Australia) delivered through a Quest 2 VR Headset (Meta, Menlo Park, CA, United States) and its hand controllers.^1,16,17 The system situates a learner in a fully immersive virtual operating room environment, and learners receive detailed verbal instructions for 26 steps to execute the procedure while the steps are highlighted on the screen ³ (Figure 2). The module also provides real-time feedback, based on its embedded metrics and data, and reports time to complete each step and total procedure. The study team had verified that the instructions in the VR simulation were aligned with the Advanced Trauma Life Support guidelines. ¹⁸

Figure 2.

Screenshots during the VR simulation. (A): A trainee is wearing a headset and controllers for VR simulation. (B): 26 steps shown on a board during the immersive VR simulation. (C): A step to identify to locate the site of chest tube insertion. VR, virtual reality.

Assessment and Outcomes

The usability of the VR simulation was evaluated using the System Usability Scale (SUS) ¹⁹ (Supplemental material 1). The SUS is composed of 10 questions asking the degree of agreement to five positive statements and five negative statements, using 5-point Likert scale, where 1 represents “strongly disagree” and 5 “strongly agree.” To calculate overall SUS score, rates from positive statements are transformed as “[Score]-1” and ones from negative statements are transformed as “5-[Score]”. The sum of each transformed score is multiplied by 2.5, so that scores ranged from 0-100, where higher scores indicate better usability; a score greater than 68 is considered to be above average. ¹⁹

Cognitive load theory describes three types of cognitive load: intrinsic, extraneous, and germane cognitive load. Intrinsic cognitive load is related to the inherent difficulty of the learning subject, whereas extraneous cognitive load is impacted by the presentation of the information and is influenced by instructional procedures and technology. Germane cognitive load refers to the construction and storage of information. Although intrinsic cognitive load is generally expected to be unaffected by instructional technology, teaching technologies with smaller extrinsic cognitive load (smaller number in this scale), and greater germane cognitive load (greater number in this scale) are considered optimal for learning.^8-10 Cognitive load was evaluated using a 10-item questionnaire (three items for intrinsic, three items for extraneous, and four items for germane cognitive load) based on a 0-10 scale, where 0 represents “not at all the case” and 10 is considered “completely the case” (Leppink’s scale) ²⁰ (Supplemental material 2). The mean scores for each domain were calculated and used for the correlation analysis with other variables.

The time taken during each VR iteration and total VR times were measured. Participants’ knowledge about chest tube insertion was assessed by a 15-item multiple-choice assessment conducted pre- and post-training.^3,15 Technical skills were assessed by an experienced surgeon using a modified Objective Structured Assessment of Technical Skills (OSATS) score (5-point, 11 items) (Supplemental material 3) ³ as participants performed chest tube insertion on a mannequin (Figure 3). In this secondary analysis, we analyzed technical skill (OSATS scores) and knowledge as they relate to usability and cognitive load.

Figure 3.

Mannequin-based simulation. (A): Participant preparing to perform a simulated chest tube insertion on a mannequin, prior to sterile draping of the surgical field. (B): Participant inserting a chest tube to the mannequin during the simulation.

Statistics

Quantitative data analysis was used to examine scores from knowledge tests, technical skills, usability and cognitive load. Numerical data are reported as median and interquartile range (IQR). Wilcoxon signed rank test was used to compare time to complete VR trials. Spearman rank correlation was used to describe the relationships between variables. Answers for open-ended questions were categorized based on our codebook, and the frequency of each category was reported.

Results

Participants required a median total of 36.5 (IQR 30.0-39.7) minutes to complete four trials of chest tube insertion in VR simulation. The median time per trial progressively improved from the 1st through the 4th iteration, from 14.7 to 5.3 minutes (P < 0.001) (Figure 4).

Figure 4.

Learning curve: time to complete 4 VR simulations. Median score significantly improved over the course of study. Box plot shows median score, inter-quartile range; whiskers show range. VR, virtual reality.

Usability of VR simulation was reported to be excellent with the SUS being 82.5 (73.8-88.8) out of 100. The medians of intrinsic, extraneous, and germane cognitive load were 3.7 (1.8-6.1), 0.2 (0-1.4), and 9.2 (6.0-10.0) out of 10, respectively.

The percentage of correct answers in the knowledge assessments increased significantly from 46.7% (40.0-53.3) pre-training to 86.7% (80.0-90.3) post-training, (P < 0.001). The OSATS score for technical skills in the post-training mannequin-based simulation was 40.5 (35.5-49.3) out of 55. There was a significant positive correlation between OSATS and SUS scores (r = 0.51, P = 0.04) (Figure 5). There was a slightly positive correlation between germane cognitive load and post-training knowledge (r = 0.37), but this was not statistically significant (P = 0.11). There was no statistically significant correlation between extraneous cognitive load and usability (r = −0.15, P = 0.41).

Figure 5.

Regression analysis: Relationship between technical performance (chest tube OSATS score) and usability (SUS score). There was significant positive correlation between OSATS score and SUS (r = 0.51, P = 0.04). OSATS, objective structured assessment of technical skill; SUS, system usability scale.

In response to the open-ended question regarding areas for improvement in the immersive VR simulation, eight participants (36.4%) cited some discrepancy between the content taught in the online learning module and the VR simulation. Five participants (22.7%) suggested that some steps should be more fully modelled in the VR simulation, such as draping and suturing, which were automated in the module. Three participants (13.6%) suggested having a self- assessment mode without instruction and would like interactions with controllers to be more like the movements in actually conducting the procedure.

Discussion

Our previous randomized controlled trial showed the benefits of using immersive VR simulation for teaching technical skills to medical students in a blended learning framework. ³ VR training added further value in learning the technical skills associated with the procedure. Since many medical students were inexperienced with the use of immersive virtual reality, there was concern that adding this new technology may increase their cognitive burden, diminishing their capacity to actually learn. The present study shows that the usability of the VR technology used in this study was excellent and the cognitive load was not overwhelming. Participants were able to become familiarized with the technology easily and quickly, and became progressively more comfortable within a few simulation trials. There was a strong correlation between SUS score and technical skill, which suggests that the perceived usability of immersive VR may impact the efficiency and effectiveness of learning. In contrast, despite the ideal theoretical cognitive load (relatively low intrinsic cognitive load, low extraneous cognitive load, and high germane load), there was no correlation between cognitive load and other measures, although extraneous cognitive load and usability should theoretically be negatively correlated. ⁷ We assume the variability in extraneous and germane load reported by the participants was too little to show statistically significant correlations with usability and other learning outcomes. Reported intrinsic cognitive load was low considering that the participants were medical students with little or no prior training experience for chest tube insertion, however the use of online material^3,15 for preparation prior to the immersive VR simulation may have helped to lower the intrinsic cognitive load.

Regarding responses to the open-ended question, more than one third of participants cited discrepancies in the content between the online learning material and VR simulation. Before conducting these studies, a panel of experts reviewed the content of both modules and confirmed they are compatible with the current ATLS guidelines. ¹⁸ This result may highlight that the participants are naïve learners, and are very sensitive to subtle changes, such as variations of the names of instruments. Simulation for basic surgical manoeuvres like draping and suturing were not the main purpose of this learning, but these skills can be taught in new different immersive VR simulation modules if there is substantial demand. The participants’ suggestion of integrating an assessment mode (without real-time guidance) can be helpful feedback to the software developer. Currently, the module includes audio and visual instructions for the steps of the procedure at all times; an assessment mode without any instructions may be a useful capability of the immersive VR simulation for learners who would like to self-assess their performance in a competency-based learning framework.

This study has some limitations. The small sample size and the analysis of a single surgical procedure may limit the generalizability of the results from this study. In addition, the measurements of usability and cognitive load were subjective and representative of the overall learning experience; self-reported outcomes may have been influenced by individual biases. The use of some objective measurements, such as counting of errors in interactions for usability, ²¹ or physiologic monitoring, eg, heart rate, ²² and brain function^22,23 could provide real-time assessment and may be useful to identify the interface elements to be improved.

Conclusion

In conclusion, usability and cognitive load of immersive VR simulation are important to evaluate. This secondary analysis of using immerse VR simulation for training chest tube insertion, found there to be excellent usability and low demands on extrinsic cognitive load, which may have contributed to its effectiveness for technical skill learning. Along with the results from the previous study, there is emerging evidence that well-designed immersive VR simulation can be useful for procedural training for medical students.

Supplemental Material

Supplemental Material - Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability

Supplemental Material for Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability by Junko Tokuno, Ryan M. Knobovitch, Fabio Botelho, Howard B. Fried Tamara E. Carver, Gerald M. Fried Ramani in Surgical Innovation.

Supplemental Material

Supplemental Material - Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability

Supplemental Material for Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability by Junko Tokuno, Ryan M. Knobovitch, Fabio Botelho, Howard B. Fried Tamara E. Carver, Gerald M. Fried Ramani in Surgical Innovation.

Supplemental Material

Supplemental Material - Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability

Supplemental Material for Immersive Virtual Reality Simulation for Medical Student Procedural Training: Assessment of Cognitive Load and Usability by Junko Tokuno, Ryan M. Knobovitch, Fabio Botelho, Howard B. Fried Tamara E. Carver, Gerald M. Fried Ramani in Surgical Innovation.