Bringing Radiology Education to a New Reality: A Pilot Study of Using Virtual Reality as a Remote Educational Tool

Abstract

French

Purpose: We investigated virtual reality (VR) during a 2-week, undergraduate, radiology elective to determine if it improved learning outcomes and user satisfaction. Methods: Eighteen students enrolled between August 2021 and February 2022. Each student had a collaborative Zoom teaching session with a preceptor using a Picture Archive and Communications System (PACS)-like viewing system Online DICOM Image Navigator (ODIN), followed by a teaching session using a VR, Digital Imaging and Communications in Medicine (DICOM) viewer (SieVRt). After each teaching session, the students independently reviewed 8 imaging cases and completed case related questions. The students completed a survey, rating their subjective experiences using ODIN and SieVRt. Results: There was no difference in total test scores between the two learning strategies. However, students did perform statistically better on two of five questions designed to test the detection/measurement capabilities of SieVRt vs ODIN. Students stated that they preferred using SieVRt over ODIN and agreed that they were able to view subtle imaging findings and abnormalities better using SieVRt. However, students found that some of the functions of SieVRt (measuring angles/lengths, and multitasking) were difficult. There were technical challenges with VR and minor undesirable physical effects (dizziness, nausea, etc.). Conclusions: Virtual reality has the potential to enhance radiology education by providing an immersive and engaging experience. Objectively, students were able to perform two tasks better with SieVRt. Subjectively, the VR platform received favourable reviews from students for a variety of features. There were reported technical and physical challenges related to using VR. Future developments in VR systems should focus on improving the user experience.

Visual Abstract

Keywords

virtual reality immersive learning radiology education remote learning medical education

Introduction

Given the availability and utilization of medical imaging, it has become pivotal in the clinical decision-making process. Therefore, it is crucial for medical students to gain a foundational understanding of basic radiologic content and concepts. One of the primary methods in which medical students strive to enhance their radiology knowledge is through the pursuit of a dedicated radiology elective. In many radiology electives, students often assume passive roles without structured goals and objectives.^1,2 In addition, there are also many challenges with integrating radiology into the clerkship experience. These challenges were made more difficult by the COVID-19 pandemic, which made in-person learning challenging given the social distancing measures. Many strategies to improve student engagement during radiology electives have been proposed. These endeavors included the implementation of a structured radiology elective with clearly outlined learning objectives, integration of a flipped classroom approach with pre/post tests and web-based tutorials, and participation in clinical radiology conferences.^2-11

Virtual reality (VR) can provide a learning platform that creates a first-person, interactive experience. This has been used in interventional radiology to enhance educational and clinical outcomes. For instance, Johnson et al showed that participants who used a simulator (consisting of a mannequin with visual input using virtual 3D glasses) to practice the Seldinger Technique performed better in both achieving key components of the task and in real-life scenarios.¹² Similar improvements in performance with the use of simulator training have been demonstrated in various vascular access procedures.^13-15

There are 3 levels of VR: (1) non-immersive VR; (2) semi-immersive VR; and (3) fully immersive VR. Non-immersive VR involves interacting with a device screen to control a virtual character or activities. This approach is used in online games such as “Second Life,” which allowed users to interact with a computer-generated virtual space using personalized avatars.^16,17 In semi-immersive VR, the user is partially but not fully immersed in VR. This is exemplified by the use of virtual dissection tables, which provide interactive 3D renderings of human anatomy on a large monitor.^18-20 Another immersive technique that has been utilized in radiology education involves the use of augmented reality such as a “Magic Mirror” system, which provides in-situ visualization of anatomy superimposed on the user’s body.^21-23 In fully immersive VR, users can be immersed in a 360°, simulated, digital, environment by wearing head-mounted displays. In this virtual environment, users can manipulate certain aspects of the simulated environment using handheld haptic controllers.²⁴

Previous studies have shown that students who are fully immersed using head-mounted displays performed better in knowledge acquisition compared to students who used a partially immersed VR environment using a computer screen.²⁵ However, literature on the use of fully immersive VR display in radiology education is lacking. Maresky et al, examined using a personal VR headset (Oculus Rift) to teach cardiac anatomy. In their study, a group of subjects were given 25 minutes of independent learning time using a VR system that displayed Computed Tomographic (CT) and Magnetic Resonance (MR) images of cardiac anatomy, while a control group of subjects spent the same amount of time in the cadaveric laboratory reviewing cardiac anatomy. Subsequent testing revealed that the VR group performed better on cardiac anatomic content and had enhanced visuospatial skills.²⁶ However, the authors of this study were not engaged in assessing imaging content and concepts but were simply using CT and MR images as a vector to teach cardiac anatomy.

Hence, VR was identified as area of potential technology to engage learners in an interactive learning experience given the circumstances related to COVID-19. We sought to determine if a personal headset mediated, VR platform, that displayed radiology images, in a reading room format, could enhance radiology knowledge acquisition in a manner that was satisfactory for the learners.

Materials and Methods

We used personal VR headsets to provide socially distanced learning during the COVID-19 pandemic. In this virtual environment, students and preceptors could engage in real-time, collaborative, interactions using both the audio and video capabilities of the system. This allowed students and preceptors to work through imaging cases without having to be in the same physical environment. To our knowledge, this is the first study assessing the use of VR headsets to provide both didactic radiology teaching and independent student learning using anonymized, clinical, radiology cases. This project was a prospective pilot study, designed to investigate radiology knowledge acquisition and student perceptions while using a personal VR headset to provide clinically based radiology education.

SieVRt (Luxsonic Technologies, Saskatoon, SK), a virtual radiology reading room displaying radiology images in the Digital Imaging and Communications in Medicine format (DICOM) was used to facilitate the delivery of radiology teaching in a VR environment. SieVRt is a server-based, virtual, environment that allows users to select imaging cases from an image repository and display them for viewing based upon a four monitor, clinical, Picture Archive and Communications System (PACS) workstation configuration. Users were granted specific login credentials to access this subscription-based teaching and learning platform designed and hosted by Luxsonic Technologies.

Image Display Technologies

Digital imaging and communications in medicine image repository radiology society of North America-medical imaging resource center (DICOM RSNA-MIRC)

The anonymized, clinical, radiology images used by the Online DICOM Image Navigator (ODIN) viewer and the SieVRt VR system were housed on a server hosting the Radiology Society of North America (RSNA), Medical Imaging Resource Center (MIRC) software. This server was formatted and hosted by the University of Saskatchewan. This version of the server was freeware and it can be accessed by a public weblink. Cases in the repository can only be created, or edited, by users with individual usernames and passwords. All the cases in this repository were created by local radiologists. The images were derived from a clinical PACS. The images stored on the MIRC server were two-pass anonymized to remove all pertinent personal and system related information. The processes and standards followed to create this image repository met Healthcare Insurance Portability and Accountability Act (HIPAA) standards. The methodology and the parameters deployed for image anonymization were detailed previously by Burbridge.²⁷

Online DICOM Image Navigator (ODIN)

The ODIN is a locally hosted, server-based software, from the University of Saskatchewan. Images are selected using an integrated search tool or using a unique dedicated case weblink. The database provides clinically based, HIPAA compliant, anonymized, DICOM, images in a manner that simulates a simple PACS. Online DICOM Image Navigator is freeware that allows any user with internet access to search for and review cases from the MIRC image repository. Features of this platform allowed users to display single or stacked images, alter the brightness and contrast (window width and level) of the displayed images, zoom and pan, perform measurements (angles, lengths, area), as well as annotate, flip, rotate images, and display images in multiple windows. Figure 1 shows an illustration of the ODIN platform. A full discussion of the origin and deployment of this platform was previously published by Burbridge et al.²⁸

Figure 1.

ODIN is a web-based image software designed to simulate PACS and allow users to manipulate and interact with imaging cases.

Virtual reality platform (SieVRt)

SieVRt is an all-in-one, web-based server software that provides a VR platform simulating a personalized, 3D, radiology, reading room. Access to SieVRt software is based upon an annual license fee. Users were provided a unique username and password. The images displayed in SieVRt were derived from cases in the MIRC image repository. A VR headset allowed students to visualize and interact with images in an immersive, virtual radiology reading room.

For the purposes of this study, 2 Omen laptops (Hewlett Packard, Palo Alto, United States) were deployed, one for the student and one for the preceptor. Each laptop was equipped with Intel (R) Core (TM) i7-10750H processor (CPU @ 2.60 GHz), 16 GB RAM, and 512 GB SSD internal storage. Each laptop was connected to an HP Reverb G2 headset (Hewlett Packard, Palo Alto, California) headset and two handheld hand controllers. The specifications for the HP Reverb G2 headset are as follows:

• Optics: Fresnel Lenses

• Display Type: 2 × LCD

• Sub Pixel Layout: RBG Strip

• Resolution: 2160 × 2160 per eye

• Avg Pixel Density: 23.58 PPD horizontal and 23.77 PPD vertical

• Refresh Rate: 90 Hz

• Field of View: 107 Degree Diagonal

• Weight: 498 g including head strap

Each workstation costs $ 3200 CAD ($2200 CAD for each Omen laptop and $1000 CAD for VR headset and controllers). The two annual educational licenses for SieVRT were provided to us at a 50% discount (original price: $600 CAD/license; discounted price: $300 CAD/license).

Figure 2 shows the laptops and VR headsets used to facilitate the VR platform.

Figure 2.

The hardware used to facilitate each virtual reality platform includes (1) Omen laptop, (2) HP Reverb G2 headset; and (3) two handheld VR controllers.

The student could work independently in the VR environment, but the system could also create real-time connectivity between a preceptor and a learner within the virtual environment. This facilitated the sharing of both live audio and video to enhance the learning experience. SieVRt can also host multi-user collaborative learning sessions but this study was limited to two users due to the limited number of personal headsets available.

SieVRt provides an integrated suite of digital tools that participants can utilize to view and interact with the DICOM images in a simulated PACS environment. Functional features of this virtual platform allowed users to display single images or stacks of sequential images, alter the brightness and contrast (window width and level) of the displayed images, zoom and pan images, perform various measurements (angles, lengths, area, and pixel densities), and rotate images. Figure 3 demonstrates the view of the virtual reading room created using SieVRt.

Figure 3.

Wearing VR headsets and using handheld controllers, users can scroll through and manipulate the image cases on the four virtual monitors in a personal reading room.

Study Design

This project was funded by a research award from the Canadian Radiological Foundation/Canadian Heads of Academic Radiology (CRF/CHAR).

This project received University of Saskatchewan Behavioral Research Ethics Board approval (BREB – 2299). All students who participated in a 2-weeks radiology elective with our Radiology department between August 2021 and February 2022 were invited to participate in the study. All students voluntarily participated in the VR learning experience.

Participants were final year medical students from the local medical school. None of the students had undertaken a previous radiology elective. All participants underwent a 2-week radiology elective, which included WebEx (Cisco, San Jose, United States) and Zoom (Zoom Video Communications, San Jose, United States) sessions for shadowing/observation sessions with radiologists, attending resident academic half-days, independent study time, and engaging with an online learning platform: Canvas (Instructure, Salt Lake City, United States).

In the last 2 days of their elective experience, students participated in both ODIN and SieVRt teaching/imaging case review sessions. For the ODIN teaching sessions, each student received individualized, interactive teaching for five imaging cases while on a Zoom call with either a PGY-3 radiology resident or a certified radiologist (Y.W. or B.B.). These sessions consisted of a discussion about radiology teaching cases, describing the salient imaging findings, and relevant imaging anatomy. The sessions also included a tutorial on how to operate the various functions of ODIN.

Students were then asked to work through eight cases independently using the ODIN platform. An online quiz accompanied the review of these eight cases and students independently provided their responses without coaching or preceptor involvement. The students were not apprised of the correct answers to the eight quiz cases.

The same students were then scheduled for the SieVRt teaching/imaging case review session the following day. The learner collaboration mode of the SieVRt platform allowed the preceptor and student to engage with one another in the VR environment with synchronization of both the video and audio features of the software. This session again consisted of reviewing five radiology teaching cases with either a PGY-3 resident or a staff radiologist (Y.W. or B.B.), as well as instruction on how to navigate and operate the various features of SieVRt. Subsequently, the students proceeded to independently work through the same eight quiz cases they had seen using the ODIN viewer, using the SieVRt platform. An online quiz accompanied the review of these eight cases and students independently provided their responses with no assistance or input from the preceptor. Again, the students were not apprised of the correct answers to these quiz questions.

To ensure that students have similar learning experiences between ODIN and VR, the same preceptor provided the two teaching sessions to the students. The preceptors used a didactic teaching format with a list of pre-planned discussion points on relevant clinical information, imaging findings and differential diagnosis. For consistency, each teaching session lasted for 30–40 minutes, and students were welcomed to ask any questions they had during the discussion. For preceptor time commitment, this required approximately 30–40 minutes for the ODIN session and 40–50 minutes for the VR session (10 minute set up time and 30-40-minutes teaching time). None of the content provided in the teaching sessions was subsequently tested in the quiz cases. For example, we did not show an ankle fracture and then have a test case about the same ankle fracture. The purpose of the teaching session was to simulate a didactic learning session and to introduce the unique functions available in the two different platforms in order for the students to be able to interact with the test cases optimally and complete the testing successfully.

The list of the eight cases and quiz questions utilized are listed in the Table 1. Student performance on the eight cases was assessed using a standardized answer key. There were three different question types provided to the students: (a) questions that tested general imaging knowledge; (b) the diagnosis of imaging pathologies (ie, the presence or absence of a specific diagnosis, such as a bone fracture); and (c) five questions that were designed to detect objective differences in image analysis in an attempt to determine if there were differences between the two imaging platforms in regard to detection and measurement capabilities. These questions assessed participant acumen in performing more quantifiable aspects of image interpretation, such as determining the number and size of visible nodules, as well as providing angle and length measurements. These are highlighted in bold in Table 1:

1. How many nodules are present in the right lower lobe of the lung?

2. Measure the traverse diameter of the largest pulmonary nodule (mm).

3. Are there lung nodules in the other lobes?

4. How many appendicoliths are present?

5. Measure the angle of the elbow joint (the angle of humeral/radial articulation).

Table 1.

List of Case Diagnoses and Quiz Questions, with Type c Questions Highlighted in Bold.

Cases	Questions
Case 1: Epidural hematoma	What is the most likely diagnosis?	Is there a fracture?	Is the middle meningeal artery affected by the fracture?	Is there subfalcine herniation?
Case 2: Pulmonary nodules	What is the most likely diagnosis?	How many nodules do you see in the RLL?	Measure the transverse diameter of the largest nodule (mm).	Are there lung nodules present in other lobes?
Case 3: Acute appendicitis	What is the most likely diagnosis?	Are there any inflammatory changes?	Is there free fluid present?	How many appendicoliths are present?
Case 4: Acute on chronic subdural hematoma	What is the most likely diagnosis?	Is there acute hemorrhage?	Is there chronic hemorrhage?	Is there compression of the ipsilateral lateral ventricle?
Case 5: Scaphoid fracture	What is the most likely diagnosis?	Is there a fracture of the radius?	Is the ulnar styloid growth plate fused?	Is the patient a child or teenager?
Case 6: Joint effusion	What is the most likely diagnosis?	Do you see a fracture?	Do you see a sail sign?	Measure the angle of the elbow joint (the angle of the humeral/radial articulation).
Case 7: Hangman fracture	What is the most likely diagnosis?	What anatomic location of the bone is fractured?	Is there pre-vertebral soft tissue?	Is there a cervical collar present?
Case 8: Pneumoperitoneum	What is the most likely diagnosis?	Do you see a Rigler sign?	Was the chest X-ray taken upright or supine?	Do you see consolidation in the lung bases?

Assessment of Student Perceptions

Students also completed an exit survey at the end of their Elective where they rated: (1) their general perceptions of using computer technology for teaching; (2) SieVRt vs ODIN for didactic teaching; and (3) the functionalities of SieVRt vs ODIN, using a seven-point Likert scale (1 = Strongly Disagree; 2 = Somewhat Disagree; 3 = Disagree; 4 = Neutral; 5 = Somewhat Agree; 6 = Agree; 7 = Strongly Agree).

Statistical Analysis

All data were analyzed using Microsoft Excel 2016 (Microsoft Corp, Redmond, United States) and GraphPad Prism 5 (GraphPad Software, San Diego, United States). Quantitative data were analyzed using unpaired and paired t tests. Students’ responses on a Likert scale were summarized using a median score and range. Categorical data were described as frequency and percentage. Student performance and survey responses were compared between ODIN vs VR using the Wilcoxson Rank Sum Test.

Results

Student Demographics

The 19 students who completed the radiology elective were final year medical students from the local College of Medicine. Nineteen students were invited to participate in the study: 18 students accepted the invitation to participate. There were 10 males and 8 females. One student declined participation in the study due to a pre-existing diagnosis of vertigo. Only one student in our cohort has used the VR before. The other students (n = 17) had no previous exposure to VR.

Student Performance

There was a total of 32 questions (8 imaging cases—4 questions per case) administered during each of the assessment sessions. Students were expected to answer all questions to the best of their ability.

There was no statistically significant difference in the pooled, group, mean scores for the students when testing general concepts using ODIN vs SieVRt (ODIN: 79.2 ± 1.9% vs SieVRt: 82.0 ± 1.3%; P = .22).

For the questions which assessed the objective performance of the image display (type c questions), the students performed better using SieVRt on two of the five questions and equivalent to ODIN for the remaining three questions. These results are summarized in Table 2.

Table 2.

Comparison of Student Performance on Questions that Assess Objective Performance of the Image Display, SieVRt vs ODIN.

Questions	SieVRt vs ODIN
How many nodules are present in the right lower lobe of the lung?	Not statistically different
Measure the traverse diameter of the largest pulmonary nodule (mm).	SieVRt was more accurate than ODIN (P = .001)
Are there lung nodules in the other lobes?	Not statistically different
How many appendicoliths are present?	SieVRt was more accurate than ODIN (P = .04)
Measure the angle of the elbow joint (the angle of humeral/radial articulation).	Not statistically different

Student Perceptions of the Elective Experience

Perceptions of the use of technology in learning

Most students stated that they enjoyed using technology for learning (16.7% somewhat agree; 38.9% agree; 44.4% strongly agree). They were confident when using technology for learning (5.6% neutral; 22.2% somewhat agree; 38.9% agree; 27.8% strongly agree). Students felt that VR will become a standard teaching tool (38.9% somewhat agree; 33.3% agree; 11.1% strongly agree). Most students also believed that VR will be useful for learning radiology (33.3% somewhat agree; 38.9% agree; 16.7% strongly agree). Student responses on the use of technology in learning are provided in Figure 4.

Figure 4.

Students’ general attitude toward the use of the technology in radiology education. Statements were ranked on a seven-point Likert scale (1 = Strongly Disagree; 2 = Somewhat Disagree; 3 = Disagree; 4 = Neutral; 5 = Somewhat Agree; 6 = Agree; and 7 = Strongly Agree).

Online DICOM image navigator vs SieVRt for didactic teaching/case quiz

Compared to ODIN, students reported higher satisfaction scores when using SieVRt during didactic teaching and independent case review sessions. Students reported that they were better able identify the image findings on the SieVRt (P = .047), and that SieVRt allowed them to concentrate and eliminate distractions (P = .006). Overall, they preferred using SieVRt over ODIN for learning purposes (P = .003) and were more likely to recommend the use of SieVRt in radiology education (P = .008).

For both of the tested modalities, students stated that they were able to interact with the preceptor in a seamless fashion (ODIN: 5.67 ± 1.13; SieVRt: 5.94 ± .76). Both SieVRt and ODIN kept students engaged in learning radiology (ODIN: 5.33 ± .97; SieVRt: 5.55 ± 1.10). In both of these areas, SieVRt showed a modest but not statistically significant advantage over oODIN (P values of .546 and .624, respectively).

These results are shown in Table 3, with statements that resulted in statistically significant difference between the two modalities in bold.

Table 3.

Student Responses on Using ODIN vs SieVRt for Didactic Teaching/Case Review.

	ODIN (Mean Rating ± Standard Deviation)	SieVRt (Mean Rating ± Standard Deviation)	P-Value
I was able to see the image findings.	5.72 ± .57	6.22 ± .73	.047
I was able to interact with the lecturer in a seamless fashion.	5.67 ± 1.13	5.94 ± .76	.546
[SieVRt/ODIN] kept me engaged in learning radiology.	5.33 ± .97	5.55 ± 1.10	.624
(SieVRt/ODIN) allowed me to concentrate better and eliminated distractions from the surrounding environment.	4.28 ± 1.36	5.78 ± 1.22	.006
I preferred using (SieVRt/ODIN) over (the other modality) for learning purposes.	3.61 ± 1.54	5.56 ± 1.04	.003
I would recommend the use of (SieVRt/ODIN) in radiology education.	4.72 ± 1.07	5.67 ± 1.03	.008

Functional Assessment of SieVRt vs ODIN

Students reported that they were able to see the images more clearly in SieVRt, compared to ODIN (P = .037).

For both SieVRt and ODIN, students stated that the platform was relatively easy to log into (ODIN: 6.17 ± .62; VR: 5.72 ± 1.02) and easy to use (ODIN: 5.72 ± 1.13; SieVRt: 5.83 ± .85). They were able to scroll through the images smoothly (ODIN: 6.00 ± .77; SieVRt: 5.56 ± 1.46) and adjust the brightness/contrast of the images easily (ODIN: 5.56 ± 1.46; SieVRt: 5.00 ± 1.57).

Students reported a modest but not statistically significant advantage in using ODIN over SieVRt to measure angles (ODIN: 5.89 ± 1.18; VR: 5.00 ± 1.57; P = .09). Students found it technically easier to measure the lengths of objects using ODIN (P = .006). A noted limitation of SieVRt was the inability to multitask (ie, access the computer desktop, interact with other software, and connect to the internet), with students reporting better functionality for ODIN in this category (P < .001). These results are summarized in Table 4.

Table 4.

Student Responses on the Functionalities of ODIN vs SieVRt.

	ODIN (Mean Rating ± Standard Deviation)	SieVRt (Mean Rating ± Standard Deviation)	P-Value
SieVRt/ODIN was easy to log into.	6.17 ± .62	5.72 ± 1.02	.138
The SieVRt/ODIN platform was easy to use.	5.72 ± 1.13	5.83 ± .85	.805
I was able to scroll through the images smoothly.	6.00 ± .77	5.56 ± 1.46	.848
I was able to adjust the brightness/contrast of the images easily.	5.56 ± 1.46	5.00 ± 1.57	.242
I found it easy to measure angles.	5.89 ± 1.18	5.00 ± 1.57	.090
I found it easy to measure the lengths of objects.	6.11 ± .96	4.61 ± 1.75	.006
Compared to (the other modality), I could see the images more clearly.	4.38 ± 1.29	5.61 ± 1.50	.037
I was able to multitask with my computer desktop while using SieVRt/ODIN.	6.05 ± .64	3.39 ± 1.75	<.001

Statements were ranked on a seven-point Likert scale (1 = Strongly Disagree; 2 = Somewhat Disagree; 3 = Disagree; 4 = Neutral; 5 = Somewhat Agree; 6 = Agree; 7 = Strongly Agree). Statements that resulted in statistically significant difference between SieVRt vs ODIN are outlined in bold text.

Adverse Physical Effects

Seven out of 18 students (38.9%) experienced minor adverse physical effects when using SieVRt. Some students reported more than one minor adverse effect. The two most common minor adverse effects were headache (n = 5) and feeling too warm while using the headset (n = 3). Other reported effects include dizziness (n = 2), nausea (n = 2), and visual disturbances (n = 2).

Other Considerations

Adapting to this new modality of teaching presented some challenges, especially in the initial stages of the project. Initially, it was difficult to establish an audio/video connection with the student during the collaborative SieVRt teaching sessions. This issue was caused by the firewall settings of the hospital’s Wi-Fi connection and was resolved by using our post-secondary institution’s Wi-Fi connection.

In addition, there were many wires and multiple connections involved in setting up the VR headsets and the laptops. A typical session would require 10–15 minutes of setup to ensure that the cables/wires were connected, and that the hardware is working appropriately, prior to the student arriving. Replacement of these wired headsets with wireless headsets will minimize set-up time and allow for a more efficient and less constrained user experience.

Discussion

In the past, radiology electives have been primarily observational experiences with medical students shadowing radiologists and residents. These electives are associated with limited clinical responsibilities and few opportunities for active engagement during the elective experience. Several studies have shown that providing students with an experience that simulates a radiologist’s workflow improved learning outcomes and learner satisfaction. Friedman et al and Strickman et al showed that students who reviewed anonymized cases on an interactive DICOM server reflected more positively on their elective experiences and performed better on musculoskeletal (MSK)-based competency exams, compared to students in the observational role.^4,29

In our project, we incorporated VR as the platform for hosting imaging cases, which allowed for a more immersive, interactive experience that eliminated distractions from the surrounding environment. To our knowledge, this is the first study examining the use of VR in clinical radiology education. Students had a favorable perception of VR for radiology teaching. They viewed VR as a potential educational tool that could be useful for learning radiology. Given the lack of literature on the use VR for undergraduate radiology education, these results provide significant insights into the student’s perspectives.

Given that our study was conducted during the COVID-19 Pandemic, it was imperative to facilitate socially distanced learning. We made use of the collaborative function conferred by the VR system, which allowed preceptors to communicate with students using real-time audio and video functionality in the VR workspace without having to be in the same physical space. This novel ability to converse and interact in the VR software allowed for one-on-one teaching with the students. The SieVRt system is also capable of providing collaborative teaching sessions to groups of students, but we did not explore this as there were only two headsets for the purposes of this project. This presents an opportunity for larger group and long-distance learning without requiring the preceptor and learner(s) to be in the same physical space, which may be utilized in the broader context to host remote teaching sessions and workshops. This includes training residents, fellows, and radiologists in various diagnostic skills such as CT colonography and lung cancer screening and to offer CME-based educational activities.

Interestingly, there were no significant differences in the overall student test scores between ODIN and SieVRt. However, students reported many benefits and advantages when using VR as a learning platform. Students reported that they were better able to see the image findings. They stated that VR allowed them to concentrate more effectively and eliminate distractions from the surrounding environment. They preferred using VR over ODIN (a traditional online PACS-like viewer) for learning purposes. Overall, students stated that they would recommend the use of VR in radiology education. This is supported by previous literature which demonstrates that the use of VR improves learner engagement and outcomes.^26,30,31 This is particularly valuable in the context of undergraduate radiology electives to stimulate student interest and motivation in learning radiology. In addition, the use of VR headsets allowed students to have their own high-resolution “monitors,” thereby eliminating the need for them to look at the clinical images “over the shoulder” of an interpreting radiologist. Similarly, Stepan et al. found that although students using VR did not show a significant improvement in neuroanatomy knowledge acquisition, they found the novel modality more engaging and thus were more motivated to learn.³²

Despite a lack of statistical difference in overall test scores, when we separated the questions designed to assess objective detection and measurement capabilities, we found that students statistically performed better on two of these five questions when using SieVRt. To our knowledge, our study is the first to examine the ability of VR to detect subtle radiological findings. Given that VR improved the ability of participants to visualize three-dimensional anatomy in the surgical literature,^30,33-35 it is not surprising that VR may confer enhanced detection and measurement capabilities in the radiology setting.

There were several drawbacks to the VR system that we tested. One of the limitations with the VR software was the difficulty in multitasking (ie, opening web browsers while in SieVRt). It would be useful to have the ability to browse the internet while in the virtual environment to allow students to research relevant imaging findings/pathologies. We are currently discussing options with the vendor to incorporate a web browser function within SieVRt that would allow students to access certain webpages (such as Radiopedia, STATDx, etc.), which may offer a solution. Another limitation was the number of wired connections required to set up the VR devices, which increased the set-up time and limited physical movement for the users. Wireless headsets would eliminate these issues.

Some students reported that they found it difficult to measure lengths of objects in SieVRt. This is likely due to the functionality of the VR software and limitation of the handheld controllers used. It was found to be difficult to stabilize the VR hand controllers when performing a measurement. Despite this, the students still performed better on the question that required them to measure the transverse diameter of the largest lung nodule using SieVRt, perhaps due to the better image resolution afforded by the VR headset and the ability to concentrate on the task at hand. This further supports the premise that the use of VR may bolster detection and measurement capabilities, particularly if vendors are able to improve the user experience related to the hand controllers.

In our study, 38.9% of the participants experienced minor adverse effects while in the VR environment including, headache, dizziness, and feeling too warm within the headset. Unfortunately, one student declined to participate in the study due to previously diagnosed vertigo. It was felt that her symptoms may be exacerbated by using a VR headset. Adverse effects while in the VR environment are commonly reported in the literature, particularly with the use of full-immersion VR headsets.^31,36 Similarly, Moro e al found that students who used VR to “dissect” a virtual cadaver reported more adverse effects compared to those who performed the same task using augmented reality or a three-dimensional model on a tablet. In their study, 40% of students reported having general discomfort, 40% reported having dizziness, and 35% reported having blurred vision.³⁷ To mitigate the adverse effects in the future, we could suggest students have scheduled short breaks during the sessions to minimize any discomfort.

It is also possible that students may become distracted by the novelty of VR technology if they have never used it before. One student in Moro et al’s study commented that he/she was “very distracted by how cool the program [VR] is” and that he/she wasn’t focused on the learning aspect.³⁷ In our study, only 1 student in our cohort have used the VR before; the other students (n = 17) had no previous exposure to VR. Contrary to previous literature, we found that the students reported being able to concentrate better in VR compared to the traditional online PACS-like viewer.

Limitations

There were limitations of our study. Firstly, this was a single institution experience with a relatively small sample size (n = 18). We were constrained by the number of elective students and number of headsets available. Additionally, there is an inherent power imbalance between students and preceptors. Although we stated that student performance on project related tasks was independent of the elective evaluation process, students may have felt that their participation could potentially alter their final evaluation. Consequently, participants might have felt compelled to answer questions in a manner that they felt the researchers were seeking.

Our cohort consisted of fourth-year medical students who were pursuing a radiology elective and likely had more interest in medical imaging. Although all students who applied for a radiology elective was accepted, these students were likely to have a stronger background knowledge and may have used additional educational resources (Radiopedia, textbooks, imaging websites, E-Anatomy, etc.) to supplement their learning during their elective. In addition, students had the ODIN sessions before the VR sessions. Although we did not provide students with the correct answers to the cases, students may have gained familiarity with the cases or looked up the answers between the two sessions. This may have confounded our objective testing. However, participants’ overall performance was equivalent across both ODIN and SieVRt, supporting that additional knowledge acquisition did not favour one technology vs the other.

Students only had one single interaction with using VR and may not have had enough opportunities with the software to develop a noticeable improvement in the test scores. In addition, the two preceptors in this study were two of the coauthors of this manuscript. Given this small sample and potential bias, we did not seek formal feedback from the preceptors. Time commitment from faculty members may present as a possible challenge as radiologist may have to take time away from clinical duties to facilitate the teaching sessions. In the future, we plan on developing a clinical case database that students can work through on their own to maximize their exposure to VR and minimize time commitment from faculty preceptors. With more exposure to VR, students may demonstrate a statistically significant improvement in objective test scores.

In our study, we did not measure the cognitive load associated with fully immersive VR. Frederiksen et al showed that compared to using a conventional simulator, fully immersive VR increased cognitive load while performing simulated laparoscopic salpingectomies, as evidenced by increased reaction time to an auditory stimulus. This was hypothesized to be due to an environment with increased complexity and increased number of elements to interact with. However, the authors did acknowledge that although fully immersive VR resulted in a higher cognitive load, it simulated the real-world clinical scenarios with more realism.³⁸

Conclusion

This is the first study, to our knowledge, examining the use of immersive VR to facilitate clinical learning in radiology. Despite some shortcomings associated with this technology, students commented positively about their use of VR to learn radiology. Many students felt more engaged by this novel teaching modality and stated that they would recommend the use of VR in radiology education. Further developments in VR systems should aim to improve the user experience by attempting to address the technical and undesirable physical effects of this virtual environment.

Footnotes

Acknowledgments

We would like to acknowledge staff members from Luxsonic Technologies Inc. (Saskatoon, SK) for their technical troubleshooting and for providing Figure 3 in our manuscript. In particular, we would like to thank Mike Weslowski, PhD., and Arjun Puri, MSc., for their help throughout the project.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors declare that they have full access to all the data in this study and the authors take complete responsibility for the integrity of the data and the accuracy of the data analysis.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research project was funded by one of the 2021 Canadian Radiological Foundation (CRF)/Canadian Heads of Academic Radiology (CHAR) Research Awards.

ORCID iD

Brent Burbridge

References

Darras

Spouge

Kang

, et al. The challenge with clinical radiology electives: Student and faculty perspectives identify areas for improvement. Can Assoc Radiol J. 2019;70(4):337-343.

Naeger

Phelps

Kohi

Elicker

Ordovas

Webb

. Reading room electives: Say goodbye to the “radi-holiday”. J Am Coll Radiol. 2013;10(6):442-448.

Tan

Bavadian

Lyons

Lochhead

Alexander

. Flipped classroom approach to teaching a radiology medical student clerkship. J Am Coll Radiol. 2018;15(12):1768-1770.

Friedman

Demertzis

Hillen

Long

Rubin

. Impact of an interactive diagnostic case simulator on a medical student radiology rotation. AJR Am J Roentgenol. 2017;208(6):1256-1261.

Nadgir

Slanetz

. Integrating evidence-based imaging into the radiology core clerkship: a proposed teaching tool of imaging strategies. J Am Coll Radiol. 2010;7(7):517-521.

Chorney

Lewis

. Integrating a radiology curriculum into clinical clerkships using case oriented radiology education. J Am Coll Radiol. 2011;8(1):58e1.

Di Salvo

Clarke

Cho

Alexander

. Redesign and implementation of the radiology clerkship: from traditional to longitudinal and integrative. J Am Coll Radiol. 2014;11(4):413-420.

Chew

. Distributed radiology clerkship for the core clinical year of medical school. Acad Med. 2002;77(11):1162-1163.

Mullins

Lieberman

Shaffer

Novelline

. Factors to consider in the revision of educational objectives for medical students in radiology clerkships. J Am Coll Radiol. 2005;2(1):55-60.

10.

Chew

Stiles

. Joseph E Whitley MD. Award. Computer-assisted instruction with interactive videodisc versus textbook for teaching radiology. Acad Radiol. 1994;1(4):326-331.

11.

Theoret

Burbridge

. Flipping the passive radiology elective by including active learning. Can Assoc Radiol J. 2020;72:621.

12.

Johnson

Guediri

Kilkenny

Clough

. Development and validation of a virtual reality simulator: Human factors input to interventional radiology training. Hum Factors. 2011;53(6):612-625.

13.

Berry

Lystig

Beard

Klingestierna

Reznick

Lönn

. Porcine transfer study: Virtual reality simulator training compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol. 2007;30(3):455-461.

14.

Knudsen

Matsumoto

Chew

, et al. A randomized, controlled, prospective study validating the acquisition of percutaneous renal collecting system access skills using a computer based hybrid virtual reality surgical simulator: Phase I. J Urol. 2006;176(5):2173-2178.

15.

Nesbitt

Tingle

Williams

, et al. Educational impact of a pulsatile human cadaver circulation model for endovascular training. Eur J Vasc Endovasc Surg. 2019;58(4):602-608.

16.

Rudolphi-Solero

Lorenzo-Alvarez

Ruiz-Gomez

Sendra-Portero

. Impact of compulsory participation of medical students in a multiuser online game to learn radiological anatomy and radiological signs within the virtual world Second Life. Anat Sci Educ. 2022;15:863.

17.

Lorenzo-Alvarez

Rudolphi-Solero

Ruiz-Gomez

Sendra-Portero

. Game-based learning in virtual worlds: A multiuser online game for medical undergraduate radiology education within second life. Anat Sci Educ. 2020;13(5):602-617.

18.

Paech

Giesel

Unterhinninghofen

Schlemmer

Kuner

Doll

. Cadaver-specific CT scans visualized at the dissection table combined with virtual dissection tables improve learning performance in general gross anatomy. Eur Radiol. 2017;27(5):2153-2160.

19.

Darras

Spouge

Hatala

, et al. Integrated virtual and cadaveric dissection laboratories enhance first year medical students’ anatomy experience: A pilot study. BMC Med Educ. 2019;19(1):366.

20.

Darras

Forster

Spouge

, et al. Virtual dissection with clinical radiology cases provides educational value to first year medical students. Acad Radiol. 2020;27(11):1633-1640.

21.

Fallavollita

Seelbach

, et al. Personalized augmented reality for anatomy education. Clin Anat. 2016;29(4):446-453.

22.

Kugelmann

Stratmann

Nühlen

, et al. An augmented reality magic mirror as additive teaching device for gross anatomy. Ann Anat. 2018;215:71-77.

23.

Bork

Stratmann

Enssle

, et al. The benefits of an augmented reality magic mirror system for integrated radiology teaching in gross anatomy. Anat Sci Educ. 2019;12(6):585-598.

24.

Pottle

. Virtual reality and the transformation of medical education. Future Healthc J. 2019;6(3):181-185.

25.

Gutiérrez

Pierce

Vergara

, et al. The effect of degree of immersion upon learning performance in virtual reality simulations for medical education. Stud Health Technol Inf. 2007;125:155-160.

26.

Maresky

Oikonomou

Ali

Ditkofsky

Pakkal

Ballyk

. Virtual reality and cardiac anatomy: Exploring immersive three-dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education. Clin Anat. 2019;32(2):238-243.

27.

Burbridge

. DICOM image anonymization and transfer to create a diagnosticradiology teaching file. Int J Radiol Imag Technol. 2020;6:065.

28.

Burbridge

Costa

Carter

. ODIN: Adaptation of an HTML 5-compatible DICOM viewer for MIRC-TFS, enhancing the incorporation of clinical images into the health science curriculum. Med Sci Educator. 2019;29:11-17.

29.

Strickland

Lowry

Petersen

Jesse

. Introduction of a virtual workstation into radiology medical student education. AJR Am J Roentgenol. 2015;204(3):W289-W292.

30.

Goh

Lohre

Parvizi

Goel

. Virtual and augmented reality for surgical training and simulation in knee arthroplasty. Arch Orthop Trauma Surg. 2021;141(12):2303-2312.

31.

Park

Lee

. Full-immersion virtual reality: Adverse effects related to static balance. Neurosci Lett. 2020;733:134974.

32.

Stepan

Zeiger

Hanchuk

, et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol. 2017;7(10):1006-1013.

33.

Cao

Cerfolio

. Virtual or augmented reality to enhance surgical education and surgical planning. Thorac Surg Clin. 2019;29(3):329-337.

34.

Seymour

Gallagher

Roman

, et al. Virtual reality training improves operating room performance: Results of a randomized, double-blinded study. Ann Surg. 2002;236(4):458-463; discussion 63-4.

35.

Aïm

Lonjon

Hannouche

Nizard

. Effectiveness of virtual reality training in orthopaedic surgery. Arthroscopy. 2016;32(1):224-232.

36.

Lavoie

Main

King

. Virtual experience, real consequences: the potential negative emotional consequences of virtual reality gameplay. Virtual Real. 2021;25:69-81, Springer.

37.

Moro

Štromberga

Raikos

Stirling

. The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ. 2017;10(6):549-559.

38.

Frederiksen

Sørensen

SMD

Konge

, et al. Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: A randomized trial. Surg Endosc. 2020;34(3):1244-1252.