VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program

Abstract

Objectives

Virtual reality (VR) offers the potential to provide a lifelike, safe, and interactive environment where healthcare providers can practice and refresh their skills. The Neonatal Resuscitation Program (NRP) is an evidence-based and standardized approach for training healthcare providers on the resuscitation of the newborn where VR can be applied. Here we describe a development study for a VR-NRP simulation. This contribution is relevant for researchers and developers in the health sector interested in the integration of VR and other extended reality (XR) technologies in medical education and training.

Methods

For the implementation of the VR simulation, we used the Unity game engine, a VR-capable laptop, and an HTC Vive Pro Head-Mounted Display. We focused on the skill of positive pressure ventilation (PPV) using a bag and mask as the main scenario for the simulation since this is a foundational skill in NRP. To validate the prototype, we compared the VR-NRP simulation with 360° immersive VR videos in a crossover study involving 30 health-care providers and students, collecting various data through questionnaires and skill assessments by NRP instructors.

Results

We described in detail the creation process by which a highly realistic VR simulation was produced reflecting the visual elements and sounds of a Neonatal Intensive Care Unit in a hospital setting. In the crossover study, we found both VR technologies were positively viewed by healthcare professionals and performed very similarly. However, the VR simulation provided a significantly increased feeling of presence. Participants found the VR simulation more useful and reported higher confidence in NRP skills such as proper mask placement and newborn response evaluation, reflecting improved experiential learning outcomes.

Conclusion

This research represents a step forward in understanding how VR technologies can be developed and applied for effective, immersive medical training, increasing the availability of NRP refresher sessions, and providing insights into similar applications.

Introduction

Virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems, encompassed under the umbrella term extended reality (XR), have been proposed as suitable technologies to facilitate medical learning. One of the reasons for this is that, when properly designed, XR helps improve learners’ ability to analyze problems and explore new concepts. By being immersive, interactive, and imaginational, VR can provide virtual learning spaces that can be accessed by all kinds of learners. ¹ VR has been studied for medical education and training for many years. This is because VR provides a lifelike, interactive environment where healthcare providers can safely practice and refresh their skills. One of the major uses of VR is in the medical field, with research showing that, by 2018, about one in every five of all VR publications were in surgery, rehabilitation, and clinical neurology categories. ² This is largely because it provides opportunities for users to learn without adverse real-world consequences. For example, medical students can be trained to perform various surgical skills using robotics and VR systems. ³ This removes the pressure of learning on real patients while simultaneously providing training for various possible surgical scenarios. VR has also been used for assistance to the elderly and is widely used in research on new ways of applying psychological treatment or training, such as for treatments and management of phobias. ⁴ Overall, the use of VR technologies has been extensively researched in surgical and anatomical education with positive results.^4–6

VR can be used to create virtual learning environments (VLEs) that provide rich teaching patterns and instructional content. VLEs can enhance education in at least three ways: (1) by presenting multiple perspectives, (2) by providing situated learning, and (3) by supporting skills transfer. In addition, education that incorporates VR technologies is perceived by learners as more engaging and enjoyable, with higher levels of satisfaction and perceived usefulness.^7,8 Research has shown that VLEs improve students’ performance significantly and that knowledge is better retained when the learning material is delivered in a stimulating way.^9,10 In recent years, head-mounted displays (HMDs) have become the primary method of experiencing VR, as these devices have become more affordable and can provide users with an immersive experience, with realistic images and sounds that replicate the actual physical environment. ¹¹ A study by Alhabi, ¹² tested the performance of engineering students using three major VR systems and found that students performed significantly better when using any VR system than with no VR. This study showed that the more students are immersed in the environment, the more they learn and the better they perform. Additionally, the HMD VR system was found to have superior results than other VR systems. HMDs are also used for training healthcare faculty and nursing students. Some medical areas where HMDs have been used include imaging training, ¹³ teaching surgical procedures, ¹⁴ training on urinary catheterizations,^15,16 and teaching about medication withdrawal. ¹⁷

The neonatal resuscitation program

According to the World Health Organization (WHO), one of the leading causes of neonatal death is birth asphyxia. Up to 10% of newborns need help to begin breathing at birth with approximately 1% requiring extensive resuscitation measures to restore cardio-respiratory function. In this context, the ability of healthcare providers to perform high-quality newborn resuscitation is essential for decreasing neonatal morbidity and mortality. ¹⁸ The need for such interventions is relatively rare but highly stressful, and even for experienced healthcare professionals, medical errors or deviations from resuscitation procedures may occur. ¹⁹ Therefore, neonatal resuscitation training is crucial for enhancing the quality of care and positive patient outcomes. ²⁰ To address this, booster and refresher sessions involving “mock codes” have been shown to improve knowledge and skill updating. ²¹ The Neonatal Resuscitation Program (NRP) is a standardized, evidence-based approach for training health care providers on the resuscitation of a newborn. It was developed in 1987 by the American Heart Association (AHA) and the American Academy of Pediatrics, to identify infants at risk of respiratory depression and provide high-quality resuscitation. ²² To maintain current Provider status, the Canadian Paediatric Society requires practitioners to successfully complete an in-person NRP Provider course at a minimum of every 2 years. ²³ Simulation usage in neonatal resuscitation has been increasing, ²⁴ as simulations help healthcare personnel maintain and refresh current knowledge. Studies have shown that resuscitation training through simulations is more effective and suitable for enhancing teamwork skills among trainees. ²⁵ We believe that VR-NRP training could be used as a complementary tool to refresh the knowledge and skills of NRP providers during the 2-year period between Provider courses. In general, the NRP program could be an ideal area for testing whether VR can be used for refreshing and training practitioners, as it has been shown that VR provides an enhanced sense of presence and immersion among learners in clinical sciences training. ²⁶

Virtual reality serious games in neonatal resuscitation

One of the ways VR is used in learning in an engaging manner is through “serious games,” i.e., entertaining and interactive activities from which users can also learn and be educated or trained in well-defined areas and tasks. ²⁷ Virtual Reality Serious Games (VRSGs) provide an immersive VR environment, which improves user experiences and knowledge acquisition. Checa and Bustillo's review of VRSGs for training and education showed that interactive experience was preferred and concluded that user satisfaction with VRSG experience was higher than any other learning methodologies. Overall, VRSGs provide an effective learning platform with high user satisfaction and have a vast potential for application in many different areas, due to its affordability and technological development, providing versatility and a feeling of presence. VRSGs have been explored in the medical sector, especially for skills improvement and knowledge acquisition for hospital staff.^27,28 In fact, there are VRSGs about NRP training aimed at improving participants’ learning and their abilities, including eHBB ²⁹ and the Compromised Neonate Program (CNP). ³⁰ While these developments are promising, little research has been done to document their development and evaluate their effectiveness, ¹⁹ as we will explore later in the Discussion.

Curran et al. ³¹ explored the use of 360° NRP training videos using VR HMDs for neonatal resuscitation training to explore healthcare providers’ experiences and perceptions and found that a high level of acceptance of the technology was reported by healthcare providers. In addition, it was found that 360° video provides an enhanced immersion experience in resuscitation scenarios, a strong sense of presence, and produces a greater level of interest. The main educational benefits reported included its usefulness for self-learning and as a supplement to traditional teaching methods and resources. In this study, we used 360° NRP training videos to compare them with a computer-generated VR simulation of an NRP training session with the following goals:

Explore the feasibility and implications of using computer-generated immersive VR simulations delivered through HMDs to support neonatal resuscitation booster or refresher learning.

Describe in detail the proposed VR simulation for NRP training to facilitate scientific reproducibility and exchange.

Explore barriers and enablers towards the integration of VR HMDs as instructional or self-learning resources for healthcare providers trained in NRP.

Describe potential avenues for future research in VR-NRP simulation.

The purpose of this article is to present a development study of a VR application to help train NRP providers, and to support the research community interested in further developing solutions for VR-mediated neonatal resuscitation training. The work we present in this article aims to fill gaps in the literature by describing the development of a new VR neonatal resuscitation application, evaluating potential NRP skill improvement via a randomized crossover study, and discussing potential areas of improvement. The insights obtained from this development process could also be applied to other medical training simulations in medical contexts, allowing others to replicate these results and improve development processes.

Methodology

Nature and timeline of the study

The conception and development of this project extended over several years at Memorial University of Newfoundland, in St John's, NL. The definition of the initial scope of the study and acquisition of funds took place between January and September of 2020. Software development started in earnest in September of 2020, and the prototype was ready for testing by June of 2021. A subsequent randomized crossover study started in July of 2021 and finished in September of 2021.

Design choices for the simulation scenario. We chose the skill of PPV with bag and mask as the simulation scenario for NRP because, in accordance to the Textbook of Neonatal Resuscitation: “Ventilation of the newborn's lungs is the single most important and most effective step in neonatal resuscitation. Learning how to provide PPV is the foundation of neonatal resuscitation.” ³² Ventilation of the newborn's lungs has been the foundation of NRP since its inception in 1987, with evidence unchanged around effective PPV.

Hardware requirements

All the development was done using an HP Omen laptop 15-ek0xxx (x64 architecture), equipped with a Windows 10 Operating System and an Intel^® Core™ i7-1075H CPU @2.60 ghz with 6 Cores and 16GB of RAM. The graphics processor was an NVIDIA GeForce RTX 2060 with 1 GB RAM. The HMD used was the HTC VIVE Pro. ³³ This HMD was at the time (2020) the one with the highest pixel resolution (1440 × 1600 pixels per eye, or 2880 × 1600 pixels combined, at 90 Hz refresh rate) and had the best (widest) field of view (110° horizontally). The total equipment cost was about $2,200CAD for the laptop and $2,200CAD for the HMD, or about $4,500CAD, taxes included. The HMD was connected directly to the laptop through a USB-C to DisplayPort adapter. When using a laptop as in our setup, there is no way to connect to the HTC Vive wirelessly. However, when using a VR-capable desktop, it is possible to get a PC wireless card to connect the HMD through a wireless adapter for the HTC Vice (sold separately). A gaming PC with support for VR displays (with a USB-C 3.0 and/or a DisplayPort) would also usually be suitable to develop and try out the prototype.

Software requirements

For this project, we used the Unity 3D game engine (version 2020.02f) ³⁴ and its development environment for the visual design of the VR simulation prototype. The Unity 3D application communicated with the HMD via the SteamVR plugin, freely available as part of the HTC Vive installation. The HTC Vive also has the advantage that it is simply an additional display connected to the PC, discoverable by the Unity engine as an available output display when turned on.

Physical space requirements

The VR room was always setup using the “room-scale” requirements of the HTC Vive. That is, a minimum space of 2 m×2.5 m (6 ft 6 in×5 ft), which did not include the space for placing the laptop and the base stations (2.0), which were at the corners of the play area. Since the VR simulation did not require the provider to walk around the infant, and all the displacements took place through teleportation, there were no issues of mobility, other than an assistant had to make sure the cable to the HMD was out of the way of the participant.

Software development

The development was done in-house by (MYA), a Master's student of Computer Science (thesis route) at Memorial University of Newfoundland. Several 3D assets were acquired from third-party online libraries. The development time was around 16 months of full-time work from initial conception until the prototype was ready for experimental trials. Since this development took place during COVID, the student was allowed to take the equipment home and the supervision took place remotely during most of the software development stage. In the initial stage, the main features of the simulation were defined. The prototype was then iteratively refined through consultation with the research supervisors (VC and OMP) and the NRP expert in the team (SW). Once the prototype was ready, a series of trials with NRP instructors were conducted and the prototype was further refined until the NRP expert was satisfied that the prototype was ready for experimental trials.

Validation

To evaluate the contributions of the VR simulations we performed a randomized controlled crossover study on the effect of VR-NRP on learning outcomes with N = 30 NRP providers. We selected 30 as the sample size as it has been suggested in the literature that a minimum of 10 samples per condition is required for estimating regression coefficients ³⁵ and 15 samples per condition is enough for performing a Student's t-test. ³⁶ We limited participation to health-care providers and respiratory therapy students who had been exposed to NRP Provider training within the last 2 years. A call for volunteers for participation was issued with posters in the corridors of the Health Sciences Centre and via emails to the group of healthcare providers who were eligible to participate. Most participants were associated with the Neonatal Intensive Care Unit (NICU) and the Health Sciences Centre, the tertiary care hospital for the province of Newfoundland and Labrador (NL).

In the study, we defined two experimental conditions: 360° NRP training videos (also referred to as 360° videos) by Curran et al., ³¹ and the VR-NRP simulation prototype introduced in this article. We chose the training videos because they are recent, focus on NRP, and use VR HMDs. An online calendar was made available for self-registration and volunteers were requested to register for an available session based on their convenience. Upon arrival, participants were assigned to either group A or group B by random sampling without replacement to ensure we had an equal number of participants (N = 15) in each group. To account for potential ordering effects, participants in group A tried the VR simulation first, and the 360° videos next, whereas participants in group B were exposed to the same conditions, but in reverse order. The duration of each condition was approximately 20 min with a 20 min washout period between conditions, which was used to fill out the questionnaires after exposure to the first condition.

The data collection tools (i.e. questionnaires) in this study were (1) sense of presence, (2) usefulness, (3) ease of use, (4) simulator sickness, and (5) confidence in NRP. Sense of presence was measured with a 17-question questionnaire adapted from Witmer et al. ³⁷ with a 7-point Likert scale, Usefulness was measured using a 6-question questionnaire adapted from Lopez-Chávez et al. ³⁸ with a 7-point Likert scale (scale LS-1-7) from 1 to 7 (1 = Extremely unlikely, 2 = Quite unlikely, 3 = Slightly unlikely, 4 = Neither, 5 = Slightly likely, 6 = Quite likely = 6, 7 = Extremely likely), ease of use was measured through a 6-question questionnaire adapted from Lopez-Chávez et al. ³⁸ also using scale LS-1-7, Simulator sickness using a 16-item questionnaire adapted from Kennedy et al. ³⁹ with a 4-point Likert scale (None, Mild, Moderate, Severe) and confidence in NRP was assessed using a 15-item questionnaire with an 11-point Likert scale from 0 to 100 (where 0 = Cannot do it; 50 = Moderately confident can do; 100 = Highly certain can do) adapted from Curran et al. ⁴⁰ These questionnaires have been published in peer-reviewed journals, were validated and shown to have good reliability by the original authors and have been regularly cited in the literature.^38–40 We also collected open-ended feedback from the participants. Additionally, using the observation checklist provided in the Supplemental material, participants were evaluated on their performance of the PPV resuscitation procedure by having NRP instructors evaluate video recordings of the participants after being exposed to one of the experimental conditions. The data were collected via three sets of questionnaires administered as follows: a pre-test for demographics and NRP self-confidence, a first post-test between conditions for presence, usefulness, ease of use, likes and dislikes, simulator illness, presence, and self-confidence. A second post-test after the second conditions for usefulness, ease of use, likes and dislikes, simulator illness, presence, and self-confidence. The statistical language R was used for the statistical analysis. The questionnaires and tools used to collect this information were pilot-tested prior to the use in the randomized crossover study and are available in the Supplemental materials accompanying this manuscript. For a detailed description of the randomized crossover study, including statistical analysis, in-depth experimental results for all questionnaires, and participant's feedback, please refer to the work by Aydin. ⁴¹

Results

Here we describe the implementation goals for the VR-NRP simulation and the prototype developed as a result of our design choices.

Unity creation process

To create an immersive hospital simulation the project was developed in Unity 2020.02f, starting with a custom 3D environment with VR cameras configured to work with an HMD, allowing users to experience the simulation in a fully immersive environment. The focus was on achieving realistic graphics to help participants feel as if they were genuinely in a hospital setting. Key elements such as baby and doctor avatars, as well as hospital equipment like an infant warmer, were modeled in resemblance to the actual equipment. This was done to provide an environment that the participants were familiar with. For example, the infant warmer shown in the simulation corresponded to the same model as the infant warmer used at the local NICU. Animations were carefully selected for the doctor avatars to ensure natural, life-like interactions. Spatial audio effects were added to ensure that sounds would shift direction based on the user's head movement, enhancing the sense of immersion and focus on the training environment. Unity's Timeline feature was extensively utilized to control animations, sound effects, and event timing, contributing to a smooth and synchronized experience. Finite state machines (FSMs) were employed for decision-making processes and triggering Timeline events, adding a layer of dynamic interaction within the simulation. Integrating Unity assets, including Wire Builder for realistic ECG cables, UMotion for precise animation control, Pulse Physiology Engine for vital signs monitoring, and Deform for physical interactions like squeezing a positive pressure device, brought a high degree of realism to the simulation. By incorporating these assets and configurations, the project provided an engaging and distraction-free environment, allowing users to focus intently on NRP training.

VR environment

The higher the realism in training simulations, the more effective the simulation is in terms of refreshing the experience from a trainee's perspective.^42,43 For this reason, we created a virtual environment to simulate the real working environment of NRP trainees (also referred to as learners, users, or providers). We drew our inspiration from the Eastern Health NICU. The walls, corridors, floors, and ceiling of the space in the simulation were designed to resemble the actual environment and we added relevant devices, posters, and furniture so that users could feel a strong sense of presence in a NICU (see Figure 1).

Figure 1.

Sample view of the NICU welcome room. In addition, there are ambient sounds played during the simulation which are typical of the interior of a medical facility's NICU.

Spatial arrangement

In this simulation, there are three rooms in total. When the program starts, the user's first room is the welcome room, followed by the tutorial room and the simulation room. This also reflects the expected sequence in which users are meant to move across rooms. All rooms are similar to each other but with different equipment and objects. For example, while there are no objects to interact with in the welcome room, the infant and doctor avatars in the tutorial room are the same as the infant and doctor avatars in the simulation room.

First room (welcome room)

The welcome room is where the program starts and where the user gains familiarity with the simulation environment, learns the use of the VR controls and checks that all equipment is working. On the wall in front of the user, there is an information canvas where the functions of the buttons on the equipment and the controller are explained (see Figure 2).

Figure 2.

Instructions on how to use the controllers are shown in the welcome room, along with a brief audio message.

Second room (tutorial room)

In this room an infant on a warmer is found in front of the user, as well as a doctor's avatar who welcomes the user. NRP information posters are found by the infant warmer that the user can refer to during the simulation (Figure 3). To draw the attention of the user to these posters, animated arrows pointing to the posters have been placed on top of them. The doctor avatar is located on the user's right side, next to the infant warmer. The doctor avatar will help the user perform PPV during the neonatal resuscitation procedure as part of a team approach to NRP. The user can move the bag valve mask with the help of the HTC VIVE Tracker held in their left hand. The head of the doctor avatar is set to follow the user, for a more realistic and natural effect. This feature was implemented with Unity's “Animation Rigging” package.

Figure 3.

MR. SOPA and Target Oxygen Saturation (SpO₂) posters were placed by the sides of the infant warmer in the tutorial room.

Third room (simulation room)

The third room is called the “Simulation Room". After the user understands the system in the welcome room and masters the equipment in the tutorial room, they are asked to teleport to this room using the main menu to start the neonatal resuscitation training simulation. The simulation starts shortly upon entering the room. There is a real-time clock on the wall and on the left side of the user, a monitor is placed that displays the infant's vital signs (heart rate and oxygen saturation). During the training simulation, the data received from the pulse oximeter attached to the infant's right wrist and ECG leads is reflected on this monitor.

NRP (7th edition) posters were placed on the infant warmer to assist healthcare providers in the neonatal resuscitation procedure. In addition, a poster of “MR. SOPA", a mnemonic for the sequential corrective actions required when ventilation is assessed to be ineffective and an important neonatal resuscitation action sequence, ⁴⁴ is placed on the side of the infant warmer. Finally, targeted pre-ductal Oxygen saturations (SpO₂) ⁴⁵ data were added on the other side of the warmer (Figure 3). Together, these pieces of information provide relevant context which might help the user remember the steps when applying neonatal resuscitation procedures in the simulation. Figure 4 shows a sample view of the simulation room from the trainee's perspective.

Figure 4.

A sample view of the simulation room from the NRP provider's perspective.

Teleporting between rooms

The user can switch between rooms using the main menu shown in Figure 5. To open the menu, the user presses the trackpad button of the HTC VIVE Controller in their right hand. After pressing the trackpad, the menu pops up in front of the user's view. There are three options in the menu: “Tutorial", “Start", and “Exit". The teleport feature of the SteamVR package ⁴⁶ is used for traveling between rooms. Teleport locations where the user will be placed and a unique name tag were determined for each room in Unity. The user can move around within different parts of the room by physically moving in real life, but when they teleport to another room, they are brought to that room's initial location. For the tutorial and simulation rooms, this location was in front of the infant warmer, which is the most suitable position to do PPV.

Figure 5.

The menu in the simulation. It follows the user's camera and moves wherever they look. Menu items highlight red when the user's controller hovers over them.

Interactive elements

Apart from the static elements, interactive elements in the simulation included the infant being treated, a supporting doctor's avatar, a blanket, a bag valve mask, and a pulse monitor.

Doctor Avatar. Teamwork is very important in the neonatal resuscitation procedure performed on newborns. As such, in this project, we selected a realistic human avatar to play the role of a doctor teammate to assist the VR user during the training. This avatar helps the user with animations in important places in the flow of the training and gives voice commands when necessary to prompt the necessary user actions. The voice commands were written in a script, and an AI-generated voice speaks out the scripts. These scripts, as well as the AI voices, can be updated within Unity's development environment. A skeletal rig is attached to the joints of this human avatar so that the model moves in a more realistic manner (Figure 6).

Figure 6.

Sample image of the Doctor avatar with stethoscope animation. He stands on the right side of the warmer and helps the NRP provider during the simulation.

Bag Valve Mask. In the event that a baby is not breathing or has a heart rate less than 100 beats per minute, a bag valve mask, one type of positive pressure device recommended by NRP (also known as Ambu bag, manual resuscitator, or self-inflating bag), is used to inflate the baby's lungs and provide oxygen as indicated. In the prototype, the face mask is designed to be held with the user's left hand, and air/oxygen can be provided by squeezing the bag with the right hand. An HTC VIVE 3D Tracker was used as a bag valve mask replacement held in the left hand of the user and the trigger on the VR controller in the right hand was pressed to hold and squeeze the bag (Figure 7). When the user squeezes the bag through the VR controller, the bag is animated through its contraction and expansion, giving air/oxygen to the baby.

Figure 7.

A sample image from the simulation. The user holds the mask to the baby's face and the positive pressure ventilation device while doing PPV.

Cardiorespiratory Monitor. Another interactive object in the simulation is the cardiorespiratory monitor (Figure 8), which is essential in neonatal resuscitation as it provides information on the infant's vital signs (heart rate, oxygen saturation). It involves 4 elements: (1) Pulse oximeter, attached to the right wrist of the infant; (2) ECG leads, placed on the infant's chest; (3) ECG cables, designed to connect the leads with the monitor, and lastly; (4) the monitor display.

Figure 8.

The monitor used to reflect the signals from the ECG sensors attached to the baby's chest is shown on the left of the image. It is updated with the infant's vitals at every second.

Sound effects

While sound is sometimes considered an afterthought in VR, we found that in the case of the VR-NRP simulation, spatial sound contributes greatly to transporting the user to the hospital setting and providing a fully immersive experience. To reflect the hospital ambience, background “hospital noise” is added and set at a clearly audible level. It is important for the user to hear the sounds from the simulation clearly because most of the information and guidance for the simulation is done through sound and voices, including the instructions and comments from the doctor's avatar. Another effect is the sound produced when the bag valve mask is squeezed while PPV is provided, which gives feedback that the bag is effectively squeezed.

Video and simulation availability

The sounds and voices in the simulation can be appreciated from the video available here: https://youtu.be/r40twTjuwTE. In addition, the executable file of the simulation itself will be posted on the corresponding author's homepage (https://www.cs.mun.ca/~omeruvia/research/research.html) as a permanent link and will be otherwise available upon request. An HTC VIVE HMD with its corresponding base stations, VR controllers, and a VIVE 3D Tracker, along with a VR-capable laptop or PC are required to try out the VR-NRP simulation. A gaming computer with VR display ports or VR-advertised capabilities is recommended.

Summary of randomized crossover study results

With respect to the demographics collected, the highlights are that 93% of participants were female, 73% reported less than 5 years of practical experience, half of them were registered nurses, and slightly over half of participants (53%) had completed their last NRP provider course within the past year. With respect to the questionnaire results, participants found both conditions moderately easy to use and reported virtually no simulator sickness effects. In addition, participants reported significantly higher perceptions of usefulness (false discovery rate (FDR)-corrected Wilcoxon–Mann–Whitney (WMW) test p = 0.014) regarding learning performance and the perceived learning effect of VR. Participants found the VR simulation was more helpful in improving learning performance and enhancing learning. They also reported an increased feeling of presence when experiencing the VR simulation (FDR-corrected WMW test p≤0.01 for 10 out of 17 questions of the presence questionnaire). Participants using the VR simulation reported higher confidence in certain NRP skills, such as proper mask placement (FDRcorrected WMW test p = 0.0388) and newborn response evaluation (FDR-corrected WMW test p = 0.0173). A blinded assessment of PPV skills showed participants exposed to VR performed significantly better in providing effective PPV (FDR-corrected WMW test p = 0.0055). Detailed experimental results are found in the work of Aydin. ⁴¹

In terms of the open-ended feedback, there was a variety of positive feedback regarding the VR simulation. Participants found that it was good for learning and provided a safe and controlled (low risk) environment. They found it was easy to use and also liked that it was “hands-on” and interactive, realistic, and immersive. Several also mentioned they appreciated the availability of the procedures, the SpO2 and MRSOPA posters by the warmer, and the guidance provided by the Doctor avatar during the procedure. Some mentioned they would have liked to see a cardiopulmonary resuscitation (CPR) scenario. Aspects that participants didn’t like included issues with the vision and/or clarity of the visual supports: while some reported that they could see the numbers on the monitor easily, others found that they couldn’t read the charts or that they were blurry when not looked straight at. Some participants found the bag mask was not realistic enough, in the sense that it was activated via the trigger of the VR controller, and that the fitting of the mask onto the infant's face couldn’t be practiced the same way as in real life. Because the avatar was guiding the users as to what to do, some participants disliked the aspect where they were not doing their own thinking as they completed the procedure. Some missed the teamwork aspect of NRP. Finally, some participants felt that it took time to become familiar with the functioning of the simulation itself but that more experience in VR would help overcome this. Detailed accounts of the participants’ feedback are found in the work of Aydin. ⁴¹

Discussion

The purpose of this development study is to support the research community in the collective goal of advancing medical training using interactive XR technologies. For this reason, we described in detail the development processes of VR-NRP and the interactive elements that allowed us to implement our design choices and achieve our implementation targets. In this section, we compare VR-NRP with related projects to highlight similarities and differences in the creation, development processes, and produced simulations.

According to a review by Makinen et al., ⁵ three types of VR technologies are being used for healthcare learning: haptic simulators, computer-based simulations, and HMD simulations. Of the three, haptic simulators were found to be used the most for healthcare learning, followed by computer-based simulators as the second most used technologies. Computer-based simulators were mostly used in nursing education or in training healthcare trainees such as residents. The most commonly used computer platform for learning was the Second Life online virtual world. Computer-based serious games such as Virtual Emergency Tele-Medicine, Body Interact, and Home Healthcare Virtual Simulation Training Systems have also been used for training purposes.^47–50 However, computer-based simulations are delivered through desktop computers that lack the advantages of fully immersive simulations made possible when wearing VR HMDs.

One example of a serious game delivered through a VR HMD is called “electronic Helping Babies Breathe” (eHBB) developed by the University of Washington, USA, and Oxford University, UK. It was tested at the National Hospital in Abuja, Nigeria. ²⁹ eHBB is accessible via mobile smartphones and low-cost VR devices such as Google cardboard. It provides a virtual environment and lets users observe the procedures of neonatal resuscitation under three different scenarios. Its purpose is to provide an opportunity for continuous learning for new skill development, as well as to help maintain training fidelity. Overall, eHBB was reported to be easy to use, educational, and to enable learning without real-life stress. ¹⁹ The eHealth4everyone team carried out usability testing of the application with nurses, midwives, and doctors specialized in the field of child birth and child health care. ²⁹ Although they mention that the users provided “crucial feedback and recommendations to the developers of the application,” ²⁹ and that the feedback was “logged and worked on to further improve the usability and performance of the application”, ²⁹ there are no references to research or literature describing the nature of this crucial feedback and/or the specifics of the recommendations provided by the specialists to the developers. There is also no mention of how the simulation was produced. In contrast, in this study, we have described in detail how the simulation was produced in a way that the methodology presented here can be adapted or replicated to help further development in this field.

Another example of a VR serious game (VRSG) aimed at improving neonatal resuscitation is Lifesaving Instructions For Emergencies (LIFE), ²⁸ developed by Oxford University, UK, and tested in Nairobi, Kenya. LIFE is both a mobile and VR game that enables users to accomplish the procedure while using their own smartphones, thereby allowing over 7000 healthcare workers and medical students to experience the game even within low-cost settings. ⁵¹ The game included a section regarding neonatal resuscitation where users must match the equipment correctly in a virtual rural hospital, ¹⁹ however, it is not clear how much support is there in this section to provide training in PPV or how much of the training is in the form of immersive VR and how much is mobile with no VR. Unfortunately, there is no research describing the development process for LIFE and no information for developers and researchers in the field to replicate LIFE's application and results. Tuti et al. ⁵² performed an evaluation on LIFE's effect of adaptive feedback on learning outcomes and found learning gains between the first two rounds of exposure to it. In this work, we complement the existing literature by presenting a development study of a highly realistic HMD simulation for NRP that takes place in an immersive VR hospital setting with visual elements and sound effects proper of an NICU, which supports an increased sense of presence.

The CNP is an immersive VR application developed at the University of Newcastle, Australia, for teaching neonatal resuscitation skills to midwifery students. ³⁰ In the CNP, learners must complete neonatal resuscitation tasks successfully in the virtual hospital environment.^53,54 When looking at comparable literature, the CNP by Jones et al. ⁵³ stands out as the most similar to VR-NRP. Both simulations are delivered via VR HMD's, setup in realistic rooms that emulate the hospital environment, and use VR controllers for interaction with the elements of the simulation. They are also immersive and interactive, turning the learning experience into an active process. One important difference between both simulations is that, in lieu of the bag valve mask ventilation, the CNP simulates the Neopuff™, a positive pressure device that enables a provider to preset and deliver ventilation parameters in a controlled manner. However, not all NICU's have a Neopuff™. PPV is a foundational NRP skill that is not practiced when using the Neopuff™ or similar devices. By contrast, VR-NRP focuses on PPV using the bag valve mask setup, so the VR-NRP simulation reflects a setup that uses more generally available equipment. Another difference is that in the CNP, there are two modes of operation, one which is guided and the other is unguided, which is used after participants have gone through the guided mode. The VR-NRP simulation corresponds to the guided mode in the CNP.

Applicability and usefulness of VR-NRP

In a pilot study of the CNP VR simulation (N = 7) Jones et al. ⁵³ found preliminary support for the notion that participating in the VR simulation helps improve confidence in neonatal resuscitation (5 out of 7 participants agreed with this proposition) and that the CNP allows for self-analysis of behaviors and actions (4/7 agreed). To document the benefits of VR-NRP, we compared it against 360° immersive videos by Curran et al. ⁵⁵ through a crossover study. By having a larger participant pool (N = 30) our study has more statistical power to support our findings. The results of the crossover study suggest that VR simulation and 360° video were both perceived as valuable and viewed positively, with overall slightly higher ratings for the VR simulation. Participants experienced a significantly increased feeling of presence and found the VR simulation to be more useful than the videos, improving experiential learning outcomes. Similar to Jones et al., ⁵³ VR-NRP participants also reported higher confidence in certain NRP skills, such as proper mask placement and newborn response evaluation, and performed significantly better in terms of providing effective PPV, which has been the foundation of NRP since its inception. The results and feedback suggest that when both techniques are used for NRP training, 360° video should be presented first, as a refresher, followed by the VR simulation, to allow for a more hands-on experience. Feedback from participants also suggests that both NRP training via VR simulation and the 360° video could be used as additional “refresher” resources for healthcare providers, adjunct to in-person courses, which are required every 2 years. In our findings, no effects of gender or any of the other demographics collected were observed.

Limitations

One of the limitations of this development study is the speed of hardware advances in VR, which makes it hard to produce software that remains current. For instance, each year a new set of VR HMDs is produced, with better specifications than previous generations and sometimes with updated requirements. In the same line, equipment that was state-of-the-art just a couple of years ago may become technically outdated or commercially unavailable, even if it works properly.

The questionnaires that we have provided in the Supplemental materials have been pilot-tested in trials prior to the crossover study. Some portions of the questionnaires have been adapted from the relevant literature and the references for these adaptations are mentioned in the questionnaires. The demographic questionnaire was designed for the particular group of people usually in charge of providing neonatal health care, so its contents are specific to NRP and may need to be adapted to other potential procedures.

Since in this study the focus is on the development process, we have limited the discussion of the crossover study details. We are planning to publish a related article that will focus on presenting the crossover study in a more succinct form. In the meantime, readers interested in the crossover study may refer to the work by Aydin, ⁴¹ which describes the crossover study in great detail.

One direction of improvement for VR-NRP is to embrace the bi-modal approach of the CNP. ⁵³ While the current simulation corresponds roughly to CNP's guided mode, participants could benefit from a mode with reduced guidance, where they would need to complete the procedure with assistance from the Doctor avatar, but without the instructions as to what steps to take next. A third mode could be an “evaluation mode", where the system would evaluate the participant's competence in completing the procedure by means of computer vision, sensing technologies, and machine learning. This would require access to the cameras and sensors in the HMD, which are currently inaccessible in commercial HMDs but could be part of future Software Development Kits.

Another direction for future research, along the lines suggested by Fealy et al., ⁵⁴ is to explore whether Mixed and Extended Reality could offer additional benefits for NRP training. We believe that adding tactile feedback or VR controller-free interaction to the VR simulation, in particular when practicing the bag mask application procedure, will help trainees experience a higher degree of realism. The hypothesis is that allowing participants to handle an actual PPV device that inflates according to the user's actual operation might be more helpful and realistic than the current setup with VR controllers. In open-ended feedback, participants alluded to the need for a more hands-on approach to VR training. We are currently working towards porting the VR-NRP simulation to the Meta Quest platform to make the simulation more accessible to a wider community. The advantages of the Meta Quest and similar platforms include the ability to execute the simulation in stand-alone mode, without a laptop or desktop requirement and no cables, the possibility of implementing direct manipulation and interaction procedures with no VR controllers, and the lower price of the newer HMDs (currently about $450-$1300 CAD), which would make VR-NRP more accessible in terms of the cost per training station.

VR technologies may offer an alternative, increasingly affordable way to provide standardized and portable refresher learning opportunities on NRP concepts. We plan to explore whether incorporating these technologies would be helpful in the context of healthcare professionals working in remote or rural areas where in-person training may be difficult to access. A field experiment or evaluation in a rural clinical setting would provide better evidence for this possibility.

Another area for future research is the ability to create a VR-NRP “game engine” that would allow for multiple programmable scenarios to be played out within the VR-NRP simulation. Programmable scenarios would potentially allow for certain elements of the simulation to be adjusted without the need to recompile the VR-NRP simulation, allowing expert users (such as NRP instructors or senior healthcare providers) to insert a variety of scripts and scenarios representing different case studies. This is a challenge from the point of view of software design & development but comes with the promise of a longer life cycle for the software. A related concept would be the creation of an Open Source or Open Access VR-NRP simulation, where a developer or enthusiast community could contribute platform improvements in a similar way as it is done for some large-scale software enterprises (e.g., the Linux operating system).

Besides NRP training, the VR training simulation can be extended to other scenarios, such as CPR and advanced cardiac life support (ACLS). In this study, we built the VR environment comprising the room, the characters, the medical equipment, and other medical aspects using in-house resources, Unity store assets, and 3D models freely available online. The scenario and simulation can be adapted for CPR and ACLS training in the pediatric and adult patient contexts. After modification of the VR environment, its effectiveness can be assessed through other studies.

Conclusion

We have presented VR-NRP, an immersive VR simulation environment designed and developed to refresh the skills of NRP providers. This study aims to help researchers and developers produce similar simulations and advance the state-of-the-art. VR-NRP was implemented in Unity using the HTC Vive Pro HMD. We compared VR-NRP with 360° NRP videos and found that VR-NRP helps improve the learning outcomes of NRP providers, increases confidence in NRP skills, and helps trainees improve their ability to provide effective PPV, a crucial resuscitation skill of NRP. In general, this research highlights the vast potential of VR technology and how it can continue to have a beneficial role and impact on healthcare education.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076251323989 - Supplemental material for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program

Supplemental material, sj-pdf-1-dhj-10.1177_20552076251323989 for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program by Mustafa Yalin Aydin, Vernon Curran, Susan White, Lourdes Peña-Castillo and Oscar Meruvia-Pastor in DIGITAL HEALTH

Supplemental Material

sj-pdf-2-dhj-10.1177_20552076251323989 - Supplemental material for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program

Supplemental material, sj-pdf-2-dhj-10.1177_20552076251323989 for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program by Mustafa Yalin Aydin, Vernon Curran, Susan White, Lourdes Peña-Castillo and Oscar Meruvia-Pastor in DIGITAL HEALTH

Supplemental Material

sj-pdf-3-dhj-10.1177_20552076251323989 - Supplemental material for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program

Supplemental material, sj-pdf-3-dhj-10.1177_20552076251323989 for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program by Mustafa Yalin Aydin, Vernon Curran, Susan White, Lourdes Peña-Castillo and Oscar Meruvia-Pastor in DIGITAL HEALTH

Supplemental Material

sj-pdf-4-dhj-10.1177_20552076251323989 - Supplemental material for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program

Supplemental material, sj-pdf-4-dhj-10.1177_20552076251323989 for VR-NRP: A development study of a virtual reality simulation for training in the neonatal resuscitation program by Mustafa Yalin Aydin, Vernon Curran, Susan White, Lourdes Peña-Castillo and Oscar Meruvia-Pastor in DIGITAL HEALTH