Uses and Considerations for Cinematic Virtual Reality in Health Care

Abstract

Background:

Through the combination of virtual reality (VR) technology with techniques from theater, filmmaking, and gaming, individuals from the Game Research and Immersive Design Laboratory (GRID Lab) at Ohio University have developed an approach to train soft skills such as communication, problem-solving, teamwork, and interpersonal skills which shows great promise.

Objectives:

The purpose of this article is to provide an overview of VR and cinematic-VR (cine-VR). This article serves as a preface to the VR research included in this special issue.

Methods:

In this article, we define VR, review key terminology, present a case study, and offer future directions.

Results:

Prior research with cine-VR has demonstrated the effectiveness in improving provider attitudes and cultural self-efficacy. While cine-VR may differ from other types of VR applications, we have been able to leverage the strengths of cine-VR to create training programs which are user friendly and highly effective. Early projects on diabetes care and opioid use disorder were sufficiently successful that the team received additional funding to pursue series addressing elder abuse/neglect and intimate partner violence. Their work has gone beyond health care and is currently being leveraged for law enforcement training as well. While this article will explore Ohio University’s approach to cine-VR training, details of their research including efficacy can be found in McCalla et al, Wardian et al, and Beverly et al.

Conclusion:

When produced correctly, cine-VR has the potential to become a mainstay component of training for soft skill applications across a multitude of industries.

Introduction

Virtual reality (VR) in health care is growing rapidly. More and more, VR is being used in the management and treatment of diabetes. For these reasons, the Journal of Diabetes Science and Technology is devoting a section of this issue to research on the VR. Below we provide a brief overview of VR and introduce a new type of VR that has the potential to revolutionize the way we educate, train, and support health care providers and trainees. In this article, we define VR, introduce cinematic virtual reality (cine-VR), describe the process of content development, share a case study, and provide areas for future development. Following this article, the journal includes three studies that share findings from VR research. The purpose of this article is to provide readers with a basic understanding of VR prior to reading these research articles.

Virtual Reality

Emerging communication technologies can make training more accessible, collaborative, interactive, and cost-effective. Utilizing emerging communication technologies in health care training can also provide better techniques for teaching complex ideas in a way that resonates emotionally with participants. Ohio University’s Game Research and Immersive Design Laboratory (GRID Lab) has been researching emerging communication technologies for 18 years, with significant efforts devoted to exploring VR. By combining the strengths of VR technology with narrative storytelling, the GRID Lab developed cine-VR. This article will help readers better understand VR by exploring how cine-VR is uniquely effective for certain types of health care training.

According to the Virtual Reality Society, VR is the process of presenting our senses with a digital environment that is explorable in some fashion. ¹ Typically, this exploration requires donning a headset (for visual occlusion of the real world), headphones (for aural occlusion of the real world), and hand controllers or sensory gloves (to recreate tactile sensation). Companies have also explored VR smells and tastes but presently most consider sight, sound, and touch as the predominant senses when discussing VR. Virtual reality developers use computer-generated assets to create virtual worlds digitally. These creations often appear “animated” or “cartoonish” in nature as they are artificial representations of human beings and their locations.

Process

Rather than creating digital characters and locations, cine-VR captures actual people in real locations by recording them with specialized cameras and microphones that record in 360°. Similar to how a Hollywood film differs from a home video recording, cine-VR goes beyond standard 360° videos by utilizing the cinematic aspects of storytelling such as the use of detailed set design, lighting, and sound design to build a world where professional actors are carefully blocked to perform.

The second key distinction between VR and cine-VR is the participant’s physical movement. In immersive environments, the ability to freely move your head to inspect your surroundings is known as “three degrees of freedom” (3-DoF); the head controls its pitch (x-axis), yaw (y-axis), and roll (z-axis), while the body remains relatively stationary. With “six degrees of freedom” (6-DoF), a participant is also able to move their body physically (forward/backward, left/right, and up/down) through space; both the head and the body are free to move about. Most VR experiences employ 6-DoF, while cine-VR uses 3-DoF.

Three degrees of freedom cine-VR experiences are viewed by the audience wearing a head-mounted display (HMD) and headphones, digitally occluding them from the world and instead providing them with a new, explorable environment. Properly executed, a cine-VR experience allows the audience (or, as we say: the participant) feel as if they are positioned where the camera was positioned—able to look and listen in any direction based on their preference. The effect makes the participant feel as if they have been transported to the location of the story and feel present in the room with the characters in the story. This feeling is called “presence” ² (see Figure 1).

Figure 1.

Seating participants in a swivel office chairs is a safe, effective method for viewing cine-VR content, as the chairs provide easy 360° rotation and limit any possible vertigo and motion sickness symptoms for participants new to VR.

The presence can be experienced in both the animated 6-DoF worlds of VR and in naturalistic 3-DoF worlds created in cine-VR. However, we have found that the contradistinction actually creates two of cine-VR’s three main strengths:

• Observational focus: By limiting movement to 3-DoF, the participant is forced to observe rather than explore, creating an emphasis on the present.

• Verisimilitude: The authenticity of cine-VR provides a more substantial experience for the participant, as animated VR training has previously left participants disappointed by a lack of realism.^3,4

• Intellectual seclusion: Using an HMD, the participants are not easily distracted by cell phones and other worldly intrusions (applies to both 3-DoF and 6-DoF).

The cinematic element of cine-VR must not be ignored. While there are meaningful use cases for basic 360° video capture, ⁵ simply recording an event using a 360° camera does not create a “cine-VR” experience. To make a 360° recording “cinematic,” the production needs to borrow techniques from film, theater, and video game production. ⁶ Actors must be professionally scripted, blocked, and lit. Camera and microphone placement must allow visuals and sound design to work hand in glove with the story design. And interactive elements, while limited, must be carefully considered for maximum impact (see Figure 2).

Figure 2.

360° production rig on the Lula Mae set in a scene about food insecurity. Participants can choose to follow the actors or to explore the empty shelves on their own.

In 2015, Chris Milk (CEO of VR company Vrse) famously claimed that 360° videos were the “ultimate empathy machine.” ⁷ As empathy is the ability to understand and share the feelings of another, then cine-VR seems predisposed to creating empathic responses, especially situation when considering its proclivity toward verisimilitude, intellectual seclusion, and creating a sense of presence. Furthermore, using professional actors in realistic environments seems to enhance its empathetic capabilities—as creating empathy is the primary role of a professional actor.

Diabetes Case Study

In 2018, an opportunity arose to leverage the GRID Lab’s cine-VR techniques for large-scale production with the ambitious goal of improving health care outcomes for people with diabetes in Appalachian Ohio. We began by meeting with subject matter experts (SMEs) in the medical field to understand the most pressing challenges, honing in on the patient-provider relationship as a significant improvement opportunity. We came to believe that if we could help providers better understand how to approach diabetes care with their patients, we could improve health care outcomes. Our writing team worked to tell stories of actual patient experiences, building a storyline around a fictional character, Lula Mae. After several drafts, we balanced medical accuracy, teaching points, and a compelling narrative. Every detail was considered when source the actors and locations to bring Lula Mae to life. The production crew consisted primarily of university students, and filming took place over the course of several weekends.

Lula Mae’s story was divided into six episodes, each less than five minutes. The first episode is a traditionally shot short film, used to bring the audience into the story via a familiar medium. The remaining five episodes are cine-VR experiences. Presented in a conference or workshop, an instructor introduces the topic and shares preliminary information about Lula Mae. Participants view episode 1 (a less than ideal interaction between Lula Mae and her primary care doctor) on a screen. After the episode, participants engage in discussion.

Episodes 2 to 4 are cine-VR experiences that offer glimpses into Lula Mae’s life, helping participants better understand the challenges she faces while managing her diabetes. Again, each episode is followed by a group discussion guided by the instructor. For the final two episodes, participants take an active role. In episode 5, participants engage in a mock conversation with the primary care provider from episode 1. Episode 6 also employs this approach and encourages the participant to re-imagine the initial provider-patient interaction, taking on the role of provider and engaging with the patient. In episodes 5 and 6, participants read the on-screen text aloud, with the content carefully timed so that characters respond accordingly. We coined the term “Guided Simulation” for these mock conversations.

Our pilot and feasibility research with the Lula Mae cine-VR simulation has shown improvements in health care providers’ cultural self-efficacy (eg, the confidence to interact with and treat patients from various cultural backgrounds) and attitudes toward diabetes (eg, health care professionals’ attitudes toward special training for diabetes, seriousness of type 2 diabetes, value of tight glucose control). ⁸ Specifically, we conducted multiple single-arm pre-post studies to assess changes in these constructs. In this special issue of Journal of Diabetes Science and Technology, we include new research findings from a study with health professional students participating in the Lula Mae cine-VR simulation (see McCalla et al). We observed improvements in cultural self-efficacy, attitudes toward diabetes, and empathy. Finally, a second research article included in this special issue also documented improvements in attitudes toward diabetes and empathy (see Wardian et al). Importantly, Wardian et al included both an immediate and long-term assessment of attitudes toward diabetes and empathy. Both showed improvements immediately after participating in the cine-VR simulation. Interestingly, the improvements’ empathy was sustained at six-week follow-up but not the attitudes toward diabetes. Additional research on this cine-VR simulation is needed with a proper control condition and long-term assessments to determine the effectiveness of this cine-VR simulation among health care providers and trainees.

Structuring our presentation episodically with discussions taking place between viewings offered two notable advantages. First, the episodes could build upon each other, with each representing a piece of the larger story (eg, each scene highlighted an individual social determinant of health that collectively painted the entire picture of Lula Mae’s complex life). Compartmentalizing the educational elements made them more accessible during the discussion sessions, allowing the instructor to provide specific facts, details, and conversation points to the group.

The second advantage is that while we kept the story content medically accurate and relevant, we were not compelled to include overt teaching points. For example, when Lula Mae exhibits a limp, no character within the story needs to vocalize that it could be a diabetes complication. Instead, the discussion sessions provide the opportunity for participants or the presenter to raise the issue of complications. We believe that this approach allows the content to feel realistic and natural, allowing for a deeper sense of presence. As participants realize the teaching points will not be force-fed to them in a didactic fashion, they pay closer attention to the story and are looking for clues, seeking the sense of accomplishment that comes from having solved a puzzle.

Upon dissemination, we realized that simplifying the experience for participants was critical. To avoid users watching the wrong episode or exiting the viewing application within the HMD, we employed software that allowed the instructor to control playback across multiple HMDs (up to 100 at a time). This software was especially beneficial during the guided simulation portion, as having the content synchronized meant everyone in the room read their lines aloud simultaneously, with the result being more choir-like than chaotic.

Identifiable Reactions

Properly done, cine-VR has been proven to create four distinctly identifiable reactions with participants: immersion, intellectual/emotional recall, empathy, and behavioral change. “Immersion” in VR occurs when the participant can suspend disbelief of the artificial environment and fully engage in the new VR presented. ⁹ In a 2018 study, Krokos et al found that immersive experiences could serve as a valuable tool for various facets of retrospective cognizance, observing that the use of HMDs creates immersion and thereby improves recall accuracy compared to viewing the content using a traditional desktop. ¹⁰

While an HMD provides maximum effectiveness, this can be seen as a limitation and a challenge. The cost of hardware for dissemination must be considered, as does the task of gathering people into a central location for trainings. In many cases, a cost-benefit analysis favors in-person training, but it should be noted that the distribution mechanism of a cine-VR experience is not limited to HMDs and in-person presentations. Cine-VR experiences can be viewed on a desktop computer, smartphone, or tablet if HMDs are cost prohibitive or present an accessibility issue. On a desktop computer, dragging a mouse replicates moving your head in an HMD. Smartphones and tablets have built-in accelerometers and gyroscopes that track movement in space, allowing the user’s physical movement of the device to replicate HMD movement. Despite availability on multiple platforms, the Kronos study found that using HMDs improves immersive experiences compared to viewing the same content on a traditional desktop. ¹⁰ It should be noted that observing cine-VR outside of a headset precludes the ability for the participants to experience a sense of presence or immersion, and the experience is no longer considered “virtual reality.”

In 2016, a series of experiments using 360° video was conducted at Stanford’s d.school where they discovered that story information, when dispersed differently within a 360° space, will create either a primarily intellectual effect or a primarily emotional effect on the audience. ¹¹ For instance, if the audience needs to look around a room in full 360° to find the story, then the story will primarily have an emotional impact on the audience. However, if the audience could discover the same story within just a 90° area, then the story will primarily have an intellectual impact on the audience. ¹² The GRID Lab used this study to develop a storytelling shorthand for cine-VR called the Christmas Carol Continuum. The tool helps us decide how to balance emotional resonance and intellectual impact within a scene. ¹³

We believe that the power of influencing emotional/intellectual recall with cinematic techniques, combined with the presence, empathy, and immersion created by cine-VR stories, can produce stronger behavioral change among participants than traditional forms of media. In 2020, Dr Liz Beverly used the Lula Mae series as part of a training program focused on simulating clinical encounters addressing social determinants of health, Appalachian culture, and diabetes. Her findings supported “the notion that cine-VR education is an innovative approach to improve cultural self-efficacy and diabetes attitudes among health care providers and administrators.” ⁸ Building on the strength of our methods, we have also developed series addressing opioid use disorder, elder abuse/neglect, intimate partner violence, and lesbian, gay, bisexual, transgender, queer, and more (LBGTQ+) health care disparities which have all been very well received. For more information on the implementation of these programs, including data collected on empathy, please see additional articles in this special issue by Beverly et al and McCall et al. Further research is warranted, but the domino effect that connects immersion, intellectual/emotional recall, empathy, and behavioral change seems promising.

Summary and Areas for Future Development

While our team’s initial applications of our cine-VR techniques were focused on improving health care outcomes through soft skill training, we believe that cine-VR is a uniquely capable training tool for almost any type of interpersonal skills. The value can be especially impactful if the topic requires high-stakes, low-frequency interaction such as difficult conversations, workplace violence, or de-escalation scenarios. We have also utilized cine-VR techniques for a concept called Tranquil VR, which leverages stunning imagery from locations such as nature preserves, zoos, or symphonies to provide respite or diversion for providers and patients. A pilot study completed in collaboration with OhioHealth, a health care system, showed that Tranquil VR “was effective in reducing subjective stress among frontline health care workers in the short-term.” ¹⁴ We continue our research in this area with active studies, both provider and patient facing.

Tracking eye movements can enhance user experience and data collection. Eye tracking in VR helps assess areas of a cine-VR experience that draw users’ attention. To accomplish this, eye-tracking headsets use cameras to record pupil movements. These cameras allow researchers to know exactly where a user looks and for how long. Tracking eyes can also be used for input controls. By focusing their gaze, users can launch an application, navigate to a specific area, or highlight specific objects, streamlining the user experience. Our team has used HMDs with eye-tracking capabilities to observe what medical residents observe in an active trauma bay. ¹⁵ Eye-tracking-capable headsets are presently more expensive, but most VR manufacturers will likely make this a standard feature in the next couple of years.

Interactive elements can be added to cine-VR experiences as well. One example includes selecting multiple-choice buttons to determine which content is next loaded. Another example uses a VR controller to identify items of importance. A third option would be to combine interactive choices with subconscious decisions illustrated through eye-tracking technologies. Developing interactive cine-VR content is not without drawbacks and is often more expensive. In our productions, the development process centers around using the Unity real-time development platform, a ubiquitous software tool predominantly used for video game development. Optimizing 360° videos on platforms like this can be challenging at the present time. A common result is that the video quality has a lower resolution than more passive cine-VR experiences, which can negatively impact participant immersion and presence.

As the tools to create cine-VR experiences have matured and distribution costs have dropped, cine-VR has become more mainstream for training. For example, as of September 2022, Walmart has used cine-VR (and VR proper) to train more than 1 million employees—from operating specialized warehouse equipment to mentally preparing associates to serve customers. ¹⁶ Large-scale implementations such as this should further drive hardware costs down and spur development of software for content creation and dissemination.

Professionals across all fields are accustomed to investing years building soft skills such as communication, problem-solving, teamwork, and interpersonal skills which are difficult to hone without real-life experience. What we are seeing with our applications of cine-VR suggests that the tools and our techniques are at a point where it is appropriate to consider how closely we can get to achieving on the job experiences from the comfort and safety of an HMD. We are still in the early days of discovering what is possible with VR technology and we invite you to explore the true potential of cine-VR.