Virtual reality as a tool supporting self-motivated training for disability support workers

Abstract

Objectives

Disability support workers play a critical role in supporting people living with disabilities, but may have limited formal training as time, resources, and funding constraints can make access to training difficult for this workforce. To prepare them for challenging work situations, virtual reality (VR) offers a safe, self-paced, and practice-based way to do self-motivated training to learn and practice relevant workplace skills. The aim of this study was to identify those aspects of a VR training experience that engages disability support workers, motivating them to choose it to build the skills and capabilities needed when working with individuals with disability who require positive behavior support (PBS).

Methods

To understand what creates an engaging user experience for disability support workers, we studied two virtual learning environments in use, both designed to teach PBS skills, with 21 health industry students and 29 employed disability support workers. Observations, a survey, and structured interviews were used to measure user experience and collect user perceptions.

Results

Analysis of collected data identified four key aspects affecting user experience in virtual learning environments for this particular cohort, narrative, environment, interaction, and immersion, finding that narrative, both relevance of the storyline and empathy with the virtual character, was key to engaging disability support workers in the VR training.

Conclusions

VR can be an effective tool for motivating disability support workers to do personal training if it is designed so that they find the virtual client engaging and the storyline meaningful and relevant to their workplace experiences.

Introduction

In Australia, disability support workers (DSWs) are frontline workers who play a critical role in our society. They not only provide support to some of the most vulnerable people in our community who are living with physical and/or mental health issues, but they can empower their clients to have a better quality of life. Training these health workers on providing effective and quality support is an important component of achieving improved health for people with disabilities. ¹ However, the lack of formal qualifications required to enter this sector, and the time, resources, and funding constraints experienced by DSWs once employed, can make access to training difficult for this workforce. This motivates the importance of combined workplace and self-motivated training to increase skills and attributes for these workers. ² To do this, organizations such aso Scope Australia: Disability Support Service Provider, one of Australia's leading providers of disability support services, are turning to virtual reality (VR) to supplement the periodic face-to-face training programs that they offer their workers.

DSWs have complex responsibilities, changing and challenging workplaces and clients. They are often doing shift work or part-time work, which can be physically and mentally taxing. To combat these challenges, Scope would like them to use the virtual learning environments to upskill, refresh their knowledge and practice alternative strategies, both as a component of their formal training sessions and in their own time. However, most of our participant DSWs reported having little or no prior experience of VR. This suggests that if we want DSWs to be motivated to use virtual learning environments, they need to be designed to be engaging, informative, and easy to use, so that they provide a positive experience for this user group. We need to understand what is required to engage this cohort, beyond existing usability focus on the realism of the virtual environment, appropriate interaction styles, and the levels of immersion within the virtual learning environment. Limited prior research mentions the importance of narrative, but existing user experience metrics for VR do not measure this. This presented a research gap, understanding the role of narrative in VR as engaging for DSWs, for us to explore. Hence, our research was guided by the following question: What aspects of the user experience in a virtual learning environment are key to engaging disability support workers to use the system?

Our aim was to create a VR experience that would motivate DSWs to enhance their skill set with positive behavior support (PBS) strategies for dealing with clients in a safe and repeatable virtual space. The DSWs need to be able to apply PBS skills to diffuse potentially threatening situations and dissipate anger and frustration in their clients in situations where this might happen. To gain these skills, DSWs need to practice applying them in realistic situations. VR supports this kind of practice-based learning. As this training can be in addition to their usual workplace commitments, the experience needs to have clear benefits for them.

To explore this, we designed and developed two virtual learning environments, the VR systems Sophie Social and Sophie Escape, purpose built to practice PBS skills. We then evaluated them firstly with health industry vocational education students and secondly with employed DSWs. Our results were drawn from observations, surveys, and interviews. The findings indicate that user engagement for DSWs is not dependent on the technical fidelity of the virtual world nor ease of interaction. While these aspects are usually understood to be important influencers of user experience, ³ we found that it actually depends on whether users find the virtual client engaging, the storyline is meaningful and relevant to their workplace experiences.

We contribute new knowledge by identifying vital aspects of user experience that are necessary to turn a virtual learning environment into an effective tool for supporting self-motivated training for DSWs. Our findings are summarized in a design framework for creating motivating virtual learning environments for DSWs. This framework encapsulates the expertise, perceptions, and priorities of this niche, and yet very important, user group. It represents key aspects of narrative to be designed for, as well as identifying aspects of environment, interaction, and immersion relevant to supporting a positive user experience for this cohort.

Background

Due to factors such as population growth, life expectancy increase, and changing social expectations, policies and training practices for DSWs in Australia have failed to keep up with community needs. The DSW sector is experiencing a growing demand for workers. At the same time, this work force has a relatively high turnover rate and is experiencing changes in employment trends, with the most common forms of employment being part-time (43%) and casual (41%). ⁴ In their work, DSWs support clients through a variety of ways that can vary from low to high intensity, including cleaning, personal care, cooking, community programs, friendship, and emotional and physical support. ⁵ There are no required certifications for DSWs; however, they usually have a certificate qualification obtained from vocational education and training providers.^5,6 The government is aware that as a result, some members of the disability support workforce might lack understanding of disability and the capacity to treat people with dignity, respect, and empathy. ⁵ This lack of formal qualifications reinforces the importance of both workplace and personal training to increase skills. To achieve this, availability, efficiency, and quality of training options need to be improved. To this end, VR offers a solution. However, VR is an emerging technology, and as a result, many people, regardless of age or gender, may be unfamiliar with its use.

To upskill the current workforce, disability support providers in Australia are currently exploring VR as a supplement to traditional face-to-face training. Virtual learning environments are being created by VR developers working with educators and university researchers to create VR training systems, as it is hoped that they will facilitate workers practicing and upskilling in their own time and at their own pace. To achieve this, DSWs need to want to use these systems. However, there has been very little research into technology acceptance by this user group, and there is currently limited input from the end users (DSWs) around designing the kind of virtual experience that would make them engaged and motivated to use it. If these already busy people are going to find the time and inclination to invest in learning how to use virtual learning environments, the systems must be useable, engaging, and provide a relevant learning experience.

VR in training

VR technology is now being used in many learning domains to provide realistic, immersive, and simulated experiences. In an increasingly digital landscape, simulations can improve the skills of workers. ⁷ Much of the current research in VR investigates potential applications. Virtual learning environments are successfully being used in human resources training,^8,9 military training, ¹⁰ and medical education.^11,12 VR offers new opportunities for educators to leverage the benefits of simulated real-world hands-on learning experiences while increasing engagement, motivation, and performance.^13–16 Virtual learning environments are designed specifically to provide experiences that prepare learners to deal effectively with future work situations.

Virtual learning environments make situated experiential education available to a broad range of learners. ¹⁷ Additionally, virtual learning environments can provide online training and complementary learning modes when face-to-face delivery is not feasible. ¹⁸ It can provide simulations of situations and environments that would otherwise be expensive or difficult to access or even situations that are unsafe for learners to practice and learn from their mistakes.

Virtual learning environments

Previous pedagogical research suggests that experiential education is effective for knowledge transfer and improves learning outcomes.^19,20 Immersive virtual learning environments offer potential benefits to adult learners by providing cost-effective and holistic experiences in a real-world simulation^21,22 with opportunities for learning through hands-on experience as well as providing a safe space to practice a broad range of skills. ²³ If designed well, they give an improved learning experience compared to traditional experiential learning such as field trips, role play, and video learning methods. ²⁴ However, to be effective, they need to be both pedagogically sound and provide positive user engagement. ¹⁴

The growing interest in the use of VR for training has inspired efforts to improve the usability and user experience of virtual learning environments. There is an abundance of research on improving usability and technical aspects of VR but very little on improving and evaluating user experience. Examples of usability and user experience metrics for evaluating VR, which are primarily based on studies evaluating 2D digital interfaces, include Nielsen and Molich, ²⁵ Norman, ²⁶ Rubin and Chisnell, ²⁷ and Tullis and Albert. ²⁸ However, more recent studies have investigated user experience when interacting with the 3D digital interfaces of virtual environments, such as Murtza et al. ²⁹ in 2017 and Tcha-Tokey et al. ³⁰ in 2016.

Evaluating usability and user experience in VR

Usability evaluation is a key phase in the process of designing interactive systems.^31–33 Usability refers to how successfully a user can accomplish their specific goal within a system. Since the mid-1990s, evaluations of digital products and services have gone beyond the purely functional, to include assessment of a user's emotional experience of them. ³⁴ User experience encompasses a person's satisfaction, enjoyment, and engagement with a system. This emphasis on the importance of user experience led to a broadening of the heuristics and metrics used in evaluations beyond efficiency and effectiveness, to include nonfunctional aspects of the experience such as satisfying, engaging, and enjoyable. ³³

Although traditional usability evaluation methods are suitable for assessing functional aspects of VR, they usually do not extend to identifying content and context-specific nuances of the user experience when interacting within a VR environment. While there is limited research that includes consideration of user experience within VR, heuristics have been developed based on understanding user perceptions of immersion, ³⁵ presence,^17,36 and flow. ³⁷ Immersion, presence, and flow have been identified as important aspects of user engagement in VR. ³⁸ Pillai and Verma ³⁹ found that users empathize with the characters in situations when there is confirmed presence and immersion in the VR experience. It therefore follows that to identify additional engaging aspects of user experience of virtual learning environments for a particular user group, we first need to establish that the system being evaluated has high levels of immersion, presence, and flow in the experience.

Engagement in virtual learning environments

The design of interactive systems is done in an iterative manner and is influenced by those aspects of usability and user experience traditionally measured in evaluations of a system. Iterative design is a methodology where the designer cycles through processes of prototyping, evaluating, and refining a system, using feedback from the evaluations to improve the usability and user experience of the next iteration of the system.

Engagement in learning environments has been linked to narrative and suspension of disbelief. ³⁸ Immersive and interactive environments facilitate learning through giving the user a sense of presence and agency. ⁴⁰ Irshad and Perkis ⁴¹ and Naul and Lui ⁴ confirm that interactive narrative experiences increase learner engagement. It has also been found that immersion can be achieved in interactive systems when the user becomes involved with the plot and demonstrates an emotional response to the character and storyline. ⁴² As existing tools for measuring user experience in VR tend not to include measurement of narrative experience, it is likely that narrative as a design intention is currently being overlooked in the design of virtual learning environments.

Method

The aim of this study was to identify those aspects of a VR training experience that engages DSWs, motivating them to choose it to build and practice the skills and capabilities needed when working with individuals with disability who require PBS. To design VR programs that facilitate knowledge transfer to real-world situations, we needed to understand the usability and user experience of VR programs to be used in this context.

To address our research question, we conducted a user evaluation of the two VR programs, Sophie Social and Sophie Escape, which were developed as part of a larger project in collaboration between Swinburne University of Technology, Melbourne, Australia and Scope to augment their face-to-face classroom training with VR-based PBS training. There was an additional expectation from Scope that DSWs would be able to practice their PBS skills using these VR programs in their own time and at their own pace, with VR equipment that could be used at home. It was therefore important that we understood the usability of the system as well as the user experience.

The team authoring this paper is focused on understanding user experience of the VR programs. They include a professor of human-computer interaction (HCI), two professors in transformative media technologies, a professor in allied health, a research fellow with a PhD in design and HCI, and a PhD candidate in design and VR. The three professors are females, and one professor and the PhD candidate are male. User studies and workshops were conducted by the four professors and the research fellow, with assistance from additional members of the larger project research team.

Participants in the study

User studies were conducted over 6 months, with two different cohorts. The two cohorts were, firstly, tertiary level students, Study 1, to get the perceptions of people invested in, but new to the field of healthcare support work. The secondly cohort, Study 2, were active DSWs, employed by Scope. These two cohorts were chosen so we could evaluate the system with respect to different levels of experience, those training and those already working in disability support. We were also interested in age difference of the cohorts, to see how the students, mean age 31, youngest 18, who claimed to be familiar with VR, measured against the face-to-face DSWs, mean age 39, youngest 26, who claimed little to no experience with VR. We did not use inclusion/exclusion criteria for participants, but sent emails to all students enrolled in the relevant educations and 150 DSWs identified by Scope and took all who volunteered. As researchers, we had no relationship with participants prior to the study.

Study 1 comprised 26 students commencing a vocational education in either Certificate III in Individual Support (Ageing, Home and Community) or in an allied health degree. The study took place at Swinburne University campus, in between other study activities. This was in March 2021.

Study 2 comprised 30 DSWs employed by Scope. With 2110 dedicated DSWs, therapists, coordinators, and business support staff reported by Scope Australia in 2021 and 150 DSWs directly approached and encouraged to participate in this project, 30 participants were lower than anticipated. Study 2 participants experienced the VR programs as part of a 6-week training course, away from their workplace. Due to the COVID-19 pandemic lockdowns, seven participants from study 2 completed the evaluation online, S2 (online), using Zoom on their home computers and a VR headset shipped to them a week before the evaluation. They were encouraged to use the off-the-shelf VR training program to familiarize with the controls. The other 23 DSWs attended Scope’s training venue to participate in a face-to-face evaluation, where a session was dedicated to use of the VR programs. This was from July to August 2021.

Table 1 shows the demographic distributions for each cohort. Of the total 56 participants, six did not complete the evaluation but withdrew at various points due to simulation sickness. If participants alerted the tester to feeling unwell or dizzy, they were instructed/helped to remove the headset immediately, retire from the evaluation, and monitored until they felt better. Their responses did not form part of the qualitative data set.

Table 1.

Demographic distributions for each user study cohort.

Cohort	Gender		Age			DNC a
	Female	Male	Range	Mean	SD
S1	19	7	18–67	30.96	14	5
S2 (online)	4	3	22–46	32.3	8.56	0
S2 (ftf)	8	15	26–57	39.91	8.63	1

^a

DNC is number of participants who “did not complete” the study due to simulation sickness.

Study materials

The Sophie Social and Sophie Escape virtual learning environments

The two immersive virtual learning environments used in this research were Sophie Social and Sophie Escape. They were developed as part of a wider research collaboration between Scope Australia and Swinburne University to deliver educationally sound VR programs for DSWs to practice their PBS skills on a virtual client. PBS involves a support worker making changes to a client's environment and building their personal skills to improve the quality of their life and the environment in which they live, to reduce the likelihood of behaviors of concern or challenging behaviors, and to reduce and eliminate the need for restrictive practices. ⁴³ The PBS educational specialists from Scope developed two scenarios covering key PBS learning points designed for DSWs to practice PBS skills in realistic home care situations.

Sophie Social addresses the need to ensure clients are included in conversations and are not ignored or excluded. In this scenario, Sophie is autistic, nonverbal and has a moderate intellectual disability and is not easily understood when she is trying to communicate with others. Sophie Escape addresses the need to allow time and space for Sophie to deal with information she is being given, and when this becomes too much at a given time, the DSW needs to manage the task demand overload.

These scenarios were developed in a design workshop involving the PBS education specialists, teachers, and administrators from Scope, alongside healthcare lecturers, digital media specialists, VR developers, and interaction designers from Swinburne University. The VR developers then took the outcomes from the design workshop and created the two VR programs, to run on Oculus Quest head-mounted displays tethered to a computer. The VR development team worked closely with PBS experts and a professional simulation actor to develop an effective performance style for Sophie, which was then motion captured and mapped onto a virtual character generator. The virtual client was programmed using Sophie's character profile to ensure that her appearance and performance accurately captured her mood and enhanced the believability of the virtual experience. This character profile for Sophie is provided to users before the VR experience, as a paper artifact covering her likes and dislikes, potential trigger points, and behaviors and mannerisms, including nonverbal vocalization style.

Both programs lead the learner through a series of unfolding events. In Sophie Social, she is trying to communicate with the user by indicating a magazine on her communication book (see Figure 1(a)). This results in a PBS decision point where the system presents the user with three choices via a visual menu. Each menu is mapped to a key PBS learning point representing effective or ineffective choices, resulting in Sophie being “more” or “less” supported. In this case, the three user responses to choose from are: “Would you like to read a magazine together?,” OR “I’ll read a magazine with you once dinner is prepared,” OR “You look bored. Would you like something to do?” (see Figure 1(b)). The user then selects the response they believe will help Sophie. If the user selects the option to read a magazine together, then they have further responses available. In this case commenting on the magazine article saying: “Wow, she looks beautiful,” OR “Which is your favourite part Sophie,” OR they can use the red button to close the dialogue box, thus not making a comment (see Figure 1(c)). At each of these PBS decision points, if an effective choice is made, Sophie is helped. If an ineffective choice is made, the situation might escalate, with Sophie becoming frustrated and potentially resulting in violent behaviors. At the end of the program, a feedback code is provided, in this case “A-L23,” so the learner can revisit their decisions and the outcomes and discuss this with their instructor (see Figure 1(d)).

Figure 1.

Scenes from Sophie Social VLE. (a) Sophie tries to communicate using her communication book, showing the environment of Sophie's home; (b) user has three available responses to Sophie, showing the interaction mechanism used in the VR system; (c) user and Sophie read a magazine together and user can comment or close the dialogue box, showing the narrative choice of the responses available and engagement with Sophie; (d) a feedback code is shown at end of the program giving user access to choices and outcomes, allowing them to reflect on and learn from their ability to apply relevant PBS strategies.

Pilot evaluation with research team members

Sophie Social was developed first, and four members of the research team, including health experts, educators, and interaction design experts, were invited to do a heuristic evaluation of the program using Nielsen's 10 Usability Heuristics, which are: (1) Visibility of System Status, (2) Match Between the System and the Real World, (3) User Control and Freedom, (4) Consistency and Standards, (5) Error Prevention, (6) Recognition Rather than Recall, (7) Flexibility and Efficiency of Use, (8) Aesthetic and Minimalist Design, (9) Help Users Recognize, Diagnose, and Recover from Errors, and (10) Help and Documentation. ⁴⁴

This pilot evaluation resulted in a report to the developers summarizing design changes and identified usability problems which were fed into the next iteration of development. However, immediately after conducting the evaluations, the two researchers with HCI expertise discussed the usefulness of the heuristics in identifying relevant aspects of the user experience in the VR program. It was decided that for evaluation with future users, aspects specifically related to user experience in VR, such as engagement and immersion, needed to be included in the evaluation tool. It was also decided that there were elements of the scenario design that were not being evaluated by this tool, and that we should find a metric to collect this data.

The user experience evaluation tool

After looking at available tools, we chose the Immersive Virtual Environment Questionnaire (IVEQ) as a validated survey tool, ³⁰ available in the public domain, to measure user experience in VR environments. After email discussions with the authors of the IVEQ, we selected 27 questions from it (9 on presence, 5 on immersion, 3 on engagement, and 10 on flow). These were relevant to our investigation because they were the sections with questions aimed at qualities which can be used to establish levels of user experience, as defined in the literature.^17,35–38 We then added 12 questions of our own to focus on evaluating the learning scenario by asking about aspects of character and story to better suit the content and context of use as practicing PBS skills for DSWs. The need for understanding the contribution of narrative to user experience was both inspired by literature surveyed as well as outcomes from the design workshop where the scenarios were developed and tested. The structure and wording of these questions were inspired by related questions in the NASA TLX, ⁴⁵ the PQ, ³⁵ the CSUQ, ⁴⁶ and work by Slater and Wilbur ³⁶ around importance of plot. These additional questions were not empirically validated, as they were used primarily to interpret the qualitative data. However, they were pilot tested with members of the broader research team in a codesign session. We call this our User Experience for Virtual Learning Environments (UX4VLE) online survey, as it includes questions relevant to learning environments. These 39 questions are listed in Appendix 1.

Study process

The aim of the research was to identify key aspects of the virtual learning experience influencing user engagement for DSWs. Our data was drawn from participant observations, evaluator note taking, audio recordings, and structured interview questions. These were not tracked and attributed to specific participants during the study. Participants were invited to “think aloud” ⁴⁷ while using the VR program so that we could access their decision-making and reactions during the evaluation. After using the virtual learning environments, participants were asked to complete the UX4VLE online survey. This study was approved by the Swinburne University ethics committee.

Study 1

Study 1 was held with Swinburne University students. Three researchers tested individual participants at our campus. A recruitment notice and a project information statement were distributed to potential students, introducing the project, research team, and explaining the goals and reason for the research. All participation was voluntary. The evaluation session was designed to take between 45 and 60 min.

At the start of the session, a researcher met with the participant and introduced them to the project, welcoming them and explaining the process. While gaining informed consent, we ensured they understood there would not be a problem if they wanted to stop at any time. Participants filled in an online pretest questionnaire including general demographics, attitude toward computers, previous experience with disability, attitude toward simulation training, and the Simulation Sickness Questionnaire. ⁴⁸

After donning the headset, participants completed a tutorial to familiarize with VR interaction modes and a second tutorial for interactions specific to our virtual learning environments. Participants first experienced Sophie Social, during which time researchers made observational notes. These sessions were not audio or video recorded as participants did not consent to this. After using Sophie Social twice, they were asked:

“What was good about the experience?”

“What was bad about the experience?”

Participants then used Sophie Escape twice and were asked the same questions afterwards. They then completed the UX4VLE posttest online survey, were thanked and given a $20 gift card.

Study 2

Study 2 was conducted with employed DSWs. There were two different procedures in this study, both using the same introduction script as study 1. For seven participants, it was conducted entirely online, study 2 (online), requiring them to practice by themselves in the week leading up to the evaluation. Time spent online was between 45 and 60 min. The other 23 participants, study 2 (ftf), were given a 30-min training session on using VR by a Scope employee, immediately prior to the evaluation. The total time duration for these participants was between 75 and 90 min.

In study 2 (online), the researcher made notes of issues on use, experience, and usability of the system as they remotely watched the participant using the VR system through a Zoom connection. The evaluation was video recorded if consent was given—one participant did not consent. After using Sophie Social twice, participants were asked the same two questions as in study 1. They then used Sophie Escape twice and were asked the same questions. At the end, they completed the UX4VLE posttest online survey and were thanked.

Due to time limitations on the training day, the 23 face-to-face DSWs were only able to use Sophie Social twice, and did not experience Sophie Escape, but were given an additional question:

• 3.

“What did you do differently the second time?”

This group was not able to “think aloud” as there were several testing stations in the same room. At the end of the evaluation, they completed the posttest UX4VLE online survey and were thanked. For all DSWs, time spent participating in the study could be claimed as work hours.

Findings

Qualitative data from study 1 and study 2 were analyzed using reflexive thematic analysis (RTA) ⁴⁹ for initial coding using an Excel spreadsheet and affinity diagramming ⁵⁰ for final clustering. We used an inductive approach for each new data set, looking for recurring patterns and codes in the data, without an initial code set, working within the six phases of analysis from Braun et al. This involved first familiarizing with the data set, then generating codes and initial themes, then creating categories to construct themes. The data from study 1 was analyzed first, then the data from study 2. The codes and initial themes from both were then combined using affinity diagramming by two members of the research team to collaboratively define, refine, and revise categories and synthesize themes. We did not specifically take note of the point of data saturation but believe our results to be valid based on our professional expertise in interpreting the data. According to Braun and Clarke, ⁵¹ in using RTA, the researcher needs to dwell with uncertainty, and therefore, judgements about how many data items or when data saturation has been reached with respect to when to stop data collection are not consistent with the values and assumptions of RTA. We are therefore confident that the limited numbers of participants did not unduly affect the insights provided by our analysis.

Quantitative support for engaging experience

The quantitative data from the posttest UX4VLE online survey confirmed that the user experience was high quality with respect to presence, immersion, engagement, and flow. Overall, ratings supporting a good user experience were higher in study 1 than study 2, and the study 2 (online) cohort was the lowest, although all were still positive (>5/10). For study 2 (online), COVID-19 rules meant that people were in enforced isolation, they were pressured by additional constraints and protocols at work and needed to provide additional support to clients in these extreme and limiting circumstances. We speculate that this might have caused more negative responses to the VR experience and was not necessarily issues with the VR. We therefore cannot draw any conclusions about design implications for the system based on this online delivery. We have reported separately on the study 2 (online) and study 2 (ftf) groups in Table 2 so that we can see the effect of doing the study online. Statistical significance of the sample size was not calculated on this data; therefore, we use it as comparative data and indicative of sufficient levels of presence and immersion for interpreting the qualitative responses.

Table 2.

User experience measures—overall scores from adapted IVEQ for the 10-point rating scale used.

	S1		S2 (online)		S2 (ftf)
	Mean	SD	Mean	SD	Mean	SD
Overall perceived level of presence	8.2	0.48	7	0.8	7.42	2.02
Overall sense of immersion	8	0.39	6.5	0.4	7.75	2.08
Overall sense of engagement	8.43	0.33	6.9	0.6	8.13	1.87
Overall feeling of flow	7.2	0.48	6.1	0.6	6.75	1.64

Qualitative results leading to new understanding

Responding to the interview questions helped participants identify and reflect on both positive and negative aspects of the experience. Analysis resulted in 12 categories: character, storyline, progression, realism, comfort, movement, mechanisms, feedback, control, presence, engagement, and flow (see Table 3).

Table 3.

Categories induced in the RTA of collected data.

Narrative	Character (virtual client) Storyline (tasks, scenario) Progression (progress, paths)
Environment	Realism (matching real world) Comfort (physical responses) Movement (navigation)
Interaction	Mechanisms (easy to use, intuitive) Feedback (instructions, responses) Control (user in control)
Immersion	Presence (sense of being there) Engagement (focused) Flow (losing sense of time)

These were then grouped into four overall themes: narrative, environment, interaction, and immersion. These themes represent key considerations in the iterative design and evaluation of user experience in virtual learning environments in the context of DSWs practicing PBS skills.

This is represented in our UX4VLE design framework shown in Figure 2.

Figure 2.

Empirically derived UX4VLE design framework for designing and evaluating user experience in virtual learning environments for DSWs.

The themes of environment, interaction, and immersion cover qualities of the user experience that are well documented and measured by existing heuristics and tools. Environment is related to the realism of the virtual world to the user, how comfortable they felt in the environment and how they navigated and moved around the space. Interaction is about the ease of use of interaction mechanisms, the feedback provided to the user in terms of instructions from the system and user responses, and how the in control the user felt in the virtual world. Immersion relates to the sense of being present in the virtual world, how engaged they felt with what was happening and whether they experienced flow by being so involved they lost sense of time.

However, our research outcomes indicate a new emphasis on the importance of narrative and the extent of its influence on user experience in a virtual learning context. This has implications for future evaluation of VR that is designed for teaching and learning. Narrative needs to be both designed for and evaluated in assessing user experience of virtual learning environments aimed at people working in the situations being simulated.

The following gives descriptions of the categories that contribute to narrative in the user experience: Character, Storyline, and Progression. Unidentified quotes are included as examples, simply to illustrate the concepts.

Character is the personality, reactions, and behavior of the avatar in the virtual world.

In our study, participants watched Sophie and interpreted her visual representation to understand her emotional state. They felt compelled to interact with Sophie and support her, saying, for example, “I wanted to, obviously, um, you know, keep Sophie happy.” They considered the different options and experimented with different ways of communicating, including moving nearer her, saying such things as, “while I was answering, I wanted to go to Sophie.” Overall, they enjoyed interacting with Sophie, actively trying to understand her needs or test different strategies to see if they could manage the situation, evidenced by comments such as, “Yeah, I’d say that you’ve triggered her, but it’s okay. You’ve got an opportunity to de-escalate or increase.” Participants identified the mood and emotions of the character by inferring things from her actions, such as, “So she reacted differently, she kind of walked away,” or noting the character calmed down because she was clapping her hands. One participant assigned traits, saying, “Sophie prefers to guess instead of asking.” Over time, they built a relationship with Sophie, showing concern for her. Many found this easier the second time through the scenario, saying things like, “the second time l chose different conversations and focused on trying to meet Sophie’s needs to support her and understand her better.” Interestingly, some of the DSWs used the opportunity of a second run to provoke the character into threatening behaviors—not usually something they could do at work, saying, “I performed actions that I would not generally do to see how Sophie would react differently” and, as one participant admitted, “I tried to trigger Sophie; I like that it gave me a second chance to de-escalate her.”

Storyline delivers the learning objectives of the scenario. The storyline needs to draw the user into practicing their PBS skills and the level of complexity needs to adapt to match the problem-solving competencies of the learner. The storyline also needs to engage the users in the learning process by being meaningful and aligned with the real work context.

In our study, several participants commented on the storyline being both realistic and aligning with their work experience, saying things like, “It showed similar scenario as in work place, the behavior, anxiety, issues,” and also, “the scenario was what/how work environment worked,” including “the part where we have to involve Sophie is something that I would also do at work.” On the other hand, some found the available responses unrealistic, as one participant with work experience noted “some of the things are just not really what you would say, or how you would go about it”. They also needed a bigger choice of actions and responses than was provided in the VR program, stating that within the given options, “Once Sophie becomes escalated there isn’t much we can do to de-escalate”.

The storyline introduced tasks within the care home environment need to be performed as part of caring for the client. These tasks were added to align with the real work situation, where the attention and effort they take add to the complexity of caring for Sophie. Our scenarios include things like cooking dinner, pouring drinks, or turning televisions on or off. These tasks were either scheduled in the storyline or done in response to a client need and could trigger reactions in the client which then needed to be dealt with.

In our study, participants found the tasks too time consuming in the virtual world and said they felt pointless and slowed them down. In Sophie Social, people found cooking dinner difficult, time consuming, and unrealistic, saying things like, “trying to prepare dinner…it's not very fun” and “the food preparation was too long.” The scenario also required the user to control the television in response to Sophie's needs. Although the DSWs confirmed that they would usually turn a television on to keep their client occupied while cooking dinner, turning it on required such a complex interaction that it took their attention away from their client, as in, “I was trying to turn the TV on, because I can’t be doing two things at once, looking after her and cooking as well.”

Progression in the virtual environment needs to happen at a pace that matches the real world, as well as giving the user the sense that they are achieving their aims. The path of the progression is guided by instructions in text boxes in the virtual world, and the user needs to select next steps using VR controllers, to move through the scenario to achieve an outcome. Depending on choices made, paths through the virtual environment can differ each time the users go through it. The path can be predetermined by instructions given to the user or variable when there are alternative user choices that can be made. These choices need to be noticeable, easy to activate, and understandable with respect to where it might lead. Alternative paths provide different consequences and outcomes, giving users the opportunity to practice by doing and to try out different skills and strategies; however, clear feedback on consequences and outcomes are necessary for improving their skills.

In our study, while some participants felt that progression improved as they got further into the scenario, others noted that at times it was unclear how to go forward, saying things like they were “feeling unsure what to do next” and “a bit confused. It wasn’t clear what I was actually meant to do.” Generally, participant's ability to progress through the program appeared to improve with experience in the virtual environment, although several had troubles identifying the end point. The DSWs were most interested in trying alternative paths, for example, saying, “I became better with navigating and making different choices for a different outcome.” The ability to take different paths was seen as “inviting, exciting and certainly kept me engaged and thinking about what to do and what best choice could be made in the scenarios.” Some commented on the options, saying it was “limited about what you can do or what better approaches you could implement” and “always having laid down options removes thinking outside the box, as we are only limited to one generic answer.” Most participants explored different options on their second run through, saying, “I just decided to just do something differently … where I didn’t in the first one.” Sometimes, paths were explored even if they knew that it might lead to problems, saying, “I chose different options just to see if it would lead me somewhere else.” They saw this as an opportunity they did not have in the workplace, the opportunity to get things wrong and practice how to recover from this. Mostly they liked the ability to practice their skills and strategies, saying, “in educational purposes you can actually play out what you would do or not do and see the difference in your approach…you can go back and learn from your mistakes.”

Discussion

Our evaluation provides insights into key aspects of virtual learning environments that influence engagement for DSWs. We identified the importance of narrative for caring about Sophie, having agency in how the storyline progresses and the ability to activate different learning situations. From literature, we know that a quality user experience supports an effective learning experience, ¹⁴ and a positive user experience requires high levels of presence, immersion, and flow. ³⁷ Hence, we used the quantitative data from the UX4VLE online survey to confirm this. In this survey, participants reported feeling present, immersed, and engaged within the simulation, finding the virtual environment a realistic representation of their working context.

We then used qualitative data from observations, notes, and interviews to understand which aspects of the experience were especially engaging. The key contribution of our UX4VLE design framework (Figure 2) is in including aspects of narrative that engage DSWs in virtual learning environments and the impact of character, storyline, and progression on engagement.

Our literature review showed that narrative was only briefly explored in previous work looking at user experience in VR. The importance of plot is mentioned by Slater and Wilbur. ³⁶ Gorini ⁵² claims that a simulated real-world equivalent for training people to operate effectively in the real world needs to provide sufficient levels of immersion and presence as well as be supported by a narrative. However, we found that current tools for measuring user experience in VR do little to assess narrative-related aspects. We argue that questions on narrative need to be added to these tools when looking at VR for learning, especially in contexts where empathy and caring for the virtual characters is important.

Overall, our findings illustrate how much emphasis our participants put on caring for Sophie despite the difficulties they were having controlling and interacting with the VR system. Therefore, narrative design needs to be prioritized in these kinds of VLEs, although this process of realization is complex in VR since character, performance style, and physical arrangement of virtual characters and users, which are key to experiencing and interpreting narrative, are created digitally.

Contrasting student and worker experiences

Our design framework, UX4VLE, is derived from both student healthcare workers and DSWs, and we found identifiable differences between these two cohorts.

Firstly, it needs to be noted that for the students, the testing took place during a typical learning day, where they understood that participating in our evaluation was in no way associated with their grading. For the DSWs, it took place outside of their usual workday, in the place where they did periodic training with their employer, as an addition to the usual face-to-face training. As an exception, the Study 2 (online) group had to do the evaluation in their own homes. All DSWs were reassured that their individual performance and feedback would be completely anonymized for their employer. Even so, some still expressed the feeling they were being tested.

For the healthcare worker students (who had mean age of 31), the experience of VR was not a novelty. They were familiar with trying out new technologies and found it a very enjoyable learning context. They embraced VR as a new way of learning. Alternatively, for most DSWs (who had a mean age of 40), VR was a very new experience. Many had problems with interaction mechanisms and the physical control device. Interestingly, they still reported a high level of enjoyment in using the virtual learning environments. In particular, they embraced the character of Sophie as someone they wanted to care for, despite issues and difficulties using the technology. Sometimes they even apologized to Sophie for being distracted by trying to control the system.

Importantly, DSWs stressed that it was vital that the story covered realistic issues and character behaviors for it to be a suitable simulation of the work situations they were likely to find themselves in. What surfaced strongly with the DSW cohort was a deeper knowledge of actual workplace issues and concerns, as well as reflections on where the scenarios aligned well with actual work issues and where it provided advantages such as practical and safe opportunities to try out alternative PBS strategies on Sophie. On the other hand, the student cohort was not worried about how things might work in practice and saw it as a gamified learning exercise.

Overall, both cohorts enjoyed the simulation, saying they preferred it to other self-driven learning methods, such as textbook learning, that did not illustrate so clearly the real-life situations they were likely to encounter. Both groups viewed this as a useful addition to their usual face-to-face learning and training methods—but not as a replacement.

The value of the design framework

As a group, DSWs are stressed and time poor for trying to upskill and gain additional learning; however, virtual learning environments offer a viable solution. VR can be used outside of work time, within working hours allocated for training or as an integrated part of periodic face-to-face training sessions. The importance of the UX4VLE design framework is that it highlights those aspects of the virtual learning environment that contribute to a positive and engaging user experience for these workers, with the intention of supporting them in self-motivated learning and practice using VR, away from formal training sessions.

Virtual learning environments gain value when they facilitate learning skills that can be directly applied in the workplace while providing an experience that feels useful and enjoyable. VR training provides learning in a way that is seen as practical for developing and practicing skills while allowing learners to enjoy themselves. These virtual learning environments represent a safe space for testing different PBS approaches without causing harm to themselves or their clients.

The design framework provides understanding of the relationship between the narrative inherent in learning scenarios and the user experience of an immersive simulated realistic work environment that they need to navigate and the interactions needed to use and control a VR program. As part of an iterative development process, this knowledge is important in both design and evaluation phases if the outcome is to be an engaging virtual learning environment.

Limitations of the study

This research was limited by several factors. Technology challenges were experienced by some participants due to unfamiliarity with technology. These challenges would improve over time, and for some, the difficulty of operating the system unaided in their own homes may have adversely affected their opinions about the VR content. This included difficulty with using current interaction mechanisms, such as handheld controllers, which may well move to more natural interaction mechanisms, such as gesture or speech, in the future.

Simulation sickness, often experienced with VR, caused six of our participants to withdraw from the study and therefore is an important consideration in using VR as an education tool that we did not address in our measurements. Additionally, our small sample size limits our generalizability of findings. This was partly due to COVID-19 restrictions and the enforced isolation, but we were also limited to those who volunteered to participate within the student groups and DSW workers associated with our research partners as part of the funding and ethics requirements of the project.

As there was no existing validated questionnaire that included aspects of narrative, such as character and storyline, we developed our own 12 questions in this area to add to the validated user experience questionnaire sections. Although developed through a design workshop, literature review, and pilot testing with the research team, it was not validated, which does limit the reliability of the responses received. However, responses were still able to influence the qualitative findings of this study by giving us insight into narrative aspects of the user experience.

We also regret the lack of DSW involvement in the design of the system. In user-centered design, it is imperative to get user input on their needs and limitations early in system design. Sadly, due to time constraints, we did not engage with the DSWs during design and development. Nor did we have the opportunity for participant checking of our results, relying on management and educational staff at Scope to review, reflect on, and verify our design framework based on their professional experience. This could be extended to opportunities for involvement of clients, which would be important for future iterations of the system to understand what DSWs need to be better at.

Conclusion

DSWs play a critical role supporting people living with disabilities and are currently in such high demand in Australia that they are entering the profession with extremely limited formal training. To ensure a skilled workforce, providers are looking for new ways to rapidly upskill their workers. These providers are exploring VR as a flexible and practice-based way to augment current periodic staff training. In this context, the use of purpose-designed virtual learning environments can provide an engaging simulated real-world experience where users can test their skills and explore alternative PBS strategies without impacting the lives of their clients or putting themselves into tricky and potentially dangerous situations. The flexibility of being able to practice and explore the virtual environment in their own time, and at their own pace, fits within with their current training commitments and their time and workplace limitations.

Our project evaluated the user experience of two purpose-designed VR programs, developed for DSWs to practice PBS skills. Our participants, 21 healthcare students and 29 employed DSWs, were observed using the VR programs and then answered interview questions and filled in our UX4VLE online survey. The survey responses were analyzed to establish an acceptable level of engagement in the VR experience. For the observations, notes, and interviews, we used RTA and discovered the importance of narrative, in particular character, storyline, and progression, as contributing to an engaging user experience for these cohorts.

Future work in this area should look at the effectiveness of using VR for experiential training in the context of caregivers and healthcare professionals generally. The focus on narrative could provide engagement with these similar user groups, where caring and empathy is an important aspect of the training. This could be expanded to different types of disabilities or other sectors of care worker training. With our student cohort, there was a very positive response to VR and interactive scenarios as a future education tool.

Our contribution to new knowledge is the empirically informed UX4VLE design framework which encapsulates the lived experience, perceptions, and priorities of healthcare students and DSWs, forming a foundation for understanding what constitutes an engaging user experience for them. In response to our findings, we argue that narrative needs the same level of focus and consideration, in both design and evaluation of virtual learning environments for DSWs practicing PBS skills, as the traditionally acknowledged aspects of environment, immersion, and interaction. We argue that a well-designed virtual learning environment can support self-motivated practice of PBS skills for DSWs by providing an engaging learning experience, thus contributing to the much-needed rapid upskilling of the current disability support workforce in Australia.