Sage Journals: Discover world-class research

Abstract

Virtual reality (VR) immerses users in others’ lives, creating empathy and understanding long after the VR scenario has finished. As VR technology has matured, VR scenarios have begun to be used in complex real-world areas such as education, health and organisational change. These scenarios can be of variable technical quality, with limited interactive capacity, unrealistic environments and clunky or absent avatars. In this study, three scenarios related to gender inequality training were constructed with glitches in the core immersive qualities of presence, interactivity and plausibility in order to understand their effect on the immersive experience. Using a multi-step in-depth series of qualitative interviews to examine the whole immersive process, the results show that immersion is not compromised but changed by glitches. Limited interactivity led to uncomfortable interactions that allowed participants to process difficult emotions; implausible situations surfaced buried norms and prejudices; and avatar variation gave rise to a sense presence that also included distance, which gave the user opportunities for critical reflection. These results point towards immersion as a robust and richly textured concept, while interactivity, plausibility and presence can best be understood as dimensions rather than goals. Totally seamless and immersive experiences may not only be utopian but also unnecessary. The glitches in low-end productions can produce powerful communication without expensive technology.

Keywords

Virtual reality immersion presence gender interactivity glitches work life plausibility

Introduction

As virtual reality (VR) enters a new space of real-world, everyday applications in areas such as healthcare, organizational change, entertainment, art and education (Crawford-Holland, 2019; Nash, 2018a; Peng, et al., 2010; Walter, et al., 2018), there is a growing interest in critically exploring and analysing virtual media productions (De la Peña et al., 2010; Golding, 2019; Hassan, 2020; Nash, 2018a, 2018b). This paper responds to calls for qualitative studies of VR to understand immersive environments from a user perspective in order to allow both creativity and new technical innovations to flourish (Murray, 2016; Parker and Saker, 2020; Shin and Biocca 2018; Sherman and Craig 2018). In our investigation, we draw on feminist redefinitions of the term ‘glitch’ to examine what we can learn about immersion in situations where users lack (even temporarily) the experience of immersion (normally a highly desirable quality of virtual reality, see Bailenson, 2018). We conducted in-depth semi-structured interviews and a focus group with experts in gender inequality training, investigating their experiences and interpretations of the three deliberately limited VR productions which are part of a wider gender equality change programme.

Immersion refers to the sensation that VR users have of being present in an alternative situation. As defined in Biocca and Levy (1995), and later expanded (c.f. Lévy, 1998, Manovich, 2001; Ryan, 2001; Biocca, et al., 2001), the concept describes a feeling of ‘being there’: the user is surrounded by an alternative reality demanding their full attention (Murray, 1997). VR designers have focused on increasing user agency and on creating intense and lifelike sensory stimuli, so that the user experiences fuller immersion, which in turn has been equated with a better outcome (see for example Ahn et al., 2013; Reeves, et al., 2015; Slater and Wilbur, 1997). This sense of immersion makes VR unique and gives it great potential in storytelling and learning. It is even argued that immersive experiences have the potential to ‘enact profound and lasting changes in us’ (Bailenson 2018: 6). However, users are not always immersed in VR environments – immersion may be compromised (or as we term it, ‘glitched’) due to a variety of factors. A glitch originally referred to an unanticipated absence due to a technological malfunction. A glitch may appear as a surprise, an absence or a mistake that the user notices (cf Menkman 2011). Any glitch becomes part of the user experience, and glitches can be used and interpreted creatively by users. Thus, glitches can be analysed as carriers of meaning, even if the meaning is unintended. In our work, we used deliberate glitches (designed-in limitations) to push for new insights into immersion.

Our research question is ‘How do glitches in immersion affect the qualitative experience of a VR scenario?’

The rest of the paper is structured as follows: first, we engage with the existing theory of immersion, dividing it into three aspects: presence, interactivity and plausibility. Next, we examine the productive capacity of glitches, before introducing the in-depth qualitative study of three VR productions with designed-in limitations that users experienced as a glitch. We then report on and analyse the experience of the users in the limited productions according to the different dimensions of immersion. Finally, in our conclusion, we suggest a conceptualising immersion as a dynamic, using the concepts of present distance, uncomfortable interactions and situational implausibility. We argue that users move in and out of immersion in order to deal with both social and technological glitches.

Elements of immersion

Our definition of VR includes 360-degree video and computer-generated animated productions where the user wears a headset or similar and is able to interact with the environment shown in the headset, using their body in some way to drive the interaction. This definition is both wide enough to encompass variations in technological solutions yet also matches what most people would call VR (cf Rose, 2018).

Immersion is frequently considered central to the VR experience, both by practitioners and scholars. While some consider immersion to be an objective sensory experience created by the headmount display (cf Slater, 2003), most scholars take a broader view. Our analysis of the literature shows that immersion is often divided into three aspects, which co-occur in immersive situations: presence, interactivity and plausibility. The most important aspect of the three is presence, which generally refers to the placement of the virtual body within the representation or narrative (for a historical review of diverse approaches to presence in relation to VR see for example Lee, 2004; Biocca and Levy, 1995; Lévy, 1998; Ryan, 2001). Presence has strong overtones of embodiment (Steuer, 1992), or the illusion of a virtual body being one’s own, an illusion in part created by the first-person perspective in many VR productions (Ratan, 2013). An embodied user at the core of a VR experience becomes part of the mediated landscape and situation to the extent that they will experience real physical reactions such as increased heartbeat or sweaty palms when looking over a cliff in a 3D-animated environment (Blascovich and Bailenson, 2011). Most VR productions strive to create a heightened sense of presence in order to create a powerful mind-body experience. Presence and the sense of embodiment are affected by states of mind (Shin and Ahn 2013; Cummings and Bailenson 2016). For example, emotional states may be contagious in VR – a user who meets an excited persona or other user may themselves become excited (Ahn et al., 2013). Studies report that personal traits such as openness to experience, emotional state and extraversion affect emotional involvement in immersive environments; presence is, at least to some extent, intentional, rather than necessarily increasing with improved technological systems (Shin, 2018; Weibel et al., 2010).

Interactivity refers to the ways the constructed context responds to the actions of the user. By context, we mean the spatial, physical and temporal dimensions of the environment as well as interactions with artefacts or other characters. It is through interactivity that the user is able to explore an environment and/or narrative, to be active and to respond to and be part of a situation; or in other words, to enact their agency (Murray, 1997). It also plays an important part in VR narrative, since narrative can be managed through the ability to reveal insights throughout the course of events and to allow for user interaction. Greater interactivity usually promotes greater identification with a particular perspective and story (Green and Jenkins, 2014; Walter et al., 2018). In addition, work in VR in journalism found that the ability to move through space is central to forming emotional impact (de la Peña et al., 2010). Research in human-computer interaction (HCI) shows that some interactivity can also lead to negative emotional impact (Bopp, et al., 2016; Höök et al., 2016). Benford et al. (2012) use the term ‘uncomfortable interactions’ to refer to interactivity that is deliberately engineered to generate discomfort (characterised by mild anxiety or fear, or ‘a degree of suffering’) in order to create “intense and memorable interactions and engag[e] with dark and challenging themes” (2012: 2005).

Finally, plausibility refers to the reasonableness of the mediated representations (Slater, 2009). In simple terms, does the logic of the narrative and the interaction within that narrative make sense to the user? Plausibility does not necessarily refer to realism, and many VR productions exist in unrealistic environments. Slater (2009) suggests that when the experiences of presence and plausibility coincide, full immersion emerges. However, plausibility is also determined by users’ previous knowledge, understanding of the world and expectations (Lambert-Beatty, 2009). For example, users may expect an able-bodied avatar, and be temporarily confounded by a disabled avatar and designers can use this for effect, for example in the game The Inpatient, in which the player is wheelchair-bound. In earlier studies, negative user bias and stereotyping have been shown to reduce in VR projects that addressed race and age discrimination through the use of dark-skinned and elderly avatars, forcing users to accept a different body as plausible (Banakou et al., 2016; Farmer et al., 2012; Maister, et al., 2013; Peck et al., 2013).

In this paper, our interest is in what happens when immersion is compromised, where presence, interactivity, or plausibility fail in some manner. In order to explore this, we rely on the concept of the glitch. Glitches can be defined as ‘accidental, highly obvious, dysfunctional events in digital systems where something is discernably wrong’, (Lesczyznski, 2020: 196). But from this definition, artists and critical scholars have extended the idea of a glitch or wrongness into both a creative and critical tool as a carrier of meaning which connects technological and social implications (Menkman, 2011). Thus glitch art (see for example Grundell Gachoud, 2020) ‘allows insight beyond the customary, omnipresent and alien computer aesthetics’, (Goriunova and Shulgin (2008: 114), while critical scholars have used the related between the social and technological to approach glitches as an opening to discuss and critique technological norms (Leszczynski, 2020; Sundén, 2015; Warfield, 2016).

Glitches therefore offer an opportunity to expand our understanding of immersion itself. By understanding and analysing the qualitative experience of designed-in limitations to presence, interactivity and plausibility, we also aim to gain an improved understanding of immersive norms. Earlier studies have suggested that violation of social norms causes users to reject or resist immersion. For example, Parker and Saker (2020) concluded that VR productions in art museums violated existing norms of public behaviour and made the experience uncomfortable for users, hindering immersion. In Popat’s (2016) study of unwanted sexual contact while in a virtual body, users withdrew from immersion to protect themselves from unwanted emotional reactions. In a similar vein, Shin and Biocca (2018) noted that the user may choose to distance themselves purposefully in order to maintain control of their experience. From these studies, we can infer that when users object or hold a critical position they may lessen their own immersion. The question remains as to whether the reverse is true and a glitch in immersion suggests a more critical position to users, or whether something else may be communicated.

Method: Qualitative study of VR Productions

Immersive experiences are a complex and personal ‘fluid state’ (Shin and Biocca, 2018: 2812). In order to evaluate the immersive experience in our glitched scenarios, we used a multi-step and highly in-depth qualitative method, where user feelings, experiences and interpretations were foregrounded.

Background to the study: the Balansa method for gender inclusiveness

The VR scenarios in this study were built by a contracted agency at the direction of the researchers to emulate real-life gender inequality situations at work, following an established Swedish method called Balansa (Eng: balance). The Balansa method was developed by Nahnfeldt (2010) based on her large-scale interview-based research in Sweden on workplace inclusiveness (see also Nahnfeldt and Lindberg, 2013; Sandlund, et al., 2011). Balansa uses abstracted real-world scenarios with gender implications; trained facilitators tell the stories of these scenarios and engage staff in discussion. Three scenarios based on the method were created, each with different limitations (described more fully later in the paper).

Participants

Our participants were three expert facilitators of discussions using the existing Balansa model, two men, aged 41 (user 1) and 57 (user 3) and one women age 58 (user 2). These three participants are the most experienced Balansa facilitators in Sweden, and were all highly familiar with the original material. Experts were recruited in order to assess experience of the VR Balansa as opposed to experience of Balansa in general. The participants were previously unknown to the researchers. All three participants had extended experiences in moderating group discussions sparked by the existing scenarios, and thus had valuable insight into how these scenarios are perceived in textual form and how the following discussion generally progressed. Male 1 had some experience of VR (very little) while the other participants had never experienced a VR production before. While the experience of the expert facilitators will be different to that of an ordinary user (i.e. someone trained in gender inequality), our aim was not to assess the effectiveness of the Balansa model but to obtain insights into the VR material. In addition, we hoped their training with gender inequality would enable them to see new possibilities within the VR material which, indeed, proved to be the case. Lack of experience with VR can sometimes result in physical discomfort such as queasiness or potentially an experience of the ‘uncanny valley’ (Kätsyri et al., 2015). Our participants did not report queasiness, but we do discuss the uncanny valley effect below in the section on plausibility. The data was collected in 2018.

Data collection

The participants first went through the scenarios individually and were observed by researchers. Participants took reflective notes at each stage of each scenario in order to help them remember their learning process. Participants were then interviewed in depth individually after each scenario and then again after all three scenarios were complete (approximately 45 min per scenario, per participant, referring to their notes). The interview questions were open and themed (Kvale, 1996; Wengraf, 2001). The first theme was plausibility: What did you see? What did you experience? How did you feel? What was your role? Did it seem real? Could you relate to the situation? Did anything seem strange, wrong, difficult to understand, or illogical? The second theme was interactivity: How could this work in your role as a process leader, would it add or change anything? What did you think about the people, where did you think you were, could you explain the scenes and how you understood them, was it coherent. What was it like being in 3D, how did you interact, how did it make you feel and so forth. Thus, we recorded and approximately 2.5 h of in-depth individual interviews per participant.

After the individual interviews were completed, the participants collectively joined in a focus group discussion of 1.5 h, in which the specific aim was to gain their professional perspective on the relation between these VR scenarios and the original oral versions. Focus groups allow for the identification of discrepancies, similarities and divergent experiences and lead to the identification of multiple meanings (Lunt and Livingstone, 1996). All individual interviews and the focus group were then transcribed and were used as the source material for our analysis. This extremely in-depth and intensive method allowed us to trace the development of the immersive experience as it happened and also gave opportunity and time for participants to reflect on its impact. Small numbers and very in-depth qualitative data are a valid research strategy when looking for insight into phenomenology and complex phenomena in virtual scenarios (see for example, Davis and Boellstorff, 2016).

Data analysis

The qualitative analysis proceeded in two cycles (cf Saldaña 2016). In the first cycle, the transcripts of both the individual and group were coded by focusing on immersive difficulties and successes. For example, the first quote, below, suggests a difficulty, while the second suggests a success:

“Of course we can learn something but it takes a while to get past the puppets or characters or whatever we should call them – you have to get past it.” (User 1, male, interview)

“I felt like I was there” (User 2, female, interview)

These quotes were then identified by stage and scenario to give context.

In the second cycle of analysis, highlighted quotes were assigned to presence, plausibility and interactivity by reviewing their content. For example, the first quote above relates to plausibility (calling people ‘puppets’ suggests they are implausible), and the second to presence. Since these dimensions of immersion are also somewhat overlapping, some quotes were assigned to multiple dimensions. Finally, since immersion is a process and can shift across the experience, we then compared different experiences of success and failure in order to understand the process of immersion and how it proceeded despite the stumbling blocks the users experienced.

The Three VR Balansa Scenarios

The Balansa method is scenario-based, although the scenarios are typically delivered verbally and discussed. For this project, we developed three VR scenarios based on existing Balansa scenarios: the office announcement, the Monday morning meeting, and the coffee break, which we describe in detail below. The scenarios represent three different levels of complexity and were also highlighted by Nahnfeld as most likely to evoke engaging discussions. The VR versions of the scenarios are organized in three stages, and in each stage new information is revealed to the participant. Further, each scenario was deliberately designed to imperfectly create immersion, creating glitches in the dimensions of interactivity, plausibility, and presence. We know that the dimensions are somewhat overlapping and that there would likely be interaction effects. In the three scenarios, each dimension of immersion is present to a varying degree, shown in Table 1.

Table 1.

Strength of immersive elements in the three scenarios.

	Announcement	Morning meeting	Coffee break
Interactivity	−	+	++
Plausibility	+	−	+
Presence	+	+	−

Note: In interactivity, the – describes a situation where the user cannot affect any part of the scenario; the + a situation where they can choose at a predetermined time to take an action; and the ++ where the users has more relative freedom to explore a scenario. In plausibility, the – describes a scenario where the environment is not realistic where the + describes a scenario where the environment is realistic. In presence, the – describes a situation where the user has no avatar; the + describes a situation where the user has an avatar. The dimensions do overlap somewhat – for example, an avatar and interactive capability may give a heightened sense of presence.

The office announcement scenario

The scenario uses 360° film footage and graphical overlays and is staged using actors, giving a realistic effect (see Figure 1). The first stage is in an office meeting room where the user is placed at a table among colleagues. The user’s presence is acknowledged by the other meeting participants, but the user has no avatar. The user cannot answer back, interact or change the course of the scenario. During the meeting, the company’s management reveals a major change to how time scheduling will be conducted in the workplace. Several people in the workforce give positive comments to the planned changes. In the second stage, the user is standing in a corridor in an unofficial discussion with the company´s management and the office administrator. They discuss problems that have emerged after the new scheduling. The administrator reveals some insights as to why these problems have occurred, pointing out that the new scheduling seems to favour a few but disfavour others. In the third stage, the user re-experiences the office meeting from the first stage, but the meeting is now overlaid with graphical representations of each meeting participants’ thoughts and reactions. These present new insights into the colleagues’ reactions, revealing that quite a few of the staff members were sceptical to the changes but did not address this in the meeting. The main aim of the scenario is to evoke a discussion concerning how re-organization can unintentionally reinforce gender bias.

Figure 1.

Screenshots from the office announcement scenario.

The Monday morning meeting scenario

This scenario is intended to surface preconceptions of gender roles in child care. The scenario is composed of a 3D computer-rendered animation and consists of three stages where the same dilemma is staged in three different environments: a car mechanic’s workshop, an architectural agency, and a high school staff room (see Figure 2). In this scenario, the degree of realism is low but the degree of interactivity is higher than in the office announcement scenario, and the user is given the perspective of a male co-worker with an avatar whose head and hands can be moved by the user. On entering the scenario, the user is part of an ongoing discussion about child care with three other male co-workers standing in an office space. The user can, if they wish, take part in the discussion by selecting different responses. However, independent of the course of actions the user takes, the conversation will end the same way – with the other men making fun of their colleague (the user) for taking part in planning and organizing child care.

Figure 2.

Screenshots from the Monday morning meeting scenario.

The coffee break scenario

As with the office announcement scenario, this scenario uses 360° film footage, graphical overlays and is staged using actors. However, in this scenario, the user is not acknowledged and has no avatar. Instead, the user has a “fly on the wall” perspective: they are not addressed and cannot change the course of events, only observe (see Figure 3). However, the user can choose to listen in on the inner thoughts of the two characters. The scenario starts with a woman placed at a table in an office break room. A second woman enters the room and sits down at the table, opposite the first woman. They have a very short conversation but then go silent. The woman who entered the room picks up a newspaper and starts turning the pages. Occasionally she looks up at the other woman who either looks at her mobile phone or out into space. After a while she receives a text message and exits the break room. In the second stage, the user is presented with the same scenario; however, if the user looks at the second woman who enters the room, the user can listen to her thoughts. The third stage presents the same scenario a third time; however, now the user can choose to listen to the thoughts of both women and read the text message. The oral version tells the narrative through the perspective of the second woman and her experiences where she thinks she has done something wrong; however, in the end, she learns that the woman reading the newspaper had her own personal concerns with her child. The VR version follows the course of events, but the user is presented with both perspectives. The aim of the scenario is to illustrate the blurry boundaries between private life and work life and how lack of communication can engender misleading assumptions.

Figure 3.

Screenshots from the coffee break scenario.

Results: Glitching in action

In this section, we examine in detail the reactions of our expert users as they experienced the glitched Balansa scenarios.

Glitches in interactivity

In our scenarios, interactivity was primarily constructed through navigation and conversation opportunities. Users navigated by eye tracking indicated by visual cues. These cues could be symbols or textual options, and users made their selections by looking at a specific item.

In the office announcement scenario, the users could not take part in the conversation, only observe their co-workers. The lack of interactivity and the inability to intervene in the conversation caused frustration among the users.

I was irritated with the manager! I thought - and you are the boss? (User 2, Female)

Then [after the announcement] you got a feeling of… there you wanted to say, stop! Now you should do like this! But I couldn’t do a thing. (User 3, male)

Not only were they prevented from expressing their thoughts about the proposed changes, they also could not voice their own observations of discomfort among their female co-workers. The limited interaction caused the users to feel powerless and angry. Similarly, in the Monday morning meeting scenario, the limited conversational replies which Ronny was able to give caused uses to feel frustration and even anger at the situation.

In both these scenarios, the participants realized that they were trapped in a situation they could not control. They could see what was going on, but not intervene, and therefore they turned their focus onto the situation itself. In the focus group discussion later, the participants pointed out that the aim of the scenarios is to discuss difficulties negotiating different social relations, and that the experience of being trapped and, for diverse reasons, unable to answer back and question others' views is central to the method as a whole. While these emotions are reflected upon when working with the oral scenarios, the VR scenarios prompted a more direct emotional experience.

When I am present [given a representation] it is really disturbing that I can’t say anything. Then I am part of the imbalance in the room. (User 2, female, group discussion)

Yes, and then I am the dilemma. That was super insightful…. And when you are not [present], then you reflect upon the situation. (User 1, male, group discussion)

Finally, in the coffee break scenario, the users felt they gained control through interactivity. The ability to enter diverse perspectives through the two women’s thoughts gave the users control to enter, exit, compare and study the situation in detail. Here, the introduction of interactivity in stage two and three functioned as a source of user power, the power to change the route through the narrative by navigating through information in a wished-for way. Thus, they could also access a deeper understanding of the ongoing dilemma. The participants said that they recognized themselves in the insecurity that may occur when you don´t understand the situation and in the ambivalence in intruding into a colleague’s personal life, as well as in the personal dilemma at the centre of the scenario. Interactive features here clearly added to the ability to take on different perspectives and added a dimension of intimacy and control.

Glitches in plausibility

Plausibility refers to how believable a situation is, and has both physical dimensions (i.e. how realistic does it seem in terms of audio and video) and other dimensions (i.e. how likely is it that you could be a in a situation of this type).

In the office announcement, all participants said they had a plausible experience, even though the scenario used unrealistic speech bubbles to illustrate the thoughts of their co-workers. The VR scenario gave a more plausible experience than the ordinary oral version with which the participants were familiar. When one of the users was asked to compare the oral scenario he described as a powerful experience:

Longer, much more extensive than reading the story in a minute. This give faces, it gives credibility, the acting in itself… One could identify joy and emotions in all of them. One can see each person… each person and their emotions, just like that. And you capture their emotions right from the start, and then reinforce them. Very powerful! (User 1, male)

Another participant described how he felt the ambience shifting in the room, allowing him to experience the tension in a subtle way.

At first it seemed as a nice work-place. They were all in a good mood. Then I thought, “this would be a nice place to work”. […] And then they announced the changes, then you could feel… something negative crawled into the room. (User 3, male)

In the Monday morning meeting scenario, users had difficulties adjusting to the constructed 3D world and the 3D-animated avatars. The scenario is designed to be sketchy and with little detail. Initially participants found it difficult to accept the avatars as trustworthy, reacting to their jerky movements and lack of facial features – that is, they reacted to the physical implausibility. We relate this both to the scenario and potentially to the user’s inexperience with VR and virtual avatars, the so-called uncanny valley in which unfamiliar human-like simulacra (such as avatars or even robots) are experienced as creepy or threatening. In fact, it is in line with previous research on the uncanny valley that this negative reaction has to do with problems in plausibility, for example when behavioural realism does not match graphical realism (cf Kätsyri et al., 2015). Thus unfamiliarity with avatars in general might contribute to the participants’ reluctance to accept the avatars in this scenario. The avatar co-workers’ facial features and body movements were unspecific and unclear which also created an opaqueness in their reactions. Were they joking, were they inclusive or exclusive? One of the participants pointed out that the sketchy faces evoked insecurity, similar to when being bullied. However, once the participants focused on the ongoing dialogue, they found the overall narrative and environmental setting plausible.

This narrative plausibility was somewhat inconsistent: in the scenario the same dialogue is placed in three different settings: at a car mechanic’s, in a teachers’ staff room and at an architecture office. The participants expressed a difference in plausibility between the three different environments. They thought that the stage situated at the car mechanics’ workshop was the most believable, ‘it felt more natural that such dialogue would appear there’ (user 1, male). But they felt it less believable when set in the teachers’ staff room. The reason given was that teachers’ working conditions would prohibit the type of problem raised in the scenario. These experiences of plausibility indicate users' preconceptions of teachers’ working conditions as well as a stereotypical conception of car mechanics’ attitudes towards gender issues.

User preconceptions were also brought to the fore in the coffee break scenario. One of the participants expressed that the actor in this scenario, who was portraying a mother worrying about her child, was not trustworthy. The user thought she was ‘too old to be a first time mum’, (user 1, male) when of course becoming a first time mother at the age of 40 is not that uncommon. By contrast, the users thought the narrative was too stereotypical in the office announcement scenario and therefore also untrustworthy. Nevertheless, the participants accepted the form of the scenario (constructed using film footage from the 360° camera) as a natural workspace.

Glitches in presence

The coffee break scenario begins with the user in the observer position. Participants witnessed the women greeting each other and then going into silence. At this stage, the users reported that they felt insecurity, ambivalence and puzzlement. They sensed a tension in the room and felt frustration at not being able to intervene and question the situation. As one of the users said, ‘Why doesn’t she [the woman entering the room] just ask if something is wrong!’ (user 1, male). However, in the first stage, they could only watch the situation unfold. In the next two stages of the scenario, they were able to call up each woman’s inner thoughts by focusing on the woman, at which point her thoughts appeared. In these phases, they reported a closer and more intimate experience, as if they were within the mind of the woman. In moving between the two women’s inner dialogue, the users became emotionally engaged with each of the women’s dilemmas.

I stepped into her thoughts, reflections and her shoes… Her dilemma became my problem. (User 3, male)

I went in and felt for her, the women with the coffee. And then I went in and felt for the woman with the phone. (User 2, female)

In comparison to working with the scenario orally, the users expressed that the VR version allowed for a much stronger emotional engagement and a deeper understanding of the situation. In the focus group interview, they pointed out that the VR version could prompt a discussion of action and reaction in more concrete way than when presenting the scenario orally. They concluded that the ‘fly on the wall’ position was central because it allowed for the shift from frustration at the first stage to a closer, more intimate experience in the following stages, intensifying the experience as well as representing the importance of acknowledging different positions in a workplace dilemma.

In the Monday morning meeting scenario, the user appears as a male avatar, ‘Ronny’. Ronny discusses child care with his colleagues, and ultimately participates in a conversation where men who care for children are belittled. Here, as the users stepped into an ongoing conversation, they were at first overwhelmed by the situation. They felt confronted by the other men who were engaged in the discussion, and at the same time, they had trouble understanding their own role in the situation. The users said they felt confused and disoriented, and had to look for cues to how their avatar functioned, as well as understand the situation itself. Thus, they felt present physically, but at the same time distant to the position they were given. They were inside, but lacked an understanding of the ‘self’. One of them even expressed that he wanted to exit the avatar, to be able to observe him and to ‘understand whose shoes I was put in’ (user 1, male). Here, the experience of being trapped in a body forced the users to reflect on whose body they had entered in the given situation.

Users also had difficulties understanding whether Ronny really enjoyed the conversation, or whether he disagreed and felt pressured to play along.

I wanted to get into Ronny’s head… to see his facial expression when he says what he says, and how he takes the comments he gets. Is he irritated, or does he also think it is fun? (User 2, female)

I had to tune into Ronny’s situation, how does he feel? (User 3, male)

To understand Ronny, the users had to use their own imagination to appreciate what he felt and why he reacted (or did not react) the way he did. After a while, they grasped the situation, identifying a discrepancy between Ronny’s behaviour and their own reaction to the ongoing situation. This discrepancy generated discomfort but also empathy for Ronny. The participants did not change their minds, but rather identified that Ronny (like themselves) was trapped in a social situation that he could not handle. In the focus group discussion, they all expressed that being trapped in a social situation generated emotional engagement and reflection which was difficult to create when reading the oral version of the scenario.

When one begins to reflect on what someone else is doing… What is happening? How will this affect his or her situation? Then it becomes really deep… and then you turn inwards. (User 1, male)

Thus, while at first they had problems with embodying their avatar and being in a first-person position, it also obliged them to reflect on the situation and to place themselves within it. The first experience of confusion and stress when entering Ronny’s body shifted and became more contemplative and compassionate over the course of the conversation.

In the office announcement scenario, the user is acknowledged by the virtual meeting participants but is unable to act or interact. All the participants felt present in the room and part of the ongoing meeting, particularly because their virtual colleagues greeted and addressed them when entering the room.

It felt like I was there. I think it was because of the person that confirmed my presence while sitting down, otherwise I think I would have thought that I was just watching. But I was really there. (User 2, female)

However, at the same time they felt like they were ‘new on the job’, since they had difficulties identifying their specific role, and did not seem to have any previous relation to their co-workers. They found themselves becoming observers, paying specific attention to their virtual colleagues’ positions, interactions and relations to each other. Thus, even in the first stage of the scenario, they noted a conflict between some of the co-workers’ body language and verbal expression, leading them to understand that not all the virtual co-workers embraced the new scheduling system, even though in their dialogue they expressed a positive attitude towards the changes.

The users thought that central to this understanding was the position they were given (or rather understood themselves to be in) as newly employed. The graphical overlay in the last stage of the scenario thus became a confirmation of their initial understanding of the situation. The mix of graphical elements onto film footage did not shift their experience of presence. It rather gave further insights to why their virtual co-workers reasoned as they did.

With the small info-bubbles you get a lot of detailed information, what they think, why they think, how they are affected or whatever it can be. So you can see more concerning each individual, more than I managed to figure out myself. (User 3, male)

Interestingly, when the users were asked to compare the different scenarios, the experience in the office announcement scenario was not as intimate as listening to the voices of the women in the coffee break scenario. While the users felt present in the office announcement scenario, the situation of an unfamiliar working space with unfamiliar co-workers also placed them at a distance emotionally.

Discussion

We anticipated that the participants would show lower levels of identification with narratives where interactivity was low, in line with previous research. However, our findings showed instead that in specific settings restricted interactivity actually promoted reflection. The users’ expectations were to enact, intervene, and question the situation, but the technological limitation prohibited action. One explanation for this is that inaction and silence may also be kinds of action, especially when the user is embodied or recognised in the VR setting.

In these situations, lack of action or speech may, paradoxically, create the feeling of acting complicitly in a morally ambiguous situation. Our participants experienced their lack of agency as very thought-provoking, even if difficult. Although the user’s reflection may originate in a technological glitch, it brings attention to the complexity of social relations. The ‘uncomfortable interactions’ identified by HCI scholars are also at play in VR scenarios when interactivity contains deliberate glitches, provoking anxiety, fear and confusion. Interactivity is not a more-is-better proposition, but rather is a dimension in which designers can make choices and critical scholars can analyse for effects on narrative.

Similarly, our results suggest that plausibility should be an area of further investigation for scholars. Just as with a lack of interaction, our results suggest that a lack of connection to the virtual body aids reflection. Earlier research into VR has already called out the power of placing the user into an ‘implausible’ body. Our data show that an implausible social situation is equally fruitful. We call this glitching of social norms situational implausibility. Situational implausibilities call attention to individuals’ mental stereotypes and preconceptions of attitudes, behaviours and working conditions. Thus, when plausibility is questioned, what may in fact be in question are norm-challenging narratives or scenarios.

Finally, our results showed that presence and distance were not quite opposite. First, we observed that being embodied in an avatar without interactivity was at times associated with emotional distance, as predicted by previous work, but not with the experience of disassociation, boredom or lack of engagement. Instead, the users wished to exit the body to take control to combat difficult emotions. Since they are unable to do so, they must take refuge in a distanced emotional position. Surprisingly, the experience of presence was strong even in the absence of an avatar, in a scenario that had more in common with other types of media forms, such as a film with interactive components. Our results show that the lack of avatar led to a feeling of intimacy, lending empathy and the ability to switch perspectives that the users found engaging. The disembodied virtual body was replaced by the imagined embodied self in the VR environment. The least amount of reflection was in the situation that was experienced as most realistic (lacking glitches), where the participants were both passive and experienced the situation as plausible (cf Kaufman and Flanagan, 2015). In summary, an element of distance – emotional, physical (i.e. absent avatar) or even intellectual – allows the user space to reflect and critique. We might call this a distant presence – the user assumes a critical, engaged stance.

Conclusions

The aim of this study was to examine the immersive experience of users who entered a VR scenario designed to limit immersion – that is, with a deliberate glitch. Immersion is a complex experience that is, to some extent, flexible. We did not find that glitches in immersion broke the users’ experience to the point where the VR scenario lost its narrative or impact. This is good news for designers working on VR with limited resources. Instead, we found that each of the aspects of immersion became stretched in unexpected ways, revealing a kind of dimensionality or texture to the immersive experience. Uncomfortable interactions, situational implausibility, and distant presence can all be a part of a complex immersive experience, which designers may use to their advantage. Equally, designers may already be communicating things they do not expect. Further research might explore whether reactions to glitches are predictable, and whether and how they vary according to cultural context, technical expertise and so forth. Reactions which today are unexpected might then be elicited by design to enhance the emotional effect of future productions.

Limited immersion caused participants to reflect critically upon their own roles, to question actions they were forced to take or not take, and to deal with difficult emotions like guilt, fear, anger or frustration as the scenario unfolded without their ‘full’ participation. In this way, the glitch broke the chain of thought (cf Menkman, 2011), interrupting the expected and demanding reflection; but also, the glitch was incorporated into the interpretative process. Critically, this interpretation is situated both in a social and an individual context. For example, the user who is frustrated by being technologically ‘silenced’ due to a glitch in interactivity may be a user who is used to being able to speak ordinarily. Equally, a wheelchair-bound user might experience a huge sense of freedom in a game with an able-bodied avatar.

Thus, we can use our new terminologies of the varieties of immersion to reflect on the communicative content of other VR productions. For example, VR productions that emphasise comfortableness, plausibility and presence with no distance are also communicating something. If these are the criteria for a ‘full’ immersion, then full immersion becomes a way of replicating a body that is powerful and in command. The glitch thus brings our attention to the techno-utopian norm contain within the idea of full immersion.

Immersion is often associated with VR’s ability to shift user perspectives (Hassan, 2020; Rose, 2018), but we argue that perspective taking may occur without perfect immersion. Rather, it can occur as the user interprets the limitations, the unexpected, and the flaws in an imperfectly immersive environment. It is in these cracks that technological and social norms shine through (Sundén, 2015). By expanding the vocabulary of immersion to allow for a variety of textures and levels of immersion, critics and designers alike will be able to have a more nuanced discussion of the communicative potential of VR productions as they become more widespread.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the VINNOVA (2016-04425).

ORCID iDs

Linda Ryan Bengtsson

Elizabeth Van Couvering

Author biographies

Dr Linda Ryan Bengtsson is a Senior Lecturer in Media and Communications at the Department of Geography, Media and Communications at Karlstad University, Sweden. She is an interdisciplinary researcher and her work investigates the relations between creative life, interactivity, media, design and place.

Dr Elizabeth Van Couvering is a Senior Lecturer in Strategic Communications at the Department of Geography, Media and Communications at Karlstad University, Sweden. She has written about the political economy and culture of platforms, including a techno-economic history of search engines and a discussion of the discursive frames of search engine construction.

References

Ahn

AMT

Bailenson

(2013) The effect of embodied experiences on self-other merging, attitude, and helping behavior. Media Psychology 16(1): 7–38.

Bailenson

(2018) Experience on Demand: What Virtual Reality Is, How it Works, and what it Can Do. London: W.W. Norton & Company.

Banakou

Hanumanthu

Slater

(2016) Virtual embodiment of white people in a black virtual body leads to a sustained reduction in their implicit racial bias. Frontiers in Human Neuroscience 10: 601.

Benford

Greenhalgh

Giannachi

, et al. (2012) Uncomfortable interactions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin Texas, USA, May 5-10, pp. 2005–2014.

Biocca

Kim

Choi

(2001) Visual touch in virtual environments: an exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence: Teleoperators and Virtual Environments 10(3): 247–265.

Biocca

Levy

(1995) Communication in the Age of Virtual Reality. Hillsdale: Erlbaum Associates.

Blascovich

Bailenson

(2011) Infinite Reality: Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution. New York: HarperCollins.

Bopp

Mekler

Opwis

(2016) Negative emotion, positive experience? Emotionally moving moments in digital games. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose California, USA, May 7-12, pp. 2996–3006.

Crawford-Holland

(2019) Virtual healing: militarizing the psyche in virtual reality exposure therapy. Television & New Media 20(1): 56–71.

10.

Cummings

Bailenson

(2016) How immersive is enough? A meta-analysis of the effect of immersive technology on user presence. Media Psychology 19(2): 272–309.

11.

Davis

Boellstorff

(2016) Compulsive creativity: virtual worlds, disability, and digital capital. International Journal of Communication 10(23): 2096–2118.

12.

de la Peña

Weil

Llobera

, et al. (2010) Immersive journalism: immersive virtual reality for the first-person experience of news. Presence 19(4): 291–301.

13.

Farmer

Tajadura-Jiménez

Tsakiris

(2012) Beyond the colour of my skin: how skin colour affects the sense of body-ownership. Consciousness and Cognition 21(3): 1242–1256.

14.

Golding

(2019) Far from paradise: the body, the apparatus and the image of contemporary virtual reality. Convergence 25(2): 340–353.

15.

Goriunova

Shulgin

(2008) Glitch. In: Fuller

(ed) Software Studies: A Lexicon. Cambridge: MIT Press, pp. 110–119.

16.

Green

Jenkins

(2014) Interactive narratives: processes and outcomes in user-directed stories: interactive narratives. Journal of Communication 64(3): 479–500.

17.

Grundell Gachoud

(2020) Tactical ambiguity: materiality, representation and interaction in evan meaney’s glitched portraits. In: Popat

Whatley

(eds) Error, Ambiguity, and Creativity. Cham: Palgrave Macmillan.

18.

Hassan

(2020) Digitality, virtual reality and the ‘empathy machine. Digital Journalism 8(2): 195–212.

19.

Höök

Jonsson

Ståhl

et al. (2016) Somaesthetic appreciation design. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose California, USA, May 7-12, pp. 3131–3142.

20.

Kätsyri

Förger

Mäkäräinen

, et al. (2015) A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness. Frontiers in Psychology 6: 390.

21.

Kaufman

Flanagan

(2015) A psychologically “embedded” approach to designing games for prosocial causes. Cyberpsychology 9(3): 5.

22.

Kvale

(1996) Interviews: An Introduction to Qualitative Research Interviewing. Thousand, Oaks Calif: Sage.

23.

Lambert-Beatty

(2009) Make-Believe: Parafiction and Plausibility. October 129(summer): 51–84.

24.

Lee

(2004) Presence, explicated. Communication Theory 14(1): 27–50.

25.

Leszczynski

(2020) Glitchy vignettes of platform urbanism. Environment and Planning D: Society and Space 38(2): 189–208.

26.

Lévy

(1998) Becoming Virtual: Reality in the Digital Age. New York: Plenum Trade.

27.

Lunt

Livingstone

(1996) Rethinking the focus group in media and communications research. Journal of Communication 46(2): 79–98.

28.

Maister

Sebanz

Knoblich

, et al. (2013) Experiencing ownership over a dark-skinned body reduces implicit racial bias. Cognition 128(2): 170–178.

29.

Manovich

(2001) The Language of New Media. Cambridge: MIT Press.

30.

Menkman

(2011) The Glitch Moment(um). Amsterdam: Institute of Network Cultures.

31.

Murray

(1997-2017) Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Cambridge: MIT Press.

32.

Murray

(2016) Not a Film and Not an Empathy Machine: How Necessary Failures Will Help VR Designers Invent New Storyforms. Immerse – Creative Discussion of Emerging Nonfiction Storytelling. https://immerse.news/not-a-film-and-not-an-empathy-machine-48b63b0eda93.

33.

Nahnfeldt

(2010) Balansamodellen: Systematisk Kvalitetsutveckling till Stöd för Balansering av arbetsliv och övrigt liv. Report. Sweden: Karlstad University.

34.

Nahnfeldt

Lindberg

(2013) Commercializing work life balance outlining a model for analyzing and promoting social science innovation. International Journal of Innovation Science 5(1): 21–30.

35.

Nash

(2018a) Virtual reality witness: exploring the ethics of mediated presence. Studies in Documentary Film 12(2): 119–131.

36.

Nash

(2018b) Virtually real: exploring VR documentary. Studies in Documentary Film 12(2): 97–100.

37.

Parker

Saker

(2020) Art museums and the incorporation of virtual reality: examining the impact of VR on spatial and social norms. Convergence 26(5–6): 1159–1173.

38.

Peck

T C

Seinfeld

Aglioti

, et al. (2013) Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and Cognition 22(3): 779–787.

39.

Peng

Lee

Heeter

(2010) The effects of a serious game on role-taking and willingness to help. Journal of Communication 60(4): 723–742.

40.

Popat

(2016) Missing in action: embodied experience and virtual reality. Theatre Journal 68(3): 357–378.

41.

Ratan

(2013) Self-Presence, explicated: body, emotion, and identity extension into the virtual self. In: Luppicini

(ed). Handbook of Research on Technoself: Identity in a Technological Society. New York: IGI Global, 322–336.

42.

Reeves

Cummings

Scarborough

, et al. (2015) Increasing energy efficiency with entertainment media: an experimental and field test of the influence of a social game on performance of energy behaviors. Environment and Behavior 47(1): 102–115.

43.

Ryan

M-L

(2001) Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Baltimore: Johns Hopkins University Press.

44.

Rose

(2018) The immersive turn: hype and hope in the emergence of virtual reality as a nonfiction platform. Studies in Documentary Film 12(2): 132–149.

45.

Saldaña

(2016) Goodall’s verbal exchange coding: an overview and example. Qualitative Inquiry 22(1): 36–39.

46.

Sandlund

Olin-Scheller

Nyroos

, et al. (2011) The performance appraisal interview – An Arena for the reinforcement of norms for employeeship. Nordic Journal of Working Life Studies 1(2): 59–75.

47.

Shin

(2018) Empathy and embodied experience in virtual environment: to what extent can virtual reality stimulate empathy and embodied experience? Computers in Human Behavior 78: 64–73. DOI: 10.1016/j.chb.2017.09.012.

48.

Shin

Ahn

(2013) Associations between game use and cognitive empathy: a cross-generational study. Cyberpsychology, Behavior, and Social Networking 16(8): 599–603.

49.

Shin

Biocca

(2018) Exploring immersive experience in journalism. New Media & Society 20(8): 2800–2823.

50.

Sherman

Craig

(2018) Understanding Virtual Reality: Interface, Application and Design. 2nd Edition. Cambridge: Morgan Kaufmann Publishers.

51.

Slater

(2003) A note on presence terminology. Presence Connect 3(3): 1–5.

52.

Slater

(2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences 364(1535): 3549–3557.

53.

Slater

Wilbur

(1997) A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence 6(6): 603–616.

54.

Steuer

(1992) Defining virtual reality: Dimensions determining telepresence. Journal of communication 42(4): 73–93.

55.

Sundén

(2015) On trans-glitch, and gender as machinery of failure. First Monday 20(4). DOI: 10.5210/fm.v20i4.5895.

56.

Walter

Murphy

Gillig

(2018) To walk a mile in someone else’s shoes: how narratives can change causal attribution through story exploration and character customization. Human Communication Research 44(1): 31–57.

57.

Warfield

(2016) Making the cut: an agential realist examination of selfies and touch. Social Media + Society 2(2): 205630511664170.

58.

Weibel

Wissmath

Mast

(2010) Immersion in mediated environments: the role of personality traits. Cyberpsychology, Behavior and Social Networking 13(3): 251–256.

59.

Wengraf

(2001) Qualitative Research Interviewing: Biographic Narrative and Semi-structured Methods. London: Sage.

Stretching immersion in virtual reality: How glitches reveal aspects of presence,interactivity and plausibility

Abstract

Keywords

Introduction

Elements of immersion

Method: Qualitative study of VR Productions

Background to the study: the Balansa method for gender inclusiveness

Participants

Data collection

Data analysis

The Three VR Balansa Scenarios

The office announcement scenario

The Monday morning meeting scenario

The coffee break scenario

Results: Glitching in action

Glitches in interactivity

Glitches in plausibility

Glitches in presence

Discussion

Conclusions

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

Author biographies

References