Virtual narratives,physical bodies: Designing diegetic sensory experiences for virtual reality

Abstract

This article presents a reflection on four virtual reality (VR) research projects to foreground questions about how physical and sensory interactions are incorporated into VR narratives. The four projects explore a range of experiences, from interactions with digitally mediated and found objects, to sensory interactions and outdoor play. Bringing these projects together here offers an opportunity to contribute insight into conceptualizations of diegetic design and the ‘design space’ of VR, providing questions and considerations towards designing sensory experiences for narratives in VR.

Introduction

For several years now, storytellers across the arts have experimented with the narrative and interactive possibilities of virtual reality (VR). These stories often push the technological capabilities of the platform, extending the virtual into complex physical and sensory environments while borrowing from traditions in immersive theatre, immersive cinema, and other narrative forms (Atkinson and Kennedy, 2018; Kennedy and Atkinson, 2018). Meanwhile, mainstream consumer VR has been trending towards increased standardization, with hardware across companies settling on similar form factors for the headset and controllers, and with content predominantly designed for at-home experiences. Intersecting these two strands of VR storytelling is work by the design research community, with small-scale projects that often receive little attention outside conference proceedings published in venues related to human-computer interaction (HCI). This article weaves together four such projects – four VR storytelling design projects that I led from 2017 to 2021 – to foreground questions about how physical and sensory experiences can be incorporated into VR narratives. These projects adopt a ‘research through design’ approach (Gaver, 2012), an inquiry-based process in which designers ask ‘what might be’ within a given design space. Whereas narrative design for the arts and/or for mainstream consumer audiences is often presented in its final form, our work is intentionally unfinished – continually restarting to ask questions about what we design, how we design, and what it might mean when we bring physical bodies and physical actions into virtual worlds.

Questions in our projects consistently revolve around sensory experiences: we argue that experiences in VR are sensory, even when they are not designed that way. While storytelling grounded each of our efforts, the work offers subtle interventions into broader design trends. In the first project (Harley et al., 2017), as dedicated handheld controllers became more common in VR, offering more generalizable forms of interaction, we designed interactive diegetic objects situated within the world of the story to ask how we might incorporate the complexity of tangible interactions in VR. In the second project (Harley et al., 2018), as start-ups and other companies advertised ‘complete’ sensory immersion using technological means, our design offered the opposite, arguing that the non-digital offers low-cost and personalizable opportunities for sensory experiences in VR. In the third project (Harley et al., 2019), as smartphone-based VR was framed as low-end VR despite its increased accessibility, we proposed that it can be its own narrative medium, deliberately calling attention to decades of storytelling with the smartphone and its related technologies. In the fourth project (Harley et al., 2021), as design research continued to expand its digital simulations in virtual worlds, we proposed a design reorientation: rather than begin with the digital, how might we begin with the physical? In the headlong rush to develop VR, I argue that our work is an attempt to slow the design process by asking what is left out, asking how we might examine and interrogate the design space of sensory VR.

Given that research through design prioritizes reporting and reflecting on process, this article continues this reflexivity by drawing connections across our four projects to propose considerations that contribute to VR storytelling and design. Throughout our projects, the goal was not to provide generalizable results that could immediately be extracted for large-scale production, and that is still not the goal here. Instead, the purpose is to use our work to raise questions about how designers and storytellers might conscientiously create for VR. As I will describe, the ‘design space’ for VR is fraught with complications and challenges, leading to questions about what is included and/or excluded in conceptualizations of sensory design.

I begin by providing a broad overview of sensory and physical design for VR storytelling, followed by a brief reflection on the methodology underpinning our research projects. I then describe each of the four projects by situating them within the context of the questions that drove the research. In the final section, I propose two overarching considerations. The first consideration offers perspectives on how diegetic design for VR can be applied to deliberately consider a range of possible sensory experiences within VR narratives. The second consideration offers an expanding conceptualization of the ‘design space’ of VR, raising questions about what it means to participate in the development of this technology, and what is included within the bounds of narrative design for VR.

Background

As consumer VR applications entered the market, the range of embodied actions and expected physical movements would depend on a balance of design and expanding technological affordances. For many of the early consumer experiences, the (technological) ability to interact did not guarantee an opportunity to interact. Bevan et al. (2019) found that for non-fiction VR titles between 2012 and 2018, the dominant experience was that of a passive observer, viewing the story without any opportunity to affect its outcome. This form of ‘passivity’ can still employ a variety of sensory modalities. For example, Notes on Blindness: Into Darkness (Middleton et al., 2016) offers a sensory experience by foregrounding an audio narrative, with visual characteristics that become present only in response to the sound design and narrative description. It is also, inevitably, a physical experience. As Atkinson and Kennedy (2018) describe, ‘[Notes on Blindness] is not an experience that you can enter and become one with, the body is required to be constantly moving, neck craning, spine twisting, sight obscured as you work to discern where to focus, what to perceive, what to attend’ (p. 14). This description offers a reminder: however intentional or unintentional the physical movement (or discomfort) may be, any analysis of VR narratives requires far more than an interpretation of the content.

Although other work in the consumer space (e.g., including narrative games, exercise games, or social experiences) would continue to offer increased physical interaction as tracking technologies improved, outside the consumer market, large installations created for festivals and events would add further complexity to the kinds of physical and sensory interactions that were possible with VR narratives. As Kennedy and Atkinson (2018) observe in their analysis of productions like Draw Me Close (Tannahill, 2017) and The Cube (Circa69, 2016), which integrate VR into live performances, these stories intentionally blur digital and non-digital storytelling traditions, providing audiences with opportunities to interact with (or experience) some sensory aspect of the narrative world. In an analysis of Alejandro G. Iñárritu’s lauded VR production, Carne y Arena (2017), Moura (2022) describes the sensory qualities as integral, including a cold waiting room where the viewer removes their shoes and socks before putting on a headset and entering a large space with sand underfoot. Situating the increasingly physical and/or sensory complexity of these types of VR narratives within their social and cultural contexts raises further questions. In addition to questions about who has the expected access and privilege to attend such exclusive events (Raessens, 2019), attempts to expand and/or leverage the experiential qualities of VR might be analysed alongside critiques of the presumably transformative power of visceral experiences in VR (Schlembach and Clewer, 2021), or critiques about whose bodies and experiences are represented and given agency (Irom, 2018; Nakamura, 2020), or examinations of the ethics of creating and witnessing VR content that depicts the suffering of others (Fisher and Schoemann, 2018; Nash, 2018).

Another analytical challenge across a range of possible VR experiences and environments is that despite identifiable differences in the types (and quality) of interaction, popular discourse can flatten distinctions between terms that are used to describe VR like immersion, presence, empathy, etc. as the language of VR is transformed into buzzwords and corporate imperatives (Evans, 2018; Foxman et al., 2021; Harley, 2022). Within HCI research, Marshall et al. (2019) refer to the differences in technologically-mediated sensory interaction as representing degrees of ‘sensory alignment’, proposing a framework to map how a sensory experience in VR is physically or digitally mediated, ultimately arguing that degrees of misalignment can also be desirable or necessary. Their work was motivated by the increasing attention to sensory VR design in HCI research, not only incorporations of sight and sound, but also touch, smell, taste, and kinaesthetic movement. As one example of this work in practice, in creating a museum installation for cultural heritage, Tennent et al. (2020) use what they call a physical ‘canvas’ for virtual reality – a ‘blank’ room with windows, display cases and other objects that provide a tactile counterpart to the virtual world. In their work, the outline of a window in physical space is something to look out from in virtual space. Elsewhere in the room, a virtual fireplace is coupled with a ceramic heater and the scent of woodsmoke. In this example, the virtual space is deliberately physical and sensorial in direct relation to the real-world environment. The work is also, in part, a response to the challenge of scaling such experiences, proposing to leverage existing spaces to bring similar efforts to wider audiences.

At first glance, some forms of sensory VR design appear to be an effort to realize early visions of VR that promised a completeness of sensory experience. The ultimate display (Sutherland, 1965) was envisioned as fully sensory, as was the holodeck (Murray, 2017): both were imagined as totalizing virtual experiences that would be indistinguishable from reality. In practice, it is a vision of technology that equates sensorial realism with good design, with the assumption that improved technological capability offers better narrative experiences. One extension of this today might be VR design that forces a player to stand because it is technologically feasible, rather than necessary or relevant to the experience (Zielasko and Riecke, 2020). Attending to the differences and complexities of sensory experiences across VR storytelling offers the consistent reminder that there is a need to examine the broader contexts of this design, including the intentional and unintentional qualities of the experience. How, for example, might we understand efforts towards embodied sensorial realism alongside evidence of physical discomfort or the need to perform difficult but required movement? How might we situate the real-world implications of this form of narrative design? For design researchers and practitioners, the range of possible experiences in VR ultimately brings into question what we simulate (or stimulate) and why, and what responsibilities we may have when extending VR into increasingly sensory and physical realities.

Annotating a ‘research through design’ process

In the following sections, I present an ‘annotated portfolio’ (Gaver and Bower, 2012) consisting of four of my collaborative design projects to begin unpacking some of the questions that revolved around our efforts to create VR narratives with sensory and physical interactions. As Gaver and Bowers (2012) describe, one of the benefits of design annotations is the opportunity to reflect on a body of work to help draw out design considerations, connections, and choices across a set of individual projects. My goal with this approach is to continue the reflexivity of our ‘research through design’ process (Gaver, 2012) to ask how our diegetic design might extend the ‘design space’ of sensory and physical interactions in VR narratives.

Gaver et al. (2022) have broadly identified two distinct trajectories within research through design: one oriented towards traditions in the arts and the other towards traditions in the sciences, and our own work follows the former orientation. We often drew on Gaver’s (2012) perspectives to emphasize the process of inquiry-based design: for Gaver, it is a reflexive, creative, and exploratory design process through which designers ask ‘what might be’ within a given design space; it is an approach that seeks questions rather than answers. Despite some similarities in our work to other approaches including discursive design (Tharp and Tharp, 2019), speculative design (Dunne and Raby, 2013), design fictions (Bleeker, 2009; Sterling, 2009), diegetic prototypes (Kirby, 2010), or even approaches that use VR as a platform for design fictions (e.g., McVeigh-Schultz et al., 2018), one distinction (though firm lines can be difficult to draw here) is that the contribution that we offer to the design research community is insight into a process that articulates and examines the contours of what we design, and why. As Gaver (2012) describes, ‘One of the valuable roles of design theory, from this perspective, is in making accessible the kinds of decisions and rationales that comprise an artefact’s embodied theory, or give dimensionality to its design space’ (p. 944).

The artifacts in our work were proofs of concept and prototypes, that is, small snippets of narrative designed for VR that also served as a prompt to ask questions about physical and sensory interactions in VR. We created these projects in a university setting, in which I led and worked alongside an interdisciplinary team of four to eight team members (ranging from undergraduate to graduate students, as well as one postdoctoral researcher who was instrumental in advising on the technical implementation). This university-based design context was well-suited to an overarching ethos that valued inquiry over generalizability, and process over outcomes. This was also reflected in the formal and informal testing that we did. When we recruited participants for the 2017, 2019, and 2021 projects, the sample size was small (N = 8, N = 7, N = 7, respectively), not seeking definitive answers on usability or experience but instead using these tests to further explore the possible implications of our design.

Our narratives ranged from designed ‘moments’ to stories that included more conventional narrative exposition, characters, and plot. Each project also offered me an opportunity to develop my own practice and understanding of diegetic design, which refers to the elements of the design that exist in the world of the story. Diegesis is a convoluted term that moves across literary theory, film, and video games, and in interaction design it is sometimes seen as a ‘bridge’ between physical reality and the reality of the story (e.g., Gupta et al., 2019). For Galloway (2006), who brings an interpretation of diegesis to games, there is no inherent value that need be associated to diegetic versus non-diegetic, as either can serve an important function in the overall experience. For me, it offered design prompts: does the interaction, or object, or sensory aspect ‘belong’ to the world of the story? If so, what narrative purpose does that element serve? These were, at first, practical considerations for design that had very little time and space to communicate a story.

In the next section, I provide an overview of each of the projects that will consist of the motivations for the work, a brief description of the narrative design, and some of the broader design questions that the project raises. Note that while I borrow an example from the related work or discourse from that time to help situate our efforts, this is not to disparage the work of others in the HCI community or to single out their work as requiring critique. Instead, I aim to show that our process was one in which we sought to ask questions about the current state of VR design, using narratives and interaction design to investigate, articulate, and respond to what we interpreted as the ‘design space’ of VR.

A portfolio: designing diegetic sensory experiences in VR

Project 1: Tangible VR

The first of our projects (Harley et al., 2017) began in early 2016, the same year as the release of consumer VR. At the time, headsets like the HTC Vive and the Oculus CV1 came with dedicated controllers that were designed to be multipurpose to facilitate a variety of interactions, feedback, and possible applications. In the days leading up to my presentation of the project in 2017, Microsoft Research had released information about a new prototype VR controller called CLAW (see Choi et al., 2018). Since CLAW was also proposed as a multipurpose controller, I used it to illustrate how our work offered a contrast to current design trends. I suggested that what this design means for the body – but specifically for the hands – is that any virtual object manipulation is mediated by the controller, which means that any sense of tactility is reduced to the feeling of the controller and its haptics. In other words, the multipurpose controller reduces the complexity of the hands in order to generalize use; in practice, however, this also specifies possible uses. For example, the CLAW has three main interaction modes, ‘touching mode’, ‘grasping mode’, and ‘gun mode’. In our design sessions, we would ask, if the three main interaction modes are touch, grasp, and shoot, what does that mean for bodies in VR? What does it assume about our hands? What does it assume about what is generalizable, or multipurpose in VR?

In response to such provocations, we designed two prototype interactions that offer specificity rather than generalizability. Set within a narrative context, the work relies on the notion of diegetic objects. We wrote the outline of a story and designed two objects from that world, a magic cube and a talking squirrel (Figure 1). In the story, the squirrel tells the player that the song of the forest is trapped inside a cube, endangering the forest animals. By playing a musical game, which requires the player to interact with all six sides of the cube, the forest is saved. The squirrel is the player’s friend and companion in the narrative and at one point is too scared to go on. The next interaction requires the player to pet the squirrel to calm it down. As an interactive object, we gave the squirrel a specificity that would not be possible with a controller. Using a plush toy roughly the size of a squirrel, the object had soft (synthetic) fur; we added clay inside for weight, a wire frame for the feeling of a spine and a delicate skeleton, a servo motor to move its head, and a vibration motor to give it a heartbeat: fast when it is scared, and slow after the player pets its fur. We argued that there are unique benefits to the use of diegetic objects in VR, with physical properties when the object is held, and virtual properties when it is put down. The project offered new questions for the design community. What happens when specificity is traded for generalizability? What happens when the complexity of physical interactions is made virtual? What is excluded when virtual worlds remain dominantly virtual?

Figure 1.

One of the students working on the first project demonstrating hand-tracking within the virtual scene (left); a cardboard box with sensors to prototype the magic cube (middle); petting the tangible/virtual squirrel (right).

Project 2: Sensory VR

In the second project (Harley et al., 2018), we were interested in what appeared to be an increase in research and industry products designed to provide a ‘sensory’ experience in VR. For example, Ambiotherm (Ranasinghe et al., 2017) proposed simulations of ‘real-world conditions’ by using fans attached to a VR headset as well as pads on the back of the user’s neck (held in place with a strap around the user’s throat) that would heat up or cool down. Later, the authors added a tube-like smell delivery mechanism to this system in a project called ‘Season Traveler’ (Ranasinghe et al., 2018). Embedded within such technological mediations, there appeared to be an assumption that a physical reality could be reduced to a limited number of sensations, and that these sensations could target specific parts of the body for generalized effect. To counter such assumptions, we proposed that the non-digital offers low-cost opportunities for ‘sensory’ experiences in VR that are vastly more complex and personalizable than those currently available using technology.

We designed two proofs of concept to develop this argument. The first was a moment at a beach in which the player turns on a space heater, sits in a lawn chair, puts on sunscreen for its distinct smell, prepares a drink, puts their feet into a plastic container of sand, puts on the headset for the visuals of the beach, and finally headphones for the sound of waves and the birds (Figure 2). For the player, the moment includes smell, sound, taste, touch, sight, the warmth of the heater, and the feeling of sitting in a lawn chair with their feet in the sand. As a design intervention, we refocused attention onto the complexity of the real: there is no haptic feedback device that can simulate the complexity of sand underfoot. It was a contrast to work that implicitly cast the player as a passive recipient of simulation, in which the player is simulated upon, rather than an active co-creator of an experience. Moreover, we suggested that the experience does not need to begin when the player first puts on the headset, and it does not need to end when they take it off.

Figure 2.

One of the authors of the second project putting her feet into the sand (left); the virtual scene for Little Red in the woods (middle); and a player eating Grandma’s bread (right).

We took these considerations into our design of a second proof of concept that presented a single moment in Little Red Riding Hood, in which Red stops in the forest on her way to check in on her grandmother. By conceptualizing the experience as co-constructed and player-driven, we suggested that the player could prepare a basket of food rather than interact with virtual food; rather than a simulation of the wind, or the simulated smell and embodied feeling of being outside, the player feels the actual wind and is outside. Framing these sensory realities as diegetic meant calling attention to the sensory world around the player: the sensory qualities of the actual environment could simulate the sensory qualities of the narrative world. There is something both serious and tongue in cheek when designing these experiences, with images of players sitting in parks chewing on bread or sitting at home in lawn chairs with their feet in the sand. Yet against a norm of technologically-driven feedback, it can be radical to imagine that the player might want to co-create an experience using low-cost, at-hand materials and environments, choosing sensory modalities that best suit their needs and preferences. It can be radical to imagine that it might be the designer’s responsibility to facilitate that kind of experience rather than mediate it entirely.

Project 3: Mobile Realities

Although our third project (Harley et al., 2019) is presented from a design perspective, it may be clearer to present its intervention here as one on discourse. In 2018, despite the variety of hardware and software products designed for VR, industry and journalist discourse often constructed the medium as a monolith entity, as if all consumer VR devices were the same. Articles or statements in the popular press that conceptualized VR as a future-tense object did not acknowledge the vast differences between headsets or experiences. In actuality, headsets ranged from smartphone-based devices to devices that relied on desktop computers capable of processing greater feats of tracking and representation. For Google and Meta, an initial strategy to popularize VR appeared to focus on the lower tier, relying on the relative ubiquity of mobile phones and the relative low-cost of Google Cardboard and Gear VR. This essentially created two versions of consumer VR: a low-end smartphone version for ‘everyone’ and a high-end desktop version for those who could afford it.

The design intervention of our project was to propose that smartphone-VR could be understood as a separate medium with its own advantages. This reframing was an attempt to deliberately call attention to decades of storytelling with the smartphone and its related technologies. We proposed that a narrative for smartphone-VR could use diegetic design to incorporate the various media and technologies of a smartphone, drawing on a history of mobile storytelling to include real-world locations, with diegetic interactions before and after the player puts on the headset. To give shape to and to ask further questions about the medium of smartphone-VR, we designed two iterations of a site-specific story across two locations using the smartphone camera, text messages, GPS, a Web site, audio narratives, AR, and VR (Figure 3). Reflecting on our use of these medium-specific qualities of smartphone-VR as well as the specificity of diegetic design for real-world locations, we concluded with a discussion of some of the wider social, cultural and political contours of this kind of mediated storytelling.

Figure 3.

A player testing the audio narrative and text messages (left); a story object that shows a clue (middle), which connects to the virtual representation of the real-world location (right).

Based on our design process and user test, we asserted that designing mediated encounters with real locations must find ways to acknowledge the politics of space, in part by including considerations for the lived realities of the city and its residents. We described how the parks where our stories took place were near social services, and lived realities within those locations can speak to social and colonial geographies that are not communicated in the idealized forms of play proposed in our proofs of concept. Although our remediation of space might foreground the player’s agency, we noted that there is a need to develop appropriate considerations for non-participants. Such considerations are not new. Mary Flanagan (2007) raised many of the same concerns in response to the location-based media projects of the early 2000s. Positioning our work as a first attempt in the design space of smartphone-VR allowed us to revisit older questions about public media use and uncover current taken-for-granted assumptions about VR play. Who gets to play, how, and where? How might we encourage design that begins with considerations for diverse players, for accessibility, for safety? How does the current use of media in public spaces influence this form of storytelling? How do these forms of play account for differing arrangements of power?

Project 4: Playing by Ear

We developed the fourth project (Harley et al., 2021) from 2019 to 2020 with an initial interest in the connections between sound and physical interactions, which turned to questions about the relationship between the physical and virtual in the design space of narrative VR. While consumer VR and some related academic work might appear to strive to achieve a full simulation of reality, there were cracks in that conceptualization. As we suggested in our previous work, the ‘real environments’ were always present; they were simply ignored to greater or lesser degrees. We saw parallels in this thinking in Marshall and Tennent’s (2019) work, which critiqued Milgram and Kishino’s (1994) mixed reality continuum and a contemporary drive towards increasing simulation, arguing (as we have in the projects discussed here) that there are advantages to engaging with all sensory realities, not only those that the headset and controllers can provide. We interpreted a focus on the simulation as one that prioritized design decisions around the capabilities of the headset, rather than a consideration for bodies and environments. Drawing inspiration from examples of tangible user interfaces (Shaer and Honecker, 2010) that remediate the physical properties of objects to produce playful sound-based interactions, we began our design process without VR, asking how the physical and tangible characteristics of the experience might contribute to the narrative.

We created two proof of concept experiences that decentred the visual experience of VR in favour of sound-based interactions that involved tangible and tactile interactions and physical movement. In the second proof of concept, for example, the player begins the experience with a diegetic note from a ‘wizard’ who says that wearing the mask (i.e., the VR headset) shows the world as he sees and hears it (Figure 4). In this open-ended experience, the player explores three zones, two of which consist of pre-made soundscapes, with one zone that offers space for a player-created soundscape. The player can move sounds from zone to zone by moving tangible objects between zones. We were interested in the ways that players interacted within the environment, and the ways that the designed characteristics appeared to encourage particular movements, whether reaching for a tangible object on a ledge, or kneeling to pick a tangible object up from the ground. We stressed that the goal was not to expect or require specific forms of mobility. As Gerling and Spiel (2021) and Spiel (2021) have argued, there is a need to better incorporate considerations for accessibility and the diversity of physical experiences. We suggested that mapping the physical affordances of the experience was a prompt to think about how designers participate in choreographing movement. We asked, what are the ways in which designers actively participate in normalizing and homogenizing movement and action? How do dominant design trends create and solidify expectations for the physicality of VR?

Figure 4.

A player wearing the VR ‘mask’ while listening to one of the tangible objects (left); a player kneeling to interact in one of the soundscape ‘zones’ (right).

The design space of diegetic VR narratives

Within a broader design context that often appears to pursue normalization and standardization, our projects asked what physical and sensory experiences might be left out or pushed aside. Each of the projects described above is imperfect, but purposely so, using design as a means to produce questions rather than products. As part of a reflexive process, those questions form the basis for the projects’ contributions. Bringing the work together here to continue that reflexivity begins to reveal thematic similarities to the questions we raised, which I will summarize by looking more closely at two concepts that underpin our projects: diegetic design, and the metaphor of the ‘design space’. Throughout our projects, we argued that it was important to develop considerations for the sensory and physical realities of VR; in this final section, I will suggest that design that attends to these realities can benefit from engaging with and extending the design space of diegetic VR narratives.

Extending diegetic design for sensory and physical interactions

Although present in each of our projects, the first project was perhaps most explicit in attempting to outline some of the possible benefits of diegetic design, arguing that it could help to communicate contextual narrative information to the player. We also drew connections to Janet Murray’s (2005) ‘threshold objects’, which are virtual or physical objects that provide a way to cross the ‘boundary’ that separates the real world and the story world. A decade before most consumer headsets were released, Murray wrote, ‘A pirate ship or a canon is a better threshold object than a VR headset because it works in both the tangible and the illusory reality. But threshold objects can also be household items if they are positioned appropriately in the virtual world’ (p. 89). By the second project, however, we began to question the notion of the ‘threshold’. If anything could become part of the story – any household object – what about everything that was left on the other side of the threshold that did not belong to the story? What does the player do with functional but non-diegetic sensations, like wearing a headset while sitting at a virtual beach? How much is the player asked to ignore to become ‘immersed’ in their virtual world? Framing diegetic design as possibly involving everything in the player’s environment requires a more deliberate consideration for everything on both sides of the threshold. For future work, one possible benefit to this approach is that it can prompt additional considerations for the player and their situated environments, with questions about how they might participate in co-creating physical and sensory experiences.

Incorporating senses that are often disregarded into our conceptualization of diegetic design is an effort to become more intentional and more holistic with our design choices: it is not only about what is included in the design, but also a direct consideration for what is excluded. It is an orientation that reasserts an active voice. The designer creating narratives for smartphone-VR includes or excludes the smartphone in the story. The designer creating interactive narratives outdoors includes or excludes various considerations for the social, environmental, and other dynamic conditions of real-world spaces. Conceptualizing the diegetic as this kind of holistic possibility is not an effort to subsume the real world into the world of the story, or to assume that diegetic objects or diegetic interactions lead to better or more ‘immersive’ stories. Similarly, it is not an effort to paper over any experiential or technological problems in VR with narrative distractions. Instead, it situates the complexity of the real-world spaces within the context of narrative design to deliberately acknowledge the reach, limits, and impacts of the choices that are being made.

Extending the ‘design space’ of VR interactions

The second consideration stems from these design choices – as diegetic design becomes more expansive, there is also an opportunity to re-examine what we understand as the ‘design space’ that we are exploring in research through design. Gaver (2012) is clear that the design space is an intentional metaphor, with designs occupying and creating ‘space around themselves’, (p. 944) which the designer can then explore, examining new or existing implications of their work. Gaver (2011) writes that because the space that design creates is also ‘like a “landscape” (rather than abstract Euclidian space), there may be areas that are impenetrable or uninhabitable’ (p. 4), requiring the designer to identify what is possible and work through imagining how that design might exist in the world. Part of what Gaver is developing in these and subsequent articles is a way to report on design and the creative process to better communicate the theory and practice of design (e.g., using workbooks and design annotations). Where, however, should we draw the boundaries of the design spaces that we hope to understand? As with our conceptualizations of diegetic design, we might ask: what do we include or exclude in the design spaces that we create?

For Gaver (2011), the space that design creates can be ‘provisional’, a kind of sandbox to explore ideas. Presented to the HCI community, our own projects initially map a space that exists within and slightly pushes the bounds of related work in HCI; as the work progresses, the design space begins to encompass more and more. As other HCI research exposes the limitations of virtual and mixed reality experiences, whether by exploring the moment people remove a VR headset (Knibbe et al., 2018), or the need to manage VR technological hurdles in live performances (Rostami and McMillan, 2022), or technological mishaps in at-home experiences (Dao et al., 2021), an extended design space for VR narratives might strive to anticipate and consider these realities of designing with VR. Current research offers relevant practice-based examples of work that negotiates similar challenges in related areas of inquiry, including design considerations that result from tensions in artistic work that strives to incorporate interactive technologies (Alaoui, 2019), or design that embraces the ‘frictions’ within mixed reality performance (Rostami et al., 2018).

Ultimately, however, as scholars explore the difficult social, cultural, historical, and technological contexts of contemporary VR (see e.g., Blackwell et al., 2019; Egliston and Carter, 2021, 2022; Nakamura, 2020), expanding design considerations for the physical and sensory realities of VR narratives highlights a need for stronger theoretical orientations. This can reflect a broader call to action in design and HCI to adapt and adopt critical and values-based perspectives (e.g., Bardzell, 2010; Bardzell and Bardzell, 2011, 2013; Flanagan and Nissenbaum, 2014; Ogbonnaya-Ogburu et al., 2020; Spiel et al., 2020) as well as more inclusive research practices (e.g., Scheuerman et al., 2019; Spiel et al., 2019). Slowing down a design process to include these considerations warrants a critical intentionality as well as a practice that crosses disciplines, which incidentally may also result in articulating the choice not to design (Sandelin and Homewood, 2020). If the design space is both created and explored, there is an urgent responsibility to this kind of work: to design is to participate in making and changing the spaces that we wish to examine.

Conclusion

To borrow from Flanagan and Nissenbaum’s (2014) notion of the ‘conscientious designer’, each project described here was an effort to understand how we might become more conscientious in the choices we make. Recurring themes included value placed on the sensory worlds beyond the headset, as well as value placed on the players and the social and technological worlds that are impacted by the sensory and physical realities of VR. The four design projects were ultimately published in conference proceedings in the field of human-computer interaction, but it is difficult to know the impact of such work outside the academic community that the work belongs to. Throughout these projects, my collaborators and I argued that the purpose of reporting on our design is not to suggest that our findings or process are replicable, but rather to raise questions about how sensory and physical design is considered for narratives when creating work that uses VR. The broader design reflection offered here concludes with two overarching considerations that can also be used to raise questions for future work. The first recommends an expanded conceptualization of diegetic design to foreground the ways that sensory realities are considered in a design project. The second recommends an expanding conceptualization of the metaphorical design space, arguing for more intentional considerations of how the design relates to and/or intersects with its broader social and technological worlds.