Expanding the magic circle: Immersive storytelling that trains environmental perception

Abstract

Scholarship on immersion in simulated environments often emphasizes cognitive immersion, or the suspension of disbelief that takes place in an illusionistic space that simulates reality, making the fact of mediation disappear in the experience. Marie-Laure Ryan writes that: “immersivity can be understood in two ways: in a properly VR sense, as the technology-induced experience of being surrounded by data, and in a narrative sense... as being imaginatively captivated by a storyworld” (230). Both of these definitions rest on the notion of cognitive immersion. Grounded in the field of post-dramatic multimedia performance, this paper will focus instead on immersive storytelling that activates the senses in a phenomenological experience. Rather than transporting the spectator into a fictional imaginary space, post-dramatic multimedia performance aims to make participants aware of their presence in the here and now (Klich and Scheer, 128). This paper will describe an immersive storytelling project that integrates virtual reality (VR) into live participatory performance events that take place outdoors. The paper is co-authored by an artist-researcher and two students who are working as research assistants on this project. We recount our creative research process in developing a pervasive game, which Montola defines as a “game that has one or more salient features that expand the contractual magic circle of play socially, spatially or temporally” (2005, 3). This game is played in a park and at key moments, inside VR environments that simulate that same park. The purpose of the game is to attune participants to the species in that particular environment.

A pebbly proscenium

“Look, there’s a snail underneath us,” says a child of about ten with sand-coloured hair. Scanning the ground beneath him, he picks up a few rocks, turning them over in his palms. A friend sitting a couple of meters away casts his gaze downward, searching all around him. He turns to look to the left. “There’s a giant butterfly!”, he cries. A third child, dressed in shorts and a T-shirt, is stretched out on the gravel nearby, and reaches his hands to the sky: “There’s a million monarchs up in the sky!” The first child interjects: “there are milkweed plants all around us too, and their silk is floating everywhere.” Their teacher, Camille-Andrée Pelletier laughs. “The advantage is that these ones won’t give you allergies!”

Reviewing video documentation of this event, I am pleased and surprised to hear these fifth-grade students correctly identifying insects and animals, using their specific names: from “butterfly,” they become more precise: “monarch.” The children are not surrounded by just any flower, but by common milkweed plants. Ms Pelletier reminds them that monarchs need milkweed to survive. My research team and I have been developing a discussion with the class about biodiversity in their neighborhood of Verdun, in Montréal, over the last several weeks.

In the opening scene above, a group of 10-year-olds is engaging with a Virtual Reality (VR) experience. After the students have removed their VR headsets, Christopher Ravenelle, a doctoral student at l’Université de Montréal (UdeM), chats with them about what they saw in the 360 videos. “I saw red plants,” says the blonde boy, “they looked like red palm trees.” “To me they looked more like trees in autumn,” responds his sandy-haired friend. “I had a great view,” laughs the blonde... “And if I told you this was filmed in Verdun?”, asks Christopher, “would that surprise you?”. All three children agree that yes, it would. “I’ve never seen… you know like a giant piece of ice,” begins one boy. “No but you know in the winter,” chimes in another, gesturing toward the river that is located just a few meters in front of them, “when you look out here, there are tons of pieces of ice that are moving down the river.” Christopher tells them that what they saw in the videos is also right here, all around them. In fact, the video footage had been recorded at this very site. The boys become serious, focused, and reflective.

Out of a group of 17 children from Notre-Dame-de-la-Paix elementary school who participated in this workshop on a sunny afternoon in September 2021, only about a quarter had ever tried VR experiences before this. Their reactions were very animated, and they were obviously excited. It was difficult to interpret their behavior inside the headsets as anything but performative, as they pointed at plants, animals, insects, and other phenomena that were invisible to the parents, teacher, and workshop facilitators who accompanied them. The students reached out, tucked their bodies into tight balls, extended their arms in welcome, grabbed and struck out at the air. Some even laid down or rolled on the ground. They sucked their breath in shock, “ooohhed and aaahhhed” in astonishment, and exchanged enthusiastic accounts of what they were seeing all around them. The adults became curious and grilled them about what they were experiencing inside these private worlds.

Although their school is located only a fifteen-minute walk away from the waterfront park where the workshop took place, some of these students had never been there before. Amazingly, some had never seen the spectacular St Lawrence River, even though this is one of the few places in Montréal where people have direct access to the St Lawrence. The concept of biodiversity, which the research team had begun to discuss with the class 2 weeks before during a preparatory workshop, evidently hit home. All of the children noted the snail in one scene, and one child took it upon himself to search the ground once he had removed his headset. He excitedly showed the real thing to his classmates. “The children were all saying, ‘wow! A snail, just like in the video!’,” reports Luana Matos De Oliveira, Masters student at UdeM and member of the research team. “I think that the videos really helped students to see the park in a new way and to notice the plants and animals around them. They all started looking around for the things they saw in the videos.”

Wearing a headset, “one child was reaching out for something but grabbed a [real] tree instead. The kids were amazed, ‘wow! A bird!’, or ‘wow! A butterfly!’. They were constantly exclaiming,” says Luana. Even though the videos that the children were watching were not synchronized as a group, students discussed their experiences throughout. The immersive videos facilitated both conversation—between classmates and adults—and encouraged unmediated interactions with plants and animals in the waterfront park afterward.

Luana notes that paradoxically, the headsets actually focused the students’ attention. Without the headsets, the children became quickly distracted and it was difficult to engage them in concentrated discussion. Because children took turns using the headsets, the narration of those experiencing the videos was observed as a performance by their classmates and by the adult facilitators, who became active audience members, asking questions to find out more about what was in the videos. This built the hype for those who were waiting for their turns.

In this event, students expressed wonder, awe, and surprise that were evoked through immersion in simulated environments (Figure 1). Further, they displayed a marked interest in the physical environment after removing their headsets. These were clearly fully embodied experiences for the children, who moved, cried, and gestured emphatically. As an artist this is a deeply satisfying reaction to my work. Like other artists since the beginning of time, I suppose, a central motivation in creating art is to provoke emotion in others. In these videos, my objective is to incite feelings of surprise by presenting a (more or less) familiar environment as though for the first time. This act of “opening the eyes” of participants is difficult to grasp or communicate, since it is a phenomenological experience of presence, and not particularly suited to words. It is an example of what performance scholar Erika Fischer-Lichte might call the “reenchantment of the world,” aroused by performances that fully command participants’ attention, creating a sense of total presence in the here and now. Importantly, there is no separation of mind and body in this form of immersion. The everyday is experienced in wonder for “spectators” who are fully present as “Embodied Mind” (2008: 116).

Figure 1.

Students experiencing immersive videos of the same park where they are located. Marie-Ève Vautrin-Nadeau, touch-ups by Jean-Philippe Gilbert. (2021).

Scholarship on immersion in simulated environments often emphasizes cognitive immersion, or the suspension of disbelief that takes place in an illusionistic space that simulates reality, making the fact of mediation disappear in the experience. Marie-Laure Ryan writes that: “immersivity can be understood in two ways: in a properly VR sense, as the technology-induced experience of being surrounded by data, and in a narrative sense... as being imaginatively captivated by a storyworld” (2015: 230). Both of these definitions rest on the notion of cognitive immersion. Grounded in the field of post-dramatic multimedia performance, this paper will focus instead on immersive storytelling that activates the senses in a phenomenological experience. Rather than transporting the spectator into a fictional imaginary space, post-dramatic multimedia performance aims to make participants aware of their presence in the here and now (Klich and Scheer, 2011: 128).

This paper will describe an immersive storytelling project, titled VerdunReality: Riparian Play (VerdunRéalité: en jeu riverain), ¹ that integrates VR into live participatory performance events that take place outdoors. The paper is co-authored by an artist-researcher and two students who are working as research assistants on this project. We recount our creative research process in developing a pervasive game, which Montola defines as a “game that has one or more salient features that expand the contractual magic circle of play socially, spatially or temporally” (2005: 3). This game is played in a park and at key moments, inside VR environments that simulate that same park. The purpose of the game is to attune participants to the species in that particular environment. The workshop with elementary school children described above was an initial test in preparation for these games. Students viewed simulations of the park that they were in, but without the interactive gaming components. Their effusive reactions are already a strong indication of the potential of these games to develop environmental perception.

This project asks: (how) can immersive and interactive games produce sensory, embodied knowledge and increase environmental awareness? It proposes that in current urbanized cultures, immersive storytelling can allow us to reimagine relationships between humans and other species and to develop deeper connections with natural environments. This is a process of attunement in which players develop heightened awareness of their environments through activities that are intended to train the senses. VerdunReality amplifies perception with the help of immersive technology. By intensifying sensory awareness, players move between the body (what Simonsen calls “the geography closest in”), and larger scales or rhythms—becoming alive to their dynamic relations with other bodies. Studies in computer-mediated communication reveal that players inhabiting the simulated experiences of animals feel heightened sensations of presence and connections to nature (Ahn et al., 2016). VerdunReality builds on that research, for example, by positioning players as monarch butterflies inside a VR game.

Below, we will define the objectives of this immersive storytelling game, situating it within the field of post-dramatic multimedia performance. We then describe the aesthetic decisions made in its development, and analyze the challenges encountered along the way. While audience reception and impact are important to fully answering the questions posed above, because VerdunReality will not be presented to the public before the summer of 2023, analysis of reception will be the topic of future research. This article will focus on analyzing the development of the piece for multigenerational audiences.

Other works

In 2022 the principal author released an immersive, interactive website: CONEY ISLAND MTL (https://coneyislandmtl.com/). This is a bilingual digital storytelling project that features a map of Verdun, Montréal, Canada. There are six stops along the St Lawrence waterfront, indicated by red pins on the map. Users can open 360 videos by either clicking on these pins, or on thumbnails displayed lower down on the homepage. The videos depict the local landscape, including plants, animals, and insects that live there. The immersive audio features a narrative account of the history of this site, as well as the voices of residents, (landscape) architects, biologists, hunters, fishers, and foragers who speak about their various interests in the place.

I selected two of these videos: “Battlefield” and “Erosion,” as components to develop within another immersive story, VerdunReality. My goal in this new work is to create a live, participatory performance, structured as a tour along the St Lawrence waterfront in Verdun. At two points along the performance route, participants will stop and sit, put on VR headsets, and view the video that corresponds with those GPS coordinates according to the map on CONEY ISLAND MTL (https://coneyislandmtl.com/). Further, I have worked with students to add interactive elements to these two scenes in order to transform them into games that can be played on Oculus Quest 2 headsets. ² The idea is that while gameplay is initiated in the simulated environment, it will continue in the actual park, with other players, once headsets are removed (Figure 2). Each performance is designed for a group of six to eight players. Our user-testing has focused on adult players, but groups of any age can be accommodated.

Figure 2.

Documentation from dress rehearsals in Verdun, Montréal ©2022, photo by Ambre Marionneau

There are particular challenges involved in developing games out of 360° videos, which I will return to below. For now, we will proceed with a very short overview of selected works that can be compared with this one, in an attempt to situate VerdunReality within an emerging genre of VR performance. In this section, Luana describes four projects that are comparable to VerdunReality on these three points:

—They are live performances that use VR

—They are participatory performances

—They are performances that engage with place, environment, and/or the senses.

While the original intention was to identify works that include all of the points above, it was difficult to find a single VR performance that incorporates all three interests. In general, many contemporary works have explored environments and places through VR, especially in recent times, when people could not travel or even leave their homes. In this context, 360° movies were a way to transport people to places where they couldn’t physically travel. These include projects such as: The 7 Wonders of The Modern World (2020); National Geographic Explore VR (2019); Antarctic Heritage Trust (2020); and Rome Reborn: The Pantheon (2019).

Comparable live performances include The Ordinary Circus Girl (2020), a production by Fheel Concepts. This work explores the five senses through mixed reality. Participants experience the circus through 360° video and sound recordings from the show. While embodying the role of the aerial artist Anna, olfactory stimulants are diffused throughout the space, and tactile elements are also employed. For example, cloths are thrown over the participants’ faces to represent the fabric used in the aerial dances. This helps participants to experience sensations that are similar to those that an aerial artist feels when performing her acrobatics. In this work, as in VerdunReality, the senses are heightened through a combination of simulated and physical provocations.

Some other pieces that help to situate this project within a developing field or genre include: Éden (2019) developed by the MxM collective, in which simulated images are loaded in real-time following the participant’s gaze through the environment; Keto (2020), a production by INVIVO collective that (like Biolum, analyzed below) explores marine life through a sensory journey; Encounters nº 1 (2021) by artists Mathieu Pradat and Poll Pebe Pueyrredón (La Prairie Productions), is an introspective sensory exploration. During this extended reality experience, participants view a film inside a VR headset while walking through a dark room where several people talk and laugh in circles. Attempts to get closer or engage with these groups are ignored. During and after the experience, many people report feeling lonely and anxious, indicating that such games can have lasting effects on the individual’s way of relating to other people and the world. VerdunReality similarly aims to evoke sensations and affective responses that are carried between virtual and physical environments.

While the works mentioned above provide reference points for this project, four immersive stories will be examined in greater detail since they resemble more closely the objectives listed at the beginning of this section:

Eyes of the Animal (2015), Marshmallow Laser Feast

This is an immersive, live performance that introduces users to the different species that live in the Grizedale Forest, located in the north of England. The idea of the producers is to mimic the sensory perspectives of animals inhabiting this forest, through VR. Users are invited to go beyond their own senses to imagine the world through the eyes of these beings. While the work now travels, it was initially sited in secluded parts of a forest in the United Kingdom. The artists were intent on aligning physical and virtual worlds, and the Rift headsets used for this installation were decorated to blend in with the natural environment. The visual effects of the film vary according to the sensory experiences of the story’s various animal protagonists. The binaural sounds in the video were recorded in the surrounding forest, using a remote sensing technology.

Describing its intentions, Barnaby Steel, co-founder and creative director of Marshmallow Laser Feast (MLF) explains that:

Coming from a photographic background, I'm tuned into the way that new perspectives on the everyday world can reconnect viewers to the wonder we experience as a child with fresh eyes. The repetition of day-to-day sensory bombardment can make us numb to the world around us, and VR can work like an unblocker, it shakes our snow globe. (Fabbula, n.d.)

MLF is perfectly aware that the idea of using VR to connect to the environment seems counter-intuitive. But these artists encourage us to avoid seeing technology and nature dualistically. “Virtual reality, this cutting edge [sic] technology that flowers from the earth through the minds and hands of some very smart people, might prove to be a powerful means of reconnection to the rock that makes this whole show possible. Using VR to connect with nature sounds ridiculous, but the experience can be profound” (Fabbula, n.d). The artists correctly point out that VR infrastructure is made of components mined from the earth. This is a thoroughly material architecture. With regards to the virtual though, it is a technology that can facilitate imagining the world from other perspectives. Steel goes on to talk about the difficulty we have in doing this, like a blind man trying to see, and as much as we try to imagine life from the points of view of other species, it is very difficult to really connect. However, VR has extraordinary potential for provoking our imaginations of other worlds and experiences (Hitti, 2019).

Eyes of the Animal is one of a handful of works that we have managed to identify that compares with VerdunReality in all its main objectives. Like MLF, we too wish to encourage participants to see the world from more-than-only human perspectives and are intent on provoking connections to the environment through the senses. To do this, it is important to locate this installation in a natural environment. The principal author is particularly interested in the parallels and overlaps between the physical places where the games are played, and the locations represented in the immersive environments. The interaction techniques that we are working on are also similar. Specifically, these include the combination of 3D assets (from Lidar scans and photogrammetry, in the case of MLF), 360° video (including from drone footage), binaural audio, and the introduction of sensory stimuli. ³ While many of the technological and aesthetic strategies are comparable, a notable difference between the two works is the use of voiceover narration in VerdunReality.

We Live in an Ocean of Air (2019), Marshmallow Laser Feast

This is a second MLF project produced by visual artist Natan Sinigaglia and composer and sound artist Mileece I’Anson. It is a continuation of their Treehugger project, and its main purpose is to allow participants to experience a poignant exchange between their bodies and nature in a live performance. According to the show’s creative director, Barnaby Steel: “We see the relationships of all bodies of nature with other bodies... We often tend to think that somehow we are separate, but here we see the connections we have. We’re taking people to experience the world beyond the limits of their perceptions. And we’re further exploring those narratives” (Brownstein, 2021).

The experience is immersive and multi-sensory, depicting the heart of Sequoia National Park, California, and showing the invisible process of breathing by making participants’ breath visible through particles, and responsive to gestures, such as hand movements. To experience it, visitors use a VR headset with headphones for binaural sound, heart rate monitors and breath sensors, scent dispersion systems and wind machines to extend connections with the natural world. “The human cardiovascular system interacts with very similar systems of nature. Our capillary networks, arteries and mitochondria exchange with the leaves, the phloem and the mycelium, placing each of our inspirations and expirations in a large system of reciprocities” (Fabbula, n. d.).

The methods and intentions of We Live in an Ocean of Air are similar to those in VerdunReality. The central idea of the project is to offer a new way of looking at and relating to nature. Philosopher of phenomenology Maurice Merleau-Ponty described the expressive space of the body’s movements as made up of gestures that link one body to another (through rhythms of breathing, for example). Through these gestures, we become “emotionally in touch” (Simonsen, 176). Moreover, the two works share an interest in raising environmental awareness through the senses, presenting a familiar phenomenon in a way that is surprising. Some of the aesthetic techniques are similar to those in VerdunReality, such as: the use of 3D animation, 360° video, real-time interactions, and the introduction of sensory stimuli, such as scents. An important difference though is that We Live... exists as an installation that is presented in an equipped room (such as a gallery).

Biolum (2021), directed by Abel Kohen, co-written by Jon Rowe and Abel Kohen

In this VR experience, the participant is taken on an expedition to the bottom of the sea, assuming the identity of a scientist named Rachael. During the exploration, the character discovers the mysterious relationships between parasites, plants and fish. Through inter-species communication, light is produced in the ocean depths. The story aims to reveal the enchanting workings of nature:

Biolum shows captivating images of the underwater world, taking viewers on an otherwise impossible journey. It also addresses the themes of pollution and climate change, explaining the decline of various species of fauna and flora. This science fiction scenario depicts humans not as the species that dominates the world as a whole, but rather as a tiny part of a much larger ecosystem. (Vena, 2021: n.p.)

Like VerdunReality, the central motivation behind this project is to engender a new way of looking at and relating to nature. Through VR, Biolum attempts to raise awareness of other-than-human habitats, ultimately provoking feelings of awe and appreciation. Some of the interaction techniques in this piece are similar to those used in VerdunReality, such as: the integration of 3D elements, 360° video, and real-time interaction. Notably different in Biolum is the fact that the videos are computer generated, rather than recordings of physical environments. Further, the narration in Biolum is based on a story that revolves around a fictional character.

Be Earth #13 (2020), XR Impact

This project was developed by the group of artists Ylva Hansdotter (director), Boo Augilar (design producer) and Paulo Gibbs (3D artist). It arose from the group’s desire to depict the impacts of human actions on the environment through VR. Based on “UN Sustainable Development action n. 13: Climatic Action,” the experience takes place in a context where participants embody the Earth (as an avatar) and become part of the Amazon rainforest. The artists use Leap Motion hand tracking, with participants’ hands becoming part of the scenario. As the story unfolds, participants are invited to resolve natural catastrophes with hand movements. These prove to be fatalistic. As the participant attempts to solve the problems, for example, stopping the strong wind, or putting out the fire, these efforts inevitably fail. This project seeks to provoke reflection in participants about their roles in nature.

Like all the works described above, apart from early iterations of Eyes of the Animal, Be Earth #13 is also presented as an installation indoors. This is a major point of separation between the works that we have studied and the goals of VerdunReality, which has been designed as a site-specific work, meaning that it was created for one specific (outdoor) location. The rationale behind the decision to design a VR performance for a particular place, as well as the limits and affordances presented by this choice, are evaluated below. For now, this brief overview of comparable projects offers insight into the intentions behind VerdunReality, while attempting to situate it within an emerging genre of VR performance.

Objectives of VerdunReality

The main objective of VerdunReality is to increase the environmental perception of participants, in a particular time and place. To do so, the work produces immersive experiences at precise moments during a live performance along the St Lawrence River in Montréal’s borough of Verdun. This multimedia storytelling experience is an extension of a larger project. As described above, CONEY ISLAND MTL is a web documentary that explores inter-species relations along Verdun’s waterfront. The website features an interactive map displaying six pins. Clicking on any of these red pins opens a 360° video. The videos depict the local landscape, with 3D renderings of birds, fish, and butterflies added to the scenes. A narrator tells stories about the history of this site, punctuated by excerpts from interviews with local residents, fishers, foragers, hunters, historians, and biologists.

Two of the six videos, “Battlefield” and “Erosion,” were selected for further development within the context of a live multimedia performance. Clémentine Brochet and Gaëlle Issa, two recent graduates from UdeM’s diploma program in art, creation and technology, were hired to add interactive elements to these two 360° videos. These were designed for Oculus Quest 2 headsets. Clémentine was in charge of programming a user interaction so that when participants fix their gaze on one of the common milkweed plants in the “Battlefield” scene and maintain that gaze for one second, silk is released from the plant, its Latin name, asclepias syriaca, appears onscreen, and a point is displayed beside the name. It is possible for participants to win up to 8 points for correctly identifying milkweed plants in the 6 minutes that it takes to experience “Battlefield” (Figure 3).

Figure 3.

Screen capture of “Battlefield” scene from CONEY ISLAND MTL https://coneyislandmtl.com/ ©2022 Natalie Doonan

Gaëlle, meanwhile, took charge of transforming the giant monarch butterfly in this same scene into an avatar. This means that when the participant enters the scene, they embody the monarch, taking on its point of view. Essentially then, the game in “Battlefield” involves imaginatively becoming a butterfly who flutters above a milkweed field, identifying the plants that are available for eating and egg-laying. While doing so, the participant listens to narration about monarch-milkweed-human relations. For instance, the narrator explains that: “Many pollinators, at one stage or another of their life cycle, feed exclusively on milkweed. Monarch butterflies can only lay their eggs on milkweed leaves and their caterpillars feed solely on this plant. The survival of monarchs is thus intimately connected with the subsistence of common milkweed” (Doonan, “Battlefield,” from CONEY ISLAND MTL). The idea is that after this video concludes, players remove the headsets and continue the game of plant identification in the physical environment of the park, alongside other participants. Each performance will accommodate up to eight players.

In the second video, “Erosion,” Clémentine programmed an interaction that allows participants to grab a torch that appears approximately midway through the video and floats at about human torso height (Figure 4). In the live performance, each participant is accompanied by a guide, who places a placard in their hand when they reach out for the 3D torch. The guide discretely blows smoke around the players from a cocktail smoke gun to further blur virtual and physical experiences. The location and movement in the video correspond with the physical site of this walk. When players remove the headsets this time, instead of holding a torch in their hand, they discover that they are in fact holding a sign that reads: “Feelin’ Hot, Hot, Hot!”, (or a variation on this heat-related message) which has been placed in their hand by their guide (Figure 5).

Figure 4.

Screen capture of “Erosion” scene from CONEY ISLAND MTL https://coneyislandmtl.com/ ©2022 Natalie Doonan

Figure 5.

Documentation from dress rehearsal VerdunReality ©2022 Natalie Doonan

Sensory immersion in VerdunReality

In both the “Battlefield” and “Erosion” scenes of this live performance, there is a slippage between virtual and physical presence. In putting on VR headsets, participants are not drawn into fictional, imaginary spaces, but are rather guided into a closer investigation of their immediate surroundings. Drawing from what was observed during the youth workshop described in the beginning of this article, using VR at key moments within a larger event seemed an effective strategy for grabbing and focusing attention on particular plants, animals, and environmental phenomena. After engaging participants in this way, the intention is to orient this targeted attention toward the physical environment.

As we saw in the section above, “Other works,” most VR experiences are designed for indoor spaces. This work is different in that VR components are used to punctuate the live performance, zooming into specific elements in the physical environment to increase participants’ sensory attunement toward certain plants, insects, animals, and environmental phenomena. The sensation of presence becomes heightened through these simulations, but carries into the actual park, where the game continues with other players. The intention is to present the work to the public in the summer of 2023. Unlike comparable works described above, this one is limited in its distribution since it is designed as a site-specific performance.

Theorists such as Marie-Laure Ryan have developed sophisticated analyses of cognitive immersion in digital storytelling, by focusing explicitly on VR as representation. As Klich and Scheer point out though, this study of immersion as a cognitive process is inadequate for post-dramatic theater and postmodern performance practice. VerdunReality as a form of immersive storytelling depends upon engagement of the entire sensorium within the immediate physical space of the performance, creating “an enhanced state of being in relation to the surrounding space and responding to immediate stimuli” (2011: 131). This understanding of immersion as corporeal resonates with the post-dramatic theater analysis proposed by Klich and Scheer, who explain that: “the concept of immersion relates to the audience members’ level of sensorial stimulation at any one moment, and their awareness of being within the present of the performance and its capacity to engage them emotionally and corporeally (heart racing, hair-raising, sweating and fidgeting)” (2011: 131).

In VerdunReality, a group of six to eight participants finds itself in a spectacular, partially groomed riverside park. The group is welcomed by a guide, who leads a 90-min tour along the bicycle path extending alongside the weedy shore. Six supporting guides, dressed as park rangers, transport VR equipment and other props in wheelbarrows. They facilitate the experience of participants, fluidly setting up commensal spaces at key moments along the route. At the first stop along the tour, a picnic blanket is gracefully spread out and participants are invited to sit and drink a cup of goldenrod tea. The supporting guides are there to ensure comfort but also safety. During the two “gaming” stops along the tour, where participants don VR headsets, each person receives personalized accompaniment. A hand is gently placed on the shoulder or back and arm, to anchor participants and to ensure against accidents.

There is great attention paid to the performative and theatrical elements of the tour. Participants are carefully guided through immersive experiences in (non)fictional and physical-virtual spaces. The VR headsets are decorated to mimic the environment. While participants of VR experiences are always on display for onlookers, this fact is not usually considered as an integral component of performance design. In VerdunReality, it is crucial. In putting on the headsets, participants are also dressing in costumes that echo elements of their immediate environment. In wearing these headdresses, players do not blend in, or become camouflaged in their surroundings, but rather embrace conspicuous roles as actors in the park environment. In some moments, this “active” engagement is pushed even further, bordering on an “activist” role. As mentioned above, players are encouraged visually to grab a virtual torch during the “Erosion” game. This corresponds to an offering made by a guide to take hold of a physical object, which turns out to be a sign reading “It’s getting hot.” Participants only see what they are holding after removing their headsets. This slippage between fiction-non-fiction and virtual-physical creates an immersion that cannot be adequately described as representational or cognitive, as it increases both agency (a term that will be defined below) and meaning of the play experience by providing a reminder of what has just been seen through the headsets: fire and smoke. This is a fully embodied performance. All of the senses are engaged throughout the tour.

This experience is designed for small groups. It requires individualized attention toward each player. The objective of attuning participants to a particular place has limited the potential reach of the work. In addition to the challenges that we have faced in reaching wider audiences due to the small scale and site-specificity of the performance, our efforts toward making it accessible have been put to the test in other ways too. A major barrier to participating in VerdunReality is the fact that its geographical trajectory proceeds along steep inclines, very narrow footpaths, and through thick, tall, grasses (common reeds). This poses an impassible obstacle for people in wheelchairs. Moreover, the design of VR headsets has been widely critiqued for being modeled on an average adult male head. Many people find the headsets uncomfortable, and the close-up screen image causes vertigo in some, making it unbearable to watch. These physical barriers may in some part be reduced or eliminated with future developments in VR technology, but the terrain itself will continue to limit access to the piece to participants who are able to sustain the inclines and narrow passes. We decided to maintain this route because the embodied experience of the waterfront path and intimate proximity with specific plants, especially milkweed and common reeds, is important to our story. However, because each person is paired with a guide, we are able to offer a modified route for anyone who is unable to manoeuvre along the narrow footpath. 

For the moment, we have plans to present VerdunReality to a small group of artists during a residency in the summer of 2023. The goal of the residency is to examine methods for encouraging participation in multimedia performances. One question for discussion on that occasion will be what audiences would most benefit from VerdunReality, and more generally from the methods developed through this project. A few performances will also be advertised locally and will be open to the public following the artist residency. This will allow us to share the work with inter-generational audiences.

We will return to a more detailed discussion of how design limits have influenced the development of the work. First though, Christopher presents the precepts of gamification and explains the benefits associated with VR and immersive virtual environments. This will help to explain the choice of VR as a tool to develop sensory attunement. Christopher will then present the concepts of meaningful play (Salen and Zimmerman, 2003), pervasive play (Montola, 2005) and critical play (Flanagan, 2013) which allow us to reflect on the complexity and permeability of the interval that exists between the player’s space and the game space.

Virtual reality and immersive virtual environments

Research on VR and Immersive Virtual Environments (IVEs) in educational and activist contexts demonstrates that they can provide an effective means of changing user behavior. In their research on internal locus of control and behavior (an individual’s tendency to attribute external causes to their own actions), Ahn and colleagues (2014) found that participants who interacted with deforestation awareness content in VR IVEs consumed 20% less paper following the study than the group who read a print description of tree cutting. In another experiment, participants in an IVE viewed a simulation that depicted the environment from the perspective of a person with color blindness. Researchers found that afterward, these participants spent on average twice as much time helping people with color blindness as other participants who only imagined being color blind after an explanation (Ahn et al., 2013). In their research aimed at improving the response time of populations living in high-risk flood areas, Zaalberg and Midden established that the use of IVEs “increased not only the motivation to evacuate from the threatened virtual polder, but also increased the motivation to buy flood insurance in the real world,” whereas other media used for raising awareness, such as shocking images of flooding, seem to have less impact on behavior (2010: 209).

Although the long-term effects of VR are still unknown (Murphy, 2017: 2), VR experiences and IVEs likely possess strong persuasive power and allow for observation of how people respond in a variety of contexts, since research shows that people respond in VR as they would outside of VR (Markowitz and Bailenson, 2019: 294). In our case, it is expected that the behaviors encouraged during the performance will be transposed into the everyday life of the participants. For example, if we succeed in stimulating an interest in recognizing plants or bird songs, it is likely that this curiosity for flora and fauna developed during the performance will be transmitted into the daily life of the players, in varying degrees. They may become more aware of their surroundings when they step onto a grassy field or when a bird sings, only for the excitement of deciphering which animal is behind the sound or what type of plants are present. But since not everyone has a curiosity about the environment to begin with, strategies are employed in VerdunReality to quickly and effectively capture the attention of participants.

The gamification of content

As Murphy explains, a VR experience must successfully reach the interest of users (2017: 11) and interest can be generated by occasioning concerns upon which emotional responses will be based, generated, and directed through different aspects of VR gaming experience (Murphy, 2017: 11). Interest in VerdunReality is generated in part through the creation of a site-specific performance. Players are able to immediately discern the real-life implications of what they hear and see in the IVEs as soon as they remove their headsets. For instance, when biologist Eugénie Potvin explains in the “Battlefield” scene that mowed park areas are food deserts for bees, this has an immediate effect on participants who find themselves sitting in just that food desert.

Beyond the benefits of site-specific performance to drive interest, gamification, which can be defined as “the transposition of elements of the game structure into non-game contexts” (Le Lay et al., 2021: 1; Deterding et al., 2011) is now seen as a functional means to enhance the appeal of serious issues. The guiding principle behind gamifying activities that are not playful in nature is that user motivation should increase if game elements (scoreboards, immediate feedback, reward systems, etc.) are introduced to create new intrinsic motivations related to the activity. Roger Caillois proposed the idea, in his book Les jeux et les hommes (2009 [1958]), that games enrich different pleasures based on systems that are built on four types of play: pretending (mimicry); competition (agôn); randomness (alea); and thrills (vertigo). The gamification of virtual experience aims to build a new appeal for serious situations based on these four types of ludic experience.

Research on the use of video games to address real-world issues demonstrates the potential of playfulness to enhance interest. For example, in their experiment on teaching construction site hazards,Greuter and Tepe (2014) developed a video game called Trouble Tower where students were asked to identify construction hazards and control measures and then identify occupational health and safety communication and reporting processes. Students testified that they had fun playing, learned things, and that this learning was more engaging than traditional learning in a lecture (Greuter and Tepe, 2014). Bang, Svahn, and Gustafsson developed an application (Power Explorer) in their project, a mobile game that turns the entire house into a playground for learning about one’s electricity consumption and highlighting the perverse downsides of overconsumption of electricity, a topic that gets little to no attention (2009: 5). Serious subjects, such as the importance of biodiversity on the Verdun waterfront, seem better received and absorbed when the context is favorable to the development of a playful attitude, instead of a purely informative, provocative, or educational context. As Luana notes above, the headsets focused the students’ attention and without them, the children became quickly distracted. This is probably because they were no longer able to have a playful attitude during the more structured return to the group.

Pedagogical studies have shown that active learning methods, for instance placing students in small groups, and using cooperative learning structures in which they are faced with realistic problems, help to foster mutual interdependence, positive student attitudes, knowledge retention and life-long learning skills (Prince, 2004). Moreover, place-based education has demonstrated the importance of a centuries-old claim that: “Knowledge of the nearest things should be acquired first, then that of those farther and farther off.” ⁴ Educational standardization annihilates the specificity of places, felling affective connections to students’ own neighborhoods (Sobel, 2004). What curator Lucy Lippard (1997) calls “the lure of the local” can resuscitate students’ passion and desire to engage in their communities. We expect that the same is true for most people, not only for students, and not only for children. The importance of localizing environmental education has underscored that a sense of place in children “fosters attachment and bonding with the natural world, and is grounded in the resources and context of the community” (Miles, 2008, 4). By developing a localized sense of place in people of all ages, environmental educators hope, by extension, to “develop care and concern for other places as well” (4). Significant here is the opening presented by place-based pedagogy to acknowledging the polyvocality of place, human and more-than-human. The presence of marginalized populations is part of what the dominant culture must strain to hear too.

Whether for educational or recreational purposes, games appear to be an attractive entry point to more serious topics, and the principles of gamification allow the idea of play to be extended to other more serious spheres. Knowing about the diversity of life in one’s neighborhood is an important step towards its preservation and respect, but becoming attuned to specific plants and animals is facilitated by an already existing interest in other species. By creating a playful event, we hope to cultivate interest in the fauna and flora of Verdun, which will then predispose participants toward active listening and situational awareness.

Moreover, Santos and colleagues (2021) demonstrated with their mobile application Shinpo that social interaction within games promoting physical education appears to have a positive effect on physical activity levels. This points to the importance of sharing the experience, whether during the activity or afterward, in order to consolidate the motivation and interest of participants. Beyond playfulness, the immediate environment must be considered as a shared space favoring exchanges, discussions and feedback. In VerdunReality, these exchanges are integrated into the performance, through the staging of commensal spaces for playing, eating, drinking, talking, and exploring, as described above.

Design challenges

While these goals may seem idealistic, it is important to address the significant challenges that we have encountered along the way. The first of these had to do with the fact that both VR games in VerdunReality were developed from 360° videos. Most VR headsets, such as the HTC Vive, Oculus Quest 2, or Valve Index, now allow manipulation of the virtual environment using twin controllers that simulate the participant’s hands (and sometimes fingers) in the IVE. Technological devices ensure that the participant’s hands and head follow the same movements in the virtual world as in the physical space, creating a synchrony of movements between the two spaces. However, the use of 360° video, while maintaining the filmic fidelity of the Montréal borough of Verdun, entails challenges in the creation of the IVE and the playfulness of its content. Interacting with a two-dimensional video in a three-dimensional virtual environment requires thinking about the interactivity of the participant in relation to an object whose content cannot be altered due to the limitations of the filmic medium.

These limitations have been addressed in a variety of ways since the arrival of Full Motion Video (FMV) during the advent of digital games. The emergence of CD-ROMs in the early 1980s allowed for a significant increase in storage space for digital games on computers and consoles. To take advantage of this increased storage space, developers began to include live-action video footage in their video games that were too large for the previous supports (cartridges). These sequences could be presented between game moments as cutscenes or directly as background scenery during game sequences: the content of the various surveillance cameras that are used in the game Night Trap (Digital Pictures, 1992); the introduction to the game Transference (SpectreVision, 2018) that is filmed in live action; the various shows of The Twilight Zone series that can be viewed on small televisions scattered throughout the world of Alan Wake (Remedy Entertainment, 2010) are also all in live action; and most of the cinematics in the real-time strategy game Command and Conquer: Tiberian Dawn (Westwood Studios, 1995) are filmed with real actors. All of this is to say that, while the use of live-action video is fairly unusual outside the old genre of FMV games, it is not uncommon to see games that employ this strategy from time to time to produce a particular aesthetic.

This strategy allows for the integration of photorealistic (or at least cinematically realistic) imagery. Developing the agency of a player during a filmed sequence, however, does not require the same design work, mainly in terms of movement. If, in a conventional game, the player can move in a two-dimensional virtual space (top to bottom, left to right), or three-dimensional (forward and backward) to explore the environment, it is not possible for the player to move in a video. He or she can move on a video (overlay), between videos (transition), zoom on a video (focus) or play with the temporality of a video (forward-backward, jump cut, etc.), but not in the video: the capture of the shots is fixed. If the left flank of a bird has not been filmed, it simply does not exist on the film and it is not possible to turn around the animal, since the camera itself has not turned around it.

However, as Ball explains, “interactivity in the context of VR is defined here as the users’ perception that they can modify the ‘form and content of a mediated environment” (Ball 2020: 1). The question of interactivity is closely linked to the concept of agency, or “potential for action,” which Janet Murray describes in the case of video games as “...the satisfying power to take meaningful action and see the results of our decisions and choices” (2017: 123). But this potential for action remains artificial in the case of a video game. As Stang states:

Videogame interactivity, and the sense of control that it elicits in players, is illusory. Player input causes the game system to react in a specific, pre-coded way and, given our current lack of true artificial intelligence able to adapt and generate content in reaction to unpredictable human behaviour, player choice is necessarily limited. (Stang, 2019: n.p.)

It is the perceived sense of agency that allows for a multitude of things, since it does not matter if participants can’t change the content of the videos. If a player feels that their actions in the virtual space are meaningful, the sense of agency is likely to emerge, even if the interactivity with (or within) the content is limited. For example, players can’t directly interact with the visible and audible flowers or birds in the 360° videos, but they can interact with certain elements in the performance that are both virtual and physical: 3D animations, props, sounds, smells, etc.). The video becomes by association and design an integral part of an immersive and interactive experience. The blending of mediated and unmediated experiences is key: the smell of a flower, the wind in the park or the heat of the fire cannot be perceived with a VR headset, but aromas are diffused in the air, wind blows, and the heat of the sun is felt poignantly. Echoes resonate throughout virtual and physical facets of the performance. “It’s getting hot” for instance, recalling simulated fire while also drawing attention to the current temperature, and climate. In other words, it is not necessary for participants to be able to interact with the 360° video content in order for them to have a sense of agency toward its content, since many aesthetic strategies are used to achieve this result.

On a related point, Murphy points out that too much realism is detrimental to capturing attention: “It can be broadly asserted that perceptually realistic environments are more likely to lead to incorrect appraisals of IVE authenticity by incurring higher cognitive processing costs. In other words, it takes longer to correctly identify perceptually realistic stimuli as not-real” (2017: 10). The fact that the games in VerdunReality are built from 360° videos as a foundation can lead to “cognitive overload.” The videos are rich in detail, with content in all directions, rather than being concentrated in front of the spectator. In adding interactive elements to “Battlefield” and “Erosion,” a main goal was thus to focus participants’ attention on specific elements in these environments. For example, in the “Battlefield” IVE, participants are tasked with identifying common milkweed amidst a field of plants.

Considering: the characteristics of the medium (VR and IVE); the advantages of gamification; and the perceptual aspect of agency, we point out three challenges in the transformation of 360° videos into interactive games: (1) what actions will the participants be able to perform and how to make these actions meaningful to the person? (2) How can we guide the experience so that it is accessible? (3) how can we work on the performative and participatory aspects of the experience while respecting other species in the physical environment? In response to these challenges, we propose the concepts of game mechanics within the framework of meaningful play. We then discuss the notion of ease in virtual worlds, and then finally underline the importance of developing connections between virtual and physical space.

Game mechanics

A game mechanic is an action verb conceptualized for the purposes of a game that will be performed to accomplish formal or personal goals. For example, the game mechanic “jump” in a platformer game allows the player to jump from platform to platform to reach the end of the level. A “shoot” game mechanic allows the player to neutralize active threats that prevent the completion of objectives. Finally, the “talk” game mechanic allows the player to talk with the characters that populate the fictional world of the game to learn more about themselves or what is happening in the story. A game mechanic must be thought through in relation to the desired experience (what do we want the player to experience?), the objectives to be accomplished (what will be required of the player?) and how it is meaningful in relation to other parts of the game.

The idea of meaningful action, or more specifically “meaningful play,” is a concept introduced by Salen and Zimmerman, which can mean either the fit between the action taken and the outcome, or when the relationships between the actions and outcomes of a game are both discernible and integrated into the larger context of the game (2003: 34). For each design project, one must ask how a particular mechanic echoes the overall game, the underlying message, and the goals of the production. In their work on raising awareness of deforestation through the use of IVEs, Ahn and colleagues questioned the counterproductive nature of asking people to cut down trees to demonstrate the dangers of deforestation, thereby encouraging the action of deforestation by performing the task that must be avoided (2014: 236). On the other hand, asking participants to cut down trees in a virtual world is an elegant solution: the harmful effects of cutting down trees on wildlife are experienced without a single trunk being damaged. Care must be taken with the implementation of game mechanics, as the agency of the participant must not contradict the overall message by the enactment of dynamic rules and models that allows arguments to be created and ideas to be conveyed in a persuasive manner (Bogost, 2008: 125).

The challenge is to develop game mechanics that do not contradict the messages that we want to convey. For instance, at first glance, it may be interesting to allow participants to pick flowers in Verdun to increase their agency in the virtual environment, but in a logic of transferring behaviors from the virtual experience to their actual surroundings, we create a conflict between our statements: we want to train environmental attunement, but we don’t want to encourage picking plants that are endangered or important to other animals and insects. It is thus important to choose wisely what players will be able to do in the IVEs, but even more so what they will not be able to do, carefully considering why they should not be able to perform certain actions.

Ease in the virtual world

Another challenge is designing the games for a range of ages and experience levels. It must be taken into consideration that not all people are accustomed to virtual worlds. As Markowitz and Bailenson state, “...the combination of highly immersive technology and the belief that people are psychologically attuned to the virtual world are two crucial ingredients for a successful VR experience” (2019: 287). It is a misconception that orienting oneself and making sense of an IVE is a given for everyone: a person who is unfamiliar with computers and technology will not have the same level of comfort as a gamer who spends several hours a week in this type of environment. Given that parts of our performative experience take place in a virtual world, this difficulty must be considered to accommodate the largest number of people.

Initiation to a new video game is usually done through a tutorial—a didactic sequence where the player is guided in their exploration of game mechanics. Similarly in VerdunReality, a guide accompanies each participant. The difference is that the guide in VerdunReality is a human being, who provides assistance in physical space. Explaining what the experience is about, answering questions, and listening to the needs of participants in real time is crucial for them to feel comfortable performing the experience in front of others, using a medium they may not be used to.

In VerdunReality, IVEs are used to attune participants to the biodiversity of Verdun, but we must take into consideration that players will also have to be attuned to the simulated environments that will be presented to them. Attunement in this work is thus a process that is enacted across different facets of reality. As Denham and Spokes argue in their article about rural retreatism in open-world video games:

…the virtual/non-virtual exist together as facets of the ‘real’: the virtual is ideally real, whereas the non-virtual – what he terms the ‘concrete present’ – is actually real. Therefore, the spatial distinction is ontologically unsound: the virtual and non-virtual are both real, but in different ways, so interconnections between the two are both possible and necessary… (Denham and Spokes, 2021: 1569)

VerdunReality being a mediated space, the work of attunement to the virtual environment happens in conjunction with attunement to the biodiversity of the Verdun waterfront.

The virtual world and live performance

The use of gamification, VR and IVEs are all key to the production of what are called “pervasive games,” which Montola defines as “a game that has one or more salient features that expand the contractual magic circle of play socially, spatially or temporally” (2005: 3). A pervasive game blurs the line between what is in the game and what is outside the game. For example, augmented reality games bring the virtual and the physical closer together by superimposing game elements onto elements of the immediate setting: in Pokémon GO for example, small monsters appear under our counters, in the street in front of our houses and in nearby parks, and it is even possible to capture them, collect them and exchange them with other people, but these little monsters do not actually appear in the physical world. This reinforces at the same time the idea that the virtual and the actual real are two facets of experience.

VerdunReality can be considered as an example of pervasive play because it increases participants’ attunement to their presence in the here and now. They are not just listening to videos about biodiversity in Verdun: they are in Verdun. The experience is not singular, but collective, shared. What surrounds the virtual environment accessible through the headset interface must be thought out in synergy with the immediate space where the participatory performance takes place.

Conclusion: enchantment in everyday life

Combining virtual and physical elements can be magical. Montola identifies a variety of opportunities afforded by pervasive games, such as interlacing games with everyday life, and as a strategy for reclaiming public spaces. Expanding games spatially, temporally, and socially allows creators to “enchance or enchant everyday life with hidden meanings,” he writes (2005, n. p.).

Since the 1990s, there has been a renewed interest in panoramic technologies in the context of exhibition spaces. A range of interactive works and IVEs have been created by media artists, engineers, and specialists in cultural heritage. Sarah Kenderdine’s work Pure Land AR, for example, is a cave-like environment that visitors can enter into and explore. Two handheld interfaces are available for people to enhance this exploration. As the tablet screen is held up toward the walls of the cave, historical paintings appear, recreating the space of the Cave 220 from Dunhuang. Groups of visitors typically gather around the two holders of the tablets, engaging in a social performance as they move collectively to discover the various parts of the cave. As Kenderdine explains, this is a fully embodied experience. This innovative presentation allows visitors access to a simulated rendition of Dunhuang’s cave paintings, which are permanently closed to the public. Hundreds of thousands of people have engaged with the work. This example demonstrates the astounding possibilities for dissemination of otherwise inaccessible experiences.

Designing immersive multimedia performances in outdoor locations however, introduces significant challenges with regards to the number of audience members that can be reached. VR headsets are unreliable. It is prudent to have backup headsets at hand since they frequently present technical glitches. Using them outdoors complicates the possibility of going ahead with performances rain or shine, since the headsets are expensive and cannot get wet. The lenses act like magnifying glasses, which burn holes into the sensors if exposed to sunlight, meaning that great care must be taken in handling them on sunny days too. There are problems with using headsets in bright environments since they are made to respond to dark spaces. Transportation solutions are important to consider since the headsets are bulky. Diligent labeling systems must be established to avoid mixing up headsets and hand controllers. Each one must be individually paired with a cell phone, which creates a host of proprietary and personal privacy issues. In the main author’s experience, pairing too many headsets with one phone causes technical problems. The development of these devices as surveillance and biometric data collection tools is even more troubling. ⁵ Furthermore, the technology is so quickly evolving that after a short time, the manufacturer tends to abdicate responsibility for “older” models so that it becomes impossible to get technical support for them when they do (all too often) break down.

Despite all of this, integration of VR into live multimedia performance holds exciting potential. The portability of the headset offers infinite possibilities for playing with (non)fictional storytelling in the spaces of everyday life. In this article we have discussed the significant challenges involved in integrating VR into site-specific, live performances. In subsequent work we intend to report on audience reception of VerdunReality. The topic of “enchantment,” for instance, deserves closer attention. Véronique Servais and Arnaud Halloy (2014) have developed an “enchantment schema” that has been used by other researchers to inquire into the quality of experience of participants in other contexts that elicit intense sensory responses (Susswein and Tasia, 2020). This avenue is worthy of pursuit since it offers a model for accounting for the marriage of embodied and narrative engagement in immersive digital storytelling.

In their schema, Servais and Halloy propose three phases involved in the difficult to describe experience of “enchantment.” The first involves a preparatory development in which participants are trained in attitudes or dispositions that will prepare them for direct observations. The second phase is what the authors describe as enchantment itself—an immediate, embodied, environmental perception in which there is an intimate connection with an object, actor, or entity. The final phase of this model involves a process of discussion with other participants, in which one’s own emotional experience resonates with others. These three phases seem to be operative in VerdunReality and could be fruitfully explored through future audience research.

At the time of writing, the VerdunReality research-creation team has concluded dress rehearsals and is preparing for user-testing in the context of an artist residency in 2023. Following that, the piece will be presented to the public during a short period. These presentations will offer the opportunity to analyze audience reception and engagement with the work. It is our intention to publish a future article focusing on experiences of enchantment amongst participants. The Magic Circle thus continues to expand.