Sage Journals: Discover world-class research

Abstract

This article reports on one stage of a project that considered twenty 8–12-years-olds use of Virtual Reality (VR) for entertainment. The entire project considered this in relation to interaction and engagement, health and safety and how VR play fitted into children’s everyday home lives. The specific focus of this article is solely on children’s interaction and engagement with a range of VR content on both a low-end and high-end head mounted display (HMD). The data were analysed using novel multimodal methods that included stop-motion animation and graphic narratives to develop multimodal means for analysis within the context of VR. The data highlighted core design elements in VR content that promoted or inhibited children’s storytelling in virtual worlds. These are visual style, movement and sound which are described in relation to three core points of the user’s journey through the virtual story; (1) entering the virtual environment, (2) being in the virtual story world, and (3) affecting the story through interactive objects. The findings offer research-based design implications for the improvement of virtual content for children, specifically in relation to creating content that promotes creativity and storytelling, thereby extending the benefits that have previously been highlighted in the field of interactive storytelling with other digital media.

Keywords

Interactive digital storytelling multimodal interaction Virtual Reality visual research methods multisensory narrative experiences

Introduction

This article focuses on children and interactive storytelling in Virtual Reality (VR). With regard to online and off-line interactive books, storytelling-based computer games and speaking toys, Garzotto (2014) found that children were engaged with these earlier digital media and content as:

Creators’ of interactive stories, individually or in groups. In both cases, interactivity has the potential of increasing enjoyment, and fostering new forms of creativity, social activities and learning. (Garzotto, 2014: 5–6)

This article builds on and extends this work to explore how children engage and play with multimedia stories in relation to the recent wave of VR technologies and content. It describes how similar findings were found in a study of children’s (aged 8–12-years-old) engagement and interaction with Virtual Reality. The Children and Virtual Reality project (CVR) (Yamada-Rice et al., 2017) researched children’s engagement and interaction with a variety of VR content on both a low-end (Cardboard) and high-end device (HTC VIVE). It was undertaken at a point when VR was rapidly emerging, and commercial companies were (and remain) cautious of the potential health and safety issues the technology might have for children. In addition, it recognised that most VR software is designed for adults, and that children often use digital content designed for older ages where content aimed at them is not available; indeed, age ratings on digital games are not an effective means of stopping younger users accessing them (Goltz, 2010; Livingstone, 2008; Stroud and Chernin, 2008). The study aimed to contribute understanding of the specific ways in which children engage and interact with games and narratives on virtual platforms as a way of exploring how to create content for younger children should the market open up for this.

This article first considers the literature in areas of interactive storytelling for children, and children and Virtual Reality. Second, it outlines the methods used to conduct the study and analyse the data including discussion of how experimental means were used to analyse the multimodal data. Specifically, drawing and animation were used as a means of knowing (e.g. Barry, 2015; Ingold, 2013; Mäkelä, 2007). The paper presents and discusses the study findings with attention to the relationship between three design features and the process of interactive storytelling in the VR journey (1) entering the story: onboarding from physical to virtual domain, (2) being in the virtual story world, and (3) affecting the story through interactive objects. By making visible the design implications of VR content for children under 12-years-old the article extends the literature on children’s interactive storytelling and contributes to the field of VR content design and development. This is significant given the growing demand for children’s VR experiences across a range of sectors including entertainment (Yamada-Rice et al., 2017), education (Southgate et al., 2019) and health (Tarrant et al., 2018).

Interactive storytelling for children

Interactive digital storytelling uses digital media to create narrative experiences that can be influenced by the user. To achieve such interaction developers need to produce products that support children’s free-play and creativity (Cassell and Ryokai, 2001). Working with categories of creativity to play (Robson, 2014), Marsh et al.’s (2015) study with mobile apps found open-ended digital content to be the best way for encouraging creativity. Further, Spierling and Szilas (2009) describe how such spaces make co-creation of stories between content designer and user possible. However, Mayr (2014) writes that there is a tension between allowing freedom and creativity with interactive stories and creating good quality narratives:

The problem is that immersion can be achieved by providing the feeling of agency or by telling an interesting story that is close to what the initial story author had in mind. Unfortunately, these two concepts seem to contradict each other. (Mayr, 2014: 59)

With VR, such conflicts can be removed through an immersive feeling of ‘presence’ afforded by the medium. This is achieved through a combination of communicative modes that are tracked to a user’s movement (described below in relation to the work of Bailey and Bailenson (2017)). In interactive VR content the user is also both author and audience, and with child-audiences specifically this immersion builds on an already established connection between play and narrative (Yamada-Rice, 2018a), and between socio-dramatic play and children’s literacy practices (Christie and Roskos, 2009; Marsh, 2014; Roskos and Christie, 2011). This article extends this work, showing how VR takes interaction in story worlds to a new level and allows for a co-existence of a pre-established story. This is achieved by developing open-ended spaces for children’s agency, and creating original narratives within immersive 360-degree environments.

Children and VR

A white paper produced by the VR content store WEARVR (2018) shows the reach and extent to which the Virtual Reality market is growing. For example, they list the current state of play (in the middle of 2018) as having 14 companies manufacturing headsets, 6000 VR content developers and 3 million VR app downloads (p. 6). The ‘Children and Virtual Reality’ report (CVR) (Yamada-Rice et al., 2017) cites market research with children aged 8 to 15 years in the UK and the US (sample size 1300 and 2000 respectively), in which more than half were familiar with what VR was and/or had used it. In relation to VR for entertainment, Rose (2018) describes it as a magnet for non-fiction content, particularly in relation to documentaries where virtual environments are seen as ‘platforms for engaging the real’ (p. 132).

The medium of VR is, however, highly multimodal and sensory-rich:

Immersive VR technology blocks out the physical world and provides rich sensory feedback that utilizes sight, sound, touch, and smell (with sight and sound most commonly used). From a psychological standpoint, VR is the feeling of non-mediation; the sensation that there is no technology between the user and his or her sensory experience. (Bailey and Bailenson, 2017: 107–108)

As with all emerging technology that has gone before, concern is expressed from some areas about the suitability of the medium for children (Bailey and Bailenson, 2017). In part, this is illustrated by the fact that most head mounted displays (HMDs) are restricted to children aged 13 years and above. It is hard to find evidence as to why this decision has been made but one explanation is that Oculus Rift set the age limit of 13+ when the company was bought by Facebook (Solomon, 2014), aligning it with the social media age restriction of that platform. This has parallels with Grimes’ (2018) research on the age limitations and bans on child players in the early days of virtual worlds and the need to critique problematic notions of suitability that can arise from the ‘child/adult dialectic that permeates much of how we organize our social structures’ (p. 638).

In addition to regulations imposed on HMDs, the appropriateness of VR content is rated in relation to users’ age using the PEGI classification system. As a result, current research on children’s use of VR is largely on VR for health (Tarrant et al., 2018) and educational contexts (Southgate et al., 2019), rather than engagement with content for entertainment or storytelling. Research in the context of health has shown that VR can have positive outcomes for children, for example, in the management of pain and anxiety:

…audio and visual immersion [that] encourages users to interact... [with the content and that as a result of this the medium has affordances that increase the ability to use it for distraction beyond that which has been noted in relation to] both passive distraction (e.g. watching television, listening to a book) and active distraction (e.g. interactive toys, electronic games). (Arane et al., 2017: 932)

The success of VR to manage health problems such as anxiety (in adult users) (e.g. Tarrant et al., 2018) lies in the ability of the medium to bring about a strong sense of presence within virtual content (Riva and Waterworth, 2014; Waterworth and Riva, 2014). That is, this potential is fundamentally tied to the extent to which virtual content can engage with and is interactive for children. Early studies into children’s responses to VR in a medical context suggest that VR can bring about a feeling of presence in ways potentially different from other platforms.

Stories, from books to films, have been described as immersive. In VR, however, the user has more agency over the direction of a story as compared to more traditional media. This article uses Rose’s (2018) description of immersion in stories as ‘forms in which a lone audience member moves from being a viewer to become an active participant or player in the story world’ (p. 133). The intention of this article is to show the benefits of children’s interaction with VR as a valuable means of storytelling for entertainment, gaming, education or disseminating information.

Methods and means of analysis

Study design

The study presented in this article are drawn from one stage (Stage 2) of a larger five stage project within the digital games industry. The stages are outlined to situate the paper in the larger study. Stage 1 drew on data from a large quantitative market survey (Dubit, 2017) to understand the extent to which children were familiar with VR technology, and the type of content they had or hoped to experience. Stage 2 focused on children’s interaction and engagement with VR in order to understand their general appeal and any noticeable features of the ways they interacted with the content. Stage 3 concerned health and safety testing with regard to potential effects on children’s vision and balance. Stage 4 brought together experts on children and digital media from industry and academia, a ‘think-tank’ to discuss the study findings. Finally, Stage 5 consisted of a 1-month ethnographic study of how VR use is situated within 8–12-year-old’s everyday lives. The methodology of Stage 2 is described in detail below.

Methods

The study was conducted with 20 children between the ages of 8 and 12-years-old. This age range was chosen as a response to the 13+ age limit imposed on VR use by tech companies. The intention was thus to understand how, if at all, research undertaken with children younger might contest or support this age limit, specifically in relation to interaction and engagement with content, as well as health and safety issues such as, effects on vision and balance (not discussed in this paper). Children under the age of 8 were not included as there was no suitable VR content available for them to use. A sample of participants with an equal gender split and diversity in terms of ethnicity and social-economic background was recruited via a commercial research recruitment company. Parents were offered an opportunity to try the VR content prior to the study, and both parents and children were guided through the study protocol, what to do if they wanted to leave the virtual experience or study, and to ask questions. Signed consent was received from each child’s guardian and verbal consent was given by the children.

VR content varies in their degree of interactivity in relation to the HMD used, for example, low-end devices like ‘Cardboard’ only offer interaction that affects the content by using eye gaze, whereas, top-end devices like the HTC VIVE track the user’s body movement, making more types of interaction possible. The study used two types of head mounted displays to monitor how the child-participants engaged with a range of content on a low-end (Cardboard) and a top-end headset (HTC VIVE).

Friendship pairs of child-participants were invited to play with VR content in a ‘Playlab’ designed for user testing. This is a child-friendly space used by the games company (in which the study took place) to test new content directly with children at various stages in its development. The room includes a couch, TV, table and chairs, toys, books and bean bags, as well as a range of recording equipment including overhead cameras for recording screen use, and wide-angled video cameras to capture the entire space. They were invited to use VR content that included both interactive games and fixed 360-degree content designed to be watched rather than played with, chosen for the way in which they differently position the user within the content as either observer or interactive user. The specific VR content included in the study (used on the high-end HTC VIVE headset) included:

Job Simulator: The 2050 archives (Owlchemy Labs). This is a low-poly interactive experience set in the future where human jobs have been replaced by robots, but users are given a chance to see what working in a range of jobs was like. These include working in a convenience store, being a chef or an office worker. Within each of these contexts there are a range of interactive objects that can be used to undertake these jobs and complete set tasks.

Fairy Garden (Dubit) is a short demo created to allow users new to VR to understand a range of possible interactions, this includes writing with light, chopping down a tree and being lifted into the sky on a bean stalk. The experience takes place in a fairy tale garden.

Adventure Time Magic Man’s Head Games (Turbo Button) is a VR experience that builds upon the Adventure Time series of comics and animations broadcast on Cartoon Network. It places the user in the fictional ‘Land of Ooo’ where their role is to chase the Magic Man character and break a curse. It includes elements such as battles with evil sandwiches.

Google Earth VR (Google) is a 360-degree experience were users can teleport to anywhere in the world and see it from new perspectives, i.e. floating above mountains. There are no interactive elements that can be altered by users in-game movements.

theBlu (Wevr) places the user on a shipwreck under the sea’s surface where they can observe the ocean’s wildlife. It is a 360-degree experience designed to be observed by the user and there are no interactive objects.

The content used on the low-end Cardboard device was as follows:

Adventure Time We See Ooo (Cartoon Network) uses the same story series as the above game. It offers the user a series of minigames, such as a snowball fight against penguins.

A car racing game called All Star Fruit Racing (XR Games) where the tracks are filled with fruit.

Funny Farm VR (Dubit) a series of mini games set on a farm, such as taking on the role of a bee collecting pollen.

Apollo 15 Moon Landing (Thomas Kole) an educational experience with photo-naturalistic graphics produced for the anniversary of the Moon landing.

Data capture and materials

Child-participants took part in a research session where individual play with any one piece of content was capped at 20 minutes with breaks in between each use of a new type of content. Each participant’s view in VR was displayed on an external monitor which could be seen by the researcher, the other child-participant in the session and their parents. Their physical movement and verbal responses were video-recorded by a wide-angled camera on a tripod. Combined this produced a record of the children’s view inside VR and an external view of children’s use of VR: total of 80-hours of data. At the end of each play session child-participants were informally interviewed about their VR experience, including what they liked or didn’t like about the content, specifics around visual aesthetics, narrative design and interactive objects; as well as questions about their observed interactions in relation to storytelling, and their use of interactive objects. The interviews, up to 20 mins long, were video-recorded.

Analysis

The data sets were multimodal taking in movement, image, sound and verbal language. Because the user was simultaneously in both physical and virtual domains there was an added layer of complexity to the data. The use of established means of multimodal analysis that transduct video footage into written descriptions proved inadequate. The interactions of the children in virtual and physical worlds required more experimental means for data analysis. In addition to identifying initial themes from the videos, data was transcribed using line drawings animated using stop-motion animation and the use of graphic narratives to understand the children’s interaction with the content (Yamada-Rice, 2018b). The stop-motion animation transcribed children’s movement and posture. This was achieved by tracing the outline of each child’s body from screenshots of the video data taken at set intervals. Once the line drawings were completed they were animated using stop-motion techniques. This method stripped away details recorded in the video data such as the VR content being used the clothes they were wearing, how they looked and details of the interior of the playlab providing a transcription purely of movement and posture (see Figure 1).

Figure 1.

Line drawing used to understand children’s movement in Google Earth VR.

This means of transcription helped to bring about knowledge of the extent to which children move, which limbs were used the most and their posture. Further, drawn to Sousanis’ (2017) argument that comics are a suitable medium for understanding and displaying complex ideas and stories, two different types of graphic narratives were used for data analysis. The first type used speech bubbles containing children’s verbal language on top of screenshots of the VR content (see Figure 2).

Figure 2.

Part of a mixed-media graphic narrative used to understand Google Earth VR interactions.

This method made it possible to consider children’s verbal responses to the designed content and any additional storytelling or play they added in relation to the specific context of the game design. In addition, to explore the complexities of physical interaction with low-poly imagined worlds and the tension of using this while simultaneously being in a physical environment, a free-hand style of graphic narrative was produced to speculate how theory of children’s interactions with the physical world could be applied to those in VR (e.g. Figure 3).

Figure 3.

Part of a graphic narrative used to understand Google Earth VR interactions.

Combined with thematic analysis Braun and Clarke (2006) these means of transcription and data analysis were able to respond to what Jones and Woglom (2016) describe as ‘a materially rich and complex manner of inquiring into and analyzing’ (Jones and Woglom, 2016: 2) children’s embodied interactions with VR. In particular, the experimental use of graphic narratives provided a means of materialising speculative ideas in a way that words or photography cannot and brought out themes from the data which more traditional methods would not have made visible.

Findings and discussion

The findings are presented in three sections related to the child-participants’ journeys in virtual story worlds: entering the story; being in the virtual story world; and affecting the story through engagement with interactive objects. Discussion centres on how the design of each stage prohibited or encouraged children’s interaction and engagement with virtual story worlds, and the implications for design.

Entering the story: The transition from physical to virtual domain

The term ‘onboarding’ refers to the first few minutes of entering gaming content that allows the player/audience to understand their role and draws them further into the narrative. This phase is critical because ‘most players that leave within a few minutes of play never return’ (Petersen et al., 2017: 377), and is a moment in which players decide whether the experience is for them (Seufert, 2014). The study findings showed that onboarding in VR is particularly important as it also relates to how safe the child-participants felt in the virtual environment and that successful onboarding into VR needs to prepare children for their transition from the physical environment into the virtual.

The stop-motion animation created to analyse movement, showed how the child-participants moved quickly through 360-degrees, looking low and high, crouching and jumping in order to explore the environment they had entered rather than entering VR with controlled movements. This has implications for how a VR story might begin, for example time can be made for the child-audience to become familiar with the virtual environment by not placing interactive or unfamiliar objects close-up on entry. This is particularly important given that when asked in the post-play interviews the majority of child-participants (18/20) described the initial experience of being in a virtual environment as strange or confusing:

Strange at first. (Hollie aged 11)

“I felt shocked at first; like I was in the TV.” (Ryan aged 12)

The video data indicates that the strangeness of entering a virtual environment directly related to the extent to which the visual aesthetics appeared realistic or not. However, rather than feeling more comfortable in photo-naturalistic environments (which Rose (2018) suggests is the preference for VR development), the children found low-poly contexts (i.e. more cartoon like imagery) easier to transition into than the photo-naturalistic experiences. The three photo-naturalistic experiences, Google Earth VR, Apollo 15 Moon Landing and theBlu, each began in an unfamiliar place – floating in space, in a rocket or on a ship wreck under the ocean. Arriving into these unfamiliar places caused the child-participants to feel uncomfortable:

Grace (aged 9):

“Oh I don’t like this.”

Holly (aged 9):

“You don’t like it?”

Researcher:

“Why?”

Grace:

“Because I don’t know where I am and I think I am going to fall.”

Holly:

“I need a big flat thing here to stand on so I can see where I am standing.”

The child-participants viewed the content as realistic enough to feel scared by the appearance of floating in space unaided and they offered ideas of what could have made them feel more comfortable (e.g. the addition of an observation platform). Their ideas focused on ‘grounding objects’ that would aid the transition from physical to virtual environment. Job simulator included such a transitional object in the form of a pair of virtual hands, that moved in sync with the players actual hands. When the child-participants entered this game, their gaze was immediately drawn to their virtual hands, this visual immersion slowed their initial movements and enabled them to understand their location within the VR content. Their comfort in VR worlds was also inhibited by the newness of the hardware, the child-participants found it hard to use the controllers as their face was covered and they could not see their hands, particularly in content that did not represent the player’s hands:

Hard to know when to pick up the controllers. (Megan aged 11)

Problems navigating the hardware delayed immersion into the virtual world and made their balance less steady as their attention was drawn back to the physical environment which they could not see as they searched around for the controllers.

Perhaps unsurprisingly, the study findings showed that children experiencing VR for the first time are more comfortable when the experience relates to a familiar context. This was demonstrated by children’s use of Google Earth VR where all initially sought or expressed a desire to find their own home. This resonates with children’s traditional literacy practices, which uses spheres of familiarity that start with the child, their bedroom and then spiral out into increasingly less familiar environments (Kenner, 2003). The experiences of the children in this study suggests that a familiar story world/gaming context allows children to later venture more easily into unknown virtual worlds. I suggest context is a dominant signifier for the meaning of VR environments (Scollon and Scollon, 2003), in unfamiliar environments child-participants were less able to predict the kind of multimodal texts that might inhabit it, or how they should respond to them in relation to their ‘real’ world experiences. In addition the child-participants were unfamiliar with the affordances of the technology. This combination of new technology and affordances of new virtual environments/objects made engagement difficult. The above example, of the child floating in space above Google Earth demonstrates this point, in that the children struggled to know how they should move through Space to get to Earth. They expressed this in relation to how to use the controllers to zoom into and transition to a place on Earth, as well as by their lack of physical movements. By contrast in theBlu, which is set under the ocean, when children wanted to move but could not do so with the hardware they moved their bodies as if trying to swim, thereby drawing on their knowledge of how they would move in similar ‘real’ world environments.

The design implications of these findings are to consider the first few minutes of the VR experience, and how children’s journey from familiar space to less familiar could provide a settling pattern for children. Children’s positive experience of virtual worlds could also be facilitated by the design of a transitional virtual object that can help children settle into the VR content. Beyond the design, if the virtual story context is likely to be unfamiliar to the child-user then social media could be used to help with the onboarding. The study illustrated that successful onboarding was achieved by children who had previously watched others use the same virtual content on YouTube. Doing so, allowed the child-participants to understand the intention of the VR experience before they engaged with it. This sped up the onboarding experience. For example, 10-years-old Rylan had watched YouTube videos of other people playing Job Simulator. When he tried this game for the first time as part of the study, he entered the virtual restaurant, looked around for the ticket machine, that he already knew existed, and began completing a customer order. His ability to do this without needing instructions showed the extent to which he was familiar with the content ahead of gameplay. Such positive outcomes can be replicated by adult involvement in facilitating children’s virtual experiences, e.g. encouraging them to watch online gameplay ahead of their own or offering their own descriptions of what to expect. In this study the researcher gave child-participants a verbal explanation of what to expect in the virtual environment and how to use the VR controllers and prompts, and that they could look the other way, close their eyes or take off the headset if they saw content that scared them. Adult verbal onboarding before play seemed to provide children with agency to manage their interaction and feel comfortable.

Being in the virtual story world

Child-participants immediately commented on the ‘look’ of the virtual worlds: they made a connection to the virtual story world through the visual aesthetic of the content. Rose (2018) describes VR as a magnet for non-fiction content, however this study suggests that this not what these children wanted. With the exception of Google Earth VR, all children said they preferred the non-photographic visuals. For example, when asked if Job Simulator would be more interesting if it looked like Google Earth all children said ‘no’. Twelve-years-old Ricky gave a reason for this:

Feels so real but also like a cartoon. Like you are really in a cartoon. (Ricky)

Ricky’s reasoning matched the child-participants overall interest in virtual worlds in that they provided the opportunity to feel fully immersed in environments that could never be visited through other means.

The comic theorist Scott McCloud (1993) suggests visual modality relates to audience engagement, which he illustrates using a sliding scale of stick figure to photo-naturalistic face. He states that with each move away from the lowest-modal representation, the additional details distance the viewer from directly relating to what they are seeing. Conversely, a simple face consisting of a circle, two dots for eyes and a line for a mouth can engage more people because it is open to interpretation in terms of gender, hair colour, age etc. In short, low-ploy images provide the possibility for audiences to project characteristics on to what they are seeing, making them more relatable, in ways that photo-naturalistic depictions do not.

The affordances of VR as an infinite 360-degree space means engagement with the visual aesthetic is affected by how the medium makes it possible to use scale in unconventional ways. Scale relates to the size of frame and social distance; two elements that are usually controlled by the medium, i.e. a lens of a camera or the framing of a painting (Kress and Van Leeuwen, 2006). However, Virtual Reality has a 360-degree frame with no horizon and social distance is engineered by the VR content designer and has no limitations to either the size of the frame or social distance. As a result, the VR creator is no longer bound by traditional conventions of scale and it is possible to alter the perceived relationship between what is viewed and the viewer. As an indication, when the children experienced Google Earth VR they explored the Earth from angles and perspectives not possible in the ‘real’ world and this altered their engagement with familiar objects. For example, children tried to pick up cars or stomp on forests (return to Figure 2):

Researcher:

“So this is Brazil.”

Grace (9-years-old):

“Oh can I touch that? I want to feel this.” [She pats her hand up and down]…“I want to pat them [the trees]. Awww I wanted to feel that.”

The above example, shows how child-participants try to make meaning of objects scaled differently from their physical counterparts through the mode of ‘touch’. With haptic feedback systems in their infancy, however, this illustrates the limited sensory engagement offered by VR.

In most forms of digital story content the visual aesthetic and use of image take priority in the dissemination of the narrative and when designing a mobile app with a limited budget funds can be saved by using stock audio. However, for VR where the sound is critical, and becomes arguably as or more important to the immersive experience as the visual mode this is not possible, although ‘generalizing the results and estimating the effect of the auditory component is difficult’ (Kern and Ellermeier, 2020: 1). The study findings showed sound formed an essential part of children’s engagement – it positioned them within the virtual environment, and, when designed well, sound guided them towards interactive elements of the story. The degree to which sound immersed the user into the experience was demonstrated best its ability to alter children’s emotional reaction to what they were viewing:

Nine-years-old Grace stands on an underwater shipwreck in the virtual content theBlu enjoying looking at the sea life, when the music changes to something more sinister. Grace looks at the whale’s face that she had previously been talking to in a friendly manner and screams. She brings her arms very close to her chest. “It’s a big whale. Oh, it is really big.” Grace steps back.

The ability of music to communicate feeling and emotional states is a well-documented, and further amplified when combined with gameplay actions (Williams and Newton, 2018: 6). The binaural sound of VR makes it possible to control the direction and distance of the sound from the player, making sound engineers and audio specialists central to the VR industry. In summary, visual and audio modes are key to connecting the user into the virtual experience and adjusting the visual modality, scale and sound scape can directly alter the users experience of the virtual narrative.

Affecting the story through interactive objects

The child-participants’ favoured VR content in this study was Job Simulator, an interactive low-poly game set in 2050, where previous human jobs are now undertaken by robots. This was the most interactive content trialled and the affordances of the interactive objects seemed to facilitate an easy shift for children to move from being the audience to storyteller in the way Rose (2018) suggests leads to a feeling of immersion. As with the process of onboarding, the study findings showed that the child-participants were drawn to virtual objects that were familiar to those from their everyday lives as they already had an idea of how to interact with them. In Job Simulator while the setting is futuristic, the story worlds are based on everyday contexts (e.g. restaurants, convenience stores) and contained familiar objects (e.g. food, and kitchen appliances). The child-participants began by interacting with these objects in the same way as if they were using them in the real world. Once they become comfortable with the story world they tested the boundaries of the objects’ affordances by using them in ways that would not be possible and/or allowed outside of VR. Eleven-year-old Hollie described this directly while using Fairy Garden; ‘I liked going up in the air and standing on a leaf. Things you can’t normally do’. This was further evident in children’s play:

Ten-years-old Aaron using Job Simulator:

[He picks up a menu] “What’s on the menu? Burgers. Wash it.” [He turns on a tap in a nearby sink and washes the menu]: “Right let’s fry this” [frying the menu]. “There you go [the menu is on fire].”

Once children had understood that they could interact with virtual objects they began to test their affordances by undertaking tasks they would not be able to do in the physical world such as setting things on fire, or blending a fire extinguisher to make soup. This gives cause to consider work on affordances which suggests the properties of an object dictate how it can be used (Norman, 1988). VR designers can take inspiration from the affordances of physical objects but not be limited by them (Colgan, 2015). Indeed, this study suggests that part of the excitement of VR is that it provides the feeling of immersion into unreal worlds with objects that might not respond as anticipated. As is described in Yamada-Rice et al. (2020) VR researchers (e.g. Matsumoto et al., 2017) are exploring how altering the way modes in virtual experiences can be juxtaposed with users’ physical touch can bring about a sense of modal illusion – caused by a gap between what is seen and what is felt. For example, touching a square physical table while viewing a virtual triangular table shapes the user’s belief that the table outside the virtual experience is also triangular. This connects to Virilio’s (1997) call to ‘dismantle the necessary conditions for sensory experience’ (p. 45): techniques which are increasingly recognised by cognitive neuroscientists such as Macknik et al. (2010).

All the child-participants enjoyed the opportunity to interact with content and produce stories based on things that they would not be able to do otherwise and were not directly part of the designers intended use or story:

Throwing things at robots, cooking a cactus, boiling soap (Job Simulator)

Going up a beanstalk into the air, shooting stuff with magic (Fairy Garden)

Talking to a whale and swimming (theBlu)

Trying to pick up cars, stamp on buildings, kick forests (Google Earth VR)

Having a conversation with a sandwich (Adventure Time We See Ooo)

The findings support those of Marsh et al. (2015), that placing interactive objects in an open-ended gaming space prolonged children’s engagement and promoted storytelling and creativity. In doing so, the content designers’ intended stories and gameplay took a back seat, but could be picked up again at any point, and challenges Mayr’s (2014) notion of the author as the most competent storyteller in interactive stories. The study findings show how both author and child-user can use the interactive virtual space to tell stories within the same content; sometimes as intended by the author, sometimes separately from each other.

Finally, in relation to interactivity, the children in this study approached all the VR content as interactive elements. This was evidenced by their desire to touch virtual objects as shown in the earlier example of touching trees in Google Earth VR. VR as a medium signified interactivity to them. When their body movements had no effect on the content of 360-degree non-interactive experiences they expressed disappointment. However, they continued to engage with such content in an embodied way, in which all parts of the body became engaged. For example, in Google Earth VR all child-participants tried to interact with the Earth using their entire body such as bending down and reaching out to touch mountains, sticking out their tongue to taste space or stomping on forests to see what would happen (Yamada-Rice, 2018b). Similarly, in theBlu, another non-interactive content, participants moved their arms back and forth throughout the experience as if swimming in the virtual ocean they were in. When the content did not respond to their touch the child-participants also expressed disappointment. This lack of response to their actions limited their role-play and storytelling.

Conclusion

The examples provided in this article illustrate how storytelling in Virtual Reality, is a multimodal experience which children engage with by using their bodies. It extends interactive storytelling research (e.g. Cassell and Ryokai, 2001; Mayr, 2014) by attending to 360-degree contexts and the use of a child’s entire body in VR, which removes a child’s vision of the physical world and temporarily envelopes them entirely within the story world in a way that is not possible with other media. Ryan (2015) suggests the closest we come to understanding the feeling of immersion in VR stories, is to think how books have immersed readers, however this feeling is intensified in VR stories which do not allow the reader to ‘look up/out’ to the physical world.

This article has shown how designers of VR content could consider specific ways in which the design of visual style, interactive objects, sound and movement can enhance the child’s virtual story experience. In relation to visual style it has pointed to children’s preference for low-poly aesthetics and that these may make it possible for a wider audience to connect by leaving some meaning conveyed by the visual mode open to interpretation in a way that photo-naturalistic aesthetics cannot. The study findings shown the importance of the mode of sound to immersive stories, particularly in relation to triggering emotional responses. This builds on work on the importance of sound in other narrative experiences such as films (Wingstedt et al., 2010) and video games (Williams and Newton, 2018) to argue that audio specialists are an important part of a VR content development team, and advocate for user testing during the process of audio production to ensure child-audiences respond emotionally in the way the content designers desire. This article has foregrounded the key role of movement for interactivity in VR, both in relation to interactive objects and non-interactive 360-degree content, and showed how children used the entirety of their bodies in their engagement with VR stories. Movement with digital stories in apps or books tends to be limited to finger movement, although some children may get to their feet in the process of reading to enact the story in embodied ways. In the context of VR however, across all content and with every child-participant movement was the norm rather than the exception. This points to the need for, more work around movement as a dominant mode of meaning-making in VR is now needed.

The study highlighted the tension created between the affordances of virtual objects not matching the properties of their physical counterparts. The expected properties of modes, in relation to the visual for example, do not remain absolute in VR, as demonstrated in relation to scale. Similarly, the study showed how Norman’s (1988) affordances of physical objects can also be altered in VR; the materials of virtual objects can thus appear visually the same as their physical counterparts but respond differently to human interaction (e.g. water can run upwards, metal can be soft, a virtual hand can pass through a window unharmed). This points to the need for designers of VR content to explore how creating interactive content with material properties different from those expected by users, might further storytelling opportunities in virtual story worlds. That is, while ‘visual structures point to particular interpretations of experience and forms of social interaction’ (Kress and Van Leeuwen, 2006: 2), the VR world need not be the same as children’s lived experiences although successful interactive storytelling nonetheless requires ‘intuitive authoring tools’ so as to make interaction possible (Spierling and Szilas, 2009). For children, as this study shows, want to start with the familiar and then when comfortable move to explore new places, objects and possibilities, thus the incorporation of new affordances and materials for virtual objects is most appropriate and effective once they have become familiar with a virtual world or after they feel grounded in a new piece of content. Design needs to carefully balance the tension between the familiar and the new via other means, including in the physical world. Further research is needed to understand this balance further and ascertain the benefits for child-users of VR with familiar objects and/or environment that allow for ‘unreal’ possibilities or those of more complex intertwinings.

Children are highly motivated to use VR which remains novel and fun (Allison and Hodges, 2000) and VR offers a unique space in which stories can be told to children or worlds can be created for children to become storytellers. Consideration and further research of the above areas, core stages of onboarding, as well as, visual aesthetics of the story world, scale, sound, movement and interactive objects for the design of VR to realise its potential to offer new story and play opportunities for children.

Finally, the article has also shown how adopting the use of experimental methods can make knowledge about users meaning-making in virtual worlds newly visible.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded by Dubit Ltd and Turner. The author received no personal financial support for the research, authorship, and/or publication of this article.

References

Allison

Hodges

(2000) Virtual reality for education? In: Proceedings of the ACM symposium on virtual reality software and technology, Seoul, Korea, October 2000, pp. 160–165. New York, NY: ACM.

Arane

Behboudi

Goldman

(2017) Virtual reality for pain and anxiety management in children. Canadian Family Physician 63: 932–934.

Bailey

Bailenson

(2017) Considering virtual reality in children’s lives. Journal of Children and Media 11(1): 107–113.

Barry

(2015) Syllabus: Notes from an Accidental Professor. Montreal: Drawn and Quarterly.

Bennett

(2010) Vibrant Matter: A Political Ecology of Things. Durham, CA: Duke University Press.

Braun

Clarke

(2006) Using thematic analysis in psychology. Qualitative Research in Psychology 3(2): 77–101.

Cassell

Ryokai

(2001) Making space for voice: technologies to support children’s fantasy and storytelling. Personal and Ubiquitous Computing 5(3): 169–190.

Christie

Roskos

(2009) Play’s potential in early literacy development. In: Tremblay

Boivin

Peters

(eds) Encyclopaedia on Early Childhood Development. Montreal: Centre of Excellence for Early Childhood Development and Strategic Knowledge Cluster on Early Child Development, pp. 1–16. Available at: http://www.child-encyclopedia.com/sites/default/files/textes-experts/en/774/plays-potential-in-early-literacy-development.pdf (accessed 28 January 2021).

Colgan

(2015) What do VR interfaces and teapots have in common? Online blogpost. Available at: https://blog.leapmotion.com/vr-interfaces-teapots-common/ (accessed 16 January 2021).

10.

Dubit (2017) Global Trends Wave 6. Unpublished market research.

11.

Garzotto

(2014) Interactive storytelling for children: a survey. International Journal of Arts and Technology 7(1): 5–16.

12.

Goltz

(2010) ESRB warning: use of virtual worlds by children may result in addiction and blurring of borders—the advisable regulations in light of foreseeable damages. Pittsburgh Journal of Technology Law and Policy 11: 1–63.

13.

Grimes

(2018) Penguins, hype, and MMOGs for kids: a critical re-examination of the 2008 “boom” in children’s virtual worlds development. Games and Culture 13(6): 624–644.

14.

Ingold

(2013) Making: Anthropology, Archaeology, Art and Architecture. London: Routledge.

15.

Jones

Woglom

(2016) On Mutant Pedagogies: Seeking Justice and Drawing Change in Teacher Education. Rotterdam : Sense Publishers.

16.

Kenner

(2003) Becoming Biliterate: Young Children Learning Different Writing Systems. Stoke-on-Trent: Trentham Books Limited.

17.

Kern

Ellermeier

(2020, February 21) Audio in VR: effects of a soundscape and movement-triggered step sounds on presence. Frontiers in Robotics and AI 7(20): 1–13.

18.

Kress

Van Leeuwen

(2006) Reading Images: The Grammar of Visual Design, 2nd edn. London/New York, NY: Routledge.

19.

Livingstone

(2008) Taking risky opportunities in youthful content creation: teenagers’ use of social networking sites for intimacy, privacy and self-expression. New Media & Society 10: 393–411.

20.

Mäkelä

(2007) Knowing Through Making: The Role of the Artefact in Practice-led Research. Knowledge and Policy 20: 157–163.

21.

McCloud

(1993) Understanding Comics: The Invisible Art. New York, NY: HarperCollins.

22.

Macknik

Martinez-Conde

Blakeslee

(2010) Sleights of Mind. New York, NY: Henry Holt and Company.

23.

Marsh

(2014) From the wild frontier of Davy Crockett to the wintery fiords of Frozen: changes in media consumption, play and literacy from the 1950s to the 2010s. International Journal of Play 3(3): 267–279.

24.

Marsh

Plowman

Yamada-Rice

, et al. (2015) Exploring Play and Creativity in Pre-Schoolers’ Use of Apps: Final Project Report. Available at: www.techandplay.org (accessed 28 January 2021).

25.

Matsumoto

Hashimoto

Mizutani

, et al. (2017) Magic table: deformable props using visuo haptic redirection. In: SIGGRAPH Asia 2017 emerging technologies, Bangkok, Thailand, November 2017, pp. 1–2. New York, NY: ACM.

26.

Mayr

(2014) Immersion in a Virtual World: Interactive Drama and Affective Sciences. Hamburg: Anchor Academic Publishing.

27.

Norman

(1988) The Design of Everyday Things. New York, NY: Basic Books.

28.

Petersen

Thomsen

Mirza-Babaei

, et al. (2017) Evaluating the onboarding phase of free-to-play mobile games: a mixed-method approach. In: CHI PLAY 2017, Amsterdam, The Netherlands, 15–18 October 2017, pp. 377–388. New York, NY: ACM.

29.

Riva

Waterworth

(2014) Being Present in the Virtual World. The Oxford Handbook of Virtuality. New York, NY: Oxford University press, pp. 205–211.

30.

Robson

(2014) The Analysing Children’s Creative Thinking framework: development of an observation-led approach to identifying and analysing young children’s creative thinking. British Educational Research Journal 40(1): 123–134.

31.

Rose

(2018) The immersive turn: hype and hope in the emergence of virtual reality as a nonfiction platform. Studies in Documentary Film 12(2): 132–149.

32.

Roskos

Christie

(2011) Mindbrain and play-literacy connections. Journal of Early Childhood Literacy 11(1): 73–94.

33.

Ryan

(2015) Narrative as Virtual Reality 2 (Parallax: Re-Visions of Culture and Society). Baltimore, MD: Johns Hopkins University.

34.

Scollon

(2003) Discourses in Place: Language in the Material World. London: Routledge.

35.

Seufert

(2014) Freemium Economics – Leveraging Analytics and User Segmentation to Drive Revenue. Burlington, MA: Morgan Kaufmann.

36.

Solomon

(2014, March 25) Facebook buys Oculus, virtual reality gaming startup, for $2 billion. Available at: https://www.forbes.com/sites/briansolomon/2014/03/25/facebook-buys-oculus-virtual-reality-gaming-startup-for-2-billion/?sh=fc834d524984 (accessed 28 January 2021).

37.

Sousanis

(2017) Thinking in comics: an emerging process. In: Cahnmann-Taylor

Siegesmund

(eds) Arts-Based Research in Education: Foundations for Practice, New York, NY/London: Routledge, Chapter 16, pp. 190–199.

38.

Southgate

Smith

Cividino

, et al. (2019) Embedding immersive virtual reality in classrooms: ethical, organisational and educational lessons in bridging research and practice. International Journal of Child-Computer Interaction 19: 19–29.

39.

Spierling

Szilas

(2009) Authoring issues beyond tools. ICIDS Proceedings. Available at: https://www.researchgate.net/publication/220920142_Authoring_Issues_beyond_Tools (accessed 18 December 2018).

40.

Stroud

Chernin

(2008) Video games and the ESRB. Journal of Children and Media 2: 1–18.

41.

Tarrant

Viczko

Cope

(2018) Virtual reality for anxiety reduction demonstrated by quantitative EEG: a pilot study. Frontiers in Psychology 9: 1280.

42.

Virilio

(1997) Open Sky. London/New York, NY: Verso.

43.

Waterworth

Riva

(2014) Feeling Present in the Physical Works and in Computer-Mediated Environments. Basingstoke: Palgrave Macmillan.

44.

WEARVR (2018) ICO white paper introducing WEAVE from WEARVR.

45.

Williams

Newton

(eds) (2018) Emotion in Video Game Soundtracking. York/Burbank, CA: Springer.

46.

Wingstedt

Brändström

Berg

(2010) Narrative music and meaning in film. Visual Communication 9(2): 193–210.

47.

Yamada-Rice

(2018a) Designing play: young children’s play and communication practices in relation to designers’ intentions for their toy. Global Studies of Childhood 8(1): 5–22.

48.

Yamada-Rice

(2018b) Licking planets and stomping on buildings: children’s interactions with curated spaces in virtual reality. Children’s Geographies 16: 529–538.

49.

Yamada-Rice

Dare

Main

, et al. (2020) Location-Based Virtual Reality Experiences for Children: Japan-UK Knowledge Exchange Network: Final Project Report. Available at: <https://ukjapanvr.wordpress.com/2020/06/13/final-project-report/> (accessed 16 January 2021).

50.

Yamada-Rice

Mushtaq

Woodgate

, et al. (2017) Children and Virtual Reality: Emerging Possibilities and Challenges. Available at: http://childrenvr.org. (accessed 28 January 2021)

51.

Yoshitake

(2015) It Might Be an Apple [Translated from Japanese]. London: Thames & Hudson.

Children’s interactive storytelling in Virtual Reality

Abstract

Keywords

Introduction

Interactive storytelling for children

Children and VR

Methods and means of analysis

Study design

Methods

Data capture and materials

Analysis

Findings and discussion

Entering the story: The transition from physical to virtual domain

Being in the virtual story world

Affecting the story through interactive objects

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

References