Abstract
The study tests the efficacy of a new sort of digital picture book. It includes camera movements to guide children’s visual attention through the pictures and the possibility to control page-turning and the pace at which the camera moves through pictures. There were 56 participants (Mage = 60.34 months, SD = 6.24) randomly assigned to three conditions: still images, camera movements (no control over pace), and camera movements (control over pace). For the 50% of children least proficient in language skills, sparingly adding well-chosen camera movements to the illustrations helps children understand the story. In addition, the camera movements’ effect can be enhanced by enabling control over the pace at which new information appears. Particularly the 50% low-language proficiency children benefited from camera movements and spending more time processing information.
Keywords
Due to a growing number of studies, we know better which enhancements in digital picture books facilitate meaning making of stories (Christ et al., 2019; Sargeant, 2015) and why 3- to 6-year-olds benefit from the enhancements (Miller & Warschauer, 2014; Roskos et al., 2009). Often studied additions to the primarily book-like designs of a new generation of digital picture books are animations or sounds that occur automatically or when a user touches a specific spot on the screen (Christ et al., 2019; Korat & Falk, 2017; Piotrowski & Kcmar, 2017; Unsworth, 2014; Xu et al., 2020). As a recent meta-analysis confirmed, these enhancements in so far providing information about the meaning of not-so-familiar words may distract children’s attention from the storyline and interfere with meaning making (Furenes et al., 2021). The more promising versions of such enhancements activate background knowledge or provide an additional explanation (e.g., clicking on people sitting in the doctor’s waiting room reveals why they are there).
This study does not focus on such standard enhancements that mainly expand the books’ information but aims at helping children process the primary sources of information in picture books—words and pictures. Words and pictures work together on an equal footing to produce a total effect in reconstructing the story (Greenhoot & Semb, 2008; Schneider & Dubé, 2005; Sipe, 2008). To help children coordinate the pictures with the narrative, the illustrations in the target digital book include camera movements. These subtle visual enhancements signify to readers which parts of the often richly illustrated spreads they should attend to optimize the match (e.g., Flack & Horst, 2017; Godwin et al., 2019). In addition, the target book enables control not only over page-turning but also over the pace of camera movements, thus enabling children to process all information before they proceed to new information.
Built-in Camera Movements in Digital Picture Books
Picture books expose children to complex narrative language and therefore prepare them better for later reading comprehension than watching films that often include much less language (Montag, 2019). This study explores technology of a particular kind added to help children process the often complex story narrative accompanied by an illustrational sequence that is indispensable in telling the story (Sipe, 2008). The target digital picture book enhancements involve camera movements such as zooming in or out and panning to support story comprehension. These additions pursue effects similar to adults’ contingent responsivity when they share a picture book with a young child. Just as adults, the enhancements focus children’s visual attention on significant incidents that directly impact the protagonist and the plot’s trajectory (e.g., Dore et al., 2018; Eng et al., 2020; Evans & Saint-Aubin, 2005; Kim & Anderson, 2008; Krcmar & Cingel, 2014). The enhancements may turn out to be a substitute for an adult pointing at details in pictures in sync with the narrative text or commenting but, of course, not for initiating discussions or the social benefits of joint attention (e.g., Aram et al., 2013; Blewitt et al., 2009; Morwane et al., 2019).
One effect of the camera movements may be that we thus reduce extraneous perceptual information forcing young children to continually switch between exploring the visuals in the picture book and processing the story narrative (Flack & Horst, 2017). In particular, younger children less advanced in narrative comprehension may benefit from such help (e.g., Li, 2020). Table 1 provides several illustrations from the target book. For instance, after presenting the first spread, the virtual camera zooms in on the main protagonist, a boy, when the narrator says: “Today I will learn how to make lightning.” The camera movement thus highlights who the storyteller is. In the same spread, the camera zooms in on a typical school when the text explains that children in this unknown world high in the sky go to school to learn about the weather. These and other enhancements in the digital version (see Table 1 for more examples) align with the multimedia learning principle temporal contiguity predicting that people’s understanding improves when words and pictures are near rather than far from each other (Mayer, 2011).
The Core Information Transferred in Three Scenes and the Techniques Used to Emphasize Pivotal Story Information
Note. The screenshots are from the target book called “Lightning(Coenraads & de Wijs, 2017).” Permission granted by Het Woeste Woud, the Netherlands.
Maybe even more critical than synchronizing the story narrative with illustrations, visual enhancements in digital picture books can benefit story comprehension by focusing on contradictions between story narrative and picture (Sipe, 2008). By focusing attention on the discrepancies between words and pictures, readers are encouraged to generate self-explanations of literary elements such as irony (Mayer, 2011). For instance, in the target book, the narrative told from the boy’s perspective emphasizes that his father allows him to use the weather-make machine. At the same time, the picture, simultaneously visible, suggests the opposite. The virtual camera guiding the reader’s visual attention highlights the discrepancy by zooming in on no entry signs and big locks attached to the weather-make machine. The camera movements thus signify to children that the picture and the text tell different stories, thus stimulating children to find explanations for the paradox (e.g., Flack & Horst, 2017; Godwin et al., 2019; Verhallen & Bus, 2010). Making children aware of such tensions allows discoveries and subtle revelations that may result in a more profound story understanding. For instance, understanding that the boy is starting an action strictly forbidden by his father, the reader may begin to develop a repertoire of possible consequences, which may help understand later story events.
The digital picture books enriched with such camera movements improved 4- and 5-year-olds’ story comprehension in previous research (e.g., Sarı et al., 2019; Smeets & Bus, 2014; Sun et al., 2019; Verhallen et al., 2006). However, the target books in those studies always framed the camera movements in a smooth, film-like presentation that may be engaging by itself. According to the arousal theory, a film-like format may attract children’s attention and help them stay attentive, thereby comprehending the story better, as several researchers claim (e.g., Richter & Courage, 2017). Consequently, not the camera movements but increased attention due to the smooth film-like presentation often enriched with music might explain the enhanced books’ effect on story comprehension. However, there is no univocal support for the hypothesis that video bringing stories alive through sound and action outperforms a book format. Studies comparing the two formats revealed mixed effects on story comprehension. For instance, Diehm et al. (2020) conclude that typically developing children between the ages of 3 and 5 years indeed produce more extended narrative retells retelling a short video than retelling the same story presented in a picture book format. However, this finding does not replicate Neuman et al. (2017), reporting that 4-year-olds from a Head Start program comprehend a story equally regardless of whether the story was a video presentation or a paper version of the same story.
Research testing the effect of parsimoniously but deliberately added camera movements in our target book, not framed in a film-like format, is not yet available. The current study sought support for the hypothesis that deliberately added camera movements aligning the above-outlined multimedia learning principles help convey the critical information about the plot’s trajectory more effectively than can be achieved with illustrational sequence and verbal text alone.
Learner-Paced Segments
The target book enables the reader to control the pace of page-turning and camera movements. Before new information appears, the story pauses, and the child has to touch the screen to make it continue. So, readers can control how soon further information appears, unlike digital picture books where the information stream automatically proceeds. The current study tested whether young children understand the story better and benefit more from incidental word learning when they control page-turning and camera movements’ pace. Although the literature does not provide evidence, control over pace may be a powerful mechanism. Control over the pace at which new information appears may reduce the risk that new information is already available while old information is not yet fully processed, thereby raising cognitive load. Research suggests that children thus have the opportunity to organize each segment of information mentally and integrate pieces of information before they proceed to new information (e.g., Kirkorian et al., 2016; Mayer & Chandler, 2001). In Mayer’s (2011) multimedia theory, one of the principles, the so-called segmenting principle, refers to the importance of controlling the pace at which new chunks of information appear.
To enable control, new chunks of visual information do not appear automatically. Instead, after each camera movement or finishing a page, the reader must tap bottom-right or on a story-relevant part of the screen to advance the story. In this way, children can adapt the story to their own (pacing) needs, even more so than during traditional picture book reading. Adults mostly turn the pages of a print book when they read to children, and adults often determine when pauses for reflection are inserted (Lawless & Brown, 1997). There is evidence that 4- and 5-year olds can use the appropriate motion and select the proper timing for touching the screen; participants younger than 4 years may struggle to do so (Xu et al., 2019). Moreover, the need to be active may be an extra incentive to pay attention, organize new information, and integrate further information into story comprehension (Gopnik & Meltzoff, 1997).
Hypotheses
Testing the efficacy of the accessory technology, we are most interested in effects on story comprehension—the main aim of the enhancements. We do not expect direct effects of the camera movements on vocabulary learning because the camera movements aim to promote pictures to reconstruct the chain of connected events comprising the narrative and not draw children’s attention to salient information essential to contextualize words (Silverman, 2013). However, camera movements may have an indirect effect on vocabulary learning. Children who understand the story better have more cues to promote incidental word learning (Lee, 2020). They can derive the meaning of unknown words from the visual and verbal context (see also Cain et al., 2004; Leung, 1992; Penno et al., 2002). There may also be an impact of reader control on vocabulary learning; if more time is available to process the story, this may benefit deriving new vocabulary from the context (Miller & Warschauer, 2014; Silverman, 2013; Silverman & Hines, 2009).
This experiment tests the following hypotheses:
Method
Design
The target book and the tests were in Dutch. We followed the logic of a value-added experiment (Mayer, 2014) and constructed three versions of the same story to test the accessory technology. All three versions included the same static illustrations (24 in all) and the oral and written text (about 300 words) but differed as follows:
A still version with only static overview illustrations (e.g., the first picture of Spread 1 in Table 1).
A version enhanced with (about) two camera movements per illustration (e.g., in Spread 1, the camera zooms in on the boy and the school; see Pictures 2 and 3 in Table 1).
In addition to camera movements, as in Version 2, this version enables reader control. With each new spread or zooming or panning within the picture, the story pauses. The reader needs to indicate when to proceed by touching the screen (e.g., there are three pauses in Spread 1, the first before the camera zooms in on the boy, the second before the camera zooms in on the school, and the third before the first picture of the classroom appears; see Table 1).
Stratified for classroom and gender, the researchers assigned each participant randomly to one of the three conditions. Because incidental word learning is not likely to occur after a single exposure to a book (Lee, 2017), each child read the allocated book version twice (Korat & Blau, 2010; Silverman, 2013). In addition, the participants were pretested on cognitive skills that might affect how they respond to the books: general language proficiency in Dutch and children’s familiarity with the book-based vocabulary. Table 2 presents an overview of all tests and activities during the two experimental sessions one week apart.
Overview of Tests and Activities During Two Sessions 1 Week Apart
All procedures performed were per the institutional and national research committees’ ethical standards and were acceptable according to the 1964 Helsinki declaration and later amendments or comparable ethical standards. The Institutional Review Board of Language Literature and Communication, Vrije Universiteit Amsterdam, approved the study protocol (January 11, 2017).
Participants
We recruited participants from two schools located in mixed neighborhoods in Amsterdam. The population of both schools involved low- and high-educated families with a Western or non-Western background. One school had four kindergarten classrooms, and the other school had five. All classrooms had mixed ages (4- to 6-year-olds). As is usual in Dutch schools, the medium of instruction was Dutch. Formal teaching of reading or writing was not part of the curriculum, and none of the participants was a conventional reader. The emphasis was on playful activities: drawing, coloring, painting, jigsaw puzzles, dressing up, playing with blocks, or playing outside in the sandbox, with bikes, on the swing, or with balls. All parents received an institutional review board–approved description of the study and an easy-to-understand consent form. We included children with another mother tongue than Dutch (the medium of instruction), but only if their teacher reported that they could have a simple conversation in Dutch. Based on a power analysis, the final sample consisted of the first 60 child participants for whom the parent returned the consent form allowing their child to participate in the study. Participants (36% girls) had a mean age of 60.43 months (SD = 6.19, range 49–72 months). During the experiment, three children dropped out because they were absent from one of the testing days. One child refused to complete the tests. Thus the final analysis related to 56 children. Their general language proficiency varied, but none of the participants scored more than 2 standard deviations (SD) below the mean on a general language test assessing general vocabulary and listening comprehension.
Procedure
Testing and the intervention, spread over two sessions (see Table 2) 1 week apart, took place in the school at a quiet location. In the first session, children completed the pretests, and in the second, the posttests. Children read the story twice: in the first session after the pretests and in the second session before the posttests. The children read the book independently. Except for the child and the examiner (the second author), no other adults or children were present during these sessions. The sessions varied in duration depending on the book’s version: sessions lasted about 30 minutes for the conditions with the still images and camera movements; when the reader was in control of the pace, the session lasted about 35 minutes. We used a touch screen interface to present the book in all three conditions, considered more user-friendly than a PC or laptop (Kucirkova, 2017). We used a Samsung Galaxy Tab 10.1 with a 25.7 cm display (1280 × 800 pixels). The adults did not initiate a conversation with the children and did not elaborate on children’s questions to ensure that all children received similar treatment. The experimenter would interfere when children were completely off-task (e.g., failure to remain seated, not attentive to the reading for a long time, or not touching the screen to proceed with the story), but all children were eager to listen to the story. It never happened that the examiner had to stimulate them to listen or continue.
Digital Picture Book
We agree with Kelley and Kinney (2016) that it is vital to intentionally design apps to be instructional in a specific way. In this study, designing a new app was made possible because the Dutch Foundation for Literature [Letterenfonds] awarded Het Woeste Woud to design a digital picture book that includes new digital storytelling techniques. So, the target digital picture book was not a recycled picture book or an existing app but created from scratch by this Dutch company according to the principles explained in the introduction (i.e., temporal contiguity, self-explanations, and segmenting).
Camera Movements
Table 1 describes how camera movements can support young children’s meaning making of the story for a few scenes. For instance, the second illustration involves a classroom within the weather school (Spread 2, Table 1). It shows a teacher sitting at her desk in front of a group of children explaining: “I will tell you about the weather, and you listen.” The camera zooms in on the boy to show the impact of this message: he is throwing paper planes and looking very bored. Thus, the camera movement highlights the story’s initiating event for upcoming actions and reactions: the boy’s disappointment about the prospect of just listening to the teacher and the need to postpone the moment that he can make lightning by himself for years.
Reader Control
The reader-paced version pauses when a new spread appears, and each time, the camera moves within an overview illustration, mostly two or three times per illustration. In all, the story is thus interrupted about 50 times. Each time the user needs to touch the screen, mostly an arrow bottom-right on-screen, sometimes at spots directly related to the events. For instance, after the boy took his father’s weather-make machine, we see him sitting at the dashboard, figuring out which button to make lightning (Spread 9, Table 1). The camera zooms in on the dashboard while the button that the boy intends to press—clearly the wrong one—highlights. The story only continues after the reader has touched the highlighted “wrong” button. While the story pauses, the reader may start wondering what the consequences will be when he presses the wrong button. Most pauses are at such locations where inferences are needed. The app developer preferred this design to breaks chosen by the reader. Due to the pauses, the story takes longer than the 464 seconds when it automatically continues.
Tests
Tests Preceding the Two Book Readings
Cito language and literacy test
We used two subtests of a standardized language and literacy test for kindergarten as indicators of children’s general language proficiency, namely listening comprehension and receptive vocabulary (Lansink & Hemker, 2012). Both tests included 12 items. The researcher read short stories to the child; after each story, the child had to select a picture matching the story from an array of three or four pictures. Cronbach’s alpha for listening comprehension equaled .86. Likewise, in the vocabulary test, the child selected a picture from three or four pictures that matched the word spoken aloud by the researcher. Cronbach’s alpha for these 12 items equaled .80. As the two tests had high loadings on the same factor (.90 and .90), we combined the two tests into one measure with a maximum score of 24 to indicate children’s language proficiency.
Book-based vocabulary
This test included 18 words from the target story. According to the Basic word list for Amsterdam Kindergarten (BAK), 15 out of 18 words rarely occur in language kindergarten children encounter (ITTA, kennisinstituut voor Taalontwikkeling, 2009). The set included the following words:
Nouns: bliksem [lightning]; regendruppel [raindrop]; weerman [weatherman]; machine [machine]; voeten [feet]; natte haren [wet hair]; knop [button]; wind [wind];
Verbs: in brand vliegen [burn]; lenen [lend]; zich vervelen [being bored]; opvrolijken [enliven]; indrukken [press]; in de hand houden [being in control]; opletten [pay attention];
Other words: handig [handy]; vertrouwen [trust]; behalve [except].
The child selected the picture that matched the word spoken out loud by the researcher. Children chose the best matching picture from four options: the correct depiction (raindrop), a phonological distractor (crop), a semantic distractor (a drop from the tap), and a random picture (cheese). The test pictures were not derived from the target book but were general pictures from the Internet. The standardized Cronbach’s alpha equaled .69.
Tests After the Two Book Readings
Book-based vocabulary
After reading the book twice, we tested each of the 18 words two ways. In addition to the receptive format of the pretest, we tested expressive knowledge. Similar to Hadley et al. (2015), we considered a definition as less workable in this age group. First, the researcher read an incomplete sentence, while the child looked at a matching picture. Then, the researcher invited the child to complete the sentence. For instance: “From the cloud falls a . . .” (raindrop). Neither the picture nor the sentence appeared in the target book. If children responded with a synonym, they were encouraged to think of another word. We always started with sentence completion to prevent an effect of the receptive test. The standardized alpha reliability for 18 items, each with a maximum score of 2 (1 point for receptive and 1 point for expressive), equaled .85.
Story comprehension
The experimenter told the children that she would go through the story while guiding the page-turning and eliciting reactions during this viewing of the book by pointing out pictures and asking a series of comprehension questions. The children answered 12 questions, nine accompanied by a picture from the story and three without a picture. The questions were designed analogous to the so-called prompted retelling approach (Paris & Paris, 2003). Seven questions targeted factual information provided in the narrative and required identifying characters, setting, initiating event, problem, and outcome resolution (e.g., The picture shows the boy walking on tiptoe to the machine: What is the little boy up to here?). The remaining five questions prompted children to make inferences about the characters’ feelings, dialogues, causal relations, predictions, and themes (e.g., What would the people in town say to each other here? Why would they say that?). We awarded their answers with 0, 1, or 2 points. The award depended on whether the child integrated information across pictures rather than described single pictures in isolation. For instance, if the child, in answer to the question about the boy walking on tiptoe to the machine described that the boy sneaks to the machine, they received 1 point; however, if the child also explained that the boy aims at picking the machine to make lightning, they received 2 points. The two authors coded the recorded answers. Intraclass correlation based on double-coding of 16 children equaled .94 (95% CI = [.84, .98]). The standardized alpha reliability equaled .80.
Duration of the Learner-Paced Session
The researcher registered the duration of the reading session in which the user was in control of the pace at which new information appeared.
Statistical Analyses
We used the R statistical environment (R Studio, Version 3.6.2; R Development Core Team, 2019) through the following packages: dplyr, tidyverse, ggplot2, ggpubr, rstatix, broom, and emmeans.
Results
Table 3 shows basic characteristics per condition. Unintentionally, more boys than girls participated in the study (7:4). Parents may have considered the digital picture book more appropriate for boys than girls, and they, therefore, may have provided more often written consent for boys than girls. The inequality in sex was similar across the three conditions, χ2(df = 2) = 2.07, p = .422. To examine whether randomization had been successful, we tested whether the three conditions had a similar median as well as variance on age in months, F(2, 50) = .93, p = .401, the Cito language and literacy test, F(2, 53) = 2.56, p = .087, and the pretested book-based vocabulary, F(2, 53) = 2.81, p = .069. Post hoc testing (Bonferroni, Tukey, and Gabriel) did not reveal significant differences between the three conditions. We also tested whether children from the two schools were similar. Since schools did not differ in Cito language and literacy test, t(54) =0.55, p =.582, or book-based vocabulary, t(54) =0.86, p = .395, we did not correct for clustering.
Means (SDs) of Child Characteristics (Gender, Age) and Relevant Cognitive Skills at Pretest per Condition
Note. Neither overall effects nor post hoc testing (Bonferroni, Tukey, Gabriel) revealed significant differences between the three conditions on age, Cito tests, and book-based vocabulary pretest. max. = maximum score.
Since we did not assign the complete test, we cannot provide standardized scores.
On average, the reader-paced version took longer than the other two versions with a fixed duration (464 seconds). The first session took, on average, 613 seconds. (SD = 102). The second session (M = 499 seconds, SD = 63) was on average 114 seconds shorter than the first session but still longer (35 sec.) than the automatically continuing versions.
Story Comprehension
We first checked the homogeneity of the regression slopes assumption. The slopes of the regression lines, formed by the covariate Cito language and literacy test and the dependent variable story comprehension, were not the same for all three conditions, indicating that the Cito language and literacy test did not meet the assumption of homogeneity of regression slopes. Therefore, we applied a median split and included the Cito language and literacy test score as a second factor (Iacobucci et al., 2015). We performed a two-way analysis of covariance (ANCOVA) to examine the effects of the condition and Cito language and literacy test (median split) on story comprehension controlling the book-based vocabulary at pretest. All assumptions for performing this ANCOVA (linearity between covariate and outcome variable, no outliers, normally distributed story comprehension scores for all three conditions and the two levels of the Cito language and literacy test [median split], and homogeneity of variance) were satisfactory. After adjustment for a book-based vocabulary score at pretest, F(1, 49) = 6.00, p = .018, there was a statistically significant difference in story comprehension at posttest between the three conditions, F(2, 49) = 4.57, p = .015, η2 = .157, Cito language and literacy test (median split), F(1, 49) = 23.12, p < .001, η2 = .321, and there was a statistically significant two-way interaction between condition and Cito language and literacy test (median split), F(2, 49) = 3.87, p = 0.027, η2 = .136.
Simple main effect analyses followed up the statistically significant two-way interaction with two aims: evaluating the effect of condition at both levels of the Cito language and literacy test (median split) and the effect of the Cito language and literacy test (median split) for each condition. After adjustment for a book-based vocabulary score at pretest, the effect of the condition was statistically significant in the low-level group of the Cito language and literacy test, F(2, 27) = 3.94, p = .032, η2 = .226, but not in the high-level group, F(2, 21) = 1.30, p = .293, η2 = .110; see Table 4 and Figure 1. Within the low-level group of the Cito language and literacy test, we tested differences between conditions contrasting the still images with the two conditions that involve camera movements and contrasting the condition that automatically continues and the condition with the pauses and reader control over the pace. As these contrasts were orthogonal, there was no need for adjustment of the alpha level. The mean score on story comprehension was significantly lower with still images than in the two conditions with camera movements, p = .014. The condition with reader control over the pace outperformed the condition that automatically proceeded, but the difference was not significant (p = .152). We contrasted the low- and high-scoring groups on the Cito language and literacy test per Condition at a Bonferroni-adjusted alpha level of .025 (0.05/2). It appeared that the 50% lowest scoring on the Cito language and literacy test lagged behind the highest scoring in the conditions with still images (p < .001) and camera movements without reader control (p < .001) but not when the reader had control over the pace (p = .397). Figure 1 shows line plots for the two Cito language and literacy proficiency levels per condition.
Means (SDs) of Comprehension and Vocabulary Scores at Posttest Per Condition and Language Proficiency Level
Note. max. = maximum score.
Line plots for the two Cito language and literacy proficiency levels per condition.
Book-Based Vocabulary
We performed a two-way ANCOVA to examine condition effects on vocabulary after controlling for the book-based vocabulary score at the pretest and the Cito language and literacy test. We checked assumptions for performing an ANCOVA: linearity between covariate and outcome variable, homogeneity of regression slopes, no significant outliers in the groups, normally distributed vocabulary score for all conditions and the two language levels, and homogeneity of variance. As the Cito language and literacy test did not meet the homogeneity of regression slopes assumption, we applied a median split. We included the Cito language and literacy test score (median split) as a second factor (Iacobucci et al., 2015). For the rest, the results of the check were satisfactory. After adjustment for the book-based vocabulary score at pretest, F(1, 49) = 49.13, p < .001, η2 = .501, there was a significant difference in the posttest book-based vocabulary score between language levels favoring the high-level group, F(1, 49) = 11.45, p < 0.001, η2 = .189. There was no statistically significant difference between conditions, F(2, 49) = 1.41, p = .253, η2 = .055, nor was the interaction between condition and Cito language and literacy test score (median split) significant, F(2, 49) = 1.60, p = .21.
Discussion
This study tested the efficacy of a new prototype of a digital picture book, enhanced with two to three camera movements per illustration to direct children’s visual attention to the right areas of the screen at the right time. The aim was to optimize children’s use of the pictures in reconstructing the chain of connected events comprising the narrative. One key finding is that this relatively small but purposive accessory technology—highlighting story incidents that directly affect the plot’s trajectory in sync with the narration—is beneficial. However these effects are only present in the 50% of children that is least proficient in language and literacy skills. The impact in this group amounts to a partial eta squared of .163, equivalent to slightly less than 1 SD (Cohen’s d = 0.88). So, still images combined with camera movements reveal substantially higher story comprehension scores than just still images. This finding aligns with previous studies with camera movements framed in a smooth film-like format (e.g., Sarı et al., 2019; Sun et al., 2020).
The second key finding concerns reader control over the pace at which new pieces of story information appear. When children are in control, they use more time to process the story even though they could use less time. For the first reading, the sessions average about 30% longer than it takes to read the story without pauses. Findings indicate that particularly the low-language proficiency children benefit from the opportunity to spend more time processing information. There is a lag in story comprehension between low and high linguistic proficiency when picture books include still images or camera movements without pauses. However, this lag is no longer present where readers control the pace at which new pieces of information appear. If children scoring low on general linguistic proficiency have more time to process visual and auditory information, they overcome the lag in story comprehension with their more linguistically proficient peers.
A third key finding is that children do not significantly advance in book-based vocabulary. There is slightly more vocabulary improvement as picture books include accessory technology, but the differences between conditions are not significant. Note that different from previous findings with digital picture books, the enhanced book does not cause adverse effects on vocabulary, similar to Sarı et al.’s results (d = −0.56). A difference with the target books in Sarı et al.’s experiment is that our book does not include music, and there are environmental sounds at only a few locations. The presence of rather loud music and sounds in the background, as in Sarı et al.’s study, may have hindered an essential step in learning new words because the music and sounds may hinder isolating words from the speech stream (Nguyen & Grahn, 2017).
How an Enhanced Book Supports Learning
Well-chosen camera movements consolidate and extend the picture book’s role as a valuable cultural tool. Since the camera movements in the target app were not part of a smooth, film-like presentation, the effects cannot result from increased attention as a film-like format may promote. Instead, the camera movements improve the coordination of words and pictures. One explanation for the effects of camera movements on meaning making is the temporal contiguity principle: the camera movements are designed to help children fixate visual elements in the illustrations that match the story text (Mayer, 2011). Due to extraneous perceptual information, young children often do not establish referential links between the spoken text and the visual items in the synchronized image (Flack & Horst, 2017; Godwin et al., 2019; Verhallen & Bus, 2010). The targeted camera movements guide children’s visual attention to crucial information in the illustrations synced with the narrative, thus helping explain the chain of connected events comprising the narrative events. In line with another multimedia learning principle, the self-reflection principle, some camera movements attract attention to conflicting information in narration and illustration, thereby stimulating reflection on such contradictions. The camera movements target children’s attention to deliberate mismatches between words and pictures to make them aware of these ironic expressions and reflect on them. For instance, the boy (the narrator) emphasizes that his father trusts him and allows him to take the weather-make machine, while the pictures suggest the opposite. The current findings also support a third multimedia learning principle: the segmenting principle, predicting that people learn more deeply when a multimedia message appears in learner-paced segments rather than as a continuous unit (Mayer, 2011).
In particular, the children low in language and literacy proficiency benefit from zooming in or out and panning with a virtual camera guiding children’s eye fixations through the pictures, probably because these children often fail to understand the narrative and therefore depend more on the illustrations. Furthermore, children low in language and literacy proficiency benefit from the extra time to mentally organize each new information segment before moving on to the next piece. Additional time diminishes the risk of not understanding the story’s setting when the rest of the story evolves. For example, the reader must realize that the story takes place in another world high in the sky, and the main character is the son of the weatherman. Otherwise, it is hard to understand why the main character wishes to make lightning. Thus, additional time reduces this group’s risk of cognitive overload due to new chunks of story information that already appear while “old” info is not yet fully understood and integrated with other pieces of information (Mayer, 2011; Mayer & Chandler, 2001).
The findings also align with the hypothesis that the accessory technology in the target digital picture book is more beneficial for story comprehension than vocabulary learning, which accords with the design principles. Camera movements do not focus children’s attention on the words’ depictions, while the computer voice speaks those aloud. For this reason, we may not see a significant increase in book-based vocabulary scores as books include camera movements. Neither do the findings support the hypothesis that children may be more successful in deriving the meaning of unknown words from the visual and verbal context and thus acquire new word meanings from listening to a story that they better understand due to the accessory techniques. This result contradicts previous research with similar digital picture books that show effects on word learning (e.g., Smeets & Bus, 2012, 2014). An explanation may be that the same book was replayed more often (3–4 times) in those studies than in the current study (2 times).
Strengths, Limitations, and Future Directions
An abundance of studies focuses on commercially published digital picture books, including enhancements that expand background information or explain difficult words (e.g., Christ et al., 2019; Lauricella et al., 2009; Richter & Courage, 2017; Sun et al., 2020). By contrast, this study targets a book designed according to multimedia learning principles that help children coordinate and process the primary sources of information in picture books—words and pictures—in reconstructing the chain of connected events comprising the narrative. However, there are as always also some limitations apart from this strength. Designing the present study, we had only one story at our disposal to test the accessory technology. Future work would benefit from including multiple experimental digital picture books created by the same design principles. Similar findings for two or more books would guarantee that the accessory technology is sufficiently defined and effects do not depend on the story content or other book features.
The main limitation is that only about half of the participants in the current sample benefited from the target book’s affordances. The 50% scoring relatively high on a general language and literacy test reached high scores without accessory technology and did not reveal any affordances’ effect. Therefore, only half of the sample enabled testing effects of the accessory technology and proved differences across conditions. For effects of about half a standard deviation (d = 0.59), as we found for the contrast between a story that automatically proceeds and reader-controlled pace, the current sample sizes of n = 8 (reader-controlled pace) and n = 13 (no reader-control over pace) are too small to prove significance. Designing the current study, we have underestimated the current sample’s language proficiency or overestimated the story’s complexity. A replication of the present experiment with the same story or a story similar in difficulty should focus on younger children or children lagging in language proficiency.
Testing the book’s effects, we focused on children reading the story without adult support. However, the possibility of being in control of the pace at which information appears may make digital picture books also more appropriate for sharing the story with an adult (Mangen et al., 2019). To this end, the target book includes well-chosen interruptions guiding adults in discussing events critical in reconstructing the plot’s trajectory (Troseth et al., 2020). The book targeted in this study may thus have a more promising format for sharing stories than many other digital picture books. For instance, Hoel and Tønnessen’s (2019) findings confirm that commercial digital books used in group-wise reading sessions hinder exchanging ideas about the story, especially when the books include special effects that easily distract children from the storyline.
Practical Implications
The primary source of information in picture books is the narrative providing children with more speech input and more lexically sophisticated speech than other caregivers–child activities, thus making book reading particularly beneficial for language and literacy development (Eng et al., 2020; Montag, 2019; Montag et al., 2015). If, soon due to all available digital devices, watching videos will replace picture book reading (e.g., Neuman et al., 2017), this should raise serious concerns. The strengths of book reading—fostering comprehension of narratives and exposure to the complex narrative language used to tell stories—will get lost (Montag, 2019). This study endorses the idea that technology helping to process visual information in books can enhance the picturebook format’s unique quality—a narrative combined with rich illustrations telling the same story as the narrative. Therefore, it seems worthwhile to provide young readers with digital picture books that include accessory technology, as realized in the prototype tested in this article. By sparingly adding well-chosen camera movements to the illustrations, digital picture books can help younger and lexically less advanced children who have problems understanding the narrative use pictures in reconstructing the chain of connected events comprising the narrative. The second accessory in the new prototype, the possibility to control the pace at which new information appears, is also helpful. When the digital picture book includes this feature, there is no longer a story comprehension gap between linguistically more and less advanced children.
Effective camera movements into picture books make high demands on designers. Apart from a deep understanding of the story and its literary qualities, it demands the ability to empathize in what children find hard to understand and stimulate young readers’ self-reflection about what happens throughout the story. Designing app versions of picture books bring, therefore unavoidably, the need to involve, in addition to an illustrator and author, an artistic director who knows what is technically and artistically possible within the digital realm but only incorporates what may contribute to children’s meaning making. This study shows that it is undoubtedly worth the investment. Note not only the fact that digital picture books open up new prospects for young children but also children’s openly expressed desires to read digital books (Barnyak & McNelly, 2016; Strouse et al., 2019). See also a survey of over 1,500 parents of children younger than 8 years in the United Kingdom, showing that children read a digital book with audio narration on their own at least once a week (Kucirkova & Littleton, 2016).
Even though enhancements in digital picture books may not replace an adult (Munzer et al., 2019), they may outweigh the lack of adult guidance and make independent (re)reading of digital picture books by young children valuable. The current study shows that well-chosen camera movements under the child’s control encourage children to process the story’s content in a more profound way even when they read the story by themselves. Independent reading routines embraced in some cultures (Pham & Lim, 2019), but mostly discouraged (e.g., Reich et al., 2016), may be advantageous for children who lack adult availability (Salmon, 2014). The enhanced digital picture books embedded in newly formed book reading routines might provide us with a way to “languageize” (cf., Golinkoff et al., 2019) children’s lives and contribute to young children’s language and literacy development and thereby increase chances for academic success and successful life.
Footnotes
Acknowledgements
This work was supported by a grant from the Dutch Reading Foundation (Stichting Lezen) to Adriana G. Bus.
Authors
ADRIANA G. BUS is professor at University of Stavanger, Norway; and honorary professor at ELTE Eötvös Loránd University, Budapest, Hungary. She aims at designing new digital book formats that benefit not only children’s understanding of narratives but also their enjoyment of literature.
ROSALIE ANSTADT graduated as a historian and worked for several years as a teacher in Montessori education.