Abstract
Informational text—resources whose purpose is to inform—is essential to daily life and fundamental to literacy. Unfortunately, young children typically have limited exposure to informational text. Two 9-week randomized controlled trials with 263 first-grade children from low-income communities examined whether free educational videos and digital games supported children’s ability to use informational text to answer real-world questions. Participants received Internet-enabled tablets and were randomly assigned to condition. Study 1 found significant positive intervention impacts on child outcomes; Study 2 replicated these findings. Combined analyses demonstrated primary impact on children’s ability to identify and use structural and graphical features of informational text. Results are discussed in the context of the scalability of educational media to support informational text learning.
Keywords
As young children develop literacy skills, gaining exposure to and learning the hallmarks of a wide range of genres—for example, fictional narrative and informational text—are essential components of being able to access and use each type (Jeong et al., 2010; Price et al., 2009). However, many children have disproportionately less exposure to informational text throughout elementary school as compared to other genres, such as narrative text (Duke, 2000; Yopp & Yopp, 2012). This gap is critical, as children—and adults—use informational text every day, whether in referencing a textbook, using a diagram, looking up a fact online, consulting an expert, or following a recipe.
Informational text is not limited to nonfiction books. The genre includes written texts and oral and visual materials, including websites and lectures. Text that is primarily intended to convey information has a set of particular characteristics that distinguish it from narrative text (Duke, 2000). Typically, informational text includes timeless verb constructions rather than past tense; a wider diversity of technical vocabulary; and a variety of structural and graphical features, such as topic headings, captions, and indexes (Jeong et al., 2010; Jones et al., 2016; Young & Goering, 2018). Informational text also often organizes information into figures, maps, and tables that support learning from the text (Duke & Kays, 1998). Lack of exposure to informational text thus means that children are less familiar with these features.
This lack of exposure limits children’s abilities to function well academically and later in adulthood. Informational text is the format in which a substantial amount of academic content, especially science and social studies, is transmitted to students, and it is a component of standardized tests beginning in Grade 4 (Duke & Bennett-Armistead, 2003; Jeong et al., 2010). Increased emphasis on informational text in academic content is reflected in the National Assessment of Educational Progress (NAEP) reading assessments across Grades 4, 8, and 12: The proportion of informational text-style passages increases from 50% to 55% to 70% on each respective level of the test (U.S. Department of Education et al., 2019). Informational text is also the basis of the NAEP science assessments (U.S. Department of Education et al., 2019). Based on this research, we can conclude that children who lack sufficient exposure to informational text are at an academic disadvantage, both in learning from texts and in testing (Duke, 2000; Duke & Bennett-Armistead, 2003; Jeong et al., 2010). These academic challenges then compound, as the ability to read and critically process informational text remains a cornerstone of secondary and post-secondary education, as well as of nonacademic adult literacy (Moss, 2008; National Governors Association Center for Best Practices & Council of Chief State School Officers [NGAC & CCSSO], 2010).
Although empirical evidence has led to federal mandates that 50% of elementary school texts should be informational, systemic changes have proven difficult to implement, especially when confronting beliefs that have taken root over decades of teaching experience (Moss, 2008). Over the course of the 1996–97 school year, only 9.8% of first-grade classrooms’ library materials could be classified as informational text of any type, and only 3.6 minutes per day, on average, were spent using informational text in written language activities; in school districts with greater proportions of students who qualify for free and reduced-price lunch, this time dropped even lower, to 1.4 minutes per day (Duke, 2000). A decade later, at the beginning of the Common Core State Standard (CCSS) rollout, only 20.2% of the materials found in second-, third-, and fourth-grade classrooms’ libraries could be classified as informational text, compared with 70.9% narrative text (Jeong et al., 2010). 1 While minutes spent with informational text rose from an average of 1 minute during teacher read-alouds in Grade 2 to an average of 16 minutes in Grades 3 and 4, the majority of time these older students spent with written informational text was taken up with filling out worksheets (Jeong et al., 2010). Eighteen years after Duke’s original analysis, MacKay et al. (2020) found that of 13,000 texts coded from 23 first-grade classrooms’ libraries, 22.8% were classified as informational or expository text, compared with 63.2% narrative text; however, they did not find significant differences based on the socioeconomic status of the schools’ populations. This finding indicates that although there has been improvement in the inclusion of informational text in classroom libraries, and presumably in its availability to children, the 50-50 ratio of informational text to narrative text has yet to be realized.
The CCSS has been in place for an additional decade since Jeong et al.’s (2010) systematic analysis of informational text in the classroom; however, little evidence suggests that the situation has changed. While there have not been more recent studies in that vein, fourth-grade NAEP reading scores have remained largely unchanged for 30 states in the years since Jeong et al.’s research, suggesting that children’s exposure has also remained unchanged; between 2009 and 2019, 13 states or jurisdictions experienced a decrease in their average NAEP scores (U.S. Department of Education et al., 2019). Assuming that performance on standardized assessments is reflective of the knowledge and skills children gain in the classroom, these gaps may be attributable to persistent lack of access to informational text.
This disparity between mandates for informational text and the reality of classrooms and school libraries may be partially a result of teachers’ concerns. MacKay et al. (2020) found that while some teachers mentioned the importance of nonfiction books, many of the teachers they interviewed after analyzing their classroom libraries did not directly mention the CCSS when asked about choosing books and subjects to include. Instead, they talked about book cost, child reading level, and child interest. Half of the teachers they spoke with specifically mentioned nonfiction texts in connection with children’s interests. However, those who did mention informational text in relation to the CCSS did not necessarily have reflective representation of informational text in their classrooms (MacKay et al., 2020). In 2018, Young and Goering found that teachers in kindergarten through Grade 2, particularly those with more than 20 years of experience, were resistant to curricular changes necessitated by the inclusion of informational text in the CCSS, as they feared that children would not like informational text and would not be able to handle the material (Young & Goering, 2018).
However, ample empirical evidence shows that children can gain content knowledge and literacy skills from informational text—and enjoy the material as well. Children can detect the differences between textual styles as early as kindergarten, as long as they have had access to these styles (Donovan & Smolkin, 2002; Duke & Kays, 1998). Many children enjoy reading informational text even more than reading narrative fiction (Caswell & Duke, 1998; Robertson & Reese, 2017). Exposure to informational text may result in students reading more because it gives them more opportunities to read about their personal interests (Caswell & Duke, 1998; Duke & Bennett-Armistead, 2003). Research has demonstrated that engaging in informational text with an authentic purpose—for example, to pursue an interest or solve a problem—increases engagement as well as comprehension (Guthrie, 2003; Purcell-Gates et al., 2002). Informational text often combines areas of interest and helps build children’s background knowledge, vocabulary, and comprehension on specific subjects (Caswell & Duke, 1998; Duke & Bennett-Armistead, 2003). Additionally, more and earlier exposure to informational text builds children’s abilities to reproduce key features of informational text in their own writing (Donovan & Smolkin, 2002; Duke & Kays, 1998). Teachers who have incorporated additional informational text into their classroom environments and curricula have found that their students gravitate toward informational text content, even in their free reading time (Young & Goering, 2018).
The benefits of exposure to informational text—and the relative lack of it—are similar in the home context. Informational text is not well represented in homes, especially when young pre-readers and early readers are dependent on parents for their exposure. One study found that only 12% of the books parents reported reading with their preschool children classified as informational text (Roberston & Reese, 2017), and another found that parents reported that an average of 14% of their books at home were informational text (Price et al., 2009). Similarly, while studying read-alouds of informational text at home, Yopp and Yopp (2006) found that over the course of a year of kindergarten, 7% of the books read aloud were informational. However, evidence suggests that sharing informational text with children at home supports language development. Parents and children naturally engage in remarks and conversations, often sparked by the text itself—termed “extratextual utterances”—that are related to but outside the text they are reading together. Parents who read informational text and storybooks with their preschool-age children made extratextual utterances more frequently, at longer lengths, and at higher levels of complexity in terms of content when reading the informational text (Mol & Neuman, 2014; Price et al., 2009), thus contributing to overall language development (Neuman et al., 2000; Weisleder & Fernald, 2013). Their utterances also featured greater diversity of vocabulary while reading informational text (Price et al., 2009).
One solution to the lack of informational text across contexts is to reach children directly, outside school and family reading time. Educational media provide one such route. Young children under the age of 8 across the socioeconomic spectrum spend an average of about 2.5 hours a day using or watching onscreen media, including television, movies, and apps (Rideout & Robb, 2020). Educational media, especially public media, are a low- to no-cost tool for parents, caregivers, and educators to explain, model, and explore new and complex information with children (McClure et al., 2017; Silander et al., 2018; Troseth et al., 2006). Educational media can consistently deliver learning content to a wide and varied audience across large geographical areas, making educational media interventions highly scalable at low cost relative to other person-to-person early childhood interventions (Kearney & Levine, 2019). The time children already spend with screens thus provides a prime opportunity to meet children where they are with engaging, intentionally designed, educational media focused on informational text.
Children can learn from a variety of media, but not all media that can support learning are intentionally educational (e.g., Nebel et al., 2016). Research has shown that children learn best from intentional, high-quality educational media. This project is grounded in Fisch’s (2000) capacity model of children’s learning from educational media, which states that certain features of educational media increase children’s allocation of working memory to the media as a whole, and to educational content specifically, thereby supporting learning. In particular, the capacity model emphasizes that children learn more from media whose educational content is closely tied to the narrative. Consistent with features highlighted in the capacity model, studies show that children learn more from media that include relatable characters and stories (Bandura, 1965; Lauricella et al., 2011; Linebarger et al., 2017), are cognitively engaging, support meaningful and socially interactive learning experiences, feature meaningful repetition of key concepts across multiple contexts, and are guided by specific learning goals (Hirsh-Pasek et al., 2015; Kirkorian et al., 2008; Schmitt et al., 2018). In addition, educational media are particularly effective when parents and children jointly engage with the content. Co-viewing and joint media engagement have been shown to support literacy (Strouse et al., 2013), math (Pasnik et al., 2015), and science learning (Pasnik, 2019). To this end, videos and games that are jointly designed by researchers, designers, and content specialists are likely to result in more joint media engagement opportunities and more effective learning (Hirsh-Pasek et al., 2015; Vahey et al., 2018).
Several research-based educational videos and interactive digital interventions have focused on children’s science and math skills as well as on social-emotional learning (e.g., Lewis Presser et al., 2015; Rasmussen et al., 2016; Rasmussen et al., 2018; Rosenfeld et al., 2019). Research supports the effectiveness of out-of-school media interventions in teaching literacy to young children (e.g., Anderson et al., 2001; Fisch et al., 1999; Fisch et al., 2005; Grindal et al., 2019; Linebarger et al., 2017; Pasnik, 2019; Pasnik et al., 2015; Schmitt et al., 2018), including to young children in low-income households (e.g., Penuel et al., 2010). A meta-analysis of high-quality studies found that children with greater exposure to educational media show greater literacy gains (Hurwitz, 2019). Furthermore, research has shown that in home and school contexts, digital resources, such as apps, activities, and videos, can support skill development that transfers to real-world contexts (Grindal et al., 2019; Huber et al., 2015). To our knowledge, however, no rigorous experimental studies have examined the effectiveness on children’s learning of educational media focused on informational text. 2
The present research focused on the first season of Molly of Denali, a PBS KIDS multi-platform media program created to help children ages 4–8 develop the skills to use informational text through videos, interactive games, and hands-on, real-world activities. Molly is an adventurous 10-year-old Alaska Native girl who uses informational text to explore the world around her, solve problems, and help her community. Molly and her friends explore their world using field guides, maps, instruction manuals, informational websites, weather reports, and more. In each episode, Molly’s adventures are enhanced by using and creating a variety of informational texts, including books (e.g., field guides and how-to manuals), online resources, historical archives, information from knowledgeable people, maps, charts, tables, and photos (Timcheck, 2018). A mobile app features digital games based in Molly’s world that are designed to provide opportunities for children to engage with and explore informational text. Resources also include hands-on activities that provide an opportunity for children to engage in related real-world informational text activities, such as creating a field guide of backyard birds, ideally with older family members.
The Molly of Denali animated series was developed and produced by GBH, the Boston-based station within the public media system, in partnership with Public Broadcasting Service (PBS) and the Corporation for Public Broadcasting (CPB) as part of their 2015–2020 Ready To Learn Initiative, funded by the U.S. Department of Education. The Ready To Learn Initiative brings free educational television and digital media resources to children ages 2–8, promoting early learning and school readiness at scale, with an emphasis on supporting children from low-income, underserved communities. The content of all Molly of Denali materials is aligned with 15 informational text learning goals adapted and expanded from the PBS KIDS Literacy-English Language Arts Learning Framework Version 4.0 (PBS KIDS, 2016). The PBS KIDS Framework is aligned with the CCSS for English Language Arts (NGAC & CCSSO, 2010) and with the Head Start Early Learning Outcomes Framework: Ages Birth to Five (U.S. Department of Health and Human Services, 2015). Additional details about the Molly of Denali resources included in the study intervention are listed in the Intervention Materials section.
The study team reports here on the combined results of two randomized controlled trials (RCTs). The study team initially intended to conduct a single RCT on the impact of access to the Molly of Denali resources with 500 families across the United States. However, the COVID-19 pandemic halted study activities after baseline data collection had been completed in person at two locations (Birmingham, Alabama, and Phoenix, Arizona); the study team completed data collection with the 127 families already enrolled (Study 1), using videoconferencing for the post-intervention assessment. The study team then replicated the same intervention in a fully online RCT (Study 2) with new participants from across the United States. Because recruitment criteria and the intervention itself were identical in the two studies, the study team combined data across the two studies to increase our statistical power to detect effects and completed analyses with the full sample. (Individual results from Study 1 and the replication, Study 2, are substantively the same and are reported in the Appendix.) This research was designed to examine how exposure to the PBS KIDS Molly of Denali videos, digital games, and hands-on activities affects children’s understanding of concepts and practices related to informational text. But because it was planned before the COVID-19 pandemic and was completed during the pandemic, this study also provides a unique window into adaptations that can allow research to continue during disruptions, such as a pandemic. All analyses addressed the following confirmatory research question.
CQ1: Does providing 9 weeks of access to Molly of Denali resources via an Internet-enabled tablet improve low-income first-grade children’s ability to use informational text skills to answer questions or solve real-world problems, as compared to providing an Internet-enabled tablet that cannot access Molly of Denali resources?
The study team also examined impacts on specific informational text skills and dispositions and whether treatment impacts differed by demographic variables, children’s use of the resources, and parent-reported child interest in the intervention materials. These analyses were guided by the following exploratory research questions.
Compared with providing an Internet-enabled tablet that cannot access Molly of Denali resources, does providing 9 weeks of access to Molly of Denali resources via an Internet-enabled tablet improve low-income first-grade children’s … EQ1. … disposition to use informational texts? EQ2. … ability to understand the purpose and use of structural features of informational text? How does the impact of providing 9 weeks of access to Molly of Denali resources on low-income first-grade children’s informational-text skills vary by … EQ3. … demographic variables? EQ4. … duration of engagement and number of times Molly of Denali resources were accessed? EQ5. … parent-reported child interest in Molly of Denali resources?
3
Participants
Families were recruited by an external recruitment firm, primarily through social media. Participants were 263 first-grade children from low-income families, with a mean age of 86.56 months (SD = 6.03; see Table 1 for demographic information). For Study 1, the study team recruited 127 children from families from Birmingham, Alabama, and Phoenix, Arizona, before the COVID-19 pandemic halted research efforts. 4 For Study 2, the study team first recontacted families who had already been recruited for the remaining sites in Study 1 (Chicago, Illinois; New York, New York; and Oklahoma City, Oklahoma) but who were not enrolled due to the pandemic-related study curtailment. The study team then expanded recruitment to families across the United States. The final sample of participants resided in the following locations: Alabama (28.9%); Arizona (19.8%); Oklahoma (17.9%), Illinois (10.6%), New York (5.7%), Alaska (5.3%), and other locations that were not part of Study 1 recruitment efforts (17.1%). The sample was relatively diverse in terms of urbanicity, with 50.2% urban participants, 21.8% suburban, and 28.0% rural. All participating families met the following eligibility requirements.
Families were low income (68.7% reported household earnings of less than $50,000 per year). A family qualified as low income if they received free or reduced-priced lunch (FRPL); if their income was determined to be equal to or less than 125% of the income requirement for FRPL, regardless of whether they indicated such a qualification; or if they participated in other government assistance programs, including the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), the Supplemental Nutrition Assistance Program (SNAP), or Temporary Assistance for Needy Families (TANF).
Only one child per household could participate. The participating child had to be enrolled in Grade 1 as of March 1, 2020; fluent in English; able to participate in gamelike activities for up to 45 minutes at a time; and not have had heavy prior exposure to Molly of Denali (defined as having watched or played 4 or more hours of Molly of Denali content in the prior 7 days, by parent report).
At least one parent was proficient in either English or Spanish. Child-facing intervention materials were available only in English; parent-facing materials were available in both languages.
Children’s Baseline Literacy Knowledge and Demographic Characteristics, Overall and by Condition, for the Full Sample
Note. The treatment and control group did not differ significantly in the combined sample. Asterisks represent comparisons between the Study 1 and 2 samples.
SD = Standard Deviation; EVT = Expressive Vocabulary Test (third edition); DIBELS = Dynamic Indicators of Basic Early Literacy Skills (eighth edition); HS = High School; GED = High School Equivalency, AA = Associate degree, BA = Bachelor’s degree.
p < .05. **p < .01. ***p < .001.
Two additional criteria applied in Study 2 because of the virtual data collection:
Participants had access to Wi-Fi or cellular signal strong enough to hold an hour-long video call. 5
Participants were able to safely receive packages (or live near a business accepting FedEx shipments), as the tablet was shipped to their residence.
Participants came from diverse racial and ethnic backgrounds: 31.9% identified as Black or African American, 28.1% as White, 24.0% as multiracial, and 13.7% as Hispanic or Latinx. Most children (83.7%) spoke English as their primary language, likely due to our recruitment criterion that one caregiver had to be proficient in English. The sample included neurodiverse children, with 16.8% having reported some form of an individualized education program (IEP), 504 plan, or therapy. However, one child was excluded from Study 1 prior to randomization because he was nonverbal, and valid one-on-one assessment was impossible. 6
No significant baseline differences existed between the treatment and control groups (see Table 2). However, some significant differences were evident between the samples from Study 1 and Study 2. Study 2 participants were statistically significantly older than Study 1 participants, b = 4.22, p < .001, g = .74; this difference was expected, given that Study 2 began 2 months after Study 1 concluded and continued to enroll first-graders. Compared with Study 2 participants, Study 1 participants were significantly more likely to be White, b = –.17, p < .01, g = –.39; to speak English as their primary language, b = –.12, p < .01, g = –.33; and to have a family annual income of less than $50,000, b = –.12, p < .05, g = –.26. We consider these differences to be a strength of the replication study, as the participants in each study represented a different sample with different demographics. Replication of findings in each sample thus demonstrates some generalizability.
Baseline Differences in ITSI Total Score, Vocabulary Score, and Demographic Characteristics, Between Treatment and Control and Between Study 1 and Study 2
Note. Parents reported child gender on the parent pre-survey. Response options were male, female, and other. All parents selected either male or female. There were no statistically significant treatment–control differences between the treatment and control groups.
Molly = Molly of Denali, ITSI = Informational Text Skills Instrument, IT = informational text, EVT = Expressive Vocabulary Test (third edition), IEP = individualized education program, HS = high school.
Overall attrition for the full sample was low, at 4.18% (3.47% for the treatment group, 5.04% for the control group, and 1.57% differential attrition). This combination of overall and differential attrition represents a tolerable threat of bias under optimistic and cautious assumptions (What Works Clearinghouse [WWC], 2017). Participating families received $125 in incentives ($25 after the pre-intervention assessment and $100 after the final meeting) and kept the study tablet.
Method
The study team designed this study to meet WWC standards (Version 4.0) without reservations and preregistered the study with the Registry of Efficacy and Effectiveness Studies (#1794.1v2). The study team successfully tested the procedures reported here, including randomization, during a 4-week pilot study with 71 families in Indianapolis, Indiana, and Anchorage, Alaska, in the fall of 2019.
For Study 1, the study team randomized children to condition during the in-person pre-intervention assessment meeting based on the order in which they completed the baseline assessments, which depended on the timing of their research appointment and the duration of their baseline assessment. Once assigned to their condition, the family received either a treatment- or control-provisioned tablet and an orientation to the study. The post-intervention assessment was conducted via video conference.
For Study 2, the study team conducted the entire study, from consent through posttest assessment, remotely via video conferencing. Participants in Study 2 met virtually with researchers three times instead of two: (a) consent, (b) pretest and orientation to the study, and (c) posttest. Rather than assignment to condition after pretest, as in Study 1, researchers assigned Study 2 participants to condition (using the same prerandomized sequence of treatment- and control-assigned ID numbers used in Study 1) based on the time-stamped order in which parents completed the pre-study survey after their one-on-one consent meeting. Researchers excluded parents who consented but did not complete the parent pre-survey within 24 hours of receipt from the study prior to randomization, and they did not receive a tablet. 7 The study team then shipped participants either a treatment- or control-provisioned tablet according to their random assignment. Study orientation was then completed after the pretest, as part of the second meeting.
For both studies, the study team kept families blind to the condition to which they were assigned, informing all participants that they were in a study of educational media use for young children. Procedures throughout the study also ensured that assessors were blind to the study participants’ assigned condition. The intervention and control conditions, described below, were identical for both studies.
Researchers provided children in
Researchers instructed families in the
One tablet showed use of the PBS KIDS Games app for 71 minutes, Amazing Amphibians for 8 minutes, and Starfall Learn to Read for 53 minutes.
A second tablet showed use of the PBS KIDS Games app for 44 minutes.
A third tablet showed use of the PBS KIDS Games app for 11 hours.
It was not possible to identify the specific PBS KIDS content accessed by these three children in the control group; in the worst-case scenario, these children exclusively engaged with Molly of Denali materials, which would weaken the likelihood of detecting between-group differences in the outcome. As such, the findings reported here are not compromised by this unintended access.
Families in the
Intervention Materials
All intervention content was produced by GBH and PBS KIDS. The study team curated the intervention materials to provide a more structured version of how children would typically encounter the publicly available Molly of Denali resources in everyday life, while intentionally allowing for natural variation in the amount of time families spent with the resources. Because natural variation was an intentional part of the design, this approach simultaneously permitted the examination of potential dosage effects and eliminated the need to ensure fidelity. Families accessed all intervention materials through a researcher-developed app available on the tablet home screen; game content also could be accessed through a separate game app, also on the home screen. The intervention focused on two types of informational text: informative/explanatory and procedural. Given the short duration of the intervention and the home context, the intervention targeted skills and knowledge that did not require extensive adult mediation. In all materials for parents, the study team emphasized the importance of repetition for young children’s learning and of parent co-engagement with the intervention resources. This realistic intervention did result in natural variation in the amount of time children spent with the intervention materials (see Results), although most children met the researchers’ recommendation to spend about 1 hour per week.
Molly of Denali Videos
Each Molly of Denali video episode consists of two animated stories, separated by a 2-minute live-action video that builds on the skill presented in the first story. To sustain children’s engagement with the intervention resources, 10 the study team organized videos into weekly bundles of content based on the focal learning goal identified by PBS: captions, procedural text, search boxes, indexes, flow diagrams, and evaluating sources. At least two researchers reviewed and cross-coded videos for content (focal and incidental) to confirm PBS’s learning goal classifications. The first week of videos provided an overview, with one video from each focal skill area; the final 2 weeks were review weeks, with bundles that included some repeat videos. The order in which bundles were released was static across participants: Many stories address more than one skill, so bundles were ordered to present stories that foreground the focal content before those that contain that content in the background. Each bundle contained approximately 70 minutes of content targeting the focal skill. Videos were available only in English.
Molly of Denali Games
The intervention included an app with three digital games focused on informational text, accessed through an interactive map-style representation of Molly’s home village; two games targeted multiple skills across content bundles, and one targeted procedural text. The map also included mini-activities (e.g., tossing a basketball) and a game intended for younger children that did not have informational text content, so the study team did not consider these to be part of the intervention. Due to software constraints, it was not possible to stagger the release of the games; all games were available at the start of the study and then throughout the study. However, the study calendar provided to treatment families did show the alignment of each game with the specific skill targeted each week. Three hands-on, real-world activities also were available, in English and Spanish, in printed form and as PDFs in the tablet app.
Instructional Videos for Parents
To support parent co-engagement, PBS developed seven short videos to orient caregivers to informational text, to introduce the Molly of Denali learning goals, and to support families’ use of the Molly of Denali materials; these videos were available on the PBSLearningMedia.org website. Parents watched the introductory video during orientation at the first meeting with researchers. Each subsequent video focused on a particular informational text concept or skill, aligned with the bundles, and modeled parents engaging their children around informational text. All videos were available through the tablet app in English and Spanish.
Measures
For Study 1, researchers collected pretest measures in person, and families completed all posttest child assessments remotely on the study tablet provided to them. For Study 2, all assessments were completed remotely via the study tablet.
Baseline Measures
Researchers measured expressive language ability at baseline to establish treatment–control equivalence on a construct that might affect performance on the informational text measure regardless of informational text knowledge, using the Expressive Vocabulary Test, third edition (EVT-3; Williams, 2019). For Study 2, the study team administered a digital version of the EVT-3, provided by the publisher, via screen share. Assessors provided prompts orally and recorded children’s responses on paper scoresheets. The prompts and the scoresheets were identical to the in-person version used in Study 1. In Study 1, researchers also estimated children’s reading abilities using the Dynamic Indicators of Basic Early Literacy Skills, eighth edition (DIBELS-8; University of Oregon, 2018). However, this measure did not translate well to remote assessment (e.g., the digital version was difficult to see, particularly for children with visual disabilities, and stimuli were easily obscured on the tablet screen), and it overlapped significantly with the EVT-3 in terms of variance explained. 11 The DIBELS-8 was thus dropped in Study 2 and excluded from combined analyses.
Informational Text Assessment
Existing measures of children’s informational text knowledge and skills either relied on open-ended questions that required too much subjective coding for a large-scale RCT (Billman et al., 2008; Witmer et al., 2014) or were designed for older children (Duke et al., 2021). Thus, the study team designed a measure, the Informational Text Skills Instrument (ITSI), to assess children’s disposition and ability to use informational text. The ITSI primarily focuses on the CCSS English Language Arts–Literacy (ELA-Literacy) standard RI.1.5: Know and use various text features (e.g., headings, tables of contents, glossaries, electronic menus, icons) to locate key facts or information in a text (NGAC & CCSSO, 2010). The ITSI consists of 37 items: 23 closed-ended items use a flipbook format (see Figure 1 for an example), and the remaining 14 items ask the child to use an informational text to answer practical and authentic questions (e.g., how to find a specific topic using the table of contents; see Figure A1 in the online Appendix for an example). Eleven items are open-ended but require very brief answers that are either right or wrong (e.g., Q: What is this called? A: A search box.). To address variations in reading ability, assessors read all key pieces of text aloud. The ITSI takes approximately 15 minutes to administer. To avoid over-alignment with the intervention, researchers included several distractor response options, added sections that measured components of informational text that were not targeted by the intervention, and included no text or graphics similar to those used in Molly of Denali.
Sample Closed-Response Item From the Informational Text Skills Assessment (ITSI)
Researchers initially developed three equivalent forms of the ITSI, targeting the same skills but using different books and stimuli. For example, Form A used a book on butterflies and Form B on plants; Form A focused on birds and Form B on fish. All questions and materials were written at the first-grade level. After minor modifications following expert review, the study team piloted the instrument with a sample of 71 children, and a confirmatory factor analysis showed a single scale with a Cronbach’s alpha of .80. Based on mean scores and subscores from the pilot, the study team selected the two most equivalent forms for use in the RCT; mean scores did not vary significantly (p = .53). Researchers randomly assigned half of the children to receive Form A at pretest and the other half to receive Form B; children then received the alternate form at posttest to mitigate the potential for learning effects or priming.
Due to the COVID-19 pandemic, the study team adapted the ITSI to be administered via video conference using an animated PowerPoint on the study tablet at Study 1 posttest. Research suggests that tablet administration yields very similar results to in-person administration of paper-based tests (Neumann & Neumann, 2018). Researchers used screen share to present the stimuli, asked the questions orally, and recorded children’s responses on paper scoresheets. The remote assessment required more verbal expression from children: Although most items remained identical, researchers converted five items that had been open-ended and that had permitted nonverbal responses (e.g., “Point to the caption”) in the in-person format to closed-ended verbal responses. For these items, researchers enclosed key elements of the page in boxes and labeled the boxes with colors and numbers to give children multiple ways to respond. Researchers chose colors that are readily distinguishable by children with colorblindness according to universal design standards. Correlations between pre- and posttest scores were higher for Study 2 (Control = .84; Treatment = .75; All = .78), where both assessments were remote, than for Study 1 (Control = .63; Treatment = .74; All = .68), where the pretest was in person and the posttest was remote. Pre-post correlations for both timepoints, however, were well within the range of typical pre-post correlations in applied settings (Estrada et al., 2019). Furthermore, overall pre-post correlations were not significantly different across the two studies based on a Fisher’s r-to-Z transformation (Weiss, 2011). This suggests that the in-person and remote assessments functioned similarly enough to justify combining analyses.
Psychometric analyses of children’s performance on the in-person and remote assessments from Study 1 and Study 2 resulted in the elimination of 11 items after examination of the item-total correlations. These items had item-total correlations less than .20 and in general were far too easy, with 80%–90% of children getting them correct, even on the pretest. Factor analysis of the resulting reduced instrument showed two factors: The first (7 items) related to the disposition to use informational text to solve a problem, and the second (19 items) assessed the ability to identify and use different structural and graphical features of informational text. The final instrument contained 26 items and yielded one total score and two subscores (one for each factor). Psychometrics for each score are reported in Table A1 in the Appendix; mean scores for each study and the combined sample are reported in Table A2.
Parent Survey
Parents in the treatment and control groups completed digital pre- and post-intervention surveys using Qualtrics. All surveys were written at approximately a fourth-grade reading level and were available in English and Spanish. Parents completed the pre-study survey before assignment to condition. The parent survey included demographic questions regarding parent and child backgrounds as well as questions regarding child engagement with the intervention materials (treatment group posttest only). The post-intervention survey in Study 1 included sections on COVID-related changes in the home that were moved to the pre-survey for Study 2. This section was intended to provide information about family context in the event of any significant demographic differences between the two samples.
Usage Data (Treatment Group Only) 12
Using back-end data from study tablets, the study team calculated children’s usage of the intervention resources. Because the literature does not agree about how best to measure usage and to permit a more robust analysis than using a single measure, usage was examined in two ways: as a total number of hours and as a total number of times. Total hours was defined as the time that each child in the treatment group spent watching all Molly of Denali videos plus the time spent playing Molly of Denali games that had informational text content, as noted in the intervention description. The number of times children accessed the Molly of Denali resources was operationalized as a count of the number of days on which children used games, videos, or both.
Data Analysis
The study team conducted linear-regression analyses to address the confirmatory research question on the impact of Molly of Denali resources on children’s ability to use informational text to answer questions or solve real-world problems. First, the study team examined bivariate correlations between the ITSI total score and family background characteristics in the full sample to identify a consistent set of child and family characteristics to include in all regression models across studies (see Table 3). Researchers selected variables for inclusion if they were correlated at p < .10 with the ITSI total score at posttest. The study team dichotomized multicategorical variables (e.g., parent education) for ease of interpretation. The final set of covariates that met these criteria was child gender, child age in months (continuous), child ethnicity (dichotomized to White or non-White), family income (dichotomized to less than $50,000 per year versus more), responding parents’ education (dichotomized to high school diploma or less versus some college or higher), child’s disability status, and urbanicity (suburban and rural versus urban). Analysis of collinearity among covariates indicated that multicollinearity was not a concern; all variance inflation factor values were below 2.9.
Bivariate Correlations Between Study Variables, Full Sample (Study 1 and Study 2)
Note. N = 263. ITSI features = Subscore for ability to understand the purpose of and use structural features of informational text; ITSI disp. = Subscore for disposition to use informational texts; EVT = Expressive Vocabulary Test; HS = high school; IEP = individualized education plan; Prior Molly minutes = parent-reported minutes of exposure to Molly of Denali in the 7 days prior to the parent pre-survey.
p < .10. *p < .05. **p < .01. ***p < .001.
The study team then estimated stepwise regression models to determine which model best predicted the informational text assessment total score (i.e., explained the highest variance in the outcome). All models included a treatment–control indicator. Model 1 included only the baseline informational text assessment total score; Model 2 added baseline EVT scores; Model 3 added child and family covariates; and Model 4 added indicators for the study site (i.e., Arizona, Illinois, New York, and Oklahoma, with all other states and Alabama 13 as the reference group). Model 5 included an indicator for Study 1 versus Study 2; this variable was included to account for significant differences between the study samples at baseline and to account for any unmeasured differences between the two samples related to time of year, COVID-19, or other unanticipated factors. Model 5 explained significantly more variance than did Model 4 and was used for all analyses of the full sample. The study team estimated this model to examine the impact of Molly of Denali resources on children’s ability to use informational text in the full sample, with the outcome variable as the total score on the ITSI. In addition, to report the magnitude of the effect in a standardized metric that could be understood relative to other findings, the study team calculated an effect size, Hedge’s g (WWC, 2017), by dividing the treatment unstandardized coefficient by the pooled standard deviation.
To answer EQ1 and EQ2, the study team estimated the same regression model that was used for the impact analyses (Model 5). The outcomes were the two subscales of the ITSI (i.e., structural features of informational text and disposition to use informational text). In addition to examining the overall treatment impact of Molly of Denali resources on children’s informational text skills, the study team examined whether the impact varied by demographic characteristics, including children’s baseline literacy scores, gender, age and ethnicity, and parents’ education level or income (EQ3). The study team created interaction terms by multiplying the treatment status by each of these characteristics; these interaction terms were entered as additional predictors in Model 5.
To answer EQ4, the study team estimated the association between usage of Molly of Denali resources and ITSI total scores. Researchers estimated the same regression model (Model 5) but entered resource usage as a predictor instead of study condition and limited the sample to the treatment group. Researchers conducted these analyses including and excluding children with extreme values of usage (more than three standard deviations above the mean).
Finally, to answer EQ5, researchers estimated the association between parent-reported interest in Molly of Denali videos and games and ITSI total scores. Parents rated their child’s interest in Molly of Denali videos and games on a scale from 1 (low interest) to 10 (high interest). Researchers estimated the same regression model (Model 5) but entered parent-reported interest as a continuous predictor instead of study condition and limited the sample to the treatment group.
Results
Consistent with the findings from the separate analyses of Studies 1 and 2 (see the Appendix), in the full sample, the treatment group outperformed the control group on ability to use informational text to answer questions or solve real-world problems, controlling for baseline EVT score, demographics (child gender, child age in months, child race, family income, parental education, and urbanicity), site, and study, b(SE) = 1.25(0.43), p = .004, g = .24. (See Table 4 for final model.) Figure 2 shows mean adjusted post-intervention assessment scores for each group in Study 1, Study 2, and the full sample.
Regression Results for Treatment Impact of Molly of Denali Resources on ITSI Total Score—Combined Sample
Note. Continuous variables are grand mean centered.
ITSI = Informational Text Skills Instrument; SE = standard error; EVT = Expressive Vocabulary Test (third edition); HS = high school; IEP = individualized education program; AZ = Arizona; AL = Alabama; IL = Illinois; NY = New York; OK = Oklahoma.
Impact of Access to Intervention Resources, by Study
EQ1 and EQ2 examined the impact of access to the Molly of Denali resources on assessment subscores. There was no significant treatment impact on disposition to use informational text, b = .06, p = .769, g = .03. However, children in the treatment group scored higher than did children in the control group on ability to understand the purpose of and to use structural features of informational text, b = 1.23, p = .001, g = .30.
EQ3 examined the extent to which the impact of providing access to Molly of Denali resources varied by demographic characteristics. There was no significant treatment by demographic interactions for EVT score, gender, parent income, parent education, or ethnicity. Older children benefited less from access to the resources than did younger children, b = −.16, p = .030.
EQ4 examined the associations between usage of Molly of Denali resources and children’s informational text skills within the treatment group. The amount of time children used the resources and the consistency with which they used them over the course of the study varied significantly. The majority of children (57%), however, accessed resources consistently over time, using games and/or videos for some amount of time during the 7 to 9 weeks of the study period. Twenty-nine percent of children accessed the resources during 4 to 6 weeks of the study period, and 14% for 3 or fewer weeks; only one of these children did not access Molly of Denali resources at all. Two children had extreme values for usage, with hours of combined video and game time totaling more than three standard deviations above the mean. Inspection of their daily usage data indicated values as high as 6.3 and 8.3 hours of usage in a single day. Although these extreme values were not consistent across the study period, these results raised some suspicion that the tracking data might have been incorrect in these cases. In addition, outlier predictor values have the potential to exert undue influence on regression results (Stevens, 1984). For these reasons, we conducted the following analyses excluding and including children with outlier usage values. Excluding these two children, we found that more combined hours spent watching Molly of Denali videos and playing the focal Molly of Denali games (M = 10.32, SD = 8.11) were associated with higher posttest scores, b(SE) = .20(.04), p < .001. This means that for every hour spent on Molly of Denali content, children scored .20 points higher at posttest (out of 27 possible points). Including all children in the analysis, usage was still associated with higher posttest scores, b(SE) = .11(.03), p = .001, but the effect had a smaller magnitude (every hour of usage was associated with a .11-point increase). Because children’s patterns of interaction with the videos and games varied considerably (see Figure 3), researchers also analyzed video and game usage separately.
Average Minutes Using Focal Resources (Videos and Games) in Each Study, by Week
For analyses with video and game usage as separate predictors, we conducted analyses excluding and including six children with extreme values: Three children had video usage that was greater than three standard deviations above the mean, and three had game usage that was three standard deviations above the mean. Excluding children with outlier values, hours children spent watching Molly of Denali videos were associated with higher posttest scores, b(SE) = .21(.06), p < .001, but hours spent playing the focal Molly of Denali games were not, b(SE) = .18(.15), p = .214. Including all children in the analysis yielded similar results with slightly smaller magnitudes; hours watching videos were associated with higher posttest scores, b(SE) = .12(.04), p = .004, but hours spent playing the focal games were not, b(SE) = .10(.12), p = .383. The results using the number of days metric of usage were very similar: An increase in the number of days on which children accessed Molly of Denali resources was also predictive of higher posttest scores, b(SE) = .15(.04), p < .001; when broken out by video and game usage, days of video usage were significant, b(SE) = .11(.04), p = .011, but days of game usage were not, b(SE) = .07(.07), p = .311. There were no outliers in analyses using the number of days metric.
Children in the treatment group who were more interested in the Molly of Denali videos and games, according to parent reports, did not benefit more than did children who were less interested, b = .18, p = .095 (EQ5).
Discussion
This research is the first to show that children can learn informational text skills from educational media. Children’s ability to use and learn from informational text is critical to their success in school and later in life. Given the lack of informational text content in elementary school curricula and libraries (Duke, 2000; Jeong et al., 2010), public media offer an inexpensive, scalable method to introduce children to the key features and functions of informational text. Two RCTs investigated the efficacy of Molly of Denali, a PBS KIDS program targeting informational text knowledge in children ages 4–8, in improving children’s ability to use informational text to answer questions and solve real-world problems. Although the effect sizes were modest, results of the 9-week intervention demonstrated statistically significant positive impacts in both studies and in combined analyses, particularly promising for a short-term, light-touch intervention that imitated children’s real-world engagement with Molly of Denali.
Although replication is an essential part of the scientific process to confirm the efficacy of an intervention, it is uncommon in education research for study findings to be replicated with a second group of participants (Makel & Plucker, 2014). Although the COVID-19 pandemic forced the study team to end enrollment into Study 1 well short of the 500 participants originally planned, it presented the opportunity to conduct a fully remote replication study with a new set of participants from across the United States. Moreover, because many of the participants in Study 2 were recruited to be in Study 1 before the pandemic, the two samples were similar in many ways. The successful replication of the Study 1 findings in Study 2, despite the pandemic, speaks not only to the impact of the Molly of Denali resources but also to the strength of the study design, including intensive logistical planning to ensure proper randomization and to preserve assessor-blind, successful conversion of the ITSI assessment to a digital format, and the provision of adequate support for families to engage in the remote research process.
This study provides further evidence that learning can take place when children engage at home with intentionally developed, high-quality educational media. This is particularly important at a time when millions of children’s in-school education has been substantially disrupted, but it is also meaningful for children who lack access to quality early learning opportunities, such as affordable early learning programs, community supports, libraries, and other resources. Our finding that older children benefited less from the intervention might be a result of longer exposure to school curricula targeting informational text; given the school disruptions many participating children faced, however, other explanations also are possible.
One further contribution of this work to the scientific community relates to how the resources were released during the intervention. Because prior studies (e.g., Grindal et al., 2019) have shown a rapid drop-off in children’s usage of intervention media, these studies used a timed release of weekly bundles of video content in an attempt to sustain engagement with the resources for the duration of the 9-week intervention. Software restrictions prevented doing the same with the digital games, however, which were available all at once and throughout the study. This approach actually mimics children’s real-world access to the content: Videos on the PBS KIDS website and many other platforms are not all available at once but cycled in and out over time, whereas the games typically are always available. While the study was not designed to compare videos and games in terms of usage or learning, the tracking software on the tablets allowed the examination of children’s relative usage under the two delivery methods. Analyses showed the expected precipitous drop in games usage but also more sustained engagement with the videos through the duration of the 9 weeks, albeit at a lower level (see Figure 3). While it is possible that the videos were just more interesting over time, the potential for staggered release to maintain interest and engagement bears further investigation.
This distinction is particularly important because analyses also demonstrated that time spent watching videos—but not time playing games—was related to treatment impacts in the studies reported here. Because prior research has demonstrated learning gains from both videos (Fisch, 2004) and digital games (see Lieberman et al., 2009, for a review), the findings here are at least suggestive that sustained engagement over time might lead to better outcomes. Of course, other structural features of the digital games used in the study—for example, a lack of difficulty progression—might have led to lower interest, less use, and thus less impact on learning.
Our findings also suggest that the intervention resources were able to produce learning gains even when families were distracted by national hardships. In Study 1, 55.2% of parents said that the pandemic made it harder to engage with the intervention resources, and 47.8% said that the pandemic made it harder to attend the study meetings and complete the surveys. To some extent, many families seemed to have adjusted to COVID-related restrictions by Study 2, as those numbers dropped to 44.2% and 29.9%, respectively. For such positive impacts to occur during such a stressful and disruptive time for families emphasizes the power of this low-cost and scalable approach.
Finally, these studies provide concrete evidence that education research can continue during a pandemic. The posttest for Study 1 and the entire replication study occurred during the COVID-19 pandemic, requiring a pivot to remote forms of participant contact and assessment. The low attrition observed for both studies suggests that remote research might be easier for families to engage in. It is also possible that the study incentive held more significance for families due to pandemic-related financial hardships. While the demands on the study team certainly were no lighter, successful navigation of these demands makes remote research in the future a real and manageable proposition. This possibility is critical, as capacity to perform these kinds of fully remote studies is essential to the continued performance of education research under conditions where in-person meetings are not possible. Remote assessment also extends participation opportunities to families who are often excluded from these studies—for example, rural families for whom travel to a central data collection destination is more of a burden. By eliminating the need for specific sites, participants can be more spread out and, thus, more representative of the general population, without requiring additional travel.
Limitations
While replication of the findings from Study 1 with a different sample drawn more broadly from across the United States suggests that these findings are generalizable to other first-graders, these studies nonetheless have some limitations. Although this study design was rigorous in terms of randomization and double-blind, sample sizes in each study were relatively small. Confirmation of our primary findings with the combined sample does suggest, though, that our findings are robust.
In addition, both studies were conducted during the COVID-19 pandemic, which may affect the generalizability of our findings. Our participant samples may favor those who had the emotional and material resources to participate in a research study during the pandemic. Given that recruitment for Study 2 was conducted in early July 2020, during a time of nationwide protests for racial equality, it is also possible that families affected by the racial justice movement were less likely to participate in a research study at that time. In the combined sample, a majority of parents (54.9%) reported decreased household income due to the pandemic, suggesting that the sample does include individuals negatively affected by the pandemic.
In addition, pandemic-related school closures could have increased demand for educational media, including the intervention resources in the treatment group. If so, these findings may be less generalizable to times when school is offered in person. During Study 1, 94.2% of parents reported school closures, and 69.9% did so in Study 2. Most parents in both studies (71.9%–83.9% by study and condition) reported that the pandemic increased the amount of time their child spent with educational technology. If this time included abnormally high usage of the intervention materials in the treatment group, it likely resulted in stronger learning outcomes; our findings show that children who spent more time using the intervention resources benefited more. Study 2 school-closure rates were lower than in Study 1; the replication of our findings in Study 2 suggests that impacts are still observable in a sample with lower rates of school closures and, thus, that school closures are not exclusively driving these results.
Because existing measures were designed for older children, relied on open-ended coding, or both, these studies used a researcher-designed measure, the ITSI. Any study relying on a researcher-derived measure runs the risk of being over-aligned with the intervention. The study team thus aligned the ITSI with the CCSS and the PBS KIDS literacy–ELA learning framework. In addition, the ITSI included distractor items and assessed content areas that were not part of the intervention (e.g., the glossary). Future research using performance on existing, standardized measures—for example, state ELA testing—is warranted to examine whether the impacts demonstrated here predict more distal learning outcomes.
Implications for Policy and Practice
The results of this research highlight the efficacy of free public media to improve children’s learning outcomes. Public media represent a low-cost, highly scalable means of reaching into the homes of millions of U.S. children and may provide learning opportunities in locations where access to early learning opportunities is limited. As such, federal programs, such as the Ready To Learn Initiative, that invest in innovative approaches to early learning through media are key to ensuring equitable access to education for young learners. In addition, the content of such media does not need to be explicitly didactic. A rich body of research shows the efficacy of innovative media in changing behavior (at least in adults) through nondirect and incidental content (see, for example, the use of telenovelas: Forster et al., 2016; Rios-Ellis et al., 2010; Singhal, 2007; Wilkin et al., 2007). Molly of Denali videos and games foreground an entertaining storyline that highlights Alaska Native culture and values while still providing rich opportunities for social-emotional and informational text learning. This research provides further evidence that educational content can be embedded into animated or fictional storylines that are appealing to young children and still result in learning.
These research findings are important, as the conversation about children’s consumption of media results in worries about children having too much screen time (Domingues-Montanari, 2017; Oswald et al., 2020). Not all screen time is the same—media differ in terms of content and in how they are consumed (Sweetser et al., 2012). While limits on screen time are important (Council on Communications and Media, 2016), these findings suggest that stringent limitations on children’s time engaging with educational media might be counterproductive to their learning outcomes. Further, adult mediation of children’s media use also can be important to extend learning and to bring media examples into the real world—for example, by helping children make the connection between Molly’s vlog and an informational video a child creates to share with friends and family.
Finally, our findings suggest that further research is needed regarding how best to use educational media to support children’s learning. For example, parents in our pilot study noted how much their children enjoyed the live-action interstitials, but the role that live action plays in reinforcing concepts is unclear. It is possible that demonstrating the application of concepts in real life makes it easier for young children to make that leap themselves. The target age range for optimal reception of educational media is also unknown; as children’s media consumption in general increases with age, it is possible that diluting educational media with other forms of entertainment-oriented media correspondingly dilutes impacts. This explanation may be why we found that older children benefitted less from the intervention. More research is also needed on the structure of learning interventions to disentangle the relative benefits of consistent, measured release of resources as compared to an all-at-once approach. While the findings here showed more benefit from the videos that were released gradually in weekly bundles than from the games that were available from the start of the study, we cannot conclude that the release mechanism was responsible for the difference. A rigorous study of learning outcomes with the same intervention materials released in different ways is warranted to ensure that interventions are using the most efficacious ways to reach children.
Supplemental Material
sj-pdf-1-aer-10.3102_00028312221113326 – Supplemental material for The Efficacy of Digital Media Resources in Improving Children’s Ability to Use Informational Text: An Evaluation of Molly of Denali From PBS KIDS
Supplemental material, sj-pdf-1-aer-10.3102_00028312221113326 for The Efficacy of Digital Media Resources in Improving Children’s Ability to Use Informational Text: An Evaluation of Molly of Denali From PBS KIDS by Joy Lorenzo Kennedy, Claire G. Christensen, Tiffany Salone Maxon, Sarah Nixon Gerard, Elisa B. Garcia, Janna F. Kook, Naomi Hupert, Phil Vahey and Shelley Pasnik in American Educational Research Journal
Footnotes
Notes
J
C
T
S
E
J
N
P
S
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
