Abstract
Preschool-age children receive little formal instruction regarding science and engineering concepts. Digital media interventions have been effective in supporting young children’s literacy and mathematics skills, but there is little evidence of their efficacy in building science and engineering skills. This article reports on an experimental study of the impact of access to science- and engineering-focused games and videos on the knowledge and skills of 4- and 5-year-old children. The intervention led to statistically significant positive impacts on children’s understanding of structure stability (d = 0.40, p < .001) and the relationship between material properties, force, and movement (d = 0.38, p < .01). These results provide causal evidence that digital media can support young children’s science learning.
Keywords
Young children regularly engage in scientific inquiry by asking questions and making predictions about the world and how things work, investigating phenomena firsthand, and generating explanations about the physical and living worlds based on evidence from their own experiences, including revising their original predictions (Fusaro & Smith, 2018; Gerde et al., 2013; Gropen et al., 2011; National Research Council, 2012). These behaviors represent key steps in developing a beginning understanding of scientific core ideas and the ability to engage with the science and engineering practices at a developmentally appropriate and accessible level. Science-focused experiences and explorations also support critical thinking, language, mathematical reasoning, executive function skills, and persistence (Bustamante et al., 2017; Kuhn, 2011; Nayfeld et al., 2013; Peterson & French, 2008; Wright & Neuman, 2014). Children who have rich and well-facilitated science experiences in preschool and early elementary school are more likely to sustain their early interests in science, have positive attitudes toward science, and have broader interpretations of what science is and who scientists are into their adolescent years (Alexander et al., 2012; Patrick et al., 2009).
However, many young children do not experience rich science- and engineering-related experiences at home or in preschool settings. Four- and 5-year-old children in center-based preschool programs spend far less time on science than other domains, such as language, literacy, and mathematics (Greenfield et al., 2009; Piasta et al., 2014). Although preschool state standards in science are increasingly being updated to reflect the K–12 Next Generation Science Standards (NGSS; National Research Council, 2013) core ideas, practices, and science’s connection to engineering and technology, many preschool educators are not yet fluent with the updated science standards and minimally trained in planning, facilitating, and assessing children’s science and engineering explorations and learning (Greenfield et al., 2009). Although children’s play with blocks, balls, and other objects regularly intersects with physical science and engineering phenomena and concepts (Greenfield et al., 2017), teachers often do not make these connections (Vitiello et al., 2019). Children also receive relatively little support for physical science and engineering explorations and learning at home compared to other developmental domains and even compared to other domains in science (Gerde et al., 2021). Many parents report struggling to answer their children’s science questions and lack the confidence and resources to engage in everyday opportunities that could deepen their child’s learning and curiosity about how things work (Silander et al., 2018).
Digital media has the potential to deliver science content to young children at a broad scale (Kearney & Levine, 2019; National Academies of Sciences, Engineering, and Medicine, 2022). Although literature suggests that educational media can improve children’s literacy (Hurwitz, 2018), mathematics (Berkowitz et al., 2015), and socioemotional outcomes (Rasmussen et al., 2016), little is known about the extent to which media can support young children’s science learning (Hirsh-Pasek et al., 2015; Young et al., 2012). Features of digital media interventions, such as engaging narratives with relatable characters, may provide an entry point for children and parents who perceive science and engineering as complex, uninteresting, or school subjects suitable for older children rather than one that can be pursued with young children in families (Lauricella et al., 2011; Lu et al., 2016; Travis, 2017). Digital media can model scientific core ideas (e.g., forces and motion) to help young children and families envision what science and engineering practices (e.g., asking questions and defining problems, planning and carrying out investigations) look like in context (Flynn & Whiten, 2008). They can also enable children to explore scientific phenomena and simulations that might be difficult or unsafe to explore in the physical world (Rutten et al., 2012).
However, research indicates that transferring learning about science and engineering concepts and practices from videos and games to the physical world can be challenging, particularly for young children who have limited real-world exploration experiences on which to draw (Bonus & Mares, 2018; Troseth, 2010). Children need scaffolding from adults to transfer their learning from media to the physical world (Rasmussen et al., 2016; Strouse et al., 2013), and this support appears particularly important for science (Young et al., 2012). For example, young children may have difficulties understanding how two-dimensional media experiences transfer to three-dimensional real-world physical science experiences (Anderson & Pempek, 2005). The use of fantastical stories with anthropomorphized animals and objects may make it difficult for young children to distinguish between fact and fiction (Bonus, 2019; Bonus & Mares, 2018; Woolley & Ghossainy, 2013).
In this study, we examine whether science media designed based on existing knowledge about effective media design—developmentally appropriate and aligned to standards, connects to children’s prior knowledge, and invites active engagement and social interactions (Hirsh-Pasek et al., 2015)—can support children in connecting science learning to the real world. A few studies have examined the relationship between media and young children’s science outcomes, including research that has found positive impacts of television series on parent reports of children’s science talk (Penuel et al., 2010), a lack of impact of a science-focused television series on children’s gender- and race-based attitudes related to STEM occupations (Aladé et al., 2022), and positive effects of educational media on children’s science outcomes through curricula-based interventions (Bachrach et al., 2012). However, much of this research is exploratory in nature because of small sample sizes (e.g., Aladé et al., 2022; Bachrach et al., 2012), reliance on parent reports (e.g., Penuel et al., 2010), or lack of experimental design (Rockman et al Cooperative, 1996). The study adds to the prior work by examining young children’s science learning outcomes in an out-of-school context and through the use of an experimental design on a large and diverse sample of children.
Intervention
The Public Broadcasting Service (PBS), in partnership with the Corporation for Public Broadcasting, developed The Cat in the Hat Knows a Lot About That! (Cat in the Hat) media resources for young children and their parents and caregivers through the 2015–2020 Ready To Learn (RTL) Initiative, funded by the U.S. Department of Education. The RTL Initiative provides free educational television and digital media resources to children ages 2 to 8 years to promote early learning, particularly for children from low-income communities.
The Cat in the Hat Knows a Lot About That! is a PBS KIDS multiplatform media property based on The Cat in the Hat’s Learning Library book series by Random House and Dr. Seuss Enterprises. This study examined the resources developed for the third season of The Cat in the Hat Knows a Lot About That!, focused on physical science and engineering content, practices, and vocabulary. Physical science is an ideal domain for young children to explore because their play with blocks, balls, and other objects in their daily experiences brings them into contact with physical science phenomena. Physical science experiences invite children to act directly on objects and observe immediate results and make close and immediate connections to early engineering design content and can take place at any time of year and in any setting. The Cat in the Hat episodes incorporate science and engineering content, model science and engineering practices, and introduce related vocabulary through narrative stories and interactive experiences. Developers designed this content to align with the NGSS while also attending to developmentally appropriate practices for young children as described in the Head Start Early Learning Outcomes Framework (U.S. Department of Health and Human Services, 2015). Each episode focuses on one to two disciplinary core ideas in physical science, and the characters explore phenomena related to these core ideas through their engagement with the practices.
At the beginning of each Cat in the Hat episode, two children, Sally and Nick, pose a science question and/or identify an engineering problem that has emerged from their backyard play and exploration. The Cat in the Hat (the Cat) appears and facilitates an exploration with Nick and Sally, helping them answer the science question or solve the engineering problem while learning about the relevant science concepts. The focal core disciplinary ideas are (a) structure and properties of matter (matter can be described and classified by its observable properties, and different properties are suited to different purposes) and (b) forces and motion (pushes and pulls can have different strengths and direction, and pushing and pulling on an object can change the speed or direction of its motion and can start it or stop it). In the process, Nick and Sally engage with the science and engineering practices, asking questions and defining problems; developing and using models; planning and carrying out investigations; analyzing and interpreting data; constructing explanations and designing solutions; engaging in argument from evidence; obtaining, evaluating, and communicating information; and generating and sharing ideas about how the world works based on evidence from their explorations (PBS KIDS, n.d.).
In addition to the videos, the developers created five digital games and five hands-on activities that align with the learning goals. The digital games provide opportunities for children to investigate scientific phenomena related to the specific core idea(s) using simulations. Each hands-on activity is also designed to support understanding of a core idea and engages children with key science and engineering practices in the physical world.
In this study, we examine the impact of providing 8 weeks of access to Cat in the Hat resources on low-income 4- to 5-year-old children’s understanding of foundational science and engineering concepts. Because it is possible that the impact of the intervention differs across these concepts, we ask four related but distinct research questions:
Research Question 1: What is the impact of providing 8 weeks of access to Cat in the Hat resources on low-income 4- to 5-year-old children’s understanding of physical science concepts and science and engineering practices?
Research Question 2: What is the impact of providing 8 weeks of access to Cat in the Hat resources on low-income 4- to 5-year-old children’s understanding of the role of material properties and forces in structural stability?
Research Question 3: What is the impact of providing 8 weeks of access to Cat in the Hat resources on low-income 4- to 5-year-old children’s understanding of the role of material properties and forces on movement down an incline?
Research Question 4: What is the impact of providing 8 weeks of access to Cat in the Hat resources on low-income 4- to 5-year-old children’s understanding of how objects can be sorted based on their material properties and uses?
Participants
Study participants were recruited from five locations: Boston, Massachusetts; Minneapolis, Minnesota; New York, New York; Phoenix, Arizona; and San Francisco, California, via a proprietary email list and through Facebook advertisements. The Facebook advertisements briefly described the study, the research organizations involved, and participation incentives and invited respondents to take a short online survey to ensure that they met geographic, age, and income requirements. Recruitment staff then called qualified respondents to gather contact information, provide study details, and confirm that potential participants met the following criteria: have a child between 4 years and 3 months to 5 years and 6 months of age, child was fluent in English, parent indicated that child could participate in game-like activities for up to 35 minutes (the pretest duration), at least one parent was proficient in English or Spanish, and an annual household income at or below the U.S. Department of Housing and Urban Development threshold to receive a Section 8 housing voucher for a family of four. 1 Possible participants were informed that they would receive a new tablet computer and $125 for participating in the baseline and postassessment data collection activities.
A total of 454 four- and 5-year old children and their parents participated in the study (see Table 1). We did not observe any statistically significant (p < .05) differences between treatment- and control-assigned children or families on any demographic characteristics, and in no case did differences exceed the What Works Clearinghouse threshold (effect size = 0.25) for equivalent samples.
Child Science Knowledge and Demographic Characteristics, Overall and by Condition
Random Assignment
The study team randomly assigned children to receive the Cat in the Hat resources or an alternate treatment (tablet and a request to have children use any educational media). Random assignment and data analysis procedures were registered with the Registry of Efficacy and Effectiveness Studies (No. 1627.1v2) prior to conducting random assignment and updated prior to conducting analysis. Study assignment, data collection, and analysis procedures were consistent with those described in the preregistration documents.
Prior to Data Collection
The study team provisioned 500 sequentially numbered tablet computers in a random sequence before in-person meetings with eligible families. Half of the tablets were provisioned for treatment and half for control. Tablets were identical, and it was not possible to determine the condition of the tablet when the tablet was powered off. All tablets were provisioned with data-enabled internet access.
Baseline Data Collection
When eligible parents and children arrived at the data collection facility, a member of the study team informed parents of study procedures, confirmed study eligibility, and asked for consent for the child to participate. All children of parents who agreed to participate were asked to complete a computer-based assessment of children’s science knowledge. Parents provided information on child and family characteristics and their child’s typical digital media use. Nine children were unable or unwilling to complete the baseline assessment. In each case, the child’s parent was given a $25 gift card, but the child was not enrolled in the study.
Random Assignment
Researchers escorted children who completed the assessment and their parent to a separate section of the facility, where they provided the family with the next sequentially numbered tablet. Through distribution of the numbered tablets, parents and children were randomly assigned to either treatment or control. To avoid the possibility that study team members’ perceptions of families would influence assignment, study team members who conducted the consent and assessments were not aware of the condition of the tablets, and the team member distributing the tablet was prohibited from interacting with study participants prior to distribution. Orientation sessions for treatment- and control-assigned families took place in separate sections of the facility to minimize interactions between treatment- and control-assigned families and ensure that assessors remained blind to condition. Table 2 provides a summary of the resources by condition.
Resources Provided to Treatment and Control Group Participants
Treatment families’ tablets included a video player app to view the Cat in the Hat videos and a games app. The video player app contained 21 full-length (11-minute) videos and nine interstitials (90-second videos designed to bridge two full-length videos in a 30-minute television slot), for a total of 3 hours and 43 minutes of content. The games app contained five digital games and 20 game-aligned hands-on activities. In addition, families received three printed science and engineering activities from the Cat in the Hat website. The resources were organized into six content areas: bridges, slides and friction, sorting objects, sounds and soundwaves, building and engineering, and five senses and making observations. See Table 3 for details regarding the resources available by content area. Families in the treatment group received a calendar with one of three sequences of the Cat in the Hat content, including 6 weeks focused on one of the six content themes and 2 weeks of free choice. The order in which families were asked to engage with the Cat in the Hat resources was randomized by week so that children’s performance on assessments would not be influenced by how recently a child engaged with a specific set of videos, games, or activities. Although the delivery of media for this intervention is more structured than how children might typically encounter media on broadcast television or cable, it is similar to the experience on a streaming platform in which children are presented with episodes in a particular order but with the ability to select which episodes to watch.
Cat in the Hat Content Themes
Treatment-assigned families were asked to use the Cat in the Hat digital media content for approximately 1 hour per week for 8 weeks or in keeping with their typical limits for their children’s media use. Researchers asked parents to have only the participating child use the tablet. Parents received two weekly text messages: one reminder to use the content and one with a link to complete a weekly media log documenting the child’s engagement with the study materials over the prior week. The purpose of the media log was to gather descriptive information from parents in the treatment group regarding engagement with the media and aspects of the resources that resonated with them and their children.
Control-assigned participants were told that they were participating in a study of children’s use of digital educational media and asked to have their child engage with educational digital media content of their choice on the tablet for 1 hour per week for 8 weeks or in keeping with their typical limits for their children’s media use. The tablets included software that blocked access to publicly available versions of the Cat in the Hat digital science learning materials. The control group orientation packet included a parent guide with information about the study and tips for using the tablet. Control group participants received text messages reminding them about their scheduled second endline data collection meeting only.
Measures
The study team assessed children’s science and engineering knowledge and science practices using two types of assessments: a modified version of a standardized assessment, Lens on Science, that focused on physical science and engineering knowledge, and the Hands-on Preschool Assessments of Science and Engineering, which are researcher-developed performance-based assessments. Table 4 provides the mean and standard deviation for each measure; the following section provides a detailed description of each assessment.
Study Measures
Note. Lens = Lens on Science.
Science and Engineering Knowledge
At baseline, all children completed the Lens on Science (Lens) a computer-administered, adaptive, item response theory-based assessment for preschool-age children (Greenfield, 2015). The Lens on Science is aligned with the Framework for K–12 Science Education (National Research Council, 2012) and is intended to assess young children’s knowledge of core ideas in four science disciplinary areas (life science, earth/space science, physical science, and engineering and technology) and science practice skills. It was designed to detect growth in learning for preschool children from low-income households. Children completed the Lens assessment on a study-provided tablet computer, wearing headphones to hear the audio instructions. The assessment takes approximately 20 minutes to 25 minutes to administer, including a 7-minute to 8-minute screener that ensures a child understands and knows how to respond to its three different item types. During the assessment itself, children are given a computer-adaptive set of items matched to their abilities based on a pool of 498 items.
Understanding Physical Science Concepts and Science and Engineering Practices
The study team worked with the Lens on Science developers to create a modified version of Lens (hereafter referred to as “Lens-Modified”) used at posttest. Specifically, researchers reduced the number of assessment items to include only items in the physical science and engineering and technology content domains, excluding items from the life science and earth and space science content domains that were not a focus of this study. From the full pool of 498 Lens items, the study team reviewed the 114 physical science items and the 43 engineering and technology items and selected a subset of 60 items that (a) assessed physical science and engineering core ideas included in the intervention resources, (b) limited redundancy, and (c) provided a range of difficulty levels to avoid possible floor and ceiling effects. To ensure the assessment was not overaligned with the intervention, researchers selected items aligned to broad concepts of physical science and engineering rather than selecting items that assessed specific topics or experiences in the intervention, such as whether a rock will sink or float. The researchers did so to ensure that the assessment was not overly aligned to the intervention and to be able to measure children’s learning in the broader domain of physical science and engineering. The resulting Lens-Modified assesses children’s understanding of a broad set of physical science and engineering core ideas. The 60 selected items included 40 physical science items and 20 engineering and technology items. The assessment was administered using the same software as the Lens. Each child was administered the same 60 items in random order.
The research team developed a set of three brief performance-based tasks, the Hands-on Preschool Assessments of Physical Science and Engineering, administered only at posttest. The three performance-based tasks are designed to assess the scientific core ideas of structure and properties of matter and forces and motion. Specifically, they assess children’s beginning understanding of the role of material properties and forces in structural stability, the role of material properties and forces on movement down an incline, and how objects can be sorted based on their material properties and uses. The tasks also incorporate six science and engineering practices, including (a) developing and using models, (b) planning and carrying out investigations, (c) analyzing and interpreting data, (c) using mathematics and computational thinking, (e) constructing explanations and designing solutions, and (f) engaging in argument from evidence. These tasks tested children’s ability to transfer learning from a digital environment to hands-on activities. Although Cat in the Hat videos and games address several physical science and engineering concepts and practices, the performance assessments focus on the three core concepts and practices that received the most substantive focus in the videos and games and that were also aligned to the NGSS.
Understanding of the Role of Material Properties and Forces in Structural Stability
The length, strength, and stability task was designed to assess a child’s understanding of how the properties of objects (e.g., size and shape) and materials (e.g., hardness and flexibility) make them suitable for building a bridge. Assessors provided children with a group of objects of different lengths and strengths and asked them to choose one of the objects to build a bridge that would support the weight of a toy car and toy figure. The assessor asks the child to select which object to use to help the toy figure get across to see their friend and how they could try out that idea (i.e., test whether the object is long enough to span the gap between the blocks and strong enough to support the toy’s weight). The assessor invites the child to test whether their choice worked. Children have opportunities to use the science and engineering practices as they plan and carry out investigations, construct explanations, engage in argument from evidence, and obtain, evaluate, and communicate information. The child’s score is based on whether the child, within three attempts, selects the suitable object, tests each chosen object, and provides an explanation about why each tested object did or did not work related to relevant properties and forces. Assessors took notes during the assessment to complete final scoring shortly after administration but did not video record children’s assessment performances. Scores range from 0 to 8 points.
Understanding of the Role of Material Properties and Forces on Movement Down an Incline
The surfaces and friction task was designed to assess a child’s beginning understanding of how the properties of materials and forces—friction in particular—influence the motion of objects on a slide. Assessors provided children with three slides with the same incline but differently textured surfaces (rough, smooth, or sticky) and asked children to choose and then test the slide that would enable a toy figure to slide the fastest. Given that all variables other than texture are equal, the toy figure will slide the fastest down the felt, more slowly down the rubber, and not at all down the steel wool. First, the assessor asks the child to look at and touch the slides and then to describe the texture of each. Next, the assessor asks the child to describe how the slides are the same and different. With the slides set up side by side, the assessor then asks the child to (a) predict which slide Sam will slide down the fastest, (b) explain that prediction, and (c) state how the child might test on which slide Sam will slide the fastest. The child is then invited to test the prediction. Finally, the assessor asks the child whether and how their prediction differs from what happened and why they think so. Children have opportunities to use the science and engineering practices as they plan and carry out investigations, analyze and interpret data, construct explanations, and obtain, evaluate, and communicate information. Scores range from 0 to 14 points.
Understanding of How Objects Can Be Sorted Based on Their Material Properties and Uses
The colors, shapes, and uses task was designed to assess a child’s beginning understanding that different objects can be described and categorized based on their observable properties and/or their functions. Assessors provided children with sets of objects and asked them to sort them based on color, shape, and function (for eating, making art, or playing). Children identify similarities and differences among colors of objects, sort objects on the basis of shape (with picture cues), complete a sort based on use, and fix a sort based on color. Children have opportunities to use the science and engineering practices as they use mathematics and computational thinking, plan and carry out investigations, and obtain, evaluate, and communicate information. Scores range from 0 to 8 points.
To assess item characteristics and internal consistency, the study team inspected means and distributions of all items, Cohen’s alpha, corrected item-total correlations, and confirmatory factor analysis (CFA) results when applicable. Based on these results, we did not include some items in final total scores. Analyses of the final items used in the length, strength, and stability scale suggest high interrater reliability (an average weighted kappa of .91) and internal consistency of Cronbach’s alpha of .486. Analyses of the original scale for this item using CFA indicated poor model fit, χ2(14, N = 287) = 69.4, p < .001, comparative fit index (CFI) = .62, root mean square error of approximation (RMSEA) = .12, and relatively low factor loadings of the three iterations of one item (less than .4). Based on this information and inconsistencies we identified in how assessors scored and recorded child responses for the item, we removed this item in the final score for this task. The final resulting scale, using a sum score, had a normal distribution (range = 0–8, M = 4.27, SD = 1.96) with no indication of skewness or kurtosis (0.11 and –1.05, respectively). Analyses of the final items used in the surfaces and friction scale suggest high interrater reliability (average weighted κ = .86) and internal consistency (Cronbach’s α = 0.68, and all corrected item-total correlations were above 0.2). Results from CFA suggest reasonably good model fit, χ2(29, N = 287) = 95.1, p < .001, CFI = .85, RMSEA = .09, and all factor loadings above .4. The final resulting scale, using a sum score, had a normal distribution (range = 0–14, M = 6.06, SD = 3.39), with no indication of skewness or kurtosis (0.03 and –0.74, respectively). Similarly, interrater reliability for the colors, shapes, and uses scored items was high (weighted κ = .94), and internal consistency was reasonable (Cronbach’s α = .572, but all items had corrected item-total correlations above 0.2; in other words, no items stood out as not fitting with the scale). Results from CFA showed good model fit, χ2(19, N = 275) = 32.2, p = .04, CFI = .92, RMSEA = .05, and all items had good factor loadings (one at .375 and all other items above .4). The final resulting scale, using a sum score, had a normal distribution (range = 0–8, M = 4.36, SD = 1.36), with no indication of skewness or kurtosis (–0.34 and –0.10, respectively).
To limit the total assessment time to no more than 35 minutes, each child was randomly assigned two of the three hands-on assessments. Both the specific performance tasks that a child was administered and the order in which the tasks were administered to the child were determined randomly prior to the study start.
Parent Survey
Parents completed presurveys and postsurveys via the survey platform Qualtrics. All surveys were written at approximately a fifth-grade reading level and available in both English and Spanish. The survey was programmed so that a respondent could choose either English or Spanish as their survey language and could toggle between the two languages. For this reason, we do not have data on which language a respondent viewed. Spanish versions were back-translated and reviewed by several native Spanish speakers for accuracy and word choice but were not equated for semantic and content equivalence. Thirty-eight participating parents in the study spoke Spanish or Spanish and English. The parent presurvey included demographic questions regarding parent and child backgrounds. We selected demographic measures based on prior research, focusing on characteristics associated with educational outcomes, engagement in science, and media usage. The survey includes items on parent education, household income, child age, gender, race and ethnicity, home language, preschool attendance, individualized education program or 504 plan status, and whether the child had watched Cat In The Hat prior to the study. Parents completed the presurvey before assignment to condition.
Use of Intervention Materials
We measured the amount of time that children used intervention videos and games using three applications embedded on intervention tablets: one application, developed by PBS KIDS, tracked which Cat In the Hat games treatment-assigned children used and the length (number of minutes) for each engagement. Researchers used two apps developed by third parties to track video and Web use for children in both the intervention and comparison groups.
Analyses
Our analysis of confirmatory outcomes used the following regression models:
where Y represents the outcome of interest, β0 represents the intercept, β1TREAT is a dummy variable equal to 1 if the child was assigned to the treatment and 0 if assigned to control, β2Pretest represents children’s score on the Lens at baseline, β3covs represents a vector of analysis-specific child and family demographic characteristics, and ε represents the error term. To mitigate concerns about p-hacking, we selected covariates using an approach that was both data driven and grounded in prior research. At the study design stage, we identified a set of child and family characteristics for which there was prior theoretical or empirical evidence of association with either children’s skill development or inequitable science opportunities (National Center for Science and Engineering Statistics, 2023). To avoid overfitting the model, we predetermined and preregistered our decision to include only those characteristics that emerged as statistically significant predictors of any one of the dependent variables in bivariate analyses. As a result, all analyses include responding parent’s highest level of education, whether the child attends a center-based preschool program, whether the child attends kindergarten, whether the child has disability, the child’s race, and whether the child speaks only English at home. For each model, we calculate a Cohen’s d effect size.
We conducted descriptive analyses documenting the extent to which treatment-assigned children used the intervention materials and control-assigned children accessed intervention content. Furthermore, we performed correlational analyses to investigate the relationship between intervention material usage and children’s learning outcomes using regression analyses, examining the association between usage and the residual values derived from the regressions that estimated impacts on the primary outcome variables, as described previously. The residual values represent the disparity between observed and predicted outcome values for each child. For each primary outcome, we conducted separate regression analyses to investigate how residuals relate to the duration of video and game usage and the percentage of all intervention games or videos that each child accessed throughout the study.
Findings
Attrition from the study was low. Among the 229 children assigned to the treatment group, 223 (97.4%) completed one or more of the posttest assessments. Rates of attrition were similar among control-assigned children, with 220 of the 225 (97.8%) control-assigned children completing at least one posttest assessment. The overall attrition rate of 2.4% and differential attrition rate of 0.4% fall within the What Works Clearinghouse standards for acceptable rates of attrition for experimental studies.
Over the 8-week study period, children in the treatment group on average played the Cat in the Hat digital games for a total of 2 hours 48 minutes and watched the Cat in the Hat videos for 4 hours 20 minutes. Video and game usage differed substantially across children: Whereas approximately one-quarter (24%) of treatment-assigned children engaged with the videos or games for less than 2 hours across the 8 weeks, over half (59%) of children engaged with these materials for more than 20 hours. Video and game usage declined sharply after the first week of the study. During Week 1, children engaged with the study videos and games for an average of 5 hours 14 minutes. Average usage of the Cat in the Hat games and videos in the second week of the study dropped to just under 2 hours and then declined to less than half an hour per week by the eighth week.
Analyses of tablet usage data suggest that about half (49%) of the children accessed at least one of the hands-on activities embedded in The Cat in the Hat Builds That app. Children accessed the hands-on activities 1.4 times on average, for an average time of under 1 minute. Among children in the control group, six were able to circumvent the app-blocking software, and one of these children accessed more than 20 minutes of Cat in the Hat content during the study.
Analyses indicate that access to Cat in the Hat resources had medium to large positive impacts on children’s learning of specific science and engineering concepts and practices. Children assigned to receive the Cat in the Hat resources experienced larger improvements in their broader physical science and engineering knowledge than did control-assigned children, but these differences were not statistically significant.
Research Question 1
Treatment-assigned children earned higher scores on average than did control children on understanding of physical science concepts and science and engineering practices as measured by the Lens-Modified assessment, but these differences were not statistically significant (d = 0.11, p = .12; see Table 5). Although this impact is not statistically significant at conventional levels, the relatively low probability (i.e., 12% likelihood vs. the 5% convention) that this effect would be found if there were no effect in the population provides promising evidence of the Cat in the Hat’s impacts on children’s broader understanding of physical science and engineering knowledge and practices. Moreover, recent research on interpreting the magnitude of effect sizes in educational interventions (Kraft, 2020) suggests that an effect size of this magnitude is meaningful given both the easy scalability and low per-child cost of providing access to the Cat in the Hat resources and particularly compared to other typically resource-heavy science interventions.
Impact of Cat in the Hat on Children’s Understanding of Physical Science Concepts and Science and Engineering Practices
~p < .10. *p < .05. **p < .01. ***p < .001.
Research Question 2
Treatment-assigned children demonstrated statistically significant greater understanding of the role of material properties and forces in structural stability than did control-assigned children, as measured by the length, strength, and stability task (d = 0.40, p < .001; see Table 6).
Impact of Cat in the Hat on Children’s Understanding of Length, Strength, and Stability
p < .001.
Research Question 3
Treatment-assigned children demonstrated statistically significant greater understanding of the role of material properties and forces on movement down an incline than did control-assigned children, as measured by the surfaces and friction task (d = 0.38, p < .01; see Table 7).
Impact of Cat in the Hat on Children’s Understanding of Surfaces and Friction
p < .05. **p < .01. ***p < .001.
Research Question 4
Treatment-assigned children earned slightly higher scores than control-assigned children on their understanding of how objects can be sorted based on their material properties and uses, as measured by the colors, shapes, and uses task (ability to sort objects by size, color, shape, and use; d = 0.15, p = .18; see Table 8), but these differences were not statistically significant.
Impact of Cat in the Hat on Children’s Sorting Ability
p < .001.
Discussion
The results of this study indicate that providing young children and their families with access to science-based media resources can support their science- and engineering-related learning. Specifically, this study found that providing families with access to the Cat in the Hat videos and digital games improved children’s understanding of physical science concepts related to matter and forces: the role of material properties (strength and length) and forces in structural stability and the role of texture on friction and movement down an incline. Children who were provided access to the Cat in the Hat resources outperformed those who did not have access on measures of how the properties of objects (e.g., size and shape) and materials (e.g., hardness and flexibility) make them suitable for different purposes and how different forces (pushes and pulls) can cause objects to move and influence the stability of a bridge. Children with Cat in the Hat access also learned more about how the properties of objects, materials, and forces (including friction) influence the motion of objects.
We observed small positive but not statistically significant effects of the Cat in the Hat on children’s understanding of sorting, and their understanding that different objects can be described and categorized based on their observable properties or functions improved as a result of using the Cat in the Hat resources. Although not statistically significant, the small positive effects on the externally developed assessment of children’s physical science and engineering knowledge and practices also provided a promising indication of the Cat in the Hat’s impact on broader science and engineering learning.
The treatment-control differences also provided evidence of children’s ability to transfer learning from a digital environment to a real-world setting. This evidence is notable in the context of a substantial body of research that has documented the challenges young children face in transferring knowledge from videos because of difficulties in separating fantasy from reality (Woolley & Ghossainy, 2013), confusion about how videos and games relate to the real world (Mares & Sivakumar, 2014), and difficulties in transferring two-dimensional experiences to a three-dimensional world (Anderson & Pempek, 2005). The results of this study offer evidence that fantasy-based storytelling can support young children’s accurate scientific understanding, although it may be important that the narrative content focuses on factual science and engineering concepts and practices.
Limitations
Assignment and data collection procedures prevented assessors and families from knowing participant treatment condition, and attrition from the study was minimal. We nevertheless encourage readers to consider the following limitations when interpreting study findings.
Generalizability of Study Sample
The sample included families with a diversity of backgrounds, but the sample is not representative of the United States as a whole. Furthermore, participating families were volunteers recruited through social media posts about a digital media study. They and their children may be different from parents who do not use social media or are not interested in participating in a digital media study.
Generalizability of Study Experiences
First, the Cat in the Hat materials are available widely through the PBS KIDS website, digital apps, and broadcast TV, but study children accessed these materials differently from how they would be accessed by children outside of the study. All digital materials were preloaded onto tablet computers accessed via icons on the tablet home screen. This display most likely encouraged treatment-assigned children to access the materials more often than they would have otherwise.
Second, all participating children received a new tablet. This provision does not limit the internal validity of the study findings because both treatment- and control-assigned children were given the new tablets. It is nevertheless likely that the novelty of the new device encouraged treatment-assigned children to use the Cat in the Hat resources more than they would have without a new device.
Third, treatment-assigned parents were encouraged to support their children’s use of the Cat in the Hat materials through reminders about study material. These reminders likely led children to use the materials more often than they would have outside the study. However, even with the encouragement and access to technology, analyses of tablet data suggest that use of the materials varied in ways that likely mirror how children engage with media in natural environments, such as a drop-off in use after several weeks into the study. In addition, treatment-assigned parents received text messages to complete a media log, and control-assigned parents did not. Although both groups received weekly reminders to use the tablets, it is possible that requests for parents in the treatment group to complete weekly surveys documenting their use of study materials encouraged treatment parents to have their children use the intervention resources more than they would have in absence of this request for documentation. It is also possible that the request to complete weekly surveys about their use of study resources led parents in the treatment group to use more educational media than the parents in the comparison group. However, there are few educational media resources that address the topic of physical science, and therefore, it seems unlikely that the survey would drive any differences in science learning outcomes.
Study Measures
We used a combination of existing measures and measures that we adapted or created for use in this study. The study team designed these measures to reflect children’s understanding of the knowledge and practices targeted by Cat in the Hat content. We do not have information on the validity and reliability of these measures in other contexts.
It is possible that the observed differences between treatment- and control-assigned children on the Hands-on Preschool Assessments of Physical Science and Engineering assessments may be partially attributable to similarities between the assessment tasks that children were asked to perform and the activities the treatment-assigned children observed in the Cat in the Hat videos. Although the assessment activities did not use the Cat in the Hat characters or explicitly reference Cat in the Hat content, the contexts for the tasks did have elements in common with the intervention materials. A child’s experience viewing fictional characters engage in a task in digital context may have contributed to the child’s ability to complete a similar task in a real-world physical context.
Dosage
The structure of our study did not permit us to examine whether the amount of time children spent watching intervention videos or playing intervention games was causally related to the impact of the intervention. We examined associations between dosage and child outcomes and did not identify a consistent relationship between the amount of time children used the Cat in the Hat games or watched the Cat in the Hat videos and their performance on posttest measures.
Implications
We see three key implications for this study. First, interacting with the Cat in the Hat resources measurably improved some aspects of children’s knowledge of science and science and engineering practices. Although the impact is evident for a subset of content knowledge and practices in physical science and engineering, the results provide consistent evidence that the intervention resources helped children learn science and engineering concepts and practices.
Second, the strong results on the friction and incline and the structure and stability performance-based measures indicate that children’s experiences manipulating materials in a digital context can transfer to mastery of those practices and knowledge in the physical world. This represents an important contribution to the research literature on how children learn from digital media. Given the scalability and low per-child cost of providing access to the Cat in the Hat resources, the positive results from this study also appear meaningful in the context of other typically resource-heavy science interventions.
Third, these findings point to the importance of careful design that aligns with key learning goals for young children. Season 3 of The Cat in the Hat Knows a Lot About That! was developed with an intensive focus on core science and engineering concepts and practices and in close collaboration with content experts, all of which likely contributed to the educational quality of the media. The positive results of this study demonstrate that learning-focused playful content can help young children learn.
The increasing ubiquity of digital media in young children’s lives makes it critical to understand how to use these tools to support a range of key skills. A growing body of research on media resources suggests that certain kinds of media experiences can help children learn literacy, math, and socioemotional skills (Hirsh-Pasek et al., 2015; Hurwitz, 2018; Kirkorian et al., 2008). This study adds new knowledge about the promise of media to support science and engineering learning. Moreover, despite the evidence about the difficulty children face in transferring knowledge to new contexts, this study indicates that engaging with two-dimensional media can help young children develop knowledge and skills that they can apply in the physical environment.
The results of the study also leave several questions unanswered. The study focused on a relatively narrow set of physical science and engineering concepts and practices. Future research should examine whether media hold promise for other concepts in science that are more abstract, such as gravity or cause and effect in ecosystems. Study results also provide some initial promising evidence that educational media have the potential for small effects on children’s general science knowledge. Because of the broad reach of children’s public media, even small effects on science learning warrant future research, and these findings suggest the need for more exploration of whether media can support far transfer of science skills and knowledge. The study does not provide guidance regarding how much video viewing or game playing is optimal for children’s science learning. Future studies might examine this question experimentally by randomly assigning children to different levels of video and game usage. In addition, an important question related to children’s ability to transfer learning from a digital environment to the physical world is the extent to which parents facilitated this transfer through, for example, real-world activities and talking about the Cat in the Hat content.
Supplemental Material
sj-pdf-1-edr-10.3102_0013189X251327187 – Supplemental material for Learning Science and Engineering From Videos and Games: A Randomized Trial of PBS KIDS The Cat in the Hat Knows a Lot About That!
Supplemental material, sj-pdf-1-edr-10.3102_0013189X251327187 for Learning Science and Engineering From Videos and Games: A Randomized Trial of PBS KIDS The Cat in the Hat Knows a Lot About That! by Megan Silander, Todd Grindal, Sarah Nixon Gerard and Tiffany Salone in Educational Researcher
Footnotes
Notes
Authors
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
