Civic Online Reasoning Across the Curriculum: Developing and Testing the Efficacy of Digital Literacy Lessons

Abstract

Given the current threat posed by toxic digital content, preparing students to evaluate online sources cannot be relegated to a single subject area—this instruction should happen across the curriculum. This article focuses on materials designed to teach students to evaluate online information across subject areas. ninth-grade biology and world geography teachers taught a series of curriculum-embedded lessons based on the following design principles: (a) Focus on a core question and strategy; (b) engage students in evaluating real online content; (c) feature cognitive apprenticeship and formative assessment; and (d) support teacher learning. We examine whether these lessons helped students become more skilled evaluators of online content. Pretest/posttest data (N =574) showed statistically significant growth in students’ ability to evaluate the credibility of online content. We analyze the role played by the curriculum design principles in this interdisciplinary intervention and explore implications for future initiatives.

Young people learn about the world online, and the Internet provides access to nearly limitless sources of information. Yet extensive research suggests that young people need support to effectively evaluate digital information (e.g., Barzilai & Zohar, 2012; Breakstone, Smith, Wineburg, et al., 2021; Breakstone et al., 2022; Coiro et al., 2015; McGrew et al., 2018). As recent events have illustrated, the inability to sort fact from fiction online can have dire consequences. From the January 6 insurrection to misinformation about COVID-19, digital illiteracy can have deleterious effects on the health of individuals and democratic society.

Secondary school classrooms are a promising place to support students to find credible information about issues that affect them and their communities. However, teaching these skills in school may be easier said than done. Many adults, including teachers, need help learning to effectively evaluate online information themselves (Fogg et al., 2003; Hargittai & Dobransky, 2017; McGrew, 2021). Further, teachers need resources to teach these evaluation strategies. In this study, we test materials designed to help teachers integrate online evaluations into biology and geography classrooms. The curriculum materials follow a set of design principles tailored to the needs of teachers of different content areas who are implementing these resources in their classrooms for the first time.

Digital Literacy

As concerns about toxic digital content have increased in recent years, curricular resources to teach web credibility have proliferated. Many are freely available online; they feature such names as “ABCs of Website Evaluation” (e.g., Adelphi Libraries, 2019; Ontell, 2009) and “CRAAP Test” (e.g., Meriam Library, 2010). However, these curricular approaches are not derived from empirical research about what experts do when evaluating online sources. These guides primarily provide checklists of questions to ask, most of which focus on surface-level features of sources. For example, in the checklist in Figure 1, students are asked whether a contact person is provided for the article, whether sources of information are identified, and whether the spelling and grammar are error-free. Questions like these treat evaluation as a straightforward process of checking for specific features and adding up these credibility cues (Breakstone et al., 2018; Meola, 2004). Even if students did have time to exhaustively complete a checklist like this for each digital source they consulted, the results could be misleading. Listing an author, providing a reference list, and using correct spelling does not mean that a source is credible. A source’s creators control all these features, and it is thus easy to assemble the elements necessary to masquerade as a reliable source based on a checklist’s standards. Students are unlikely to discover the truth about a source by considering aspects determined by its creators.

Figure 1.

“Credible or Incredible?” worksheet from Media Education Lab lesson “Who Do You Trust?”

Thousands of websites feature checklist-based tools for evaluating Internet sources. Not surprisingly, students’ reasoning about online information often mirrors these checklists’ guidance. Students focus on easy-to-manipulate features of sites. They judge sites based on their appearance, including visual appeal, organization, and the presence of advertisements (Barzilai & Zohar, 2012; Kohnen et al., 2020). They give undue weight to websites’ top-level domain, often believing that a .org or .edu site is preferable to a .com (Breakstone, Smith, Wineburg, et al., 2021; Breakstone et al., 2022; Kohnen et al., 2020). While students focus on elements of websites that have little relation to credibility, they often do not ask more relevant questions regarding whether information is credible: Who is behind the information? Is the author trustworthy and an expert on the topic at hand? (e.g., Barzilai & Zohar, 2012; Breakstone, Smith, Wineburg, et al., 2021; Coiro et al., 2015; McGrew et al., 2018).

Teaching Skilled Approaches to Digital Evaluations

This study investigated a curriculum designed to teach students to investigate digital content based on research with skilled evaluators. To identify evaluative approaches that are effective on the open Internet, we investigated how professional fact-checkers, university-based historians, and college students searched for and evaluated online information on contentious topics (Wineburg & McGrew, 2019). Asked to evaluate an unfamiliar website, students and historians mostly read vertically, scrolling up and down on the original site. In contrast, fact-checkers did not dive into reading a site’s contents or assessing its appearance; instead, they prioritized investigating the source and did so by reading laterally. They quickly left the page and opened new browser tabs to investigate the source by reading what other reliable sources said about it (Wineburg & McGrew, 2019). Investigating web sources through lateral reading helped fact-checkers avoid being taken in by the narrative a site presented about itself via its contents, appearance, or About page. Instead, they used one of the web’s strengths—its abundance of information—to learn more about the source.

We used fact-checkers’ strategies as the basis for a curricular approach to teaching students how to be better consumers of digital information. We called this approach civic online reasoning. We featured the term civic to differentiate from broader media literacy efforts and to emphasize the important role that finding credible information plays in democratic decision-making. Civic online reasoning does not seek to prepare students to become professional fact-checkers. Instead, it teaches students some of the evaluative approaches that allowed fact-checkers to find better information more efficiently. Civic online reasoning features three questions that were at the heart of fact-checkers’ approach. First, students should ask, “Who is behind this information?” They should read laterally to probe the author or organization presenting the information and consider their qualifications, motivations, and perspective on the issue at hand. When students locate sources they decide may be trustworthy, they should ask, “What is the evidence?” and examine whether sufficient evidence is provided from reliable sources to support the claims made. Finally, students should routinely ask, “What do other sources say?” If they are uncertain about a source or claim, they should turn to the wider resources of the Internet and seek additional, reliable sources. As they search, students should exercise click restraint, or slow down on search results, to make a wiser choice about where to begin their research (McGrew, 2022; Wineburg & McGrew, 2019).

Studies with middle school, high school, and college students have indicated that students can become more skilled evaluators of digital content through explicit civic online reasoning instruction. A series of studies (Addy, 2020; Breakstone, Smith, Connors, et al., 2021; Brodsky, Brooks, Scimeca, Galati, et al., 2021; Brodsky, Brooks, Scimeca, Todorova, et al., 2021; McGrew et al., 2019) have shown that college students improve in their approaches to evaluating online information, including reading laterally more often, after completing modules focused on these skills. McGrew (2020) and Wineburg and colleagues (2022) demonstrated that civic online reasoning lessons taught in high school classes helped students conduct more effective evaluations. Similarly, Kohnen et al. (2020) taught a lesson that featured lateral reading with a group of middle school students and reported that students were more likely to attempt to read laterally after completing it. Taken together, these studies point to the potential of civic online reasoning for teaching students to evaluate unfamiliar Web sources.

The findings from these studies also point to the need for continued research. Although the length of the interventions varied, from a single 90-minute lesson (Kohnen et al., 2020) to a series of eight lessons (McGrew, 2020), student performance left substantial room for improvement. For example, Brodsky, Brooks, Scimeca, Galati, et al. (2021) found that just over half of participants successfully read laterally on one out of four posttest items after they have completed three instructional modules. Wineburg et al. (2022) reported that after high school students completed six civic online reasoning lessons, their scores improved from an average of 2.9 points on pretest to 5.1 points on a posttest; however, the score was out of a possible 14 points. These results speak to the scale of the issue at hand.

Strategies like lateral reading rely on procedural and substantive knowledge, as well as underlying literacy skills, which students should develop as they learn to read laterally. Procedurally, students need to know that lateral reading requires them to ignore clues contained within an unknown source, leave the site, open new tabs, and search for additional information about the source (Wineburg & McGrew, 2019). Substantively, lateral reading is only successful if students can identify reliable sources to use as they read laterally. If they rely on questionable sites to learn about a source, they may draw erroneous or unwarranted conclusions. Finally, lateral reading relies on students’ reading comprehension: To search for, read, and interpret clues about a site’s reliability, students need to be able to read and process information accurately (e.g., Brodsky, Brooks, Scimeca, Galati, et al., 2021; Leu et al., 2018). For example, Brodsky, Brooks, Scimeca, Galati, et al. (2021) reported gains in students’ lateral reading after an asynchronous intervention to teach that approach. They administered a reading comprehension measure at pretest and found that students with higher reading comprehension scores were more likely to read laterally at posttest.

The scope of existing interventions likely also plays a role in student performance. Each of the existing interventions to teach civic online reasoning took place in a single subject area in part of a single school year. The duration of these interventions pales in comparison to the amount of time young people spend online—an average of more than 7 hours a day (Rideout & Robb, 2019). Students likely need more frequent and varied experiences to develop the skills required to effectively evaluate digital content.

Cross-curricular integration of civic online reasoning materials would provide students opportunities to learn online evaluation strategies across the school curriculum. It would increase students’ exposure to effective evaluative approaches, give them opportunities to practice with varied content, and provide them with more teacher feedback on their progress. Teachers would also be able to collaboratively learn with colleagues about how best to teach civic online reasoning. However, lessons that can be flexibly integrated across subject areas are in short supply. In this study, we asked, Do curriculum-embedded lessons in civic online reasoning based on four design principles help students become more skilled evaluators of online content?

Methods and Materials

Setting and Participants

The study took place at a suburban comprehensive high school in the midwestern United States. The school enrolls more than 3,000 students. A group of ninth-grade educators from the school attended a free, daylong professional development workshop on civic online reasoning led by the authors as part of a statewide initiative to support civics instruction. After the workshop, several of the educators approached the authors about the possibility of integrating civic online reasoning curriculum materials into their ninth-grade curriculum. Across an entire school year, the authors worked with this leadership team to develop civic online reasoning materials for use in ninth-grade biology and world geography classes (see Figure 2). The leadership team included a librarian, the social studies department chair, a biology teacher, and two geography teachers. All of these educators attended the daylong professional development workshop except for one of the geography teachers. The lessons were implemented in general and honors biology classes as well as in general geography classes. The school offers Advanced Placement geography classes to ninth graders, but no civic online reasoning lessons were taught in those classes.

Figure 2.

Timeline of civic online reasoning lessons.

All ninth-grade biology teachers in the school (n = 6, including the biology teacher from the leadership team) integrated five hour-long civic online reasoning lessons into their general and honors biology courses. (The five biology teachers who were not part of the leadership team attended a 1-hour online professional development workshop with the authors. The biology teacher on the leadership team supported her colleagues on the use of the lessons throughout the school year.) There was less implementation of civic online reasoning lessons in geography classes than had been originally planned due to pandemic-related circumstances. Both geography teachers from the leadership team taught civic online reasoning lessons. One taught three lessons to their general geography students. Two lessons each took a full hour-long class period, and one, “China: Lateral vs. Vertical Reading,” took 15 minutes. The second geography teacher on the leadership team taught one hour-long lesson (“India: Lateral Reading Review”) and the 15-minute lesson to their general geography students. The other ninth-grade geography teachers (who were not part of the leadership team) did not feel ready to take on a new curricular initiative during the pandemic. This arrangement meant that some students took part in civic online reasoning lessons in biology and geography classes, while others did so only in their biology class.

The teachers (N = 8) had an average of 15.5 years of teaching experience (SD = 8.2; Mdn = 19.5; range: 4–26) and had been teaching at the school for an average of 13.9 years (SD = 9.7; Mdn = 19.5; range: 2–24). All were certified to teach in their subject area and held a bachelor’s degree in a related field. Seven of the eight teachers also held a master’s degree in education.

Design Principles for a Flexible Civic Online Reasoning Curriculum

As part of prior research projects, the research team developed civic online reasoning curricular materials (see cor.stanford.edu). In this intervention, we created new lessons for use in ninth-grade biology and world geography classes following the curriculum’s design principles: (a) Focus on a core question and strategy; (b) engage students in evaluating real online content; (c) feature cognitive apprenticeship and formative assessment; and (d) support teacher learning. In this section, we introduce the civic online reasoning curriculum design principles that shape this study’s curricular resources.

Focus on a Core Question and Strategy

Civic online reasoning resources focus on three questions to ask about online information: (a) Who is behind this information?; (b) What is the evidence?; and (c) What do other sources say? Unlike checklists that provide long lists of questions or evaluation criteria that students are unlikely to remember, focusing on a small set of questions and strategies makes them more manageable for students to learn.

Starting with just a few civic online reasoning questions and strategies facilitated initial interactions with the teachers in this study. In collaboration with the teachers, we narrowed our focus further: We selected one question and one strategy to teach and practice over the course of the year. Teachers reasoned that their incoming ninth graders had likely received little instruction on evaluating online information and wanted a unified theme for the cross-curricular collaboration. We chose to focus on the question “Who is behind this information?” and the corresponding strategy of lateral reading. Over the course of the year, lessons addressed different aspects of lateral reading, including contrasting it with vertical reading, practicing lateral reading on social media, and discussing different features of reliable sources (see Figure 2 for an overview of lessons).

Engage Students in Evaluating Real Online Content

The civic online reasoning lessons prioritize engaging students in evaluating real online content on social and political issues. Because the goal of civic online reasoning is to prepare students to make informed decisions, it is important that students gain experience evaluating authentic content. However, existing research on digital evaluations often does not ask students to investigate sources on the open web. Instead, researchers have shown students printouts of websites (Walraven et al., 2013), adapted paper sources to make them appear web-like (Macedo-Rouet et al., 2019), and prompted students to “search for information” on a hard-copy version of a website (Pérez et al., 2018, p. 57). Although methodologically rigorous, these studies do not ask students to engage with the messy reality of the open Internet. As a result, students miss out on opportunities to practice evaluating real digital sources related to contentious topics of social or political importance—precisely the kinds of sources students need to vet in civic life. Civic online reasoning lessons feature authentic, live web content, including websites, social media posts, and search engine results pages. The lessons provide students with links to these varied sources and ask students to search on the open Internet to evaluate them. Further, the lessons purposefully expose students to a range of sources, from those that are very trustworthy to those that lack credibility, so that students gain experience in evaluating content of varying quality.

Civic online reasoning lessons are intended to be modular. Each lesson contains specific digital sources (see online Appendix A for a complete lesson plan), but those sources can be replaced by teachers with other examples more relevant to course content while still following the lesson plan. For example, a lateral reading lesson could be adapted for a unit on Russia in a world geography class by including an Instagram post featuring a speech by Vladimir Putin that requires students to read laterally about the organization behind the Instagram account. In this study, we relied on this design feature and worked with teachers to select content that fit course topics. The modularity of the lessons meant that we could still use many portions of the lessons, including lesson overviews, short lectures, and discussions, while exchanging the online examples students evaluated. Importantly, we carefully adhered to the tenets of selecting actual online content that varied in type (e.g., websites and social media posts) and ranged in credibility from trustworthy to questionable.

Feature Cognitive Apprenticeship and Formative Assessment

The civic online reasoning lesson plans include aspects of cognitive apprenticeship (cf. Collins et al., 1989; Dennen, 2004). Lessons that introduce evaluation strategies (e.g., lateral reading) incorporate cognitive modeling, which allows teachers to make expert cognitive processes visible to novice learners (Collins et al., 1991; De La Paz et al., 2016). The lessons then incorporate opportunities for guided practice with scaffolding that gradually fades as students develop expertise. Lessons often end with discussions of what students learned as they practiced lateral reading. Drawing on sociocultural learning theory, our lessons center student discourse to support students in sharing, practicing, and internalizing sophisticated approaches to evaluation (Vygotsky, 1978).

For a multi-subject area collaboration, tracking student learning was critical. To make decisions about where to go next in terms of teaching lateral reading, we relied on formative assessments embedded in the lessons. Each short task asks students to write a brief evaluation of a digital source (see online Appendix B for a sample task). These assessments provide teachers with windows into student thinking. Early in the year, they helped us tailor lessons to students’ strengths and needs (Black & Wiliam, 1998; Popham, 2006; Shepard, 2008). As the year progressed, analyzing and sharing student performance on a formative assessment completed in one subject area helped other teachers make curricular adjustments before they taught a civic online reasoning lesson in their subject area. Based on what we learned from the formative assessments, we worked together to teach gradually more complex aspects of lateral reading through cognitive apprenticeship. Initial lessons modeled the process of lateral reading. Subsequent lessons focused on practicing more complex components of lateral reading, including elements of reliability to consider and specific online resources to use (see online Appendix C for descriptions of each lesson).

Support Teacher Learning

We designed the lesson plans to be educational for teachers because we recognized that they may need support to learn civic online reasoning themselves (Davis & Krajcik, 2005; McGrew, 2021). The beginning of each lesson plan contains a short description of the lesson’s learning objectives and explains their importance. Each component of the lesson plan includes detailed guidance for teachers, including suggested answers to guiding questions and questions for class discussions (see online Appendix A for a complete lesson plan). At the same time, the lessons are designed to be classroom-ready, with materials for teachers (e.g., modeling scripts, discussion questions) and students (e.g., links to online content and graphic organizers). Teachers received guidance in how to use these resources during the professional development they attended. For example, as part of the workshop, a member of the research team demonstrated how to model lateral reading during a sample lesson that teachers participated in and then reflected on.

We only worked directly with one of the biology teachers who taught the civic online reasoning lessons. The other five biology teachers did not attend the initial professional development session or regularly meet with the research team. To support these teachers to teach the civic online reasoning lessons, we relied on the educative nature of the lessons—as well as support from their colleagues.

Curriculum Development Process

Over the course of the year in which this study took place, the research team met with teachers on the leadership team (a librarian, the social studies department chair, one biology teacher, and two geography teachers) to develop civic online reasoning curriculum materials tailored to their biology and geography courses. The group first met in the summer to discuss the focus for the year and identify places in the curriculum to integrate civic online reasoning. Subsequently, during the school year, the leadership team met with researchers approximately twice a month over Zoom to discuss previous lessons that they had implemented and to review materials for upcoming lessons.

For each curricular topic identified by teachers as a site for civic online reasoning instruction, the research team drafted a lesson plan that incorporated online sources related to that topic. The teachers would then add instructional activities to finalize the lesson. For example, teachers often added activities from Peardeck, an interactive instructional app, to facilitate student participation throughout the lesson.

The first biology lesson to introduce lateral reading illustrates how civic online reasoning was integrated into the curriculum. The biology curriculum begins with an introductory unit focused on the scientific process. During this unit, students complete a variety of activities related to caffeine. To introduce students to the skill of lateral reading, the research team located an article from the website foodinsight.org for students to investigate (see Figure 3). The article details the benefits of caffeine consumption and links to various scientific studies. Students were asked to indicate on Peardeck how reliable the site is as a source to learn about caffeine. Students then watched a screencast of a member of the research team explaining how they would use lateral reading to identify who was behind the website. At first glance, the site seems credible. The author of the article has a PhD. The article links to scientific studies. Foodinsight.org’s About page indicates that it is a project of the International Food Information Council, which is a 501(c)(3) nonprofit organization committed to communicating “science-based information about health, nutrition, food safety and agriculture.” The screencast demonstrated how lateral reading about the International Food Information Council reveals that the organization works on behalf of food and beverage corporations, which have vested interests in portraying caffeine in a positive light. The teacher then reviewed lateral reading with students before asking them to practice the skill with another article about caffeine from the website theodysseyonline.com, a crowd-sourced website without editorial oversight that has been criticized for specializing in “clickbait” (Porter, 2017). After students completed their lateral reading in small groups, they shared the sources they used to learn more about theodysseyonline.com and the teacher offered feedback.

Figure 3.

Foodinsight.org article about caffeine.

The initial geography lesson provided students with another opportunity to practice lateral reading. Like the biology lesson about caffeine, the teacher presented students with an unfamiliar website and asked them to evaluate whether it is a trustworthy source of information about sanitation. In this case, it was the website for World Toilet Day, a project of United Nations Water (see Figure 4). This website was chosen deliberately to give students the chance to analyze a trustworthy source and avoid the possibility that students would begin to doubt the credibility of all online sources. After students completed their evaluations, the teacher reviewed the skill of lateral reading. Students then shared how they evaluated the website and discussed how they might have been misled if they had just evaluated the site based on surface-level features (e.g., unusual .info URL, website tagline of “Leaving No One Behind”). If students struggled with the lateral reading, teachers had the option to show students a screencast of a member of the research team reading laterally about the World Toilet Day website. Next, students practiced reading laterally about two infographics. One came from The Times of India, a large, well-established news organization in India with various practices that help ensure the quality of the news it produces (e.g., professional journalists, editors), and the other was an unsourced blog post. The whole class then discussed their evaluations of those sources based on what they learned from their lateral reading.

Figure 4.

World Toilet Day website.

Pretest and Posttest

To gauge student learning, students completed a pretest at the beginning of the school year and a posttest at the end of the school year. Students completed the assessments by using the survey platform Qualtrics. There were two versions of the assessment: Form A and Form B. At pretest, Qualtrics randomly assigned students one of the two forms. At posttest, students completed the other form. Students needed to complete the pretest and the posttest to be included in the data set. The assessment included six items: three constructed-response questions and three multiple-choice questions. Each item asked students to evaluate real online sources. Although the curriculum featured examples connected to biology and geography, the pretest and the posttest included sources about social and political issues (e.g., climate change, gun control, student debt, foreign policy). This design provided evidence of whether students could evaluate the trustworthiness of Internet sources on a range of content. Moreover, misinformation proliferates online about these kinds of topics. To engage in civic life, students need the ability to evaluate these types of sources.

The questions were the same on both forms of the assessment but featured different online sources. For example, on Form A, students were shown the website friendsofscience.org and asked whether it is a trustworthy source of information about global warming. The organization describes itself as “a non-profit organization run by dedicated volunteers comprised mainly of active and retired earth and atmospheric scientists, engineers, and other professionals.” However, a search online reveals a variety of sources indicating that the group denies humans’ role in global warming and that the group receives funding from oil companies. On Form B, students were shown the website of another organization (co2science.org) funded by the fossil-fuel industry that rejects the scientific consensus on global warming and were asked whether it is a trustworthy source of information about global warming. In both cases, students needed to read laterally to correctly answer the question.

Scoring

Members of the research team scored the constructed-response items by using rubrics developed as part of prior research projects (McGrew et al., 2018). The rubrics contained three levels: Beginning—0; Emerging—1; and Mastery—2. In Mastery responses, students demonstrated clear proficiency in the targeted reasoning processes. Emerging responses were either partially correct but incomplete or included elements of proficient as well as problematic reasoning. Beginning responses revealed incorrect or irrelevant reasoning about online sources. Students’ writing quality did not factor into the scoring rubrics; it was possible for students to earn a Mastery score with an incomplete sentence, as long as it demonstrated an understanding of the concept the task sought to measure. Student responses for the pretest and the posttest were combined by form (e.g., all responses for Form A were grouped together, regardless of whether they were from the pretest or the posttest). We used a random number generator to assign all responses a unique number and then custom-sorted by that number so scorers were blind to the order of administration (pre/post). One member of the research team scored all the constructed-response items. A randomly generated subset of 20% of student responses was scored independently by a second rater for each form, and weighted kappa was used to estimate inter-rater reliability for constructed-response items (Form A κ = .977 [95% CI, .961 to .993], p < .001; Form B κ = .971 [95% CI, .953 to .990], p < .001). In total, the assessment had 9 possible points: 2 points for each of the three constructed-response questions and 1 point for each of the three multiple-choice questions.

Results

We used a repeated-measures analysis of variance (ANOVA) to investigate whether there was a significant difference in the pretest and the posttest scores. Students averaged 2.25 points out of 9 possible points on the pretest. On the posttest, they averaged 3.75 points. First, we tested whether there was a significant treatment effect for all students (N = 574) who completed both the pretest and the posttest. Results indicated that students performed significantly better at posttest (M_pre = 2.24, M_post = 3.75; F (1,572) = 299.91, p < .001). Next, we explored whether results differed depending on the order in which students took the two forms. The interaction between order and time effect was nonsignificant (F (1,572) = 4.50, p = 0.152). This finding supports an inference that a similar amount of learning occurred regardless of the order in which the forms were administered, so order effects were ignored (Figure 5).

Figure 5.

Pre- and posttest scores by test for order.

Civic online reasoning lessons were completed in general biology (n = 304) and honors biology (n = 270) classes. We explored whether there was a difference in the effect of the intervention for students across class type. Tests of equality of covariances and error variances suggested a violation of the assumptions of a parametric repeated-measures ANOVA analysis, so we used a non-parametric ANOVA-type statistic (ATS) to test the null hypothesis that students in the honors biology group had the same pattern of pre- and posttest scores as students in the general biology group (see Erceg-Hurn & Mirosevich [2008] for a concise overview of ATS statistics and their use in non-parametric analyses). A significant ATS for the interaction between honors biology versus general biology and time (pretest versus posttest) indicated a rejection of the null hypothesis and evidence of a difference in how the effect occurred in the different class types.

The results show significant main effects of time (ATS _time = 301.88, p < .001) and class type (ATS _class = 149.44, p < .001), which indicate that students’ scores were significantly different from pretest to posttest and that there were significant differences in scores across class type (general vs. honors). The results also indicate a significant interaction between class type and time (ATS _{class x time} = 13.14, p < .001), which supports a rejection of the null hypothesis that students’ scores exhibited the same patterns from pretest to posttest across class type (general versus honors). Figure 6 includes score box plots for both class types at pretest and posttest. It also charts the relative treatment effects for students in both class types. Figure 6 indicates that students in both class types saw a significant increase in scores after the intervention, but the increase was more substantial for students in honors biology.

Figure 6.

Pre- and posttest score box plots and relative treatment effects for students in honors and general biology.

Finally, we tested whether having civic online reasoning lessons in biology and geography resulted in greater improvement than did having lessons in only biology class. Of the 574 students in the sample, 470 completed lessons in biology class only, and 104 received additional lessons in their general geography class.

We conducted a repeated-measures ANOVA to test whether there was an added effect for students who completed additional civic online reasoning lessons in geography. Results reveal a significant difference in students’ scores from pretest to posttest (M_pre = 2.24, M_post = 3.75; F (1,572) = 169.221, p < .001) but also indicated that the interaction between lessons in geography class and time was nonsignificant (F (1,572) = .265, p = .61). These results suggest that students improved significantly overall and that a similar amount of learning occurred regardless of whether students completed additional lessons in geography class. Figure 7 shows average scores at pretest and posttest for students who received extra lessons in geography and for students who completed lessons only in biology. The similarity of improvement across groups is reflected in the parallel slopes for the group that completed lessons only in biology and the group that completed lessons in biology and in geography. The chart also shows that students in the biology-only group scored higher at both pre- and posttest than did students who completed lessons in biology and geography. This disparity in average scores may reflect a difference in the proportion of honors-level students across the two groups. Only 30% of students who completed lessons in biology and in geography were enrolled in honors biology. In contrast, 51% of the students who received lessons only in biology were enrolled in the honors class.

Figure 7.

Pre- and posttest scores by whether students received extra lessons in geography class.

Limitations

Two major contextual factors shaped this study and the claims that result from it. First, the effects of the COVID-19 pandemic changed the course of this intervention. Instead of fully integrating civic online reasoning lessons into science and social studies, only the science teachers fully integrated the curricular materials. In social studies, the two teachers in the leadership team implemented lessons. They taught just a few lessons, one of which was quite short. As a result, the study does not provide evidence about the effects on student learning of a truly cross-curricular intervention. Studies that take place during more typical school years may shed more light on student learning when lessons are thoroughly integrated across different courses.

Second, the close partnership between teachers and researchers in this initiative was atypical. The teachers in the leadership group met with the curriculum designers on a regular basis and could reach out with questions at any time. Moreover, members of the research team, who designed the original civic online reasoning lessons, created curriculum materials tailored to the topics teachers were covering in their classes. Such close collaboration is atypical when new curricular initiatives are implemented. Future studies might examine how teams of teachers collaborate to teach civic online reasoning in their courses with less direct support from a research team.

Although this intervention has yielded statistically significant results, its design could be strengthened. First, we do not include measures of students’ reading comprehension or academic achievement. It is possible that students with higher reading levels are able to critically analyze online content more proficiently. Additionally, students’ literacy skills may have improved over the course of the school year and influenced their performance on the posttest. Finally, students who receive higher grades in school may have been more inclined to carefully complete the pre- and posttest. Future studies that include such covariates could help us understand how these factors interact as students learn to evaluate digital content. Second, collecting information regarding students’ knowledge of and opinions about the topics covered in assessment items could help us understand how these factors influence students’ evaluations. Third, future studies should consider the durability of students’ improvement in evaluating information and the transferability of their evaluation skills to new contexts—particularly their online behavior outside school. Finally, this study does not include a randomized controlled trial, which would have controlled for confounding variables and supported stronger inferences about the effects of the lessons on student learning. This study arose from an effort to help an entire school implement civic online reasoning instruction. Because developing strategies for school-wide implementation was the goal, random assignment for an experiment would have needed to be at the school level to account for data nesting. Such a multisite trial was beyond the scope of this study. However, future studies should implement a randomized controlled trial to further test the efficacy of the strategies we developed for the present study. A multisite trial would also strengthen the generalizability of the findings.

Discussion

Over the course of the school year, ninth-grade students who took part in a series of curriculum-embedded civic online reasoning lesson plans showed significant growth in their ability to evaluate online sources. This study joins other interventions in secondary schools (e.g., Kohnen et al., 2020; McGrew, 2020; Pavlounis et al., 2021; Wineburg et al., 2022) that suggest that explicit instruction in strategies like lateral reading helps students improve their evaluative approaches. However, this study is unique in its approach to designing and teaching the civic online reasoning lessons. The lessons were not divorced from the curriculum. Instead, teachers helped students evaluate online content directly related to course content, from nutrition to sanitation. This study also moved beyond a focus on teachers in a single subject area and considered how civic online reasoning lessons could be integrated across multiple content areas. Finally, the study focused on developing students’ lateral reading over the course of an academic year, which allowed lessons to gradually build in complexity as students developed fluency with the strategy.

Although students’ scores improved significantly from pretest to posttest, performance at posttest was far from ideal. On average, students still answered fewer than half of the posttest questions correctly. Students who completed lessons in both biology and geography did not perform better on the posttest than students who learned civic online reasoning in only biology. This result may be due to the limited number of civic online reasoning lessons in the geography classes. Alternatively, the intervention itself may need adjustments to make it more robust instead of simply adding more lessons. Further research that compares different approaches to cross-curricular interventions would help ensure the efficacy of future efforts at curriculum design and implementation.

This study provides evidence that curriculum-embedded civic online reasoning instruction can work, but broader integration is likely necessary to adequately prepare students for the demands of our present digital age. Integrating civic online reasoning instruction into core subject areas and across grade levels would help ensure that students have varied opportunities to hone their digital evaluation skills. Such an effort would require large-scale investments in curriculum and professional development. Teachers need curriculum materials aligned with the content they teach and professional development to help them effectively use the lessons. Such a development effort would require a robust research base. The civic online reasoning design principles that guide this study’s curriculum development point to areas for future research, which we explore below.

Design Principles in Action

Focus on One Civic Online Reasoning Strategy

The project’s focus on lateral reading affected both the teacher-researcher collaboration and teachers’ approaches to instruction. For the team of teachers and researchers, it allowed for focused collaboration over the course of the year by streamlining lesson selection and modification. It also allowed the team to discuss a lesson in one subject area and to consider implications for the next lesson on lateral reading—often in a different subject area. Focusing on lateral reading also affected teachers’ work with students. Teachers emphasized using common language (e.g., contrasting lateral reading with vertical reading) and tackled a similar set of student misconceptions about evaluating online information over the course of the year. A similar model could be used to integrate instruction in civic online reasoning across subjects and grade levels. For example, after a focus on “Who’s behind this information?” and lateral reading in ninth grade, tenth grade teachers could focus on the question of “What’s the evidence?” They could continue to reinforce the skill of lateral reading while also helping students learn to recognize authoritative sources of evidence in various disciplines. Eleventh- and twelfth-grade teachers could then teach “What do other sources say?” and engage students in more complex, open-ended research projects that build on the evaluation skills students learned in ninth and tenth grades.

The study’s findings also point to the need for continued research on how to sequence instruction focused on lateral reading, taught in multiple subject areas, over the course of a school year. Although students had additional exposure to lateral reading in geography, it did not make a significant difference in their performance. What could teachers do in manageable amounts of time to improve students’ lateral reading? Research could probe the pacing of lessons, what aspects of lateral reading to focus on, and how lessons taught initially in one subject area could be reviewed in others.

Engage Students With Actual Online Content

The civic online reasoning lessons in this study rely on actual online resources selected to fit within the content of each subject area. Students had opportunities to see and practice lateral reading with real Internet sources that were relevant to what they were studying in class. This approach exposed students to a range of content that varied in credibility.

In this study, the researchers took primary responsibility for locating appropriate online materials. Of course, this kind of close researcher collaboration is not possible at scale. A library of different subject-specific materials and guidance about how to adapt materials for particular content areas would support teachers seeking to integrate civic online reasoning instruction across the curriculum. Further, research could probe how teachers go about selecting online content themselves—and the effectiveness of the examples they use—as a way to design supports for teachers in this area.

Feature Cognitive Apprenticeship and Formative Assessment

The civic online reasoning lessons incorporated modeling in unique formats, including screen-recorded videos for students to watch, live teacher modeling, and lectures with screenshots for students to review. These varied formats helped teachers present different aspects of lateral reading over time and allowed students to see more expert approaches in action. Cognitive modeling was always followed by opportunities for students to practice with guiding questions or graphic organizers to focus them on the elements of lateral reading the modeling had featured. Importantly, our decisions about the elements of lateral reading on which to focus were guided by analyses of formative assessments from prior lessons. Teachers incorporated more traditional (e.g., written responses to short assessments) as well as creative (e.g., students submitted brief screen recordings of reading laterally about a social media source of their choosing) formative assessments. These responses helped the leadership team quickly get a sense for how students were doing and where to go next.

Results from this study suggest that students could use even more explicit instruction and opportunities to practice civic online reasoning. In this intervention, less time was spent on civic online reasoning in geography classes than in biology classes, which limited students’ chances to practice lateral reading. Future research should investigate approaches for supporting teachers in modeling their thinking, facilitating guided practice, and formatively assessing student learning. Further, research should investigate how professional development supports can help teachers integrate civic online reasoning into their existing courses.

Support Teacher Learning

Research is needed to explore how teachers use civic online reasoning curriculum materials without attending relevant professional development. If civic online reasoning is to be widely taught across subject areas, such research is particularly important because many teachers who have not participated in formal professional development will need to implement the lessons. It is possible that high quality materials designed to support teacher learning could fill some of the gaps for teachers; whether that is true needs to be investigated.

Conclusion

This study adds to a growing body of research showing that targeted interventions in typical educational settings can help students become more skilled evaluators of digital information. In contrast to previous research, this study embedded civic online reasoning instruction in core academic courses by aligning the curricular materials with course subject matter. This allowed teachers to readily take up the instructional content. For such an approach to be adopted widely, teachers across subject areas will need access to materials that match the content they teach. A national open-resource library of discipline-specific curricular resources would go a long way toward making meaningful civic online reasoning instruction a reality. In addition, teachers across subject areas will need support in their use of these new materials. Curricular materials designed to support teacher learning may help in this regard. Professional development delivered at scale could also be effective. Such initiatives will require significant research and development to ensure the quality of these critically important resources.

Supplemental Material

sj-docx-1-ero-10.1177_23328584231176451 – Supplemental material for Civic Online Reasoning Across the Curriculum: Developing and Testing the Efficacy of Digital Literacy Lessons

Supplemental material, sj-docx-1-ero-10.1177_23328584231176451 for Civic Online Reasoning Across the Curriculum: Developing and Testing the Efficacy of Digital Literacy Lessons by Sarah McGrew and Joel Breakstone in AERA Open