Associations Between Online Instruction in Lateral Reading Strategies and Fact-Checking COVID-19 News Among College Students

Abstract

College students, and adults in general, may find it hard to identify trustworthy information amid the proliferation of false news and misinformation about the COVID-19 pandemic. In Fall 2020, college students (N = 221) in an online general education civics course were taught through asynchronous assignments how to use lateral reading strategies to fact-check online information. Students improved from pretest to posttest in the use of lateral reading to fact-check information; lateral reading was predicted by the number of assignments completed and students’ reading comprehension test scores. Students reported greater use, endorsement, and knowledge of Wikipedia at posttest, aligning with the curriculum’s emphasis on using Wikipedia to investigate information sources. Students also reported increased confidence in their ability to fact-check COVID-19 news. While confidence was related to perceived helpfulness of the assignments, it was only weakly associated with lateral reading. Findings support the effectiveness of the online curriculum for improving fact-checking.

Keywords

lateral reading fact-checking COVID-19 news infodemic false news online education general education higher education curriculum media technology hierarchical linear modeling descriptive analysis regression analyses

While online platforms allow for fast and widespread sharing of information about the coronavirus pandemic (COVID-19), these technologies also allow for the proliferation of false news and deliberate misinformation, often intentionally designed to mislead or push an agenda (World Health Organization [WHO], 2020a). In Fall 2020, we used an online curriculum to teach “SIFT” fact-checking strategies (Stop, Investigate the source, Find better coverage, and Trace claims, quotes, and media to the original context; Caulfield, n.d; 2019) to college students in a general education civics course. SIFT strategies rely on lateral reading, that is, leaving the initial content to investigate sources and verify claims using trusted sources (Wineburg & McGrew, 2017). We examined whether the curriculum improved students’ use of lateral reading when fact-checking online content and students’ perceptions of how it affected their fact-checking of online news about the COVID-19 pandemic.

The COVID-19 “Infodemic”

In a February 2020 report on the COVID-19 outbreak, the WHO noted the presence of a “massive ‘infodemic’- an overabundance of information—some accurate and some not—that makes it hard for people to find trustworthy sources and reliable guidance when they need it” (WHO, 2020b, p. 2). COVID-19 misinformation includes blatantly false information, but it more often takes the form of information that is “reconfigured” (i.e., online content that is mislabeled or miscaptioned, or manipulated and repurposed in various ways; Brennen et al., 2020). Misinformation related to the COVID-19 pandemic has covered a wide range of topics (e.g., the actions of public authorities, virus spread and transmission, conspiracy theories) with potentially dire consequences for individual and public health (WHO, 2020a). For example, exposure to misinformation about the COVID-19 vaccine has increased vaccine hesitancy among adults in the United Kingdom and the United States (Loomba et al., 2021).

While misinformation has accompanied previous disease outbreaks, the rapid spread of information via social media platforms has posed an unprecedented challenge in the context of COVID-19 news (Zarocostas, 2020). Misinformation about COVID-19 has circulated faster on social media platforms (e.g., Facebook, Twitter, and YouTube) than the companies owning these platforms can flag or remove it (Frenkel et al., 2020). Fact-checking organizations have struggled to keep up with the onslaught of questions about the accuracy of COVID-19-related information (Izadi, 2020). Understanding the COVID-19 infodemic remains an area of active investigation (see Tsao et al., 2021, for a brief review). Areas of inquiry include characterizing COVID-19-related false news and misinformation (e.g., Brennen et al., 2020), how it spreads via different social media platforms (e.g., Cinelli et al., 2020; Pulido et al., 2020), and strategies for reducing its spread (e.g., Pennycook et al., 2020).

Surveys of American adults conducted by the Pew Research Center provide some insight into how different age-groups navigated news and information about COVID-19 at the height of the pandemic in 2020. Younger adults (ages 18–29 years) were consistently less likely than older adults to report following COVID-19 news very closely (e.g., 24% of younger adults vs. 37% of the total adults surveyed in November 2020; Pew Research Center, 2020a), while also relying more heavily on social media for news about COVID-19 (30% of younger adults vs. 19% of all the adults surveyed in April 2020; Pew Research Center, 2020b). In April 2020, only 26% of younger adults felt very confident that they knew how to find out if news about COVID-19 was accurate (Pew Research Center, 2020c). Most (52%) were only somewhat confident, and 21% were not too confident or not at all confident. These rates were comparable with those of all the U.S. adults surveyed. Given younger adults’ reliance on social media as a source of news and information about COVID-19, this age-group may benefit from direct instruction in strategies for fact-checking online information. Teaching these strategies as part of general education college courses is one means of reaching individuals in this age-group.

Reading Laterally to Fact-Check Online Information

In response to concerns about the spread of inaccurate information related to the COVID-19 pandemic, the current study sought to teach college students how to fact-check online information using strategies observed in expert fact-checkers. When Wineburg and McGrew (2017) asked a small sample of historians, college students, and expert fact-checkers to evaluate the trustworthiness of online social and political information, they found that fact-checkers tended to read laterally by “[leaving] a website and [opening] new tabs along a horizontal axis in order to use the resources of the Internet to learn more about a site and its claims” (p. 38). Expert fact-checkers were more effective and efficient at judging credibility than either historians or college students, who mostly read vertically by closely examining the features of the original content or website.

One of the sources that expert fact-checkers quickly turn to is Wikipedia. For example, Wineburg and McGrew (2017) observed one fact-checker “taking bearings” when evaluating an organization’s article about children’s bullying by using Wikipedia to learn more about the organization’s agenda. Similarly, in the context of using lateral reading to fact-check information about the COVID-19 pandemic, one could use Wikipedia to find out whether an organization posting vaccine-related information on social media had a hidden agenda. Although both teachers and students may be concerned about the reliability of information on Wikipedia (Garrison, 2018; Konieczny, 2016), Wikipedia articles are as accurate as those in the Encyclopaedia Britannica (Giles, 2005). Wikipedia articles are expected to be written from a neutral point of view, with information accompanied by citations to reliable sources (Wikipedia, n.d.-a). These citations can point students to other sources that they can use to verify claims. Furthermore, vetted Wikipedia editors with administrative privileges are able to protect controversial Wikipedia articles from public editing (Wikipedia, n.d.-b), making it less likely that the information is intentionally manipulated.

However, without direct instruction in lateral reading strategies and opportunities to practice the skills, college students are unlikely to read laterally. Additionally, students’ self-reported information verification strategies do not align with their observed behaviors (Brodsky et al., 2020; Hargittai et al., 2010; List & Alexander, 2018). Multiple recent studies have used performance-based tasks to assess students’ use of lateral reading. McGrew et al. (2018) asked middle school, high school, and college students to evaluate the credibility of online social and political content across media formats. Across tasks, college students rarely used lateral reading to research sources and verify claims. A follow-up study again found that, rather than reading laterally, college students examined features of the online information, like the domain name, About Us page, links, and aesthetics (i.e., vertical reading strategies; Wineburg et al., 2020).

Small-scale efforts to teach college students to read laterally have been moderately successful. College students who received a two-session in-class intervention were more likely to read laterally at posttest than students who did not receive the curriculum (McGrew et al., 2019). Similarly, students in an asynchronous online course who completed online modules teaching lateral reading strategies were more likely to use lateral reading at posttest than at pretest (Breakstone et al., 2021).

In 2018, the American Association of State Colleges and Universities launched the Digital Polarization Initiative (DPI; American Democracy Project, n.d.), which oversaw the piloting of an in-person fact-checking curriculum in colleges and universities across the United States. The curriculum taught college students four “moves” (i.e., strategies) for fact-checking online information, three of which relied heavily on lateral reading (Caulfield, 2017b). Each of these three moves involved reading laterally by leaving the original information to learn more about the information or its source. For the move “look for trusted work,” students were taught to research a claim by using independent sources like Google News and fact-checking sites. To “find the original,” students learned to use a reverse-image search to find out whether an image had been reconfigured. Last, for the move “investigate the source,” students used Wikipedia and Google News to learn about a source and whether it had a hidden agenda or biases that could influence the trustworthiness of the information.

Our institution was one of 11 colleges selected to pilot the DPI curriculum using performance-based assessments similar to those employed by McGrew et al. (2018). In Fall 2018, multiple sections of a general education civics course received the in-person curriculum and were compared with “business-as-usual” control sections at pre- and posttest (see Brodsky, Brooks, Scimeca, Todorova et al., 2021). At posttest, students in the DPI sections were more likely to read laterally and correctly assess trustworthiness than students in the control sections, with 61.0% of students in the DPI sections doing so on at least one of four problems. However, students did not consistently engage in lateral reading across posttest problems. Rather, they read laterally and accurately assessed the online content on just one out of four problems, on average, and some did not read laterally at all. Moreover, despite the curriculum’s encouragement of Wikipedia use, students in the DPI sections made only small gains in their self-reported use of Wikipedia and no gains in their trust of Wikipedia at posttest. Students’ inconsistent use of lateral reading and Wikipedia for fact-checking information may reflect a variety of factors, including attendance and class participation, motivation, and time constraints. These initial results indicate that additional practice with feedback is needed for students to demonstrate the metacognitive awareness, strategy selection, and automaticity of experts (Chi, 2006; Healy et al., 2014).

Research Objectives

In light of the rapid spread of COVID-19 misinformation and challenges associated with determining what information to trust about the pandemic, the current study examined the efficacy of a fully online curriculum for teaching college students how to read laterally to evaluate online information. The curriculum presented four fact-checking moves using the acronym SIFT: Stop, Investigate the source, Find better coverage, and Trace claims, quotes, and media to the original context (Caulfield, n.d; 2019). While students learned about all four SIFT strategies, we focused our assessment efforts explicitly on students’ use of the three SIFT strategies that rely on lateral reading (investigating the source, finding better coverage, and tracing information to the original context).

The study had several goals. First, we examined the extent to which students improved on the skills and content knowledge emphasized in the online SIFT curriculum. Specifically, we were interested in students’ reported use of lateral reading to fact-check online content and their use, endorsement, and knowledge of Wikipedia. Given the ongoing COVID-19 infodemic, we were also interested in how students’ use of lateral reading related to their self-reported confidence in their ability to fact-check COVID-19-related online news and the perceived helpfulness of the SIFT curriculum for fact-checking COVID-19-related online news.

Throughout the study, we chose to take into account individual differences in students’ reading comprehension. Prior studies of lateral reading for fact-checking online information have not considered this factor, even though reading laterally inherently relies on students’ ability to comprehend and integrate text-based information from a variety of sources. Furthermore, the current study was conducted at an open-admissions public college serving a diverse student body. As students’ reading comprehension skills were expected to vary widely (Perin, 2013), we included a measure of reading comprehension as a potential predictor of individual differences in responses to the online curriculum.

Method

Participants

The participants were college students (N = 221; M_age = 19.6 years, SD = 4.5, range = 16–45) enrolled in 10 online sections of a general education civics course taught by seven instructors. The course was offered at a Hispanic-serving public institution located in the northeastern United States. The college has an open-enrollment policy, operates two campuses, and runs classes days, evenings, and weekends. Table 1 shows the self-reported demographics of the sample. About 40% of participants were first-generation students (i.e., neither parent attended college). Data were collected for purposes of course outcomes assessment, with the research protocol classified as exempt by the institutional review board. All participants completed pretest and posttest assessments.

Table 1

Participants’ Self-Reported Demographics (N = 221)

Characteristics	Percentage of students
Gender
Female	64.7
Male	34.4
Prefer not to respond	0.9
Race/ethnicity (not mutually exclusive)
American Indian/Alaska Native	0.0
Asian/Asian American	11.3
Black/African American	10.9
Latinx, Chicanx, Hispanic, or Spanish origin	16.7
Middle Eastern/North African	16.3
Native Hawaiian/Other Pacific Islander	0.0
White	45.3
Some other race	1.4
Prefer not to say	3.2
Unavailable/unknown	0.5
Either parent attended college
Yes	61.5
No	38.5
Native language
English	67.9
Spanish	6.8
Other	25.3

The Online SIFT Curriculum

Over the course of the semester, the students completed five online homework assignments, which were graded based on completion. The pretest (Assignment 1) was administered in Week 2 of a 15-week semester and served to assess the students’ baseline knowledge and skills in using lateral reading to evaluate online news items spanning a broad range of topics. The posttest (Assignment 5) was administered in Week 15 to assess gains. Materials were presented in the following order on the pretest and posttest: lateral reading problem set; demographic questions (pretest only); online news consumption questions;¹ use of online news sources to find information about the COVID-19 pandemic;² confidence in fact-checking COVID-19 news; questions about Wikipedia use, endorsement, instruction, and quality of information; and a reading comprehension test (pretest only). On the posttest, students were also asked about their familiarity with the sources of the online content used in the pretest and posttest and about the perceived helpfulness of the online assignments for fact-checking online news about COVID-19. All the materials are described below or are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

Assignments 2, 3, and 4 were administered in Weeks 4, 9, and 13 of the semester, respectively. All three assignments provided SIFT instruction via videos and practice problems. Across problems, the students were asked to assess a variety of types of online content (i.e., social media posts, online articles, and images) and received feedback on how to use SIFT strategies to fact-check the information. In Assignment 2, the students watched a series of videos about the SIFT moves and answered questions about information from the videos. They were then guided through using the SIFT moves to investigate the trustworthiness of online content about COVID-19-related public health policy issues overseen by state governments. In Assignment 3, the students reviewed the SIFT moves and practiced using them to assess content related to controversial aspects of the 2020 U.S. Census. In Assignment 4, the students again reviewed the SIFT moves and evaluated online content about the effects of COVID-19 lockdowns on the economy and the effects of announcements about the COVID-19 vaccine on global and U.S. markets.

At the end of each assignment, the students were asked to locate three new articles related to the topic of the assignment and use the SIFT moves to evaluate their trustworthiness. For 6 of the 10 course sections, the online assignments included an additional, brief video and four true/false questions about using Wikipedia to fact-check online content. The five assignments are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021). The assignments were administered via Qualtrics software. The instructors posted a link to each assignment to the learning management system.

Lateral Reading Problems

Two sets of lateral reading problems (Problem Sets A and B) were used to assess students’ use of lateral reading at pretest and posttest. The assignment of problem sets to pre- and posttest was counterbalanced across instructors; that is, three instructors’ sections completed Set A at pretest and Set B at posttest, while four instructors’ sections completed Set B at pretest and Set A at posttest. In each problem set, problems were presented in random order.

Each problem set comprised four problems exemplifying different types of online content: an image, a social media post, a website with medical news, and a news article. The online content used in Set A was identical to one of the problem sets in the original DPI curriculum (Brodsky, Brooks, Scimeca, Todorova et al., 2021). The online content used in Set B was updated with content from the Four Moves blog (Caulfield, 2017a), a repository of lateral reading prompts maintained by the author of the original DPI curriculum. The content of Set B was selected to represent the diversity of information sources that students may encounter online, while also matching the formats and content types in Set A. The trustworthiness of the content varied across problems within each set.

Students could use multiple SIFT moves to fact-check each problem. For example, for the Set A problem featuring an article published on newser.com titled “School District Arms Students With Rocks” (http://www.newser.com/story/256977/school-district-arms-students-with-rocks.html; left panel of Figure 1), students could use Wikipedia to investigate the source (Newser) or look for better coverage by searching keywords from the article’s headline (e.g., “school arms students rocks”) in Google or Google News. For the Set B problem consisting of a tweet claiming to show a photograph of megabats (right panel of Figure 1), students could choose to trace the photograph to get additional context by reverse searching the image using Google. Alternatively, they might try to find better coverage of the topic by typing keywords (e.g., “black flying-fox” or “megabat”) in Google or Google News.

Figure 1.

Screenshots of two of the lateral reading problems.

For each problem, the students first rated how provoked they were by the content (1 = not at all provoked to 5 = very provoked) and then indicated their level of trust in the content using a Likert-type scale ranging from 1 = very low to 5 = very high or −9 = unsure. They were then prompted to explain their process for determining their level of trust by responding to the prompt “How did you decide on your level of trust?” using an open-response textbox. At the end of the posttest, the students were asked to indicate their familiarity with each of the sources used in the two problem sets (1 = not at all familiar, 2 = a little familiar, 3 = somewhat familiar, 4 = moderately familiar, and 5 = very familiar). Table 2 presents information about each content type, problem set, and online content used.

Table 2

Content Type, Online Content, and Correct Trust Assessment for Problem Sets A and B

Content type	Problem set	Online content	Correct trust assessment
Image	A	Photograph on Imgur claiming to show mutated flowers near the Fukushima Daiichi Nuclear Power Plant in Japan (https://imgur.com/gallery/BZWWx)	Low (1)Very low (2)
Image	B	Photograph claiming to show a child detained in a cage by U.S. Immigration and Customs Enforcement (https://www.snopes.com/fact-check/toddler-cage-photo/)	Low (1)Very low (2)
Tweet	A	Tweet from MoveOn.org stating that “2 out of 3 gun owners would be more likely to vote for a candidate that supported background checks” (https://twitter.com/MoveOn/status/666772893846675456?lang=en)	Low (2)Moderate (3)High (4)
Tweet	B	Tweet from Chris Walker (@WildRedlands) claiming to show a photograph of megabats in the Black Swamp Wetlands in Cleveland (https://twitter.com/WildRedlands/status/1034718288062738433)	Moderate (3)High (4)Very high (5)
Medical news	A	Article published on BioNews titled “Majority of Breast Cancer Patients Do Not Need Chemotherapy” (https://www.bionews.org.uk/page_136385)	Moderate (3)High (4)Very high (5)
Medical news	B	Article about the measles vaccine from the National Vaccine Information Center (https://www.nvic.org/vaccines-and-diseases/Measles.aspx)	Very low (1)Low (2)Moderate (3)
News article	A	Article published on newser.com titled “School District Arms Students With Rocks” (http://www.newser.com/story/256977/school-district-arms-students-with-rocks.html)	Moderate (3)High (4)Very high (5)
News article	B	Article published on azcentral.com titled “Grand Canyon Tourists Exposed for Years to Radiation in Museum Building, Safety Manager Says” (https://www.azcentral.com/story/news/local/arizona/2019/02/18/grand-canyon-tourists-exposed-radiation-safety-manager-says/2876435002/)	Very low (1)Low (2)Moderate (3)

Scoring of Lateral Reading Problems

We scored each open response for the presence or absence of lateral reading using an automated approach validated in our previous work (Brodsky, Brooks, Scimeca, Todorova et al., 2021). Each response was searched for keywords that indicated use of lateral reading. We sometimes used truncated versions of keywords to increase the likelihood of capturing different forms of a word (e.g., “googl” captured “Google,” “Googling,” and “Googled”). We used the same keywords as Brodsky, Brooks, Scimeca, Todorova et al. (2021).³ We also added a keyword for SIFT since the original list of keywords was developed prior to the use of the “SIFT” acronym in the fact-checking curriculum. A response was scored as “1” to indicate that the student reported reading laterally if it contained one or more of the target keywords; it was scored as “0” to indicate no lateral reading if the student did not use any of the keywords in the response.

To assess the reliability of the automated scoring procedure, we first used the keywords to score problem sets from a previous semester that had been manually scored for the presence or absence of lateral reading. The only difference in the content of the problem sets between semesters was that the tweet about megabats in Problem Set B in the current study came from a different Twitter account. There was 93.5% agreement between the automated and manual scoring; Cohen’s kappa = .72, which is considered substantial (Landis & Koch, 1977). Having established adequate reliability for the automated scoring procedure, we used the keywords to score the responses for the current study and also manually scored all responses. In the current study, there was 94.1% agreement between the automated and manual scoring (Cohen’s kappa = .80), which is comparable with our previous work (Brodsky, Brooks, Scimeca, Todorova et al., 2021). All disagreements were resolved in favor of the manual coding.

In addition to scoring the responses for evidence that the students used the skill of lateral reading, we also used a more conservative scoring scheme to determine whether or not the students used lateral reading and accurately assessed the trustworthiness of the content (see also Brodsky, Brooks, Scimeca, Todorova et al., 2021). To do this, we scored responses on the Likert-type scale asking about the trustworthiness of the online content as “0” for incorrect trust assessment and “1” for correct trust assessment (see Table 2). We then combined the keyword and trust assessment scores such that “0” indicated no use of lateral reading or use of lateral reading but with an inaccurate trust assessment and “1” indicated use of lateral reading with an accurate trust assessment.

Confidence in Fact-Checking COVID-19 Pandemic News

The students responded to the question “If you wanted to check the accuracy of news that you saw online about the COVID-19 pandemic, how confident are you that you would know what steps to take?” on a Likert-type scale of 1 = not at all confident, 2 = a little confident, 3 = somewhat confident, 4 = mostly confident, and 5 = very confident. This item was adapted from a similar item asked by the Pew Research Center as part of a survey conducted in April 2020 (Gottfried, 2020).

Wikipedia Use

The students responded to the following questions about their use of Wikipedia on a Likert-type scale of 1 = very unlikely to 5 = very likely: “How likely are you to use Wikipedia to look up information for academic purposes (e.g., for a school assignment)?” and “How likely are you to use Wikipedia to look up information for nonacademic purposes (e.g., for personal research)?”

Wikipedia Endorsement

The students responded to the question “How likely are you to recommend Wikipedia as a source of information to one of your classmates?” on a Likert-type scale of 1 = very unlikely to 5 = very likely.

Wikipedia Instruction

The students indicated the extent to which they agreed with the statement “My teachers have discouraged me from using Wikipedia as an information source” on a Likert-type scale of 1 = strongly disagree to 5 = strongly agree.

Wikipedia Quality of Information

The students indicated the extent to which they agreed with nine statements (three reverse scored) about the quality of information on Wikipedia on a Likert-type scale of 1 = Strongly disagree to 5 = Strongly agree. Scale items and reliability analyses are presented in the Results section.

Reading Comprehension

The students read Passage C from the June 2018 New York State Regents High School Examination in English language arts and answered six multiple-choice comprehension questions from the exam (New York State Education Department, 2018). This passage was excerpted from a 2012 article in The Wall Street Journal about fostering socioeconomic antifragility in response to unexpected events (Taleb, 2012, as cited in New York State Education Department, 2018).

Perceived Helpfulness of the Curriculum

The students indicated their agreement with five statements (two reverse scored) about the perceived helpfulness of the online SIFT curriculum for fact-checking online news about the COVID-19 pandemic. The response scale ranged from 1 = Strongly disagree to 5 = Strongly agree. Scale items and reliability analyses are presented in the Results.

Results

The results are organized by research questions. All the analyses were run in R (Version 3.6.2; R Core Team, 2018; RStudio Team, 2016).

Preliminary Analyses for Changes in Lateral Reading

As a first step in evaluating whether the curriculum increased the use of lateral reading to fact-check online information from pretest to posttest, we conducted preliminary analyses to determine the factors associated with the use of lateral reading at pretest and posttest. The preliminary analyses examined the possible effects of the instructor, curricular emphasis on the use of Wikipedia (i.e., inclusion of the additional video and four multiple-choice questions), problem set used at pre- and posttest (i.e., Set A or B), and problem content type (i.e., image, tweet, medical news, or news article). Within each problem set, we also looked at whether lateral reading was associated with student ratings of provocative content and familiarity with the source of the online content used in each problem. We conducted all the preliminary analyses twice, once with reported use of lateral reading as the dependent variable and once with reported use of lateral reading and accurate assessment of the trustworthiness of the content as the dependent variable. All these preliminary analyses are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

Table 3 presents a breakdown of students who reported reading laterally (top panel) and students who read laterally with an accurate trust assessment (bottom panel) on each problem at pretest and posttest. The score for using lateral reading and accurately assessing trustworthiness is more conservative than the lateral reading score because it takes into account the accuracy of a student’s response on the Likert-type scale asking about the trustworthiness of the online content. Only 12.2% of students reported reading laterally on one or more problems at pretest, and only 9.5% read laterally and made an accurate trust assessment on at least one pretest problem. None of the preliminary analyses were significant in accounting for the variation in lateral reading at pretest.

Table 3

Percentage of Students Who Read Laterally for Each Problem and Percentage Who Read Laterally and Accurately Assessed the Trustworthiness of the Information for Each Problem by Problem Set and Time (N_AB = 126, N_BA = 95, N = 221)

Problem	Pretest			Posttest
Problem	Set A	Set B	Sets A and B	Set A	Set B	Sets A and B
Percentage of students who read laterally
Image	4.0	3.2	3.6	21.1	36.5	29.9
Tweet	3.2	6.3	4.5	17.9	38.9	29.9
Medical news	5.6	1.1	3.6	24.2	42.9	34.8
News article	5.6	4.2	5.0	24.2	44.4	35.7
Percentage of students who read laterally and accurately assessed trustworthiness
Image	2.4	2.1	2.3	9.5	24.6	18.1
Tweet	2.4	5.3	3.6	14.7	27.8	22.2
Medical news	4.8	0.0	2.7	15.8	36.5	27.6
News article	4.8	4.2	4.5	17.9	27.8	23.5

In contrast, 52.5% of students reported reading laterally on at least 1 problem at posttest, with 48.9% reporting lateral reading on at least one more problem at posttest than at pretest. On average, the students reported reading laterally on 1.30 (SD = 1.53) of the 4 posttest problems. Lateral reading at posttest varied by problem set (see Table 3, top panel) and instructor. Additionally, students’ familiarity with the sources of problems in Set B was related to their use of lateral reading, though this finding should be interpreted with caution since the image problem in Set B did not include a source and all the students were lacking familiarity ratings for this problem. No other preliminary analyses were significant.

Posttest results were similar with lateral reading and accurate trust assessment as the dependent variable: 46.6% of students reported reading laterally and accurately assessing the trustworthiness of at least 1 problem, and 44.3% made gains of at least 1 problem from pretest to posttest. On average, students reported reading laterally and accurately assessing the trustworthiness of the information on 0.91 (SD = 1.18) of the 4 problems at posttest. Matching the previous set of analyses, lateral reading with an accurate trust assessment at posttest varied by problem set (see Table 3, bottom panel), instructor, and familiarity (again, only for Set B). Additionally, use of lateral reading with an accurate trust assessment varied by content type, such that use of lateral reading with an accurate trust assessment was more likely to occur for the medical news than for the image. No other preliminary analyses were significant.

Since problem set was completely confounded with instructor (i.e. Instructor 2’s students received the easier Set B at posttest, while Instructor 3’s and Instructor 6’s students received the harder Set A at posttest), we could not retain both variables as predictors in our multivariate analyses. We chose to retain the problem set as the predictor since the online implementation of the curriculum was designed to reduce instructor effects. We also did not include familiarity with a problem’s source as a predictor since the data were incomplete for the problems in Set B. All of the nonsignificant factors in the preliminary analyses were excluded from further analysis.

Online Assignment Completion

Most of the students (72.0%) completed all three online homework assignments. Of the remaining students, 3.6% completed between two and three assignments (i.e., partial completion of an assignment), 14.9% completed two assignments, 1.4% completed between one and two assignments, 5.4% completed one assignment, and 2.7% completed no assignments. On average, the students completed 2.62 assignments (SD = 0.71). Given the different levels of engagement with online homework, we included the number of completed assignments (range 0–3) as a predictor in the models.

Students’ Reading Comprehension

The students’ performance on the pretest measure of reading comprehension varied widely. On average, they answered 3.75 (SD = 1.58, Mdn = 4) out of 6 questions correctly, with one student’s score imputed using the mean. Given the diversity of the students’ reading comprehension skills, we included the number of reading comprehension questions answered correctly (range 0–6) as a predictor in the models.

Changes in Lateral Reading

To determine whether the students improved from pretest to posttest in their reported use of lateral reading to fact-check online information, we ran mixed-effects logistic regression models. The models were run with a binomial logit link using the glmer function of the lme4 package (Bates et al., 2014) in R (R Core Team, 2018; RStudio Team, 2016). In Model 1, the dependent variable (coded as “1” or “0” for each problem) indicated whether or not the student reported reading laterally to determine the level of trustworthiness (see Table 4, top panel). The model included an intercept-only random effect for students. The fixed effects were the problem set (Set A vs. Set B), content type (image, tweet, medical news, or news article), time (pretest vs. posttest), number of online assignments completed (range 0–3), and pretest reading comprehension score (range 0–6). Controlling for the problem set and content type, the students were more likely to read laterally on the posttest than on the pretest. Table 5 presents sample responses for the problems to help illustrate the dearth of lateral reading at pretest and the diversity of SIFT moves used by the students at posttest. Students who completed more assignments and those with higher reading comprehension scores were more likely to have read laterally as well.

Table 4

Mixed-Effects Logistic Regression Models Predicting the Use of Lateral Reading on Each Problem and the Use of Lateral Reading and Accurate Assessment of Trustworthiness of Each Problem at Pretest and Posttest (N = 221)

Predictor	B (SE)	Odds ratio [95% CI]	z	χ²
Model 1: Predicting lateral reading on each problem
Intercept	−8.78 (0.94)	0.0002 [0.00002, 0.001]	−9.34***	—
Problem set (A = 0)^a	0.72 (0.21)	2.05 [1.35, 3.11]	3.36***	11.18***
Content type (image = 0)^a	—	—	—	4.79
Tweet	0.06 (0.24)	1.06 [0.66, 1.71]	0.24	—
Medical news	0.31 (0.24)	1.37 [0.86, 2.19]	1.31	—
News article	0.45 (0.24)	1.57 [0.98, 2.50]	1.89†	—
Number of assignments completed	0.75 (0.27)	2.12 [1.26, 3.59]	2.81**	8.50**
Pretest reading comprehension score	0.41 (0.11)	1.51 [1.22, 1.87]	3.82***	15.25***
Time (pretest = 0)	3.33 (0.25)	28.03 [17.09, 45.99]	13.20***	303.20***
Model 2: Predicting lateral reading and accurate assessment of trustworthiness of each problem
Intercept	−8.02 (0.84)	0.0003 [0.00006, 0.002]	−9.53***	—
Problem set (A = 0)^a	0.64 (0.21)	1.90 [1.26, 2.87]	3.06**	9.30**
Content type (image = 0)^a	—	—	—	8.13*
Tweet	0.39 (0.26)	1.48 [0.89, 2.45]	1.53	—
Medical news	0.68 (0.25)	1.97 [1.20, 3.23]	2.69**	—
News article	0.54 (0.25)	1.71 [1.04, 2.82]	2.12*	—
Number of assignments completed	0.59 (0.24)	1.80 [1.13, 2.87]	2.49*	6.72**
Pretest reading comprehension score	0.34 (0.09)	1.41 [1.17, 1.69]	3.62***	13.60***
Time (pretest = 0)	2.60 (0.25)	13.48 [8.31, 21.88]	10.54***	168.25***

Note. The full Model 1 with all the fixed effects fit significantly better than the null model with only the random effect of students, χ²(7) = 396.33, p < .001. For each fixed effect, we compared the fit of the full model with the fit of the same model with the fixed effect excluded. This allowed us to determine if including the fixed effect significantly improved the model fit. All the variables except content type improved the fit of Model 1. The full Model 2 with all the fixed effects fit significantly better than the null model with only the random effect of students, χ²(7) = 243.59, p < .001. All the variables improved the fit of Model 2, as compared with the fit of the same model with the fixed effect excluded.

Baselines were set based on the lowest number of problems read laterally at posttest.

†

p < .06. *p < .05. **p < .01. ***p < .001.

Table 5

Sample Responses Indicating the Absence of Lateral Reading at Pretest and Presence of Lateral Reading at Posttest for Each Content Type

Content type	Sample pretest response	Sample posttest response
Photo (AB)	This student examined the use of sources and prior beliefs about the platform: “Very low. No sources, and their username is questionable. You can post anything on imgur without it being true or accurate.”	This student found better coverage: “At CNN, a trusted news source says the picture [was] taken out of context.”
Photo (BA)	This student relied on their gut response: “I am not sure if I can trust just watching the photo.”	This student traced the photograph and found better coverage: “I used search by image and found this on Snopes.”
Tweet (AB)	This student relied on their gut response: “I wouldn’t completely trust this source.”	This student traced the photograph: “I did a reverse image search and found out that this is true and that they do exist.”
Tweet (BA)	This student relied on prior beliefs about the platform: “Because I saw where it was posted and I don’t trust what gets posted on Twitter.”	This student investigated the source: “I searched on Wikipedia with their name, and it shows that they support certain people.”
Medical news (AB)	This student examined the use of sources and the quality of the content: “The story above has a lot of references to other articles as well as evidence showing the number of patients tested as well as answers and outside information that all correlates with one another to prove the story’s legitimacy and trustworthiness.”	This student investigated the source: “According to wiki, they have been reported to be fearmongers.”
Medical news (BA)	This student examined the quality of the references: “The reference at the bottom of the website is reassuring that their information is [legitimate] since it came from a government website.”	This student investigated the source:“How I came to the conclusion that I trust this article is when I look up its website on wiki in order to see [if] this was a valid website. . .”
News article (AB)	This student relied on prior beliefs and experiences: “My level of trust in news articles depends on the level of relative truth it seems to offer. In this situation, a U.S. school offering punishments is highly unlikely as that is against our law. Also the news source does not sound reliable. “Newser” is not a reputable source that I have heard of.”	This student found better coverage: “I checked the source and googled the headline and found other reputable sources such as CNN and FOX news reporting similar cases.”
News article (BA)	This student examined the website features: “The website has too many ads and random videos; however, the story seems somewhat legitimate. I would have to do more research to find out for sure.”	This student found better coverage: “I looked it up on Google and found the same story covered by NY Post, Guardian, and other trustworthy news outlets.”

Note. Problem set order is presented in parentheses next to content type. The responses in each row are from the same student.

In Model 2, the dependent variable (coded as “1” or “0”) indicated whether the student read laterally and accurately assessed trustworthiness on each problem (see Table 4, bottom panel). Controlling for the problem set and content type, students were more likely to have read laterally and reached an accurate trust assessment on the posttest than on the pretest. Additionally, the number of assignments completed and reading comprehension scores were associated with using lateral reading and accurately assessing trust.

Changes in Wikipedia Use, Endorsement, and Knowledge

On the pretest, 77.8% of students Agreed or Strongly agreed with the statement “My teachers have discouraged me from using Wikipedia as an information source.” Given the curriculum’s emphasis on using Wikipedia to learn more about a source, we examined whether the students made gains in their self-reported Wikipedia use and endorsement, and knowledge of Wikipedia quality from pretest to posttest. Table 6 presents the frequencies of responses at pretest and posttest to the two items assessing likelihood of Wikipedia use and the one item assessing Wikipedia endorsement. Wilcoxon signed rank tests indicated that the students self-reported a greater likelihood of using Wikipedia to look up information for academic and nonacademic purposes at posttest than at pretest. They were also more likely to recommend Wikipedia as an information source to one of their classmates at posttest (see Table 6).

Table 6

Percentage of Students Who Indicated Each Response for Wikipedia Use and Endorsement by Time (N = 221)

Item		Very unlikely(1)	Unlikely(2)	Neither likely nor unlikely(3)	Likely(4)	Very likely(5)	Mdn	V
How likely are you to use Wikipedia to look up information for academic purposes (e.g., for school assignments)?	Pre	45.2	23.1	14.0	14.5	3.2	2	1076***
	Post	13.1	14.5	27.1	34.4	10.9	3
How likely are you to use Wikipedia to look up information for nonacademic purposes (e.g., for personal research)?	Pre	25.8	22.2	20.4	24.0	7.7	3	2519***
	Post	13.6	13.6	22.2	33.9	16.7	4
How likely are you to recommend Wikipedia as a source of information to one of your classmates?	Pre	49.8	18.1	19.0	10.4	2.7	2	1261.5***
	Post	15.8	13.6	34.4	27.6	8.6	3

Note. For each question, each row sums to 100%, with slight deviations due to rounding.

***

p < .001.

To further examine these changes, we ran a series of mixed-effects ordinal logistic regression models with a logit link using the clmm function of the ordinal package (Christensen, 2019) in R (R Core Team, 2018; RStudio Team, 2016).⁴ The models included an intercept-only random effect for students. Our fixed effects were time (pretest vs. posttest), number of online assignments completed (range 0–3), and pretest reading comprehension score (range 0–6).

Model 1 used students’ self-reported likelihood of using Wikipedia to look up information for academic purposes as the ordinal dependent variable. The students reported a higher likelihood of using Wikipedia for academic purposes at posttest than at pretest (odds ratio [95% CI] = 8.50 [5.45, 13.27], z = 9.42, p < .001). Students who completed more of the online assignments also reported a higher likelihood of using Wikipedia for academic purposes (odds ratio [95% CI] = 1.53 [1.03, 2.27], z = 2.09, p = .036). Model 2 used students’ self-reported likelihood of using Wikipedia to look up information for nonacademic purposes as the ordinal dependent variable. The students reported a higher likelihood of using Wikipedia for nonacademic purposes at posttest than at pretest (odds ratio [95% CI] = 3.32 [2.26, 4.87], z = 6.14, p < .001). Students with higher reading comprehension at pretest also reported a higher likelihood of using Wikipedia for nonacademic purposes (odds ratio [95% CI] = 1.31 [1.10, 1.57], z = 2.96, p = .003). Model 3 used students’ self-reported likelihood of recommending Wikipedia as a source of information to one of their classmates as the ordinal dependent variable. The students reported a higher likelihood of recommending Wikipedia at posttest than at pretest (odds ratio [95% CI] = 8.49 [5.40, 13.36], z = 9.26, p < .001). The other variables in the model were not significant.

Table 7 presents descriptives for the nine-item scale assessing knowledge of Wikipedia quality at pretest and posttest. The nine-item scale showed adequate internal consistency (Cronbach’s α = .78). However, two items (“Wikipedia is a biased source of information” and “Wikipedia articles express the writer’s point of view and are not neutral”) had low item-rest correlations (.26 and .10, respectively). After removing these two items, reliability increased to α = .82, and all item-rest correlations were greater than .30 (Field et al., 2012). We averaged the students’ responses to the seven-item scale to create a single Wikipedia knowledge score.

Table 7

Percentage of Students Who Indicated Each Response for Knowledge of Wikipedia Quality by Time (N = 221)

Item		Strongly disagree(1)	Disagree(2)	Neither agree nor disagree(3)	Agree(4)	Strongly agree(5)	M (SD)	Mdn
Wikipedia has guidelines for writing articles.	Pre	6.8	16.3	49.3	24.0	3.6	3.01 (0.91)	3
Wikipedia has guidelines for writing articles.	Post	4.1	6.3	42.5	36.7	10.4	3.43 (0.91)	3
Wikipedia articles are reviewed to ensure quality.	Pre	10.9	30.3	39.4	18.1	1.4	2.69 (0.94)	3
Wikipedia articles are reviewed to ensure quality.	Post	3.2	10.0	39.8	38.5	8.6	3.39 (0.90)	3
Wikipedia is a trustworthy source of information.	Pre	30.8	32.6	23.5	12.2	0.9	2.20 (1.04)	2
Wikipedia is a trustworthy source of information.	Post	5.4	9.0	37.6	39.8	8.1	3.36 (0.95)	3
Wikipedia has a good reputation as a source of information.	Pre	28.1	38.0	19.9	10.9	3.2	2.23 (1.07)	2
Wikipedia has a good reputation as a source of information.	Post	9.5	23.1	29.9	30.8	6.8	3.02 (1.09)	3
Wikipedia is a biased source of information. (reverse scored)^a	Pre	3.2	14.9	47.5	24.0	10.4	3.24 (0.94)	3
	Post	7.7	30.8	43.9	16.3	1.4	2.73 (0.87)	3
Wikipedia articles express the writer’s point of view and are not neutral. (reverse scored)^a	Pre	1.8	18.1	50.7	25.8	3.6	3.11 (0.80)	3
	Post	7.2	29.0	51.1	11.3	1.4	2.71 (0.81)	3
The information on Wikipedia is linked to sources where you can read more about the topic.	Pre	4.5	14.9	33.5	38.9	8.1	3.31 (0.98)	3
	Post	1.4	5.4	29.0	43.4	20.8	3.77 (0.89)	4
Wikipedia checks articles to make sure that the content is not plagiarized.	Pre	10.9	29.0	44.8	13.6	1.8	2.67 (0.91)	3
	Post	2.3	11.3	46.2	32.6	7.7	3.32 (0.86)	3
Wikipedia articles often contain inaccurate information. (reverse scored)	Pre	1.4	10.9	32.1	35.3	20.4	3.62 (0.97)	4
	Post	3.2	30.3	42.5	19.9	4.1	2.91 (0.89)	3

Note. For each item, each row sums to 100%, with slight deviations due to rounding.

Items dropped due to low item-rest correlation.

We ran a mixed-effects linear regression model using the lmer function of the lme4 package (Bates et al., 2014) in R (R Core Team, 2018; RStudio Team, 2016).⁵ Our dependent variable was students’ mean score on the scale assessing their knowledge of Wikipedia quality. We included an intercept-only random effect for students. Our fixed effects were time (pretest vs. posttest), number of online assignments completed (range 0–3), and pretest reading comprehension score (range 0–6). The students’ knowledge of Wikipedia quality was more accurate at posttest than at pretest, B (SE) = 0.70 (0.05), t = 13.48, p < .001. Both the number of online assignments completed, B (SE) = 0.12 (0.05), t = 2.45, p = .015, and pretest reading comprehension scores, B (SE) = 0.06 (0.02), t = 2.79, p = .006, were positively associated with accurate knowledge about Wikipedia quality.

Changes in Confidence in Fact-Checking Online COVID-19 News

Table 8 shows students’ self-reported confidence in their ability to fact-check online news about the COVID-19 pandemic. At the beginning of the semester, only 27.6% of students were mostly or very confident in their ability to fact-check online news about the COVID-19 pandemic. However, at the end of the semester, 55.2% of students reported being mostly or very confident in their fact-checking ability.

Table 8

Percentage of Students Who Indicated Each Response for Confidence in Knowing the Steps to Take to Check the Accuracy of Online News About the COVID-19 Pandemic at Pretest and Posttest (N = 221)

Item	Pretest, %	Posttest, %	V
If you wanted to check the accuracy of news that you saw online about the COVID-19 pandemic, how confident are you that you would know what steps to take?
Not at all confident (1)	7.2	1.8
A little confident (2)	26.7	11.8
Somewhat confident (3)	38.5	31.2
Mostly confident (4)	22.6	36.2
Very confident (5)	5.0	19.0
Mdn	3	4	1678.5***

Note. For each question, each column sums to 100%, with slight deviations due to rounding.

***

p < .001.

To examine the differences in self-reported confidence in fact-checking online news about the COVID-19 pandemic, we ran a mixed-effects ordinal logistic regression model with a logit link using the clmm function of the ordinal package (Christensen, 2019) in R (R Core Team, 2018; RStudio Team, 2016).⁶ The model included an intercept-only random effect for students and fixed effects of time (pretest vs. posttest), number of online assignments completed (range 0–3), and pretest reading comprehension score (range 0–6). The students’ confidence was higher at posttest than at pretest (odds ratio [95% CI] = 4.92 [3.29, 7.36], z = 7.75, p < .001). Pretest reading comprehension score was a significant positive predictor of confidence (odds ratio [95% CI] = 1.26 [1.07, 1.48], z = 2.79, p = .005), whereas the number of online assignments completed was unrelated to confidence (odds ratio [95% CI] =1.06 [0.75, 1.51], z = 0.35, p = .729).

Perceived Helpfulness of the Curriculum for Fact-Checking COVID-19 News

On the posttest, we asked the students to report on the helpfulness of the SIFT curriculum for fact-checking online news about the COVID-19 pandemic. Table 9 presents posttest descriptives for items assessing perceived helpfulness of the curriculum. As the scale showed adequate internal consistency (Cronbach’s α = .78), the students’ responses to the five items were averaged to create a single perceived helpfulness score. On average, the students reported that they agreed with statements about the curriculum’s helpfulness (M = 3.76, SD = 0.64 on a scale of 1 = strongly disagree to 5 = strongly agree).

Table 9

Percentage of Students Who Indicated Each Response for Perceived Helpfulness of the Online SIFT Curriculum for Fact-Checking Online News About the COVID-19 Pandemic at Posttest (N = 221)

Item	Strongly disagree(1)	Disagree(2)	Neither agree nor disagree(3)	Agree(4)	Strongly agree(5)	M (SD)	Mdn
Made it easier for me to fact-check news that I see online about the COVID-19 pandemic	0.0	4.1	20.4	53.4	22.2	3.94 (0.77)	4
Made me more confident in my ability to fact-check news that I see online about the COVID-19 pandemic	0.0	2.7	21.3	50.2	25.8	3.99 (0.76)	4
Made me less sure of my ability to fact-check news that I see online about the COVID-19 pandemic (reverse scored)	15.4	38.5	31.7	11.8	2.7	2.48 (0.98)	2
Made me more likely to fact-check news that I see online about the COVID-19 pandemic	1.4	5.9	20.4	51.6	20.8	3.85 (0.87)	4
Made me disinterested in fact-checking news that I see online about the COVID-19 pandemic (reverse scored)	14.5	40.3	29.9	12.2	3.2	2.49 (0.99)	2

Note. For each item, each row sums to 100%, with slight deviations due to rounding. SIFT = Stop, Investigate the source, Find better coverage, and Trace claims, quotes, and media to the original context.

Table 10 presents zero-order correlations between various outcome variables assessed at posttest: number of problems read laterally and accurately assessed, knowledge of Wikipedia quality, confidence in fact-checking COVID-19 online news, ratings of the perceived helpfulness of the online curriculum for fact-checking COVID-19 online news. All of the variables were significantly associated, with the magnitude of the correlations in the small-to-moderate range.

Table 10

Zero-Order Correlations of Outcome Variables at Posttest (N = 221)

Variable	M	SD	1	2	3	4
1. Number of problems read laterally and accurately assessed	0.91	1.18	—	Pearson’s r	Kendall’s τ	Pearson’s r
2. Knowledge of Wikipedia quality	3.34	0.65	.27***	—	Kendall’s τ	Pearson’s r
3. Confidence in fact-checking online news about COVID-19	3.59	0.99	.22***	.25***	—	Kendall’s τ
4. Perceived helpfulness of the online curriculum for fact-checking online news about COVID-19	3.76	0.64	.35***	.41***	.46***	—

Note. All the correlations were significant at Bonferroni-corrected α = .00833.

***

p < .001.

Discussion

Over the course of the Fall 2020 semester, college students in an online general education civics course completed asynchronous homework assignments to improve their information fact-checking skills. The instruction introduced the lateral reading strategies used by expert fact-checkers using the SIFT mnemonic: Stop, Investigate the source, Find better coverage, and Trace claims, quotes, and media to the original context (Caulfield, n.d; 2019). We examined whether the online SIFT curriculum increased the students’ reported use of lateral reading to fact-check online content and their use, endorsement, and knowledge of Wikipedia. In light of the COVID-19 infodemic (WHO, 2020b), we also assessed the curriculum’s impact on the students’ confidence in fact-checking online news about the COVID-19 pandemic and the perceived helpfulness of the curriculum.

In keeping with previous assessments of college students’ lateral reading (Brodsky, Brooks, Scimeca, Todorova et al., 2021; McGrew et al., 2018; Wineburg et al., 2020), we found that students rarely read laterally before receiving direct instruction (i.e., only 12.2% of students read laterally on at least one of the four pretest problems). However, as in previous in-person and online intervention studies (Breakstone et al., 2021; Brodsky, Brooks, Scimeca, Todorova et al., 2021; McGrew et al., 2019), the students made gains after receiving direct instruction in this skill, with more than half (52.5%) reading laterally on at least one problem at posttest. Similar gains were found on the more conservative measure of reading laterally and reaching an accurate trust assessment (i.e., from 9.5% of students at pretest to 46.6% at posttest). It is possible that the lower rate of reading laterally and reaching an accurate trust assessment, as compared with lateral reading alone, may be due to students reporting their initial trust assessment on the Likert-type scale and then failing to update their assessment after reading laterally. Additionally, psychological factors, such as confirmation bias (Nickerson, 1998), may have led students to draw conclusions that reinforce their existing beliefs even if they were successful in finding alternative perspectives. In the context of scientific issues, Sinatra and Lombardi (2020) argued that students need to be taught both how to evaluate the credibility of online information sources and how to reexamine their own judgments about the plausibility of the claims and evidence presented by those sources. They wrote, “We teach students to question the motivations and intentions of the information source, but what about their own motivations and intentions as they search?” (p. 128).

Overall, the current results are encouraging for the use of online, asynchronous assignments to teach college students how to read laterally. At the same time, the percentage of students who read laterally and made accurate trust assessments was lower in this study than in the in-person implementation of the DPI curriculum (46.6% vs. 61.0% of students at posttest; Brodsky, Brooks, Scimeca, Todorova et al., 2021). Additionally, as in the previous study, even after completing the online curriculum, the students did not use the lateral reading strategies consistently across the posttest problems. We interpret this finding in the context of skill acquisition and development of expertise. Acquiring a skill is initially an effortful, conscious process, which may eventually become automated but only through prolonged training and practice (Ericsson, 2006; Healy et al., 2014). For lateral reading strategies to become habitual, students need to apply what they have learned in the homework assignments to fact-check the online information they encounter in their daily lives. Unfortunately, the only information we have on the students’ application of the curriculum was the number of assignments completed, which was associated with lateral reading. It is also possible that we underestimated the extent of the students’ lateral reading, as our assessments relied on their written descriptions of how they determined the trustworthiness of the online information, and these descriptions varied in detail and clarity. Future studies should screen-record students as they engage in fact-checking information to gain insight into their use of strategies (see Brodsky et al., 2020; Wineburg & McGrew, 2017), though such an approach may not yet be viable at scale. Future iterations should also consider how best to leverage the synchronous portion of courses (whether in-person or online) to further reinforce and extend the use of lateral reading skills.

Given that the online curriculum was largely text-based, we included a brief reading comprehension test in the pretest and found that the measure predicted the use of lateral reading strategies at posttest. In light of this finding, we encourage researchers to include measures of reading ability and associated factors—for example, vocabulary knowledge, frequency and types of reading, domain-specific knowledge, and general reasoning (Freed et al., 2017)—in future fact-checking assessments and curricular interventions for college students. Including such measures will help explain the variability in students’ responses to the curriculum and identify students in need of further support.

Notably, while all SIFT instruction took place online and asynchronously to ensure fidelity of implementation, posttest performance still appeared to vary significantly by instructor. It is possible that some instructors engaged students with key concepts from the assignments during their synchronous class meetings to a greater extent than others. Alternatively, as most of the instructors taught only one class, the effect of instructor may reflect differences in the students enrolled (full-time vs. part-time, associate vs. baccalaureate) and other related factors (campus affiliation, day/evening/weekend classes). Instructor effect was also confounded with counterbalancing of problem sets, which may not have been of equivalent difficulty despite our efforts to create balanced sets of problems at pretest and posttest.

The SIFT curriculum appeared to have successfully increased students’ self-reported use, endorsement, and accuracy of Wikipedia knowledge. These gains suggest that college students, like high school students (McGrew & Byrne, 2020), benefit from direct instruction and repeated practice in using Wikipedia as a starting point for their lateral reading (McGrew et al., 2017). Future iterations of the curriculum should include additional explicit instruction on Wikipedia’s standards and policies for ensuring and maintaining the quality of articles. As in previous studies (e.g., Garrison, 2018), the students in this study reported being discouraged from using Wikipedia by their teachers. Therefore, it is also possible that they may have underreported their use of Wikipedia at pretest due to social desirability.

Similar to the younger adults surveyed by the Pew Research Center (2020c), most of the students at pretest were not very confident in their ability to fact-check news related to the COVID-19 pandemic. Their confidence in their fact-checking ability for COVID-19-related news improved from pretest to posttest and was positively associated with the perceived helpfulness of the curriculum. However, their confidence at posttest was only weakly associated with the number of problems read laterally and accurately assessed, highlighting the need for performance-based assessment of students’ information verification strategies (Hargittai et al., 2010; List & Alexander, 2018). Overall, these findings are in keeping with the disproportionately large gains in confidence relative to actual ability observed in novices after just a brief period of instruction (Sanchez & Dunning, 2018).

Conclusion

Previous research indicates that college students benefit from direct instruction on how to use lateral reading strategies to fact-check online information. The current study demonstrated that a lateral reading curriculum can be effective when delivered fully online via asynchronous assignments. From pretest to posttest, students made gains in their use of lateral reading and accurate assessment of the trustworthiness of online content and in their knowledge, use, and endorsement of Wikipedia for fact-checking purposes. The lateral reading curriculum increased students’ confidence in their ability to fact-check news about the COVID-19 pandemic. Students’ confidence was positively associated with perceived helpfulness of the online homework assignments for fact-checking COVID-19-related news but only weakly associated with their reported use of lateral reading and correct trust assessments at posttest. Teaching students to read laterally to fact-check online information is a promising first step in helping them discern trustworthy information online amidst the proliferation of false news and misinformation about the COVID-19 pandemic and other issues.

Footnotes

Acknowledgements

The instructional materials, de-identified data, analysis code, analysis output, and supplemental analyses and tables for the current study are available in the openICPSR repository at .

1.

Items assessing online news consumption and an analysis comparing responses at pretest and posttest are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

2.

Items assessing the use of online news sources to find information about the COVID-19 pandemic and an analysis comparing responses at pretest and posttest are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

3.

The keyword search terms were “wiki,” “googl,” “snope,” “politifact,” “cnn,” “national geographic,” “revers,” “search,” “researched,” “researching,” “look up,” “looked up,” “looking up,” “look for,” “looked for,” “looking for,” “look into,” “looked into,” “looking into,” “look it up,” “looked it up,” “looking it up,” “investigat,” “original,” “other websites,” “other sites,” “four moves,” “four factors,” “fact check,” “hoax,” “debunk,” and “SIFT.”

4.

The full models are available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

5.

The full model is available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

6.

The full model is available in our openICPSR repository (Brodsky, Brooks, Scimeca, Galati et al., 2021).

ORCID iDs

Jessica E. Brodsky

Patricia J. Brooks

Authors

JESSICA E. BRODSKY is a doctoral student at the Graduate Center and the College of Staten Island, City University of New York. Her research interests include assessing and fostering media literacy, algorithm literacy, and fact-checking skills in college students and adolescents.

PATRICIA J. BROOKS is a professor of psychology at the College of Staten Island, City University of New York. Her research interests are in two broad areas: (1) individual differences in language learning and development and (2) effective teaching and learning.

DONNA SCIMECA is the core program coordinator at the College of Staten Island, City University of New York and teaches in the college’s Core 100 Program.

PETER GALATI is the assistant to the core program coordinator at the College of Staten Island, City University of New York and teaches in the college’s Core 100 Program. His research interests include civics and law, and historical understandings of human engagement with sharks.

RALITSA TODOROVA is a lecturer at Lehman College, City University of New York. Her research focuses on adolescent development, primarily in educational settings.

MICHAEL CAULFIELD is the director of Blended and Networked Learning at Washington State University Vancouver. His work focuses on civic digital literacies.

References

American Democracy Project. (n.d.). Digital Polarization Initiative. American Association of State Colleges and Universities. https://www.aascu.org/AcademicAffairs/ADP/DigiPo/

Bates

Maechler

Bolker

Walker

(2014). lme4: Linear mixed-effects models using Eigen and S4. CRAN R Package, 1(7), 15–23. https://doi.org/10.18637/jss.v067.i01

Breakstone

Smith

Connors

Ortega

Kerr

Wineburg

(2021). Lateral reading: College students learn to critically evaluate internet sources in an online course. Harvard Kennedy School (HKS) Misinformation Review. https://doi.org/10.37016/mr-2020-56

Brennen

J. S.

Simon

Howard

P. N.

Nielsen

R. K.

(2020, April 7). Types, sources, and claims of COVID-19 misinformation [Fact sheet]. Reuters Institute for the Study of Journalism. https://reutersinstitute.politics.ox.ac.uk/types-sources-and-claims-covid-19-misinformation

Brodsky

J. E.

Barshaba

C. N.

Lodhi

A. K.

Brooks

P. J.

(2020, April 17). Dissociations between college students’ media literacy knowledge and fact-checking skills [Paper session]. AERA Annual Meeting, San Francisco, CA, United States. http://tinyurl.com/saedj5t (Conference canceled)

Brodsky

J. E.

Brooks

P. J.

Scimeca

Galati

Todorova

Caulfield

(2021). Associations between online instruction in lateral reading strategies and fact-checking COVID-19 news among college students (openicpsr-145661). openICPSR. https://doi.org/10.3886/E145661V1

Brodsky

J. E.

Brooks

P. J.

Scimeca

Todorova

Galati

Batson

Grosso

Matthews

Miller

Caulfield

(2021). Improving college students’ fact-checking strategies through lateral reading instruction in a general education civics course. Cognitive Research: Principles and Implications, 6, Article 23. https://doi.org/10.1186/s41235-021-00291-4

Caulfield

(n.d.). Check, please! Starter course. Notion. https://www.notion.so/ae34d043575e42828dc2964437ea4eed

Caulfield

(2017a). Four moves: Adventures in fact-checking for students. https://fourmoves.blog/

10.

Caulfield

(2017b). Web literacy for student fact-checkers . . . and other people who care about facts. Pressbooks. https://webliteracy.pressbooks.com/

11.

Caulfield

(2019, June 19). SIFT (the four moves). Hapgood. https://hapgood.us/2019/06/19/sift-the-four-moves/

12.

Chi

M. T. H.

(2006). Two approaches to the study of experts’ characteristics. In Ericsson

K. A.

Charness

Hoffman

R. R.

Feltovich

P. J.

(Eds.), The Cambridge handbook of expertise and expert performance (pp. 21–30). Cambridge University Press. https://doi.org/10.1017/CBO9780511816796.002

13.

Christensen

R. H. B.

(2019). ordinal: Regression models for ordinal data (R package Version 2019.12-10). https://CRAN.R-project.org/package=ordinal

14.

Cinelli

Quattrociocchi

Galeazzi

Valensise

C. M.

Brugnoli

Schmidt

A. L.

Zola

Zollo

Scala

(2020). The COVID-19 social media infodemic. Scientific Reports, 10(1), Article 16598. https://doi.org/10.1038/s41598-020-73510-5

15.

Ericsson

K. A.

(2006). The influence of experience and deliberate practice on the development of superior expert performance. In Ericsson

K. A.

Charness

Hoffman

R. R.

Feltovich

P. J.

(Eds.), The Cambridge handbook of expertise and expert performance (pp. 683–704). Cambridge University Press. https://doi.org/10.1017/CBO9780511816796.038

16.

Field

A. P.

Miles

Field

(2012). Discovering statistics using R. Sage.

17.

Freed

E. M.

Hamilton

S. T.

Long

D. L.

(2017). Comprehension in proficient readers: The nature of individual variation. Journal of Memory and Language, 97, 135–153. https://doi.org/10.1016/j.jml.2017.07.008

18.

Frenkel

Alba

Zhong

(2020, March 8). Surge of virus misinformation stumps Facebook and Twitter. The New York Times. https://www.nytimes.com/2020/03/08/technology/coronavirus-misinformation-social-media.html

19.

Garrison

J. C.

(2018). Instructor and peer influence on college student use and perceptions of Wikipedia. The Electronic Library, 36(2), 237–257. https://doi.org/10.1108/EL-02-2017-0034

20.

Giles

(2005). Internet encyclopaedias go head to head. Nature, 438, 900–901. https://doi.org/10.1038/438900a

21.

Gottfried

(2020, May 28). Around three-in-ten Americans are very confident they could fact-check news about COVID-19. Pew Research Center. https://www.pewresearch.org/fact-tank/2020/05/28/around-three-in-ten-americans-are-very-confident-they-could-fact-check-news-about-covid-19/

22.

Hargittai

Fullerton

Menchen-Trevino

Thomas

K. Y.

(2010). Trust online: Young adults’ evaluation of web content. International Journal of Communication, 4, 468–494. https://ijoc.org/index.php/ijoc/article/view/636

23.

Healy

A. F.

Kole

J. A.

Bourne

L. E.

Jr. (2014). Training principles to advance expertise. Frontiers in Psychology, 5, 131. https://doi.org/10.3389/fpsyg.2014.00131

24.

Izadi

(2020, April 15). There are so many coronavirus myths that even Snopes can’t keep up. The Washington Post. https://www.washingtonpost.com/media/2020/04/15/coronavirus-misinformation-snopes/

25.

Konieczny

(2016). Teaching with Wikipedia in a 21st-century classroom: Perceptions of Wikipedia and its educational benefits. Journal of the Association for Information Science and Technology, 67(7), 1523–1534. https://doi.org/10.1002/asi.23616

26.

Landis

R. J.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310

27.

List

Alexander

P. A.

(2018). Corroborating students’ self-reports of source evaluation. Behaviour & Information Technology, 37(3), 198–216. https://doi.org/10.1080/0144929X.2018.1430849

28.

Loomba

de Figueiredo

Piatek

S. J.

de Graaf

Larson

H. J.

(2021). Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01056-1

29.

McGrew

Breakstone

Ortega

Smith

Wineburg

(2018). Can students evaluate online sources? Learning from assessments of civic online reasoning, Theory & Research in Social Education, 46(2), 165–193. https://doi.org/10.1080/00933104.2017.1416320

30.

McGrew

Byrne

(2020). Teaching Wikipedia: Supporting students in lifelong learning. In Gresalfi

Horn

I. S.

(Eds.), The interdisciplinarity of the learning sciences: 14th International Conference of the Learning Sciences (Vol. 3, pp. 1565–1568). International Society of the Learning Sciences. https://repository.isls.org//handle/1/6367

31.

McGrew

Ortega

Breakstone

Wineburg

(2017). The challenge that’s bigger than fake news: Civic reasoning in a social media environment. American Educator, 41(3), 4–9. https://eric.ed.gov/?id=EJ1156387

32.

McGrew

Smith

Breakstone

Ortega

Wineburg

(2019). Improving university students’ web savvy: An intervention study. British Journal of Educational Psychology, 89(3), 485–500. https://doi.org/10.1111/bjep.12279

33.

New York State Education Department. (2018, June). The University of the State of New York Regents High School Examination in English language arts. https://www.nysedregents.org/hsela/618/reela62018-exam.pdf

34.

Nickerson

R. S.

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

35.

Pennycook

McPhetres

Zhang

J. G.

Rand

D. G.

(2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7), 770–780. https://doi.org/10.1177/0956797620939054

36.

Perin

(2013). Literacy skills among academically underprepared students. Community College Review, 41(2), 118–136. https://doi.org/10.1177/0091552113484057

37.

Pew Research Center. (2020a). American News Pathways Project data tool [Data set]. https://www.pewresearch.org/pathways-2020/covidfol/age/us_adults/

38.

Pew Research Center. (2020b). American News Pathways Project data tool [Data set]. https://www.pewresearch.org/pathways-2020/covidsm/age/us_adults/

39.

Pew Research Center. (2020c). American News Pathways Project data tool [Data set]. https://www.pewresearch.org/pathways-2020/covidinfocheck/age/us_adults/

40.

Pulido

C. M.

Villarejo-Carballido

Redondo-Sama

Gómez

(2020). COVID-19 infodemic: More retweets for science-based information on coronavirus than for false information. International Sociology, 35(4), 377–392. https://doi.org/10.1177/0268580920914755

41.

R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org

42.

RStudio Team. (2016). RStudio: Integrated development for R. RStudio Inc. http://www.rstudio.com/

43.

Sanchez

Dunning

(2018). Overconfidence among beginners: Is a little learning a dangerous thing? Journal of Personality and Social Psychology, 114(1), 10–28. https://doi.org/10.1037/pspa0000102

44.

Sinatra

G. M.

Lombardi

(2020). Evaluating sources of scientific evidence and claims in the post-truth era may require reappraising plausibility judgments. Educational Psychologist, 55(3), 120–131. https://doi.org/10.1080/00461520.2020.1730181

45.

Tsao

S. F.

Chen

Tisseverasinghe

Yang

Butt

Z. A.

(2021). What social media told us in the time of COVID-19: A scoping review. Lancet Digital Health, 3(3), e175–e194. https://doi.org/10.1016/S2589-7500(20)30315-0

46.

Wikipedia. (n.d.-a). List of policies. https://en.wikipedia.org/wiki/Wikipedia:List_of_policies

47.

Wikipedia. (n.d.-b). Protection policy. https://en.wikipedia.org/wiki/Wikipedia:Protection_policy

48.

Wineburg

Breakstone

Ziv

Smith

(2020). Educating for misunderstanding: How approaches to teaching digital literacy make students susceptible to scammers, rogues, bad actors, and hate mongers (Stanford History Education Group Working Paper No. A-21322). https://purl.stanford.edu/mf412bt5333

49.

Wineburg

McGrew

(2017). Lateral reading: Reading less and learning more when evaluating digital information (Stanford History Education Group Working Paper No. 2017-A1). https://ssrn.com/abstract=3048994

50.

World Health Organization. (2020a, September, 23). Managing the COVID-19 infodemic: Promoting healthy behaviours and mitigating the harm from misinformation and disinformation. https://www.who.int/news/item/23-09-2020-managing-the-covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-and-disinformation

51.

World Health Organization. (2020b, February, 2). Novel coronavirus (2019-nCoV): Situation Report-13. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200202-sitrep-13-ncov-v3.pd

52.

Zarocostas

(2020). How to fight an infodemic. The Lancet, 395(10225), 676. https://doi.org/10.1016/S0140-6736(20)30461-X