Given growing interest in the potential importance of news literacy around the world, a theoretically grounded and empirically validated measure of news literacy is essential. Building on existing theory, we developed and validated a 15-item true/false measure of news literacy knowledge. This measure comprehensively operationalizes the five C’s of news literacy—context, creation, content, circulation, and consumption—in a concise, adaptable, knowledge-based format. Using item response theory and differential item functioning analysis, we followed a three-survey process with representative U.S. samples, developing and assessing 80 true/false items in Study 1 (N = 1,502) to reduce to 43 items in Study 2 (N = 1,273). The final reduced set of 15 items was evaluated and validated in Study 3 (N = 681) along with related measures of civics and current events knowledge, which were positively predicted by the news literacy knowledge measure. While this measure is designed and tested in the U.S. context, our process of operationalizing these complicated concepts and the novel true/false format facilitates its applicability to those interested in studying news literacy around the globe.
AjzenI. (1985). From intentions to actions: A theory of planned behavior. In KuhlJ.BeckmannJ. (Eds.), Action control: From cognition to behavior (pp. 11–39). Springer. https://doi.org/10.1007/978-3-642-69746-3_2
2.
AmazeenM. A.BucyE. P. (2019). Conferring resistance to digital disinformation: The inoculating influence of procedural news knowledge. Journal of Broadcasting & Electronic Media, 63(3), 415–432. https://doi.org/10.1080/08838151.2019.1653101
AshleyS.CraftS.MakslA.TullyM.VragaE. K. (2023). Can news literacy help reduce belief in COVID misinformation?Mass Communication and Society, 26(4), 695–719. https://doi.org/10.1080/15205436.2022.2137040
6.
AshleyS.MakslA.CraftS. (2013). Developing a news media literacy scale. Journalism & Mass Communication Educator, 68(1), 7–21. https://doi.org/10.1177/1077695812469802
7.
AshleyS.MakslA.CraftS. (2017). News media literacy and political engagement: What’s the connection?Journal of Media Literacy Education, 9(1), 79–98. https://doi.org/10.23860/JMLE-2017-9-1-6
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, Statistical Methodology, 57, 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
10.
BoedekerP. (2021). Nonlinear mixed-effects growth models: A tutorial using ‘saemix’ in R. Methodology, 17, 250–270. https://doi.org/10.5964/meth.7061
11.
BooneW. J.StaverJ. R. (2020). Principal Component Analysis of Residuals (PCAR). In BooneW. J.StaverJ. R. (Eds.), Advances in Rasch analyses in the human sciences (pp. 13–24). Springer International Publishing. https://doi.org/10.1007/978- 3-030-43420-5_2
12.
ChanM.LeeF. L.ChenH. T. (2021). Examining the roles of multi-platform social media news use, engagement, and connections with news organizations and journalists on news literacy: A comparison of seven democracies. Digital Journalism, 9(5), 571–588. https://doi.org/10.1080/21670811.2021.1890168
13.
ChoE.KimS. (2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207–230. https://doi.org/10.1177/1094428114555994
14.
CraftS.AshleyS.MakslA. (2016). Elements of news literacy: A focus group study of how teenagers define news and why they consume it. Electronic News, 10(3), 143–160. https://doi.org/10.1177/193124311665671
15.
CraftS.AshleyS.MakslA. (2017). News media literacy and conspiracy theory endorsement. Communication and the Public, 2(4), 388–401. https://doi.org/10.1177/2057047317725539
16.
Cunliffe-JonesP.GayeS.GichungeW.OnumahC.PretoriusC.SchiffrinA. (2021). The state of media literacy in Sub-Saharan Africa 2020 and a theory of misinformation literacy. In Cunliffe-JonesP.DiagneA.FinlayA.GayeS.GichungeW.OnumahC.PretoriusC.SchiffrinA. (Eds.), Misinformation policy in Sub-Saharan Africa (pp. 5–96). University of Westminster Press. https://doi.org/10.16997/book53.a
17.
DogruelL.MasurP.JoeckelS. (2022). Development and validation of an algorithm literacy scale for internet users. Communication Methods and Measures, 16(2), 115–133. https://doi.org/10.1080/19312458.2021.1968361
18.
DrasgowF.LissakR. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied Psychology, 68, 363–373. https://doi.org/10.1037/0021-9010.68.3.363
19.
EbelR. L.FrisbieD. A. (1991). Essentials of educational measurements (5th ed.). Prentice Hall.
20.
EdgerlyS.VragaE. K. (2019). News, entertainment, or both? Exploring audience perceptions of media genre in a hybrid media environment. Journalism, 20(6), 807–826.
21.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates Publishers.
GeersS.BoukesM.MoellerJ. (2020). Bridging the gap? The impact of a media literacy educational intervention on news media literacy, political knowledge, political efficacy among lower-educated youth. Journal of Media Literacy Education, 12(2), 41–53. https://doi.org/10.23860/JMLE-2020-12-2-4
HameleersM. (2022). Separating truth from lies: Comparing the effects of news media literacy interventions and fact-checkers in response to political misinformation in the US and Netherlands. Information, Communication & Society, 25(1), 110–126. https://doi.org/10.1080/1369118X.2020.1764603
26.
HayesA. F.CouttsJ. J. (2020). Use Omega rather than Cronbach’s Alpha for estimating reliability. But. . ., Communication Methods and Measures, 14(1), 1–24. https://doi.org/10.1080/19312458.2020.1718629
27.
HermannE.MorganM.ShanahanJ. (2021). Television, continuity, and change: A meta-analysis of five decades of cultivation research. Journal of Communication, 71(4), 515–544. https://doi.org/10.1093/joc/jqab014
28.
HobbsR. (2017). Measuring the digital and media literacy competencies of children and teens. In BlumbergF. C.BrooksP. J. (Eds.), Cognitive development in digital contexts (pp. 253–274). Academic Press. https://doi.org/10.1016/B978-0-12-809481-5.00013-4
29.
HobbsR. (2021). Media literacy in action: Questioning the media. Rowman & Littlefield Publishers.
30.
HobbsR.FrostR. (2003). Measuring the acquisition of media literacy skills. Reading Research Quarterly, 38, 330–352. https://doi.org/10.1598/RRQ.38.3.2
31.
HuL.-T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
JenkinsJ.NielsenR. K. (2020). Proximity, public service, and popularity: A comparative study of how local journalists view quality news. Journalism Studies, 21(2), 236–253. https://doi.org/10.1080/1461670X.2019.1636704
JesseeS. A. (2015). “Don’t Know” responses, personality, and the measurement of political knowledge. Political Science Research and Methods, 5(4), 711–731. https://doi.org/10.1017/psrm.2015.23
36.
JohnsonN. R.PaalK.WaggonerE.BleierK. (2021). Scales for assessing news literacy education in the digital era. Journalism & Mass Communication Educator, 76(2), 156–175. https://doi.org/10.1177/1077695820930980
Jones-JangS. M.MortensenT.LiuJ. (2021). Does media literacy help identification of fake news? Information literacy helps, but other literacies don’t. American Behavioral Scientist, 65(2), 371–388. https://doi.org/10.1177/0002764219869406
39.
KahneJ.BowyerB. (2019). Can media literacy education increase digital engagement in politics?Learning, Media and Technology, 44(2), 211–224. https://doi.org/10.1080/17439884.2019.1601108
40.
KajimotoM. (2016). Developing news literacy curricula in the age of social media in Hong Kong, Vietnam and Myanmar. Journalism Education, 5(1), 136–154.
KimJ.OshimaT. C. (2013). Effect of multiple testing adjustment in differential item functioning detection. Educational and Psychological Measurement, 73, 458–470. https://doi.org/10.1177/0013164412467033
KurzM.RosendahlJ.RodeckJ.MuehleckJ.BergerU. (2002). School-based interventions improve body image and media literacy in youth: A systematic review and meta-analysis. Journal of Prevention, 43, 5–23. doi.org/10.1007/s10935-021-00660-1
45.
LinacreJ. M. (2002). What do infit and outfit, mean-square and standardized mean?Rasch Measurement Transactions, 16(2), Article 878. https://www.rasch.org/rmt/rmt162f.htm
46.
LopesP.CostaP.AraujoL.ÁvilaP. (2018). Measuring media and information literacy skills: Construction of a test. Communications, 43(4), 508–534. https://doi.org/10.1515/commun-2017-0051
47.
LordF. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203056615
48.
LuC.HuB.BaoM.WangC.BiC.JuX. (2024). Can media literacy intervention improve fake news credibility assessment? A meta-analysis. Cyberpsychology, Behavior, and Social Networking27(4), 240–252. https://doi.org/10.1089/cyber.2023.0324
49.
LyonsB. A.MontgomeryJ. M.GuessA. M.NyhanB.ReiflerJ. (2021). Overconfidence in news judgments is associated with false news susceptibility. Proceedings of the National Academy of Sciences, 118(23), Article e2019527118. https://doi.org/10.1073/pnas.2019527118
McGrewS.BreakstoneJ.OrtegaT.SmithM.WineburgS. (2018). Can students evaluate online sources? Learning from assessments of civic online reasoning. Theory & Research in Social Education, 46(2), 165–193. https://doi.org/10.1080/00933104.2017.1416320
53.
McLeodJ. M.PanZ. (2005). Concept explication and theory construction. In DunwoodyS.BeckerL. B.McCleodD. M.KosickiG. M. (Eds.), The evolution of key mass communication concepts honoring Jack M. McLeod (pp. 13–76). Hampton.
54.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (pp. 13–103). Macmillan.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
57.
PalanS.SchitterC. (2018). Prolific.ac – A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27. https://doi.org/10.1016/j.jbef.2017.12.004
58.
PeerE.BrandimarteL.SamatS.AcquistiA. (2018). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research, Journal of Experimental Social Psychology, 70, 153–163. https://doi.org/10.1016/j.jesp.2017.01.006.
PotterW. J. (2022). Analyzing the distinction between protectionism and empowerment as perspectives on media literacy education. Journal of Media Literacy Education, 14(3), 119–131. https://doi.org/10.23860/JMLE-2022-14-3-10
61.
PrimackB. A.GoldM. A.SwitzerG. E.HobbsR.LandS. R.FineM. J. (2006). Development and validation of a smoking media literacy scale for adolescents. Archives of Pediatrics & Adolescent Medicine, 160(4), 369–374. https://doi.org/10.1001/archpedi.160.4.369
62.
PuringtonA.MasurP. K.BazarovaN.ZouE. W.WhitlockJ. (2023). The youth social media literacy inventory: Development and validation using item response theory in the US. Journal of Children and Media, 17(4), 467–487. https://doi.org/10.1080/17482798.2023.2230493
63.
RosenbaumJ.BeentjesJ. W. J.KonigR. P. (2008). Mapping media literacy: Key concepts and future directions. Annals of the International Communication Association, 32(1), 313–353. https://doi.org/10.1080/23808985.2008.11679081
64.
SchulzA.FletcherR.NielsenR. K. (2024). The role of news media knowledge for how people use social media for news in five countries. New Media & Society, 26(7), 4056–4077. https://doi.org/10.1177/14614448221108957
SwartJ. (2023). Tactics of news literacy: How young people access, evaluate, and engage with news on social media. New Media & Society, 25(3), 505–521. https://doi.org/10.1177/14614448211011447
69.
TamboerS. L.DaalmansS.MolenaarI.BosseT.KleemansM. (2022). How to increase news literate behaviors via interventions: Eight guidelines by early adolescents. Journalism & Mass Communication Educator, 77(4), 363–375. https://doi.org/10.1177/10776958221096198
70.
TamboerS. L.KleemansM.DaalmansS. (2020). ‘We are a neeeew generation’: Early adolescents’ views on news and news literacy. Journalism, 23(4), 806–822. https://doi.org/10.1177/1464884920924527
71.
TamboerS. L.KleemansM.MolenaarI.BosseT. (2023). Developing a model of news literacy in early adolescents: A survey study. Mass Communication and Society, 26(1), 74–98. https://doi.org/10.1080/15205436.2022.2048027
72.
ThissenD.SteinbergL.GerrardM. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99(1), 118–128. https://doi.org/10.1037/0033-2909.99.1.118
ValenzuelaS.MuñizC.SantosM. (2024). Social media and belief in misinformation in Mexico: A case of maximal panic, minimal effects?The International Journal of Press/Politics, 29(3), 667–688. https://doi.org/10.1177/19401612221088988
75.
VragaE. K.TullyM. (2021) News literacy, social media behaviors, and skepticism toward information on social media. Information, Communication & Society, 24, 150–166. https://doi.org/10.1080/1369118X.2019.1637445
76.
VragaE. K.TullyM.KotcherJ. E.SmithsonA.-B.Broeckelman-PostM. (2015). A multi-dimensional approach to measuring news media literacy. Journal of Media Literacy Education, 7(3), 41–53. https://doi.org/10.23860/jmle-7-3-4
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
