This paper examines whether proactive efforts to educate people about disinformation through advertisements can successfully increase skepticism towards false headlines or if such efforts do more harm than good by inadvertently increasing belief in false information. We analyze a survey experiment that employed three different advertisements that directly addressed “fake news.” We find that all advertisements were effective at increasing skepticism towards “fake news” headlines. We also find no evidence of backfire effects occurring. However, subsequent analysis using Bayesian additive regression trees (BART) finds significant heterogeneity within these treatment effects. While all advertisements were effective, each ad was effective in different ways despite common themes and content. This suggests a more complicated understanding of the counter-disinformation process and highlights BART’s utility in public opinion research.
AllenJenniferHowlandBairdMobiusMarkusRothschildDavidWattsDuncan J.. 2020. “Evaluating the Fake News Problem at the Scale of the Information Ecosystem.” Science Advances6 (14): eaay3539.
2.
AtheySusanImbensGuido. 2016. “Recursive Partitioning for Heterogeneous Causal Effects.” Proceedings of the National Academy of Sciences113 (27): 7353–7360.
3.
BerinskyAdam J.2017. “Rumors and Health Care Reform: Experiments in Political Misinformation.” British Journal of Political Science47 (2): 241–262.
4.
Beltramini (1982) - BeltraminiRichard F. “Rating Scale Selection in Advertising Research: Does it Matter Which One is Used?.” Current Issues and Research in Advertising. 1982; 5(1): 39-58.
5.
BeltraminiFRichardKennethR. Evans. 1985.. Perceived Believability of Research Results Information in Advertising. Journal of Advertising14(3): 18–31.
6.
BeltraminiRichard F.1988. Perceived Believability of Warning Label Information Presented in Cigarette Advertising. Journal of Advertising17(2): 26–32.
CadelagoChristopherHessonTed. 2018. “Why Trump Is Talking Nonstop about the Migrant Caravan.” Politico23: 2018.
9.
CappellaJoseph N.JamiesonKathleen Hall. 1994. “Broadcast Adwatch Effects: A Field Experiment.” Communication Research21 (3): 342–365.
10.
ChipmanHugh A.GeorgeEdward I.McCullochRobert E.. 2010. “BART: Bayesian Additive Regression Trees.”Annals of Applied Statistics4: 266–298.
11.
ClaytonKatherineBlairSpencerBusamJonathan A.ForstnerSamuelGlanceJohnGreenGuyKawataAnnaKovvuriAkhilaMartinJonathanMorganEvanSandhuMorganSangRachelScholz-BrightRachelWelchAustin T.WolffAndrew G.ZhouAmandaNyhanBrendan. 2020. Real Solutions for Fake News? Measuring the Effectiveness of General Warnings and Fact-Check Tags in Reducing Belief in False Stories on Social Media. Political Behavior42, 1073–1095.
12.
CohenGeoffrey L.2003. “Party Over Policy: The Dominating Impact of Group Influence on Political Beliefs.” Journal of Personality and Social Psychology85 (5): 808–822.
13.
EckerUllrich K. H.LewandowskyStephanSwireBrionyChangDarren. 2011. “Correcting False Information in Memory: Manipulating the Strength of Misinformation Encoding and its Retraction.” Psychonomic Bulletin & Review18: 570–578.
14.
EckerUllrich K. H.LewandowskyStephanChangEe PinPillaiRekha. 2014. “The Effects of Subtle Misinformation in News Headlines.” Journal of Experimental Psychology: Applied20 (4): 323–335.
15.
FreelonDeenWellsChris. 2020. Disinformation as Political Communication. Political Communication37(2): 145–156.
16.
FreezeMelanieBaumgartnerMaryBrunoPeterGundersonJacob R.OlinJoshuaRossMorgan QuinnSzafranJustine. 2020. “Fake Claims of Fake News: Political Misinformation, Warnings, and the Tainted Truth Effect.” Political Behavior43: 1433–1465.
17.
GalesicMirtaBosnjakMichael. 2009. “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.” Public Opinion Quarterly73 (2): 349–360.
18.
GerberAlan S.HuberGregory A.WashingtonEbonya. 2010. “Party Affiliation, Partisanship, and Political Beliefs: A Field Experiment.” American Political Science Review104 (4): 720–744.
19.
GilensMartin. 2001. “Political Ignorance and Collective Policy Preferences.” American Political Science Review95 (2): 379–396.
20.
GrahamMatthew H.2023. “Measuring Misperceptions?” American Political Science Review117 (1): 80–102.
21.
GrahamMatthew H.2020. “Self-awareness of Political Knowledge.” Political Behavior42 (1): 305–326.
22.
GreenDonald P.KernHolger L.. 2012. “Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees.” Public Opinion Quarterly76 (3): 491–511.
23.
GrimmerJustinMessingSolomonWestwoodSean J.. 2017. “Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods.” Political Analysis25 (4): 413–434.
24.
GrinbergNirJosephKennethFriedlandLisaSwire-ThompsonBrionyLazerDavid. 2019. “Fake News on Twitter During the 2016 US Presidential Election.” Science363 (6425): 374–378.
25.
HaglinKathryn. 2017. “The Limitations of the Backfire Effect.” Research & Politics4 (3): 1–5.
26.
HastieTrevorTibshiraniRobertFriedmanJerome H.FriedmanJerome H.. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Vol. 2. New York: Springer.
27.
HillJennifer L.2011. “Bayesian Nonparametric Modeling for Causal Inference.” Journal of Computational and Graphical Statistics20 (1): 217–240.
28.
HillJenniferLineroAntonioMurrayJared. 2020. “Bayesian Additive Regression Trees: A Review and Look Forward”. Annual Review of Statistics and Its Application.; 7: 251–278.
29.
ImaiKosukeRatkovicMarc. 2013. “Estimating Treatment Effect Heterogeneity in Randomized Program Evaluation.” Annals of Applied Statistics7 (1): 443–470.
30.
KlinePaul. Handbook of Psychological Testing. London: Routledge, 2013.
31.
KrasnoJonathan S.GreenDonald P.. 2008. “Do Televised Presidential Ads Increase Voter Turnout? Evidence from a Natural Experiment.” The Journal of Politics70 (1): 245–261.
32.
KrosnickJ. A.AlwinD. F.. 1989. “Aging and Susceptibility to Attitude Change”. Journal of Personality and Social Psychology; 7(3): 416–425.
33.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology5 (3): 213–236.
34.
KrosnickJon A.FabrigarLeandre R.. 1997. “Designing Rating Scales for Effective Measurement in Surveys.” in Survey Measurement and Process Quality, edited by Lars Lyberg et. al., Hoboken NJ: Wiley p. 141–164.
35.
KuklinskiJames H.QuirkPaul J.SchwiederDavid W.RichRobert F.. 1998. ““Just the Facts, Ma'am”: Political Facts and Public Opinion.” The Annals of the American Academy of Political and Social Science560 (1): 143–154.
36.
KünzelSören R.SekhonJasjeet S.BickelPeter J.YuBin. 2019. “Metalearners for Estimating Heterogeneous Treatment Effects Using Machine Learning.” Proceedings of the National Academy of Sciences116 (10): 4156–4165.
37.
LewandowskyStephanEckerUllrich K. H.SeifertColleen M.SchwarzNorbertCookJohn. 2012. Misinformation and its Correction: Continued Influence and Successful Debiasing. Psychological science in the public interest; 13(3): 106–131.
38.
LazerDavid M. J.BaumMatthew A.BenklerYochaiBerinskyAdam J.GreenhillKelly M.MenczerFilippoMetzgerMiriam J.NyhanBrendanPennycookGordonRothschildDavidSchudsonMichaelSlomanSteven A.SunsteinCass R.ThorsonEmily A.WattsDuncan J.ZittrainJonathan L.. 2018. “The Science of Fake News.” Science359 (6380): 1094–1096.
39.
LuskinRobert C.SoodGuaravParkYul MinBlankJoshua. 2018. “Misinformation about Misinformation? Of Headlines and Survey Design.” Unpublished manuscript.
40.
MackieDiane M.Gastardo-ConacoM. CeciliaSkellyJohn J.. 1992. “Knowledge of the Advocated Position and the Processing of In-Group and Out-Group Persuasive Messages.” Personality and Social Psychology Bulletin18 (2): 145–151.
41.
McKaySpencerTenoveChris. 2021. “Disinformation as a Threat to Deliberative Democracy.” Political Research Quarterly74 (3): 703–717.
42.
MoravecPatricia L.MinasRandall K.DennisAlan. 2019. “Fake News on Social Media: People Believe What They Want to Believe When it Makes No Sense at All.” Management Information Systems Quarterly43 (4): 1343–1360.
43.
NelsonThomas E.OxleyZoe M.ClawsonRosalee A.. 1997. “Toward a Psychology of Framing Effects”. Political Behavior; 19(3): 221–246.
44.
NyhanBrendanReiflerJason. 2010. “When Corrections Fail: The Persistence of Political Misperceptions.” Political Behavior32 (2): 303–330.
45.
O’CassAron. 2002. Political Advertising Believability and Information Source Value During Elections. Journal of Advertising; 31(1): 63–74.
46.
PennycookGordonBearAdamCollinsEvan T.RandDavid G.. 2020. “The Implied Truth Effect: Attaching Warnings to a Subset of Fake News Headlines Increases Perceived Accuracy of Headlines Without Warnings.” Management Science66 (11): 4944–4957.
47.
RubinJennifer. Evan McMullin Makes a Splash by Going After Trump and Putin. Washington Post. January 25, 2017.
48.
SarisWillem E.GallhoferIrmtraud N.. 2014. Design, Evaluation, and Analysis of Questionnaires for Survey Research. Hoboken: John Wiley & Sons.
49.
ScheufeleDietram A.KrauseNicole M.. 2019. “Science Audiences, Misinformation, and Fake News.” Proceedings of the National Academy of Sciences116 (16): 7662–7669.
50.
TourangeauRoger. 1987. “Attitude Measurement: A Cognitive Perspective.” In Social Information Processing and Survey Methodology, 149–162. New York, NY: Springer.
51.
TourangeauRogerRipsLance J.RasinskiKenneth. 2012. The Psychology of the Survey Response. Cambridge: Cambridge University Press.
52.
VehovarVasjaToepoelVeraSteinmetzStephanie. 2016. Non-probability Sampling. In The Sage Handbook of Survey Methods, 149–162. Los Angeles: Sage.
53.
WagerStefanAtheySusan. 2018. “Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests.” Journal of the American Statistical Association113 (523): 1228–1242.
54.
WoodThomasPorterEthan. 2019. “The Elusive Backfire Effect: Mass Attitudes’ Steadfast Factual Adherence.” Political Behavior41: 135–163.
55.
YeagerDavid S.KrosnickJon A.ChangLinChiatJavitzHarold S.LevenduskyMatthew S.SimpserAlbertoWangRui. 2011. “Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-probability Samples.” Public Opinion Quarterly75 (4): 709–747.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
