Disinformation presented in multiple modalities (textual, visual, and auditory modes; multimodal disinformation) has become a serious concern. This study examines how disinformation, portrayed using an image or video format, may be more powerful than text-only disinformation. In particular, we examined the impact on affective mechanisms, as well as the moderating role of perceived issue relevance. Through an online experiment with modality conditions and a control group (text-only disinformation vs image-plus-text disinformation vs video-plus-text disinformation vs control; N = 413), results indicate that while anxiety is a critical mechanism that explains the overall effects of disinformation on misperceptions, video-plus-text disinformation turns out to increase misperceptions directly or indirectly through anxiety. Video-plus-text disinformation (vs control) showed a significant interaction with perceived issue relevance; that said, the difference in anxiety decreased between those with low and high perceived issue relevance in the video-plus-text disinformation. Implications are discussed in light of the realism heuristic, affect heuristic, and modality-biased processing in explaining the emotional impact of multimodal disinformation.
AhnSJ (2015) Incorporating immersive virtual environments in health promotion campaigns: a construal level theory approach. Health Communication30(6): 545–556.
2.
BagoBRosenzweigLRandD (2021) Emotion may predict susceptibility to disinformation but emotion regulation does not seem to help. PsyArXiv Preprints. DOI: 10.31234/osf.io/hm9vu.
BergkvistL (2015) Appropriate use of single-item measures is here to stay. Marketing Letters26(3): 245–255.
5.
BergkvistLRossiterJR (2007) The predictive validity of multiple-item versus single-item measures of the same constructs. Journal of Marketing Research44(2): 175–184.
6.
BodeLVragaEK (2015) In related news, that was wrong: the correction of misinformation through related stories functionality in social media. Journal of Communication65: 619–638.
7.
BrennenJSSimonFMNielsenRK (2021) Beyond (mis) representation: visuals in COVID-19 disinformation. The International Journal of Press/Politics26(1): 277–299.
8.
BroniatowskiDAJamisonAMQiS, et al. (2018) Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. American Journal of Public Health108(10): 1378–1384.
9.
ChenLZhangYYoungR, et al. (2021) Effects of vaccine-related conspiracy theories on Chinese young adults’ perceptions of the HPV vaccine: an experimental study. Health Communication36(11): 1343–1353.
10.
ChmielewskiMKuckerSC (2020) An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science11(4): 464–473.
11.
ChouWYSBudenzA (2020) Considering emotion in COVID-19 vaccine communication: addressing vaccine hesitancy and fostering vaccine confidence. Health Communication35(14): 1718–1722.
12.
DanVParisBDonovanJ, et al. (2021) Visual mis-and disinformation, social media, and democracy. Journalism & Mass Communication Quarterly98(3): 641–664.
EntmanRM (1993) Framing: Toward clarification of a fractured paradigm. Journal of Communication43(4): 51–58.
15.
FinucaneMLAlhakamiASlovicP, et al. (2000) The affect heuristic in judgments of risks and benefits. Journal of Behavioral Decision Making13(1): 1–17.
16.
FisherJTHoppFRWeberR (2019) Modality-specific effects of perceptual load in multimedia processing. Media and Communication7(4): 149–165.
17.
FongARoozenbeekJGoldwertD, et al. (2021) The language of conspiracy: a psychological analysis of speech used by conspiracy theorists and their followers on Twitter. Group Processes & Intergroup Relations24(4): 606–623.
GandhiCKPatelJZhanX (2020) Trend of influenza vaccine Facebook posts in last 4 years: a content analysis. American Journal of Infection Control48(4): 361–367.
20.
GeiseSBadenC (2015) Putting the image back into the frame: modeling the linkage between visual communication and frame-processing theory. Communication Theory25(1): 46–69.
21.
GermaniFBiller-AndornoN (2021) The anti-vaccination infodemic on social media: a behavioral analysis. PLoS ONE16(3): e0247642.
22.
GertsDShelleyCDParikhN, et al. (2021) “Thought I’d share first” and other conspiracy theory Tweets from the COVID-19 infodemic: exploratory study. JMIR Public Health and Surveillance7(4): e26527.
23.
GuidryJPAustinLLO’DonnellNH, et al. (2020) Tweeting the# flushot: beliefs, barriers, and threats during different periods of the 2018 to 2019 flu season. Journal of Primary Care & Community Health11: 2150132720932722.
24.
HameleersMPowellTEVan Der MeerTG, et al. (2020) A picture paints a thousand lies? The effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication37(2): 281–301.
25.
HarveyAMThompsonSLacA, et al. (2019) Fear and derision: a quantitative content analysis of provaccine and antivaccine Internet Memes. Health Education & Behavior46(6): 1012–1023.
26.
HayesAF (2017) Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-based Approach, 2nd ed.New York: Guilford Press.
27.
KennedyRCliffordSBurleighT, et al. (2020) The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods8(4): 614–629.
28.
KimHSeoYYoonHJ, et al. (2021) The effects of user comment valence of Facebook health messages on intention to receive the flu vaccine: the role of pre-existing attitude towards the flu vaccine and psychological reactance. International Journal of Advertising40(7): 1187–1208.
LangA (2000) The limited capacity model of mediated message processing. Journal of Communication50(1): 46–70.
31.
LazarusRS (1991) Emotion and Adaptation. Oxford: Oxford University Press.
32.
LearyMRTateEBAdamsCE, et al. (2007) Self-compassion and reactions to unpleasant self-relevant events: the implications of treating oneself kindly. Journal of Personality and Social Psychology92(5): 887–904.
33.
LeeEJKimYW (2016) Effects of infographics on news elaboration, acquisition, and evaluation: prior knowledge and issue involvement as moderators. New Media & Society18(8): 1579–1598.
34.
LeeJ (2022) When web add-on correction comes with fear-arousing disinformation in public health crisis: focusing on the role of risk perception in belief in disinformation. Journal of Applied Communication Research50(1): 70–90.
35.
LeeJShinSY (2022) Something that they never said: multimodal disinformation and source vividness in understanding the power of AI-enabled deepfake news. Media Psychology 25(4): 531–546.
36.
LeeJOttTDeavoursD (2021) Combating misinformation in risk: emotional appeal in false beliefs. In: LuttrellRXiaoLGlassJ (eds) Democracy in the Disinformation Age. New York: Routledge, pp. 165–181.
37.
LuY (2019) Incidental exposure to political disagreement on Facebook and corrective participation: unraveling the effects of emotional responses and issue relevance. International Journal of Communication13: 874–896.
38.
LyuJCLe HanELuliGK (2021) COVID-19 vaccine–related discussion on Twitter: topic modeling and sentiment analysis. Journal of Medical Internet Research23(6): e24435.
39.
MartelCPennycookGRandDG (2020) Reliance on emotion promotes belief in fake news. Cognitive Research: Principles and Implications5: 47.
40.
MessarisPAbrahamL (2001) The role of images in framing news stories. In: ReeseSDGandyOH JrGrantAE (eds) Framing Public Life. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 215–226.
41.
MetzgerMJFlanaginAJMeddersRB (2010) Social and heuristic approaches to credibility evaluation online. Journal of Communication60(3): 413–439.
42.
MolinaMDSundarSS (2019) Technological affordances can promote misinformation: what journalists should watch out for when relying on online tools and social media. In: KatzJEMaysKK (eds) Journalism and Truth in an Age of Social Media. Oxford: Oxford University Press, pp. 182–197.
43.
NyhanBReiflerJ (2012) Misinformation and fact-checking: Research findings from social science. Media Policy Initiative Research Paper. New American Foundation.
44.
O’KeefeDJ (2003) Message properties, mediating states, and manipulation checks: claims, evidence, and data analysis in experimental persuasive message effects research. Communication Theory13(3): 251–274.
45.
PanWLiuDFangJ (2021) An examination of factors contributing to the acceptance of online health misinformation. Frontiers in Psychology12:630268.
PaschenJ (2019) Investigating the emotional appeal of disinformation using artificial intelligence and human contributions. Journal of Product & Brand Management29(2): 223–233.
48.
PennycookGEpsteinZMoslehM, et al. (2021) Shifting attention to accuracy can reduce misinformation online. Nature592(7855): 590–595.
49.
PettyRECacioppoJT (1979) Issue involvement can increase or decrease persuasion by enhancing message-relevant cognitive responses. Journal of Personality and Social Psychology37(10): 1915–1926.
50.
PettyRECacioppoJT (1986) The elaboration likelihood model of persuasion. In: PettyRECacioppoJT (eds) Communication and Persuasion. New York: Springer, pp. 1–24.
51.
PowellTEBoomgaardenHGDe SwertK, et al. (2015) A clearer picture: the contribution of visuals and text to framing effects. Journal of Communication65(6): 997–1017.
52.
PowellTEBoomgaardenHGDe SwertK, et al. (2019) Framing fast and slow: a dual processing account of multimodal framing effects. Media Psychology22(4): 572–600.
53.
RobertsonCTMourãoRR (2020) Faking alternative journalism? An analysis of self-presentations of “fake news” sites. Digital Journalism8(8): 1011–1029.
54.
SallamMDababsehDYaseenA, et al. (2020) Conspiracy beliefs are associated with lower knowledge and higher anxiety levels regarding COVID-19 among students at the University of Jordan. International Journal of Environmental Research and Public Health17(14): 4915.
SlovicPFinucaneMLPetersE, et al. (2007) The affect heuristic. European Journal of Operational Research177(3): 1333–1352.
57.
SmithCAEllsworthPC (1987) Patterns of appraisal and emotion related to taking an exam. Journal of Personality and Social Psychology52(3): 475–488.
58.
SunYChiaSCLuF, et al. (2022) The battle is on: factors that motivate people to combat anti-vaccine misinformation. Health Communication37(3): 327–336.
59.
SundarSS (2008) The MAIN model: a heuristic approach to understanding technology effects on credibility. In: MetzgerMJFlanaginAJ (eds) Digital Media, Youth, and Credibility. Cambridge, MA: MIT Press, pp. 72–100.
60.
SundarSSMolinaMDChoE (2021) Seeing is believing: is video modality more powerful in spreading disinformation via online messaging apps?Journal of Computer-Mediated Communication26(6): 301–319.
61.
TandocECJrLimDLingR (2020) Diffusion of disinformation: how social media users respond to disinformation and why. Journalism21(3): 381–398.
62.
TörnbergP (2018) Echo chambers and viral misinformation: modeling fake news as complex contagion. PLoS ONE13(9): e0203958.
63.
Van ProoijenJW (2018) The Psychology of Conspiracy Theories. Abingdon: Routledge.
64.
VosoughiSRoyDAralS (2018) The spread of true and false news online. Science359(6380): 1146–1151.
65.
YadavAPhillipsMMLundebergMA, et al. (2011) If a picture is worth a thousand words is video worth a million? Differences in affective and cognitive processing of video and text cases. Journal of Computing in Higher Education23(1): 15–37.
