As generative artificial intelligence (AI) becomes increasingly integrated into news production, concerns about the credibility of synthetic news content are intensifying. While some studies report that audiences find AI-generated news less credible than human-written content, others reveal mixed or even reversed findings. This meta-analysis synthesizes findings across 31 studies (41 effect sizes) to assess how the source label (AI vs. human) influences news users’ evaluations of source credibility and message credibility. Results show a small but statistically significant penalty for AI-labeled (vs. human-labeled) news on both credibility measures. To better understand variability across studies, we put forth three moderators: (1) the value-ladenness of the topic, (2) participants’ source orientation, and (3) actual authorship of the text. Although the first two moderators showed descriptive trends consistent with expectations, only the third reached statistical significance, with effects being more pronounced when the news articles were actually written by human only (vs. AI). These findings contribute to ongoing debates about the role of automation in journalism by clarifying when and why audiences mind the source.
SunG, ZhouYH. AI in healthcare: Navigating opportunities and challenges in digital communication. Front Digit Health2023;5:1291132.
2.
RazaMM, VenkateshKP, KvedarJC. Generative AI and large language models in health care: Pathways to implementation. NPJ Digit Med2024;7(1):62; doi: 10.1038/s41746-023-00988-4
3.
KumarK. The impact of generative AI on financial forecasting and decision-making. Trans Recent Dev Ind IoT2024;16(16).
4.
OyewoleAT, AdeoyeOB, AddyWA, et al.Automating financial reporting with natural language processing: A review and case analysis. World J Adv Res Rev2024;21(3):575–589.
GraefeA, HaimM, HaarmannB, et al.Readers’ perception of computer-generated news: Credibility, expertise, and readability. Journalism2018;19(5):595–610.
12.
LiuB, WeiL. Machine authorship in situ: Effect of news organization and news genre on news credibility. Digit Journal2019;7(5):635–657.
13.
TandocECJr, YaoLJ, WuS. Man vs. machine? The impact of algorithm authorship on news credibility. Digit Journal2020;8(4):548–562.
14.
WaddellTF. A robot wrote this? How perceived machine authorship affects news credibility. Digit Journal2018;6(2):236–255.
15.
McCroskeyJC. An Introduction to Rhetorical Communication. 7th ed. Allyn and Bacon: Needham Heights, MA; 1997.
16.
AppelmanA, SundarSS. Measuring message credibility: Construction and validation of an exclusive scale. J Mass Commun Q2016;93(1):59–79; doi: 10.1177/1077699015606057
17.
JiaC, JohnsonTJ. Source credibility matters: Does automated journalism inspire selective exposure? Int J Commun2021;15:22.
18.
JangW, ChunJW, KimS, et al.The effects of anthropomorphism on how people evaluate algorithm-written news. Digit Journal2023;11(1):103–124.
19.
KimY, LeeH. Towards a sustainable news business: Understanding readers’ perceptions of algorithm-generated news based on cultural conditioning. Sustainability2021;13(7):3728.
20.
WuY. Is automated journalistic writing less biased? An experimental test of auto-written and human-written news stories. Journal Pract2020;14(8):1008–1028.
21.
LaniusC, WeberR, MacKenzieWI.Jr., Use of bot and content flags to limit the spread of misinformation among social networks: A behavior and attitude survey. Soc Netw Anal Min2021;11(1):32.
22.
LongoniC, FradkinA, CianL, et al.News from generative artificial intelligence is believed less. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022; pp. 97–106.
23.
WangS, KimKJ. Content moderation on social media: Does it matter who and why moderates hate speech? Cyberpsychol Behav Soc Netw2023;26(7):527–534.
24.
LeeEJ. Minding the source: Toward an integrative theory of human–machine communication. Hum Commun Res2024;50(2):184–193.
25.
ChaikenS. Heuristic versus systematic information processing and the use of source versus message cues in persuasion. J Pers Soc Psychol1980;39(5):752–766; doi: 10.1037//0022-3514.39.5.752
26.
ChenS, ChaikenS. The heuristic-systematic model in its broader context. In: ChaikenS, TropeY., eds. Dual-Process Theories in Social Psychology. Guilford Press: New York, NY; 1999:73–96.
27.
NassC, SteuerJ, TauberER. Computers are social actors. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1994:72–78; doi: 10.1145/191666.191703
28.
LeeEJ, KimHS, SuhYJ, et al.Something’s fishy about it: How opinion congeniality and explainability affect motivated attribution to artificial intelligence versus human comment moderators. Cyberpsychol Behav Soc Netw2022;25(8):496–503.
29.
WalterN, CohenJ, HolbertRL, et al.Fact-checking: A meta-analysis of what works and for whom. Polit Commun2020;37(3):350–375.
30.
BurgoonJK, BonitoJA, LowryPB, et al.Application of expectancy violations theory to communication with and judgments about embodied agents during a decision-making task. Int J Hum-Comput Stud2016;91:24–36.
31.
PettyRE, CacioppoJT. Message elaboration versus peripheral cues. In: PettyRE, CacioppoJT. Communication and Persuasion: Central and Peripheral Routes to Attitude Change. Springer: New York (NY); 1986. pp. 141–172.
32.
PornpitakpanC. The persuasiveness of source credibility: A critical review of five decades’ evidence. J Applied Social Pyschol2004;34(2):243–281.
33.
LeeEJ. Authenticity model of (mass-oriented) computer-mediated communication: Conceptual explorations and testable propositions. J Comput Mediat Commun2020;25(1):60–73.
34.
McGuireWJ. Personality and attitude change: an information-processing theory. In: GreenwaldAG, BrockTC, OstromTM., editors. Psychological foundations of attitudes. Academic Press: New York, NY; 1968. pp. 171–196.
35.
MolinaMD, SundarSS. When AI moderates online content: Effects of human collaboration and interactive transparency on user trust. J Comput Mediat Commun2022;27(4):zmac010.
36.
HenestrosaAL, GrevingH, KimmerleJ. Automated journalism: The effects of AI authorship and evaluative information on the perception of a science journalism article. Comput Hum Behav2023;138:107445.
37.
JiaC, LiuR. Algorithmic or human source? Examining relative hostile media effect with a transformer-based framework. MaC2021;9(4):170–181.
38.
HerboldS, Hautli-JaniszA, HeuerU, et al.A large-scale comparison of human-written versus ChatGPT-generated essays. Sci Rep2023;13(1):18617.
39.
BorensteinM, HedgesL, HigginsJ, et al.Comprehensive Meta-Analysis Version 4. Biostat: Englewood, NJ; 2022.
40.
BorensteinM, HedgesLV, HigginsJP, et al.Introduction to Meta-Analysis. John Wiley & Sons: Chichester, UK; 2009.
41.
DuvalS, TweedieR. A nonparametric “trim and fill” method of accounting for publication bias in meta‐analysis. J Am Stat Assoc2000;95(449):89–98.
42.
ZhengY, ZhongB, YangF. When algorithms meet journalism: The user perception to automated news in a cross-cultural context. Comput Hum Behav2018;86:266–275.
43.
SundarSS, KimJ. (2019). Machine heuristic: When we trust computers more than humans with our personal information. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Paper 538, p. 9). ACM; doi: 10.1145/3290605.3300768
