Restricted accessResearch articleFirst published online 2026-1
Behind the black box: The moderating role of the machine heuristic on the effect of transparency information about automated journalism on hostile media bias perception
Facing historically low levels of public trust, journalists had been increasingly interested in the potential of artificial intelligence to produce news content. Some have suggested that Automated Journalism (AJ) may reduce Hostile Media Biases (HMB), where partisans perceive balanced articles as slanted against their side. However, empirical evidence for the hypothesis remains limited and inconclusive. In this study, we examine whether the effectiveness of AJ at reducing HMB perceptions could be enhanced by disclosure of transparency information about how the algorithm works. We conducted an online experiment (N = 264 US adults) in which participants were randomly assigned to read a balanced news article about gun control written by different authors (AJ, AJ + transparency information, journalist, student, no author). Our findings indicate that AJ transparency, on average, did not significantly reduce HMB compared to AJ along. A significant interaction effect was identified: participants who strongly endorsed the machine heuristic were less likely to perceive the content in the AJ transparency condition, but not that of other conditions, as biased. Theoretical and practical implications are discussed.
AbbeyJDMeloyMG (2017) Attention by design: using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management53–56(1): 63–70. DOI: 10.1016/j.jom.2017.06.001.
2.
AndersonC (2013) Towards a sociology of computational and algorithmic journalism. New Media & Society15(7): 1005–1021. DOI: 10.1177/1461444812465137.
3.
ArpanLMRaneyAA (2003) An experimental investigation of news source and the hostile media effect. Journalism & Mass Communication Quarterly80(2): 265–281. DOI: 10.1177/107769900308000203.
4.
BoykoffMTBoykoffJM (2004) Balance as bias: global warming and the US prestige press. Global Environmental Change14(2): 125–136.
CarlsonMRobinsonSLewisSC (2021) News after Trump: Journalism’s Crisis of Relevance in a Changed Media Culture. Oxford: Oxford University Press.
7.
ChadhaKKoliskaM (2015) Newsrooms and transparency in the digital age. Journalism Practice9(2): 215–229. DOI: 10.1080/17512786.2014.924737.
8.
ChaikenSLibermanAEaglyAH (1989) Heuristic and systematic information processing within and beyond the persuasion context. In: UlemanJSBarghJA (eds) Unintended Thought. New York: Guilford, 212–252.
9.
ClerwallC (2014) Enter the robot journalist: users’ perceptions of automated content. Journalism Practice8(5): 519–531. DOI: 10.1080/17512786.2014.883116.
10.
CloudyJBanksJBowmanND (2021) The str(AI)ght scoop: artificial intelligence cues reduce perceptions of hostile media bias. Digital Journalism11: 1577–1596. DOI: 10.1080/21670811.2021.1969974.
11.
CraftSHeimK (2008) Transparency in journalism: meanings, merits, and risks. In: The Handbook of Mass Media Ethics. Oxfordshire: Routledge.
12.
CurryALStroudNJ (2021) The effects of journalistic transparency on credibility assessments and engagement intentions. Journalism22(4): 901–918. DOI: 10.1177/1464884919850387.
13.
de Fine LichtKde Fine LichtJ (2020) Artificial intelligence, transparency, and public decision-making. AI & Society35(4): 917–926. DOI: 10.1007/s00146-020-00960-w.
14.
DiakopoulosNKoliskaM (2017) Algorithmic transparency in the news media. Digital Journalism5(7): 809–828. DOI: 10.1080/21670811.2016.1208053.
15.
GarfinkelSMatthewsJShapiroSS, et al. (2017) Toward algorithmic transparency and accountability. Communications of the ACM60(9): 5. DOI: 10.1145/3125780.
16.
GraefeA (2016) Guide to automated journalism. DOI: 10.7916/D80G3XDJ.
17.
GraefeABohlkenN (2023) Automated journalism: A meta-analysis of readers’ perceptions of human-written in comparison to automated news.Media and Communication8(3): 50–59. DOI: 10.17645/mac.v8i3.3019.
18.
GraefeAHaimMHaarmannB, et al. (2018) Readers’ perception of computer-generated news: credibility, expertise, and readability. Journalism19(5): 595–610. DOI: 10.1177/1464884916641269.
19.
GuntherACLiebhartJL (2006) Broad reach or biased source? Decomposing the hostile media effect. Journal of Communication56(3): 449–466. DOI: 10.1111/j.1460-2466.2006.00295.x.
20.
GuntherACSchmittK (2004) Mapping boundaries of the hostile media effect. Journal of Communication54(1): 55–70. DOI: 10.1111/j.1460-2466.2004.tb02613.x.
21.
HansenGJKimH (2011) Is the media biased against me? A meta-analysis of the hostile media effect research. Communication Research Reports28(2): 169–179. DOI: 10.1080/08824096.2011.565280.
22.
HassellHJGHolbeinJBMilesMR (2020) There is no liberal media bias in which news stories political journalists choose to cover. Science Advances6(14): eaay9344. DOI: 10.1126/sciadv.aay9344.
23.
HofeditzLMirbabaieMHolsteinJ, et al. (2021, June) Do you trust an AI-journalist? A credibility analysis of news content with AI-authorship. In: ECIS, Marrakech, Morocco, 14–16 June 2021.
24.
HovlandCIJanisILKelleyHH (1953) Communication and Persuasion. New Haven, CT: Yale University Press.
25.
HulmanADollerupOLMortensenJF, et al. (2023) ChatGPT-versus human-generated answers to frequently asked questions about diabetes: a Turing test-inspired survey among employees of a Danish diabetes center. PLoS One18(8): e0290773.
26.
JamiesonKHCappellaJN (2008) Echo Chamber: Rush Limbaugh and the Conservative Media Establishment. Oxford: Oxford University Press.
27.
JangWKwakDHBucyE (2022) Knowledge of automated journalism moderates evaluations of algorithmically generated news. New Media & Society. DOI: 10.1177/14614448221142534.
28.
JiaC (2023) Effects of Issue Involvement and Algorithmic Literacy on Individuals’ Perceptions of News Recommended by Algorithms: A Machine Heuristic and Systematic Processing Model. USA: SSRN. [Preprint] DOI: 10.2139/ssrn.4431856.
29.
JiaCLiuR (2021) Algorithmic or human source? Examining relative hostile media effect with a transformer-based framework. Media and Communication9(4): 170–181.
30.
Jones-JangSMParkYJ (2022) How do people react to AI failure? Automation bias, algorithmic aversion, and perceived controllability. Journal of Computer-Mediated Communication28(1): zmac029. DOI: 10.1093/jcmc/zmac029.
31.
KarlssonMClerwallCNordL (2014) You Ain’t Seen Nothing Yet: transparency’s (lack of) effect on source and message credibility. Journalism Studies15(5): 668–678. DOI: 10.1080/1461670X.2014.886837.
32.
KizilcecRF (2016) How much information? effects of transparency on trust in an algorithmic interface. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery, 2390–2395. DOI: 10.1145/2858036.2858402.
33.
KotzianPStoeberTHoosF, et al. (2020) To be or not to be in the sample? On using manipulation checks in experimental accounting research. Accounting Research Journal33(3): 469–482. DOI: 10.1108/ARJ-06-2019-0128.
34.
KovachBRosenstielT (2021) The Elements of Journalism, Revised and Updated 4th Edition: What Newspeople Should Know and the Public Should Expect. New Bremen, OH: Crown.
35.
KundaZ (1990) The case for motivated reasoning. Psychological Bulletin108: 480–498. DOI: 10.1037/0033-2909.108.3.480.
36.
LaddJM (2011) Why Americans hate the media and how it matters. In: Why Americans Hate the Media and How it Matters. Princeton, NJ: Princeton University Press. DOI: 10.1515/9781400840359.
LeeE-J (2023) Minding the source: toward an integrative theory of human–machine communication. Human Communication Research50: 184–193.
39.
LiuB (2021) In AI we trust? Effects of agency locus and transparency on uncertainty reduction in human–AI interaction. Journal of Computer-Mediated Communication26(6): 384–402. DOI: 10.1093/jcmc/zmab013.
40.
LiuBWeiL (2018) Reading machine-written news: effect of machine heuristic and novelty on hostile media perception. In: KurosuM (ed) Human-Computer Interaction. Theories, Methods, and Human Issues. USA: Springer International Publishing, Vol. 10901, 307–324. DOI: 10.1007/978-3-319-91238-7_26.
MolinaMDSundarSS (2022a) Does distrust in humans predict greater trust in AI? Role of individual differences in user responses to content moderation. New Media & Society26: 3638–3656. DOI: 10.1177/14614448221103534.
43.
MolinaMDSundarSS (2022b) When AI moderates online content: effects of human collaboration and interactive transparency on user trust. Journal of Computer-Mediated Communication27(4): zmac010. DOI: 10.1093/jcmc/zmac010.
44.
NickersonRS (1998) Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology2(2): 175–220. DOI: 10.1037/1089-2680.2.2.175.
45.
NobleSU (2018) Algorithms of oppression: how search engines reinforce racism. In: Algorithms of Oppression. New York, NY: New York University Press.
46.
OffitPA (2015) Deadly choices: how the anti-vaccine movement threatens us all. New York City: Basic Books (AZ).
47.
O’KeefeD (2003) Message properties, mediating states, and manipulation checks: claims, evidence, and data analysis in experimental persuasive message effects research. Communication Theory13: 251–274. DOI: 10.1111/j.1468-2885.2003.tb00292.x.
48.
O’NeilC (2017) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New Bremen, OH: Crown.
49.
OphirYWalterDJamiesonPEJamiesonKH (2023) Factors Assessing Science’s Self-Presentation model and their effect on conservatives’ and liberals’ support for funding science. Proceedings of the National Academy of Sciences120(38): e2213838120. DOI: 10.1073/pnas.2213838120.
50.
PerloffRM (2015) A three-decade retrospective on the hostile media effect. Mass Communication & Society18(6): 701–729. DOI: 10.1080/15205436.2015.1051234.
51.
RaderECotterKChoJ (2018) Explanations as mechanisms for supporting algorithmic transparency. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery, 1–13. DOI: 10.1145/3173574.3173677.
52.
ReidSA (2012) A self-categorization explanation for the hostile media effect. Journal of Communication62(3): 381–399. DOI: 10.1111/j.1460-2466.2012.01647.x.
53.
RinehartA (2022) Artificial Intelligence in Local News. New York, NY: The Associated Press.
54.
RomerDJamiesonKH (2021) Conspiratorial thinking, selective exposure to conservative media, and response to COVID-19 in the US. Social Science & Medicine291: 114480. DOI: 10.1016/j.socscimed.2021.114480.
55.
SchmidtPBiessmannFTeubnerT (2020) Transparency and trust in artificial intelligence systems. Journal of Decision Systems29(4): 260–278. DOI: 10.1080/12460125.2020.1819094.
56.
SundarSS (2008) The MAIN model: A heuristic approach to understanding technology effects on credibility. Cambridge, MA: MacArthur Foundation Digital Media and Learning Initiative. Chicago, 73–100.
57.
SundarSS (2020) Rise of machine agency: a framework for studying the psychology of human–AI interaction (HAII). Journal of Computer-Mediated Communication25(1): 74–88. DOI: 10.1093/jcmc/zmz026.
58.
SundarSSKimJ (2019) Machine heuristic: when we trust computers more than humans with our personal information. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery, 1–9. DOI: 10.1145/3290605.3300768.
59.
TandocECYaoLJWuS (2020) Man vs. Machine? The impact of algorithm authorship on news credibility. Digital Journalism8(4): 548–562. DOI: 10.1080/21670811.2020.1762102.
60.
TsfatiYCohenJ (2005) Democratic consequences of hostile media perceptions: the case of gaza settlers. Harvard International Journal of Press/Politics10(4): 28–51. DOI: 10.1177/1081180X05280776.
61.
ValloneRPRossLLepperMR (1985) The hostile media phenomenon: biased perception and perceptions of media bias in coverage of the Beirut massacre. Journal of Personality and Social Psychology49(3): 577–585. DOI: 10.1037/0022-3514.49.3.577.
62.
Van der KaaHAKrahmerEJ (2014) Journalist versus news consumer: The perceived credibility of machine written news. Computation+ Journalism Symposium 2014. New York, USA: Columbia University.
63.
VosTPCraftS (2017) The discursive construction of journalistic transparency. Journalism Studies18(12): 1505–1522. DOI: 10.1080/1461670X.2015.1135754.
64.
WaddellTF (2018) A Robot Wrote This? how perceived machine authorship affects news credibility. Digital Journalism6(2): 236–255. DOI: 10.1080/21670811.2017.1384319.
65.
WaddellTF (2019) Can an algorithm reduce the perceived bias of news? Testing the effect of machine attribution on news readers’ evaluations of bias, anthropomorphism, and credibility. Journalism & Mass Communication Quarterly96(1): 82–100. DOI: 10.1177/1077699018815891.
66.
WangS (2021) Moderating uncivil user comments by humans or machines? The effects of moderation agent on perceptions of bias and credibility in news content. Digital Journalism9(1): 64–83. DOI: 10.1080/21670811.2020.1851279.
67.
WangRThimmanayakanapalyaSSOphirY (2024) The growing partisan politicization of non-political online spaces: A mixed-method analysis of news app reviews on Google Play between 2009 and 2022. New Media & Society0(0). DOI: 10.1177/14614448241237765.
68.
WesterståhlJ (1983) Objective news reporting: general premises. Communication Research10(3): 403–424.
69.
WölkerAPowellTE (2018) Algorithms in the newsroom? News readers’ perceived credibility and selection of automated journalism. Journalism22: 86–103. DOI: 10.1177/1464884918757072.
70.
WuY (2020) Is automated journalistic writing less biased? An experimental test of auto-written and human-written news stories. Journalism Practice14(8): 1008–1028. DOI: 10.1080/17512786.2019.1682940.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
