Algorithmic decision-making systems are ubiquitous in digital media, but the public has been largely unable to negotiate the role of algorithms in society. Building from the concept of attitude-behavior consistency for political behavior, we develop a framework for fostering algorithmic literacy to develop well-informed attitudes toward algorithms. As algorithms are increasingly relevant to broad societal effects, an integrative approach is needed for a full account of how the public makes sense of algorithms and their role in society. We designed and tested a novel intervention that combines algorithmic literacy with personalized user experiences to see how each component influenced attitudes toward algorithms. We found these methods jointly informed attitudes, but the intervention’s efficacy was dependent on participants’ individual differences in technology use.
AlvaradoOHeuerHVanden AbeeleV, et al. (2020) Middle-aged video consumers’ beliefs about algorithmic recommendations on YouTube. Proceedings of the ACM on Human-Computer Interaction4(CSCW2): 1–24.
2.
AnannyMCrawfordK (2018) Seeing without knowing: limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society20(3): 973–989.
3.
AnikAIBuntA (2021) Data-centric explanations: explaining training data of machine learning systems to promote transparency. In: Proceedings of the 2021 CHI conference on human factors in computing systems, Yokohama, Japan, 8–13 May, pp. 1–13. New York: ACM. Available at: https://doi.org/10.1145/3411764.3445736
4.
AustinEWPinkletonBEHustSJT, et al. (2005) Evaluation of an American Legacy Foundation/Washington State Department of Health Media Literacy Pilot Study. Health Communication18(1): 75–95.
5.
BarlowBWebbABarlowA (2021) Maximizing the visual translation of medical information: a narrative review of the role of infographics in clinical pharmacy practice, education, and research. Journal of the American College of Clinical Pharmacy4(2): 257–266.
BasingerSJLavineH (2005) Ambivalence, information, and electoral choice. American Political Science Review99(2): 169–184.
8.
BeerD (2017) The social power of algorithms. Information, Communication & Society20(1): 1–13.
9.
BishopS (2019) Managing visibility on YouTube through algorithmic gossip. New Media & Society21(11–12): 2589–2606.
10.
BitzerTWienerMMoranaS (2021) Algorithmic transparency and contact—tracing apps—an empirical investigation. In: Proceedings of the twenty-seventh Americas conference on information systems, Montreal, QC, Canada, 9–13 August, pp. 1–10. Association for Information Systems.
BucherT (2012) Want to be on the top? Algorithmic power and the threat of invisibility on Facebook. New Media & Society14(7): 1164–1180.
14.
CaslerKBickelLHackettE (2013) Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior29: 2156–2160.
15.
ChenT-WSundarSS (2018) This app would like to use your current location to better serve you: importance of user assent and system transparency in personalized mobile services. In: Proceedings of the 2018 CHI conference on human factors in computing systems —CHI ’18, Montreal, QC, Canada, 21–26 April, pp. 1–13. New York: ACM. Available at: https://doi.org/10.1145/3173574.3174111
16.
ChengH-FWangRZhangZ, et al. (2019) Explaining decision-making algorithms through UI: strategies to help non-expert stakeholders. In: Proceedings of the 2019 CHI conference on human factors in computing systems —CHI ’19, Glasgow, 4–9 May, pp. 1–12. New York: ACM. Available at: https://doi.org/10.1145/3290605.3300789
DeVitoMA (2021) Adaptive folk theorization as a path to algorithmic literacy on changing platforms. Proceedings of the ACM on Human-Computer Interaction5: 339.
21.
DeVitoMABirnholtzJHancockJT, et al. (2018) How people form folk theories of social media feeds and what it means for how we study self-presentation. In: Proceedings of the 2018 CHI conference on human factors in computing systems, Montreal, QC, Canada, 21–26 April, pp. 1–12. New York: ACM. Available at: https://doi.org/10.1145/3173574.3173694
22.
DiakopoulosNKoliskaM (2017) Algorithmic transparency in the news media. Digital Journalism5(7): 809–828.
23.
DietvorstBJSimmonsJPMasseyC (2015) Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General144(1): 114–126.
24.
DiFranzoDChoiYHPuringtonA, et al. (2019) Social media TestDrive: real-world social media education for the next generation. In: Proceedings of the 2019 CHI conference on human factors in computing systems, Glasgow, 4–9 May, pp. 303:1–303:11. New York: ACM. Available at: https://doi.org/10.1145/3290605.3300533
25.
DogruelL (2021) Folk theories of algorithmic operations during internet use: a mixed methods study. The Information Society37(5): 287–298.
26.
DogruelLMasurPJoeckelS (2021) Development and validation of an algorithm literacy scale for internet users. Communication Methods and Measures. Epub ahead of print 13 September. DOI: 10.1080/19312458.2021.1968361.
27.
DowneyAB (2015) Think Python, 2e: How to think like a computer scientist. O’Reilly Media.
28.
DzindoletMTPetersonSAPomrankyRA, et al. (2003) The role of trust in automation reliance. International Journal of Human-Computer Studies58(6): 697–718.
29.
EibandMSchneiderHBilandzicM, et al. (2018) Bringing transparency design into practice. In: 23rd International conference on intelligent user interfaces, Tokyo, Japan, 7–11 March, pp. 211–223. New York: ACM. Available at: https://doi.org/10.1145/3172944.3172961
30.
ElaldıŞÇifçiT (2021) The effectiveness of using infographics on academic achievement: a meta-analysis and a meta-thematic analysis. Journal of Pedagogical Research5(4): 92–118.
31.
EslamiMKarahaliosKSandvigC, et al. (2016) First I “like” it, then I hide it: folk theories of social feeds. In: Proceedings of the 2016 CHI conference on human factors in computing systems —CHI ’16, San Jose, CA, 7–12 May, pp. 2371–2382. New York: ACM. Available at: https://doi.org/10.1145/2858036.2858494
32.
EslamiMRickmanAVaccaroK, et al. (2015) “I always assumed that I wasn’t really that close to [her]”: reasoning about invisible algorithms in news feeds. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, Seoul, Republic of Korea, 18–23 April, pp. 153–162. New York: ACM. Available at: https://doi.org/10.1145/2702123.2702556
33.
EslamiMVaccaroKKarahaliosK, et al. (2017) “Be careful; things can be worse than they appear”: understanding biased algorithms and users’ behavior around them in rating platforms. Proceedings of the Eleventh International AAAI Conference on Web and Social Media11(1): 63–71.
34.
EslamiMVaccaroKLeeMK, et al. (2019) User attitudes towards algorithmic opacity and transparency in online reviewing platforms. In: Proceedings of the CHI conference on human factors in computing systems, Glasgow, 4–9 May, pp. 1–14. New York: ACM. Available at: https://doi.org/10.1145/3290605.3300724
35.
FazioRHChenJMcDonelEC, et al. (1982) Attitude accessibility, attitude-behavior consistency, and the strength of the object-evaluation association. Journal of Experimental Social Psychology18(4): 339–357. DOI: 10.1016/0022-1031(82)90058-0.
36.
FazioRHEiserJRShookNJ (2004) Attitude formation through exploration: Valence asymmetries. Journal of Personality and Social Psychology87(3): 293–311. DOI: 10.1037/0022-3514.87.3.293.
37.
FifeD (2020) The eight steps of data analysis: A graphical framework to promote sound statistical analysis. Perspectives on Psychological Science15(4): 35. DOI: 10.1177/1745691620917333.
38.
FletcherRNielsenRK (2018) Are people incidentally exposed to news on social media? A comparative analysis. New Media & Society20(7): 2450–2468.
39.
GillespieT (2010) The politics of “platforms.”New Media & Society12(3): 347–364.
40.
GlotfelterA (2019) Algorithmic circulation: how content creators navigate the effects of algorithms on their work. Computers and Composition54: 102521.
GranABBoothPBucherT (2020) To be or not to be algorithm aware: a question of a new digital divide?Information, Communication & Society24(12): 1779–1796.
43.
GrawitchMJMaloneyPWBarberLK, et al. (2013) Examining the nomological network of satisfaction with work–life balance. Journal of Occupational Health Psychology18(3): 276–284.
44.
HargittaiEGruberJDjukaricT, et al. (2020) Black box measures? How to study people’s algorithm skills. Information, Communication & Society23: 764–775.
45.
HuffCTingleyD (2015) “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research and Politics2: 1–12.
46.
JensonJDroumevaM (2016) Exploring media literacy and computational thinking: a game maker curriculum study. Electronic Journal of E-Learning14(2): 111–121.
47.
JhaverSBruckmanAGilbertE (2019) Does transparency in moderation really matter? User behavior after content removal explanations on Reddit. Proceedings of the ACM on Human-Computer Interaction3(CSCW): 150.
48.
KangHShinW (2016) Do smartphone power users protect mobile privacy better than nonpower users? Exploring power usage as a factor in mobile privacy protection and disclosure. CyberPsychology, Behavior & Social Networking19(3): 179–185.
49.
KimKMoonS-I (2021) When algorithmic transparency failed: controversies over algorithm-driven content curation in the South Korean digital environment. American Behavioral Scientist65(6): 847–862. https://doi.org/10.1177/0002764221989783
50.
KizilcecRF (2016) How much information? Effects of transparency on trust in an algorithmic interface. In: Proceedings of the 2016 CHI conference on human factors in computing systems —CHI ’16, San Jose, CA, 7–12 May, pp. 2390–2395. New York: ACM. Available at: https://doi.org/10.1145/2858036.2858402
LeeMKBaykalS (2017) Algorithmic mediation in group decisions: fairness perceptions of algorithmically mediated vs. discussion-based social division. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing —CSCW ’17, Portland, OR, 25 February–1 March, pp. 1035–1048. New York: ACM. Available at: https://doi.org/10.1145/2998181.2998230
53.
LiaoQVGruenDMillerS (2020) Questioning the AI: informing design practices for explainable AI user experiences. In: Proceedings of the 2020 CHI conference on human factors in computing systems, Honolulu, HI, 25–30 April, pp. 1–15. New York: ACM. Available at: https://doi.org/10.1145/3313831.3376590
54.
LiaoTTysonO (2021) “Crystal is creepy, but cool”: mapping folk theories and responses to automated personality recognition algorithms. Social Media + Society7(2): 1–11.
55.
LloydA (2019) Chasing Frankenstein’s monster: information literacy in the black box society. Journal of Documentation75(6): 1475–1485.
56.
MaratheSSSundarSNije BijvankM, et al. (2007) Who are these power users anyway? Building a psychological profile. In: 57th Annual conference of the International Communication Association, San Francisco, CA, 23 May.
NewhagenJEBucyE (2003) Routes to media access. In: BucyENewhagenJE (eds) Media Access: Social and Psychological Dimensions of New Technology Use. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 3–23.
59.
OduorKFWiebeEN (2008) The effects of automated decision algorithm modality and transparency on reported trust and task performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting52(4): 302–306.
60.
OsoskySSandersTJentschF, et al. (2014) Determinants of system transparency and its influence on trust in and reliance on unmanned robotic systems. Unmanned Systems Technology XVI9084: 90840E.
61.
PerrinAAndersonM (2019) Share of U.S. adults using social media, including Facebook is mostly unchanged since 2018. Available at: https://pewrsr.ch/2VxJuJ3 (accessed 31 March 2021).
62.
PinkletonBEAustinEWChenY-CY, et al. (2012) The role of media literacy in shaping adolescents’ understanding of and responses to sexual portrayals in mass media. Journal of Health Communication17(4): 460–476.
63.
PinkletonBEAustinEWCohenM, et al. (2008) Effects of a peer-led media literacy curriculum on adolescents’ knowledge and attitudes toward sexual behavior and media portrayals of sex. Health Communication23(5): 462–472.
64.
PolackP (2020) Beyond algorithmic reformism: forward engineering the designs of algorithmic systems. Big Data & Society7(1): 1–15.
65.
R Core Team (2016) R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available at: https://www.R-project.org/.
66.
RaderECotterKChoJ (2018) Explanations as mechanisms for supporting algorithmic transparency. In: Proceedings of the 2018 CHI conference on human factors in computing systems, Montreal, QC, Canada, 21–26 April, pp. 1–13. New York: ACM.
RiederBMatamoros-FernándezACorominaÒ (2018) From ranking algorithms to ‘ranking cultures’: Investigating the modulation of visibility in YouTube search results. Convergence24(1): 50–68.
69.
ShinDParkYJ (2019) Role of fairness, accountability, and transparency in algorithmic affordance. Computers in Human Behavior98: 277–284.
70.
ShinDRasulAFotiadisA (2021) Why am I seeing this? Deconstructing algorithm literacy through the lens of users. Internet Research. Epub ahead of print 23 September. DOI: 10.1108/INTR-02-2021-0087.
71.
SilesIMeléndez-MoranA (2021) “The most aggressive of algorithms”: user awareness of and attachment to TikTok’s content personalization. In Paper presented at the 71st annual conference of the international communication association (ICA), Denver, CO, 27–31 May. Available at: http://www.kerwa.ucr.ac.cr/handle//1066983230
SnowbergEYarivL (2018) Testing the Waters: Behavior across Participant Pools (National Bureau of Economic Research Working Paper Series: 24781). Cambridge, MA: National Bureau of Economic Research. Available at: http://www.nber.org/papers/w24781
74.
SpringerAWhittakerS (2019) Progressive disclosure: empirically motivated approaches to designing effective transparency. In: Proceedings of the 24th international conference on intelligent user interfaces —IUI ’19, Marina del Rey, CA, 17–20 March, pp. 107–120. New York: ACM. Available at: https://doi.org/10.1145/3301275.3302322
75.
SundarSSMaratheSS (2010) Personalization versus customization: the importance of agency, privacy, and power usage. Human Communication Research36(3): 298–322.
76.
SwartJ (2021) Experiencing algorithms: how young people understand, feel about, and engage with algorithmic news selection on social media. Social Media + Society7(2): 1–11.
77.
TriftsVAghakhaniH (2019) Enhancing digital entertainment through personalization: the evolving role of product placements. Journal of Marketing Communications25(6): 607–625.
78.
ValtonenTTedreMMäkitaloK, et al. (2019) Media literacy education in the age of machine learning. Journal of Media Literacy Education11(2): 20–36.
79.
WachterSMittelstadtBDMRussellC (2018) Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law and Technology31(2): 841–887.
80.
WangRHarperFMZhuH (2020) Factors influencing perceived fairness in algorithmic decision-making: algorithm outcomes, development procedures, and individual differences. In: Proceedings of the 2020 CHI conference on human factors in computing systems, Honolulu, HI, 25–30 April, pp. 1–14. New York: ACM. Available at: https://doi.org/10.1145/3313831.3376813
81.
WarschauerM (2003) Technology and Social Inclusion: Rethinking the Digital Divide. Cambridge, MA: MIT Press.
82.
WickhamH (2009) ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag.
83.
ZhongB (2013) From smartphones to iPad: power users’ disposition toward mobile media devices. Computers in Human Behavior29(4): 1742–1748.
84.
ZuboffS (2019) The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. New York: PublicAffairs.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
