Restricted accessResearch articleFirst published online 2024-4
Not All Bots are Created Equal: The Impact of Bots Classification Techniques on Identification of Discursive Behaviors Around the COVID-19 Vaccine and Climate Change
As concerns about social bots online increase, studies have attempted to explore the discourse they produce, and its effects on individuals and the public at large. We argue that the common reliance on aggregated scores of binary classifiers for bot detection may have yielded biased or inaccurate results. To test this possibility, we systematically compare the differences between non-bots and bots using binary and non-binary classifiers (classified into the categories of astroturf, self-declared, spammers, fake followers, and Other). We use two Twitter corpora, about COVID-19 vaccines (N = 1,697,280) and climate change (N = 1,062,522). We find that both in terms of volume and thematic content, the use of binary classifiers may hinder, distort, or mask differences between humans and bots, that could only be discerned when observing specific bot types. We discuss the theoretical and practical implications of these findings.
AbbasA. H. (2022). Politicizing COVID-19 vaccines in the press: A critical discourse analysis. International Journal for the Semiotics of Law, 35(3), 1167–1185. https://doi.org/10.1007/s11196-021-09857-3
2.
AldayelA.MagdyW. (2022). Characterizing the role of bots’ in polarized stance on social media. Social Network Analysis and Mining, 12(1), 30. https://doi.org/10.1007/s13278-022-00858-z
3.
AllemJ.-P.FerraraE. (2018). Could social bots pose a threat to public health?American Journal of Public Health, 108(8), 1005–1006. https://doi.org/10.2105/AJPH.2018.304512
4.
AllemJ.-P.FerraraE.UppuS. P.CruzT. B.UngerJ. B. (2017). E-cigarette surveillance with social media data: Social bots, emerging topics, and trends. JMIR Public Health and Surveillance, 3(4), Article e98. https://doi.org/10.2196/publichealth.8641
5.
AssenmacherD.CleverL.FrischlichL.QuandtT.TrautmannH.GrimmeC. (2020). Demystifying social bots: On the intelligence of automated social media actors. Social Media + Society, 6(3), 205630512093926. https://doi.org/10.1177/2056305120939264
6.
BarbonS.JrCamposG. F. C.TavaresG. M.IgawaR. A. M. L. P.JrGuidoR. C. (2018). Detection of human, legitimate bot, and malicious bot in online social networks based on wavelets. ACM Transactions on Multimedia Computing, Communications, and Applications, 14(1s), 26:1–26:17. https://doi.org/10.1145/3183506
7.
BenklerY. (2008). The wealth of networks: How social production transforms markets and freedom. In The wealth of networks. Yale University Press. https://doi.org/10.12987/9780300127232
8.
BenklerY.FarisR.RobertsH. (2018). Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press.
9.
BennettW. L.SegerbergA. (2013). The logic of connective action: Digital media and the personalization of contentious politics. Cambridge University Press.
10.
BessiA.FerraraE. (2016). Social bots distort the 2016 U.S. Presidential election online discussion. First Monday. https://doi.org/10.5210/fm.v21i11.7090
11.
BleiD. M. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3(2003), 993–1022.
12.
BlondelV. D.GuillaumeJ.-L.LambiotteR.LefebvreE. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008. https://doi.org/10.1088/1742-5468/2008/10/P10008
13.
BolsoverG.HowardP. (2019). Chinese computational propaganda: Automation, algorithms and the manipulation of information about Chinese politics on Twitter and Weibo. Information, Communication & Society, 22(14), 2063–2080. https://doi.org/10.1080/1369118X.2018.1476576
14.
BoshmafY.MuslukhovI.BeznosovK.RipeanuM. (2011). The socialbot network: When bots socialize for fame and money. Proceedings of the 27th Annual Computer Security Applications Conference, ’11, 93. Orlando, FL, USA. https://doi.org/10.1145/2076732.2076746
BroniatowskiD. A.JamisonA. M.QiS.AlKulaibL.ChenT.BentonA.QuinnS. C.DredzeM. (2018). Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. American Journal of Public Health, 108(10), 1378–1384. https://doi.org/10.2105/AJPH.2018.304567
ChangH.-C. H.FerraraE. (2022). Comparative analysis of social bots and humans during the COVID-19 pandemic. Journal of Computational Social Science, 5(2), 1409–1425. https://doi.org/10.1007/s42001-022-00173-9
ChenC.-F.ShiW.YangJ.FuH.-H. (2021). Social bots’ role in climate change discussion on Twitter: Measuring standpoints, topics, and interaction strategies. Advances in Climate Change Research, 12(6), 913–923. https://doi.org/10.1016/j.accre.2021.09.011
22.
ChenW.PachecoD.YangK.-C.MenczerF. (2021). Neutral bots probe political bias on social media. Nature Communications, 12(1), 5580. https://doi.org/10.1038/s41467-021-25738-6
23.
CresciS.LilloF.RegoliD.TardelliS.TesconiM. (2019). Cashtag piggybacking: Uncovering spam and bot activity in stock microblogs on twitter. ACM Transactions on the Web, 13(2), 11:1–11:27. https://doi.org/10.1145/3313184
24.
CresciS.PetrocchiM.SpognardiA.TognazziS. (2019). Better safe than sorry: An adversarial approach to improve social bot detection. Proceedings of the 10th ACM Conference on Web Science, Amsterdam, Netherlands, 47–56. https://doi.org/10.1145/3292522.3326030
25.
DavisC. A.VarolO.FerraraE.FlamminiA.MenczerF. (2016). BotOrNot: A system to evaluate social bots. Proceedings of the 25th international conference companion on world wide web-16 Companion, Geneva, Switzerland, 273–274. https://doi.org/10.1145/2872518.2889302
26.
De MeoP.FerraraE.FiumaraG.ProvettiA. (2011). Generalized Louvain method for community detection in large networks. 2011 11th International Conference on Intelligent Systems Design and Applications, Cordoba, Spain, 88–93. https://doi.org/10.1109/ISDA.2011.6121636
27.
DiMaggioP.NagM.BleiD. (2013). Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of U.S. government arts funding. Poetics, 41(6), 570–606. https://doi.org/10.1016/j.poetic.2013.08.004
28.
DuanZ.LiJ.LukitoJ.YangK.-C.ChenF.ShahD. V.YangS. (2022). Algorithmic agents in the hybrid media system: Social bots, selective amplification, and partisan news about COVID-19. Human Communication Research, 48(3), 516–542. https://doi.org/10.1093/hcr/hqac012
29.
DunnA. G.SurianD.DalmazzoJ.RezazadeganD.SteffensM.DydaA.LeaskJ.CoieraE.DeyA.MandlK. D. (2020). Limited role of bots in spreading vaccine-critical information among active twitter users in the United States: 2017–2019. American Journal of Public Health, 110(S3), S319–S325. https://doi.org/10.2105/AJPH.2020.305902
30.
FanR.TalaveraO.TranV. (2020). Social media bots and stock markets. European Financial Management, 26(3), 753–777. https://doi.org/10.1111/eufm.12245
31.
FerraraE. (2017). Disinformation and social bot operations in the run up to the 2017 French presidential election. First Monday. https://doi.org/10.5210/fm.v22i8.8005
32.
FerraraE.ChangH.ChenE.MuricG.PatelJ. (2020). Characterizing social media manipulation in the 2020 U.S. presidential election. First Monday. https://doi.org/10.5210/fm.v25i11.11431
33.
FerraraE.VarolO.DavisC.MenczerF.FlamminiA. (2016). The rise of social bots. Communications of the ACM, 59(7), 96–104. https://doi.org/10.1145/2818717
34.
GallwitzF.KreilM. (2022). Investigating the validity of botometer-based social bot studies. (arXiv:2207.11474). arXivhttps://arxiv.org/abs/2207.11474
35.
GorwaR.GuilbeaultD. (2020). Unpacking the social media bot: A typology to guide research and policy. Policy & Internet, 12(2), 225–248. https://doi.org/10.1002/poi3.184
36.
GruzdA.MaiP. (2020). Going viral: How a single tweet spawned a COVID-19 conspiracy theory on Twitter. Big Data & Society, 7(2), 205395172093840. https://doi.org/10.1177/2053951720938405
37.
HagenL.NeelyS.KellerT. E.ScharfR.VasquezF. E. (2020). Rise of the machines? Examining the influence of social bots on a political discussion network. Social Science Computer Review. 40(2), 0894439320908190. https://doi.org/10.1177/0894439320908190
38.
HegelichS.JanetzkoD. (2016). Are social bots on twitter political actors? Empirical evidence from a Ukrainian social botnet. Tenth International AAAI Conference on Web and Social Media. Cologne, Germany. https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/view/13015
39.
HofeditzL.EhnisC.BunkerD.BrachtenF.StieglitzS. (2019). Meaningful use of social bots? Possible applications in crisis communication during disasters. In Proceedings of the 27th European Conference on Information Systems (ECIS). Stockholm, Sweden. https://aisel.aisnet.org/ecis2019_rp/138
40.
HurtadoS.RayP.MarculescuR. (2019). Bot detection in reddit political discussion. Proceedings of the Fourth International Workshop on Social Sensing, Montreal, Canada, 30–35. https://doi.org/10.1145/3313294.3313386
JacobiC.van AtteveldtW.WelbersK. (2016). Quantitative analysis of large amounts of journalistic texts using topic modelling. Digital Journalism, 4(1), 89–106. https://doi.org/10.1080/21670811.2015.1093271
43.
JamiesonK. H. (2018). Cyberwar: How Russian hackers and trolls helped elect a president: What we don’t, can’t, and do know (Illustrated edition). Oxford University Press.
44.
JamisonA. M.BroniatowskiD. A.QuinnS. C. (2019). Malicious actors on twitter: A guide for public health researchers. American Journal of Public Health, 109(5), 688–692. https://doi.org/10.2105/AJPH.2019.304969
45.
KahanD. M.PetersE.WittlinM.SlovicP.OuelletteL. L.BramanD.MandelG. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change, 2(10), 732–735. https://doi.org/10.1038/nclimate1547
46.
KellerT. R.KlingerU. (2019). Social bots in election campaigns: Theoretical, empirical, and methodological implications. Political Communication, 36(1), 171–189. https://doi.org/10.1080/10584609.2018.1526238
47.
KlingerU.Lance BennettW.KnüpferC. B.MartiniF.ZhangX. (2022). From the fringes into mainstream politics: Intermediary networks and movement-party coordination of a global anti-immigration campaign in Germany. Information, Communication & Society, 1–18. https://doi.org/10.1080/1369118X.2022.2050415
LuceriL.DebA.BadawyA.FerraraE. (2019). Red bots do it better: Comparative analysis of social bot partisan behavior. Companion Proceedings of the 2019 World Wide Web Conference. Perth, Australia. (1007–1012). https://doi.org/10.1145/3308560.3316735
50.
MartiniF.SamulaP.KellerT. R.KlingerU. (2021). Bot, or not? Comparing three methods for detecting social bots in five political discourses. Big Data & Society, 8(2), 205395172110335. https://doi.org/10.1177/20539517211033566
51.
MillerJ. M.SaundersK. L.FarhartC. E. (2016). Conspiracy endorsement as motivated reasoning: The moderating roles of political knowledge and trust. American Journal of Political Science, 60(4), 824–844. https://doi.org/10.1111/ajps.12234
52.
OphirY.WalterN.WalterD.VelhoR. M.LokmanogluA. D.PrudenM. L.AndrewsE. A. (2022). Vaccine hesitancy under the magnifying glass: A systematic review of the uses and misuses of an increasingly popular construct. Health communication, 1–15. https://doi.org/10.1080/10410236.2022.2054102
53.
OrabiM.MouhebD.Al AghbariZ.KamelI. (2020). Detection of bots in social media: A systematic review. Information Processing & Management, 57(4), 102250. https://doi.org/10.1016/j.ipm.2020.102250
54.
RatkiewiczJ.ConoverM.MeissM.GonçalvesB.PatilS.FlamminiA.MenczerF. (2011). Truthy: Mapping the spread of astroturf in microblog streams. Proceedings of the 20th International Conference Companion on World Wide Web, Hyderabad, India, 249–252. https://doi.org/10.1145/1963192.1963301
55.
RauchfleischA.KaiserJ. (2020). The false positive problem of automatic bot detection in social science research. PLoS One, 15(10), Article e0241045. https://doi.org/10.1371/journal.pone.0241045
56.
RossiS.RossiM.UpretiB. R.LiuY. (2020). Detecting political bots on twitter during the 2019 Finnish parliamentary election. Proceedings of the 53rd Hawaii International Conference on System Sciences, Hawaii, USA. (10).
57.
SantiaG. C.MujibM. I.WilliamsJ. R. (2019). Detecting social bots on facebook in an information veracity context. Proceedings of the International AAAI Conference on Web and Social Media, 13, Palo Alto, CA, USA, 463–472. https://doi.org/10.1609/icwsm.v13i01.3244
58.
SayyadiharikandehM.VarolO.YangK.-C.FlamminiA.MenczerF. (2020). Detection of novel social bots by ensembles of specialized classifiers. Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York, NY, USA, 2725–2732. https://doi.org/10.1145/3340531.3412698
59.
ShaoC.CiampagliaG. L.VarolO.YangK.FlamminiA.MenczerF. (2018). The spread of low-credibility content by social bots. Nature Communications, 9(1), 4787. https://doi.org/10.1038/s41467-018-06930-7
60.
ShiW.LiuD.YangJ.ZhangJ.WenS.SuJ. (2020). Social bots’ sentiment engagement in health emergencies: A topic-based analysis of the COVID-19 pandemic discussions on twitter. International Journal of Environmental Research and Public Health, 17(22), 8701. https://doi.org/10.3390/ijerph17228701
StieglitzS.BrachtenF.RossB.JungA.-K. (2017). Do social bots dream of electric sheep? A categorisation of social media bot accounts. (arXiv: 1710.04044). arXivhttps://doi.org/10.48550/arXiv.1710.04044
TorusdağM. B.KutluM.SelçukA. A. (2020). Are we secure from bots? Investigating vulnerabilities of botometer. 2020 5th International Conference on Computer Science and Engineering (UBMK). Diyarbakir, Turkey. 343–348. https://doi.org/10.1109/UBMK50275.2020.9219433
65.
TsvetkovaM.García-GavilanesR.FloridiL.YasseriT. (2017). Even good bots fight: The case of Wikipedia. PLoS One, 12(2), Article e0171774. https://doi.org/10.1371/journal.pone.0171774
66.
TuckerJ. A.TheocharisY.RobertsM. E.BarberáP. (2017). From liberation to turmoil: Social media and democracy. Journal of Democracy, 28(4), 46–59. https://doi.org/10.1353/jod.2017.0064
VarolO.FerraraE.DavisC.MenczerF.FlamminiA. (2017). Online human-bot interactions: Detection, estimation, and characterization. Proceedings of the International AAAI Conference on Web and Social Media, 11(1), 280–289. Article 1https://doi.org/10.1609/icwsm.v11i1.14871
WalterD.OphirY. (2019). News frame analysis: An inductive mixed-method computational approach. Communication Methods and Measures, 13(4), 248–266. https://doi.org/10.1080/19312458.2019.1639145
71.
WalterD.OphirY. (2023). Trolls without borders: A comparative analysis of six foreign countries’ online propaganda campaigns. Human Communication Research. https://doi.org/10.1093/hcr/hqad022
72.
WalterD.OphirY.JamiesonK. H. (2020). Russian twitter accounts and the partisan polarization of vaccine discourse, 2015–2017. American Journal of Public Health, 110(5), 718–724. https://doi.org/10.2105/AJPH.2019.305564
73.
WalterD.OphirY.LokmanogluA. D.PrudenM. L. (2022). Vaccine discourse in white nationalist online communication: A mixed-methods computational approach. Social Science & Medicine, 298, 114859. https://doi.org/10.1016/j.socscimed.2022.114859.
74.
WangG.MohanlalM.WilsonC.WangX.MetzgerM.ZhengH.ZhaoB. Y. (2012). Social turing tests: Crowdsourcing sybil detection. (arXiv: 1205.3856). arXivhttps://arxiv.org/abs/1205.3856
75.
WojcikS.MessingS.SmithA. W.RainieL.HitlinP. (2018). Bots in the twittersphere. [Report]. Pew Research Center. https://apo.org.au/node/141291
WoolleyS. C.HowardP. N. (2018). Computational propaganda: Political parties, politicians, and political manipulation on social media. Oxford University Press.
78.
XuY.SunY.HagenL.PatelM.FallingM. (2021) Evolution of the pandemic communication network among serial participants on TwitterNew Media & Society. 14614448211050928. https://doi.org/10.1177/14614448211050928
79.
YanH. Y.YangK.-C.MenczerF.ShanahanJ. (2021). Asymmetrical perceptions of partisan political bots. New Media & Society, 23(10), 3016–3037. https://doi.org/10.1177/1461444820942744
80.
YangK.-C.FerraraE.MenczerF. (2022). Botometer 101: Social bot practicum for computational social scientists. ArXiv: 2201.01608 [Cs]https://arxiv.org/abs/2201.01608
81.
YangK.-C.VarolO.DavisC. A.FerraraE.FlamminiA.MenczerF. (2019). Arming the public with artificial intelligence to counter social bots. Human Behavior and Emerging Technologies, 1(1), 48–61. https://doi.org/10.1002/hbe2.115
82.
YuanX.SchuchardR. J.CrooksA. T. (2019). Examining emergent communities and social bots within the polarized online vaccination debate in twitter. Social Media + Society, 5(3), 205630511986546. https://doi.org/10.1177/2056305119865465
83.
ZhangM.QiX.ChenZ.LiuJ. (2022). Social bots’ involvement in the COVID-19 vaccine discussions on twitter. International Journal of Environmental Research and Public Health, 19(3), 1651. https://doi.org/10.3390/ijerph19031651
84.
ZhangY.ChenF.LukitoJ. (2022). Network amplification of politicized information and misinformation about COVID-19 by conservative media and partisan influencers on twitter. Political Communication, 40(1), 24–47. https://doi.org/10.1080/10584609.2022.2113844
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
