Trust and Safety on Social Media: Understanding the Impact of Anti-Social Behavior and Misinformation on Content Moderation and Platform Governance

Abstract

The Special Issue on Trust and Safety on Social Media delves into two pressing and interlinked concerns: the growing prevalence of anti-social behavior and the widespread presence of misinformation within and across various social media platforms. The collection of articles featured in the issue collectively examines factors that contribute to these concerns and proposes potential strategies to mitigate their negative impact on social media users and society. The articles included in the issue are extended versions of research first presented at the 2022 International Conference on Social Media & Society (#SMSociety), organized by the Social Media Lab at Toronto Metropolitan University.

Introduction

The Special Issue on Trust and Safety on Social Media is a collection of peer-reviewed articles that examine the rise of anti-social behavior, misinformation, and other forms of problematic content within and across various social media settings, contexts, and user groups. The special issue emerged from the presentations and deliberations by interdisciplinary scholars at the 2022 International Conference on Social Media & Society, organized by the Social Media Lab at Toronto Metropolitan University. The issue explores two dangerous and interconnecting trends: the rise of anti-social behavior and the spread of misinformation online. Its aim is to help the public, policymakers, and platform operators better understand the factors contributing to the rise of these minacious trends on social media.

In recent years, anti-social behaviors, such as trolling, harassment, and bullying have surged online. For instance, the percentage of adults (above 18 years of age) in the United States who have reported being harassed online has sharply increased from 23% in 2022 to 33% in 2023 (ADL Center for Technology & Society, 2023). According to the same survey, the increase is even more pronounced among teens (13–17 years old), going from 36% to 51%. This rise in online anti-social behavior has impacted various groups of people, including journalists (e.g., Holton et al., 2023; Lewis et al., 2020), politicians (e.g., Anne et al., 2023; Tenove et al., 2023), academics (e.g., Gosse et al., 2023; Houlden et al., 2022), women (e.g., Esposito & Breeze, 2022; Kumar et al., 2021), LGBTQ+ (e.g., Mkhize et al., 2020; Strand & Svensson, 2022), ethnic minorities (e.g., Chaudhry & Gruzd, 2020; Li & Nicholson Jr., 2021), religious minorities (e.g., Ferguson et al., 2021; Wahlström et al., 2021), and other vulnerable groups. The motives for engaging in online anti-social behavior are varied, as studies have indicated. Social media users may participate in such acts for various reasons, including seeking revenge, seeking amusement, seeking social approval, displaying sadistic tendencies, or simply due to a lack of empathy (e.g., Santre, 2023; Soares et al., 2023; Vismara et al., 2022; Volkmer et al., 2023).

The diverse targets of and motives for engaging in anti-social behavior make content moderation a challenge. Platforms have been testing and building both automated and manual approaches to mitigate problems on the social web (e.g., Gibson, 2023; Horta Ribeiro et al., 2023; Papaevangelou & Smyrnaios, 2022). For example, Twitter tested (with some success) a feature that prompted users to reconsider their messages if they were potentially harmful (Katsaros et al., 2022). Other platforms, such as Reddit, rely more on community-led moderation and use jargon-free community rules to promote “healthier” conversations by enhancing civility and other pro-social behaviors (e.g., Del Valle et al., 2020; Trujillo & Cresci, 2022).

The situation surrounding the spread of misinformation online is not much different. Even though measuring the extent of misinformation on social media can be challenging due to its various forms and often ephemeral nature, researchers have extensively documented its impact, spread, and prevalence across social media platforms (e.g., Chen, Xiao & Kumar, 2023; van der Linden, 2022). This phenomenon is particularly evident in discussions of politically polarized topics, such as gun control (Williams, 2022), climate change (Falkenberg et al., 2022), abortion (Pagoto et al., 2023), vaccination (Gruzd et al., 2023), refugees (Zhen et al., 2023), and more recently, the COVID-19 pandemic (Gruzd et al., 2021). In its most dangerous form, misinformation turns into disinformation when it is deliberately used to deceive, polarize, or radicalize the population. Disinformation has been weaponized by malicious actors during events, such as the 2020 US election (e.g., Chang et al., 2021; Lee & Jones-Jang, 2022) and, more recently, Russia’s full-scale invasion of Ukraine (Gruzd et al., 2022; Milmo, 2022). Previous efforts to understand the reasons for the spread of misinformation and propose mitigation strategies include identifying and flagging false claims, fake accounts, coordinated sharing of misleading links, and manipulated media (e.g., Calleja et al., 2021; Carnahan & Bergan, 2022; Gruzd et al., 2022). In addition, there have been some attempts to implement intervention strategies, such as accuracy prompts, debunking, friction, inoculation, media literacy, and self-reflection tools (e.g., Koch et al., 2023; Scharrer et al., 2022; Singh & Banga, 2022; Traberg et al., 2022; Vivion et al., 2022). Further research is necessary to fully grasp the phenomenon of misinformation and develop effective countering strategies, given the constantly evolving techniques of misinformation spreaders and changing nature of social media platforms and social norms.

Building on the previous research in these areas, the special issue proposes an integrated and multidisciplinary approach toward addressing the challenges posed by the growth of anti-social behavior and the proliferation of misinformation on social media. The articles in the issue demonstrate the complex and intertwined nature of these matters; for example, in cases where online harassment is employed as part of strategic disinformation campaigns, and vice versa. The issue incorporates a diverse range of works, including both conceptual and empirical case studies, and examines both human and algorithmic aspects of content moderation and platform governance. It also takes a critical view of user practices, community engagement, and the network effects of anti-social behavior. Below is a brief overview of the works included in the issue.

Platform Governance

The issue starts with three papers on platform governance (Haythornthwaite, 2023; Nagappa, 2023; Zuckerman & Rajendra-Nicolucci, 2023). By examining the role of platform and community governance, these studies offer potential strategies for improving online moderation practices and fostering healthier online spaces.

In “Moderation, Networks, and Anti-Social Behavior Online,” Haythornthwaite discusses the challenges of moderating extreme content on major social media platforms. The author examines the questions of how to define and identify anti-social behavior online and how to effectively use automation and human review to manage offending content. The paper suggests a framework of three layers (environment, community, and crowd) and emphasizes the importance of understanding the network impact of anti-social behavior.

In “From Community Governance to Moderation and Back Again: Re-examining Pre-Web Models of Online Governance to Address Trust and Safety’s Crisis of Legitimacy,” Zuckerman and Rajendra-Nicolucci argue that the issues of content moderation on social media are central to democratic participation. They review early models of social media content moderation, considering whether the “free speech” and “public health” approaches to moderation might have obscured an earlier model of community-led content moderation. The authors advocate for a community moderation approach to social media, which they argue could address persistent challenges of social media moderation and provide valuable training in democratic participation.

In “Narratives of Change to Platform Governance on DTube, an Emerging Blockchain-based Video-sharing Platform,” Nagappa discusses the emergence of blockchain-based social media platforms as alternatives to mainstream social media platforms. The paper focuses on the changes to one such platform called DTube and its governance structure. It highlights how user practices, rather than technology, steer platform functions.

Computational Techniques for Investigating Anti-Social Behavior and Disinformation and Misinformation

The next set of three articles are focused on methodology and present advanced computational techniques for investigating anti-social behavior and the dissemination of disinformation and misinformation on social media (Angus et al., 2023; Giglietto et al., 2023; Steinfeld, 2023).

In their paper, “Computational Communication Methods for Examining Problematic News-Sharing Practices on Facebook at Scale,” Angus et al. present a novel approach for analyzing the spread of “fake news” and other problematic information on Facebook. By analyzing networks of content sharing between public pages, groups, and external sources, the proposed method can pinpoint the most prominent sources of problematic content. The result is a comprehensive understanding of the impact that misinformation and disinformation can have on societies that are interconnected through social media.

In “A Workflow to Detect, Monitor and Update Lists of Coordinated Social Media Accounts Across Time: The Case of 2022 Italian Election,” Giglietto et al. propose a methodology for detecting coordinated social media accounts in the context of political elections. The authors applied their technique to uncover various instances of potentially coordinated accounts engaged in discussions about the 2022 Italian election, including politically motivated, click-driven, and religiously motivated operations.

In “How Do Users Examine Online Messages to Determine If They Are Credible? An Eye-Tracking Study of Digital Literacy, Visual Attention to Metadata and Success in Misinformation Identification,” Steinfeld investigates how users assess the credibility of online information and the association between user attention to metadata, digital literacy, and the identification of misinformation. The author found that users with advanced digital literacy tend to focus more on information metadata and are better equipped at spotting online misinformation.

Complexities of Managing Online Content and Behavior across Various Platforms

The issue concludes with five case studies that offer insights into the complexities of managing online content and behavior across various platforms, contextual backgrounds, and user demographics. These case studies have been authored by B. Chen, Lukito, and Koo (2023), Salles et al. (2023), Morales (2023), Hodson and O’Meara (2023), and Musiyiwa and Jacobson (2023).

In their research article, “Comparing the #StopTheSteal Movement across Multi-platform: Differentiating Discourse on Facebook, Twitter, and Parler,” Chen, Lukito, and Koo analyze the discourse around the #StopTheSteal movement on Facebook, Twitter, and Parler in the aftermath of the 2020 US Presidential election and leading up to the Capitol Riot. The authors employ Snow and Benford’s Social Movement Frames typology and specifically explore the presence of violence cues. Their findings indicate that Parler, an alternate platform, was more inclined toward inciting violence through the use of aggressive language, thereby exacerbating the call to action, as opposed to Facebook or Twitter.

In “The Far-Right Smokescreen: Environmental Conspiracy and Culture Wars on Brazilian YouTube” by Salles et al., the authors examine how the conservative YouTube channel “Brasil Paralelo” used platform affordances and political alignment to gain social relevance on environmental conspiracies. The study, which used topic modeling and network analysis, discovered that far-right rhetoric opposing environmentalism is employed as a contemporary culture war weapon. This narrative often contains unfounded allegations concerning politics, gender, religion, and other ideological themes.

In the article on “Ecologies of Violence on Social Media: An Exploration of Practices, Contexts, and Grammars of Online Harm,” Morales argues for the need to understand the ways that violence is performed and communicated on social media. Using a case study of young adults in Colombia, the author demonstrates the complexity of violence on social media platforms, which is often multifaceted, overlapping, and interconnected. Morales proposes a framework for understanding and addressing harmful practices as ecologies of violence, which includes examining the practices, contexts, and grammars that contribute to online harm. The author stresses the importance of recognizing and addressing this complexity to build online and offline “cultures of peace.”

Hodson and O’Meara’s study, “Curating Hope: The Aspirational Self and Social Engagement in Early-Onset Cancer Communities on Social Media,” examines online communities and discourse among young cancer patients and caregivers on social media. The authors note that these communities offer more than just information and emotional support, they also inspire hope and confront the traditional division between online authenticity and the aspirational self often present on social media.

Finally, in their work titled “Sponsorship Disclosure in Social Media Influencer Marketing: The Algorithmic and Non-Algorithmic Barriers,” Musiyiwa and Jacobson explore the obstacles that hinder compliance with sponsorship disclosure on social media. The study reveals that both algorithmic and non-algorithmic challenges exist, making it difficult to ensure proper disclosure. Such challenges include the deprioritization of disclosure by algorithms and time-consuming disclosure procedures. The researchers suggest various strategies that influencers can employ to effectively disclose their sponsored content upfront and in a prominent manner.

Closing Thoughts

Overall, the diverse perspectives and innovative approaches presented in all 11 papers contribute to a deeper understanding of the challenges and opportunities in managing social media platforms, fostering healthier online spaces, and guiding future research in this rapidly evolving field. While the papers tackle various issues, their collective message emphasizes the importance of a comprehensive strategy that incorporates automated and manual review, community governance, and user literacy to effectively combat emerging challenges in digital spaces.

As the editors of the special issue, our hope is that it will reach a broad audience of social media stakeholders, including policymakers, developers, and platform owners and users. Researchers from a range of fields, such as communication, information technology, media studies, sociology, psychology, computer science, and other fields, will benefit from the theoretical frameworks, methodologies, and findings provided in these papers. Social media platform developers and managers can use the insights gained from the issue to inform design decisions related to content moderation, community governance, and user engagement. In addition, policymakers and regulators, educators, digital literacy advocates, and the general public may use this research to develop a deeper understanding of issues related to online behavior, moderate content, misinformation, and user safety.