(In)visible moderation: A digital ethnography of marginalized users and content moderation on Twitch and Reddit

Abstract

Research suggests that marginalized social media users face disproportionate content moderation and removal. However, when content is removed or accounts suspended, the processes governing content moderation are largely invisible, making assessing content moderation bias difficult. To study this bias, we conducted a digital ethnography of marginalized users on Reddit’s /r/FTM subreddit and Twitch’s “Just Chatting” and “Pools, Hot Tubs, and Beaches” categories, observing content moderation visibility in real time. We found that on Reddit, a text-based platform, platform tools make content moderation practices invisible to users, but moderators make their practices visible through communication with users. Yet on Twitch, a live chat and streaming platform, content moderation practices are visible in channel live chats, “unban appeal” streams, and “back from my ban” streams. Our ethnography shows how content moderation visibility differs in important ways between social media platforms, harming those who must see offensive content, and at other times, allowing for increased platform accountability.

Keywords

Content moderation digital ethnography marginalization Reddit social media platforms Twitch visibility

Get full access to this article

View all access options for this article.

References

Alford

(2021) How transgender Redditors are being driven from the site. Jezebel, 7 September. Available at: https://jezebel.com/transgender-redditors-are-being-driven-from-the-site-by-1847256024

Angwin

ProPublica Grassegger

(2017) Facebook’s secret censorship rules protect white men from hate speech but not black children. Propublica, 28 June. Available at: https://www.propublica.org/article/facebook-hate-speech-censorship-internal-documents-algorithms

Are

(2020) How Instagram’s algorithm is censoring women and vulnerable users but helping online abusers. Feminist Media Studies 20(5): 741–744.

Asarch

(2021) Hot tub livestreamers broadcasting in bikinis on Twitch have divided the platform’s community. Insider. Available at: https://www.insider.com/twitch-hot-tub-meta-controversy-explained-2021-5

Biddle

Ribeiro

Dias

(2020) Invisible censorship: TikTok told moderators to suppress posts by “ugly” people and the poor to attract new users. The Intercept, 16 March. Available at: https://theintercept.com/2020/03/16/tiktok-app-moderators-users-discrimination/

Binns

Veale

Van Kleek

, et al. (2017) Like trainer, like bot? Inheritance of bias in algorithmic content moderation. In: Ciampaglia

Mashhadi

Yasseri

(eds) Social Informatics. Berlin: Springer, pp. 405–415.

Blunt

Wolf

(2020) Erased: the impact of FOSTA-SESTA and the removal of backpage. Hacking//hustling. Available at: https://hackinghustling.org/erased-the-impact-of-fosta-sesta-2020/

Botella

(2019) TikTok admits it suppressed videos by disabled, queer, and fat creators. Slate, 4 December. Available at: https://slate.com/technology/2019/12/tiktok-disabled-users-videos-suppressed.html

Bradford

Grisel

Meares

, et al. (2019) Report of the Facebook data transparency advisory group. The Justice Collaboratory, Yale Law School, New Haven, CT, April.

10.

Braun

Clarke

(2006) Using thematic analysis in psychology. Qualitative Research in Psychology 3(2): 77–101.

11.

Buolamwini

Gebru

(2018) Gender shades: intersectional accuracy disparities in commercial gender classiﬁcation. Proceeding of Machine Learning Research 81: 77–91.

12.

Burrell

(2009) The field site as a network: a strategy for locating ethnographic research. Field Methods 21(2): 181–199.

13.

Cai

Wohn

(2019) Categorizing live streaming moderation tools: an analysis of Twitch. International Journal of Interactive Communication Systems and Technologies 9(2): 36–50.

14.

Cai

Wohn

Almoqbel

(2021) Moderation visibility: mapping the strategies of volunteer moderators in live streaming micro communities. In: Proceedings of IMX 2021: ACM international conference on interactive media experiences, 2123 June, New York, pp. 61–72.

15.

Caplan

(2018) Content or Context Moderation? Artisanal, Community-Reliant, and Industrial Approaches. New York: Data & Society Research Institute.

16.

Chandrasekharan

Pavalanathan

Srinivasan

, et al. (2017) You can’t stay here: the efficacy of Reddit’s 2015 ban examined through hate speech. Proceedings of the ACM on Human-Computer Interaction 1: 31.

17.

Crawford

Gillespie

(2014) What is a flag for? Social media reporting tools and the vocabulary of complaint. new media & society 18(3): 410–428.

18.

D’Anastasio

(2021) Twitch sues users over alleged “hate raids” against streamers. Wired, 10 September. Available at: https://www.wired.com/story/twitch-sues-users-over-alleged-hate-raids/

19.

Diaz

(2021) Twitch bans Amouranth and Indiefoxx after yoga ASMR streams. Polygon, 21 June. Available at: https://www.polygon.com/22543478/twitch-amouranth-indiefoxx-yoga-asmr-stream-hot-tub-meta-ban

20.

Electronic Frontier Foundation (2019) EFF Project Shows How People are unfairly “Tossed Out” by Platforms’ Absurd Enforcement of Content Rules. San Francisco, CA: Electronic Frontier Foundation.

21.

Ferrari

(2021) Twitch’s #BikiniGate as seen by streamers: “It’s just to criticize women..” Madmoizelle, 25 May. Available at: https://www.madmoizelle.com/le-bikinigate-vu-par-les-streameuses-cest-juste-pour-critiquer-les-femmes-1127836

22.

Feuston

Taylor

Piper

(2020) Conformity of eating disorders through content moderation. Proceedings of the ACM on Human-Computer Interaction 4: 40.

23.

franzke

Bechmann

Zimmer

, et al. (2020) Internet Research: Ethical Guidelines 3.0. Chicago, IL: Association of Internet Research.

24.

Gadde

Beykpour

(2018) Setting the record straight on shadow banning. Twitter Blog, 26 July. Available at: https://blog.twitter.com/en_us/topics/company/2018/Setting-the-record-straight-on-shadow-banning

25.

Gebel

(2020) Black creators say TikTok still secretly hides their content. Digital Trends, 21 July. Available at: https://www.digitaltrends.com/social-media/black-creators-claim-tiktok-still-secretly-blocking-content/

26.

Gerrard

Thornham

(2020) Content moderation: social media’s sexist assemblages. new media & society 22(7): 1266–1286.

27.

Gilbert

(2020) “I run the world’s largest historical outreach project and it’s on a cesspool of a website.” Moderating a public scholarship site on Reddit: a case study of r/AskHistorians. Proceedings of the ACM on Human-Computer Interaction 4: 19.

28.

Gillespie

(2017) Governance of and by platforms. In: Burgess

Poell

Marwick

(eds) SAGE Handbook of Social Media. Thousand Oaks, CA: SAGE, pp. 254–278.

29.

Gillespie

(2018) Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. New Haven, CT: Yale University Press.

30.

Gray

Leonard

(eds) (2018) Woke Gaming: Digital Challenges to Oppression and Social Injustice. Seattle, WA: University of Washington Press.

31.

Grayson

(2021a) Hot tub streams are waning, but some people still think they’re destroying Twitch. Kotaku, 23 April. Available at: https://kotaku.com/hot-tub-streams-are-waning-but-some-people-still-think-1846749371

32.

Grayson

(2021b) Marginalized streamers beg Twitch to “do better” in wake of hate raids, poor pay. The Washington Post, 11 August. Available at: https://www.washingtonpost.com/video-games/2021/08/25/twitch-hate-raids-streamers-discord-cybersecurity/

33.

Grayson

(2021c) Twitch’s “Hot Tub Meta” has sparked off yet another debate about women’s attire. Kotaku, 2 April. Available at: https://kotaku.com/twitchs-hot-tub-meta-has-sparked-off-yet-another-debate-1846600932

34.

Guynn

(2019) Facebook while black: users call it getting “Zucked,” say talking about racism is censored as hate speech. USA Today, 24 April. Available at: https://www.usatoday.com/story/news/2019/04/24/facebook-while-black-zucked-users-say-they-get-blocked-racism-discussion/2859593002/

35.

Habib

Musa

Zaffar

, et al. (2019) To act or react: investigating proactive strategies for online community moderation. arXiv. Available at: https://arxiv.org/abs/1906.11932

36.

Haimson

Dame-Griff

Capello

, et al. (2021a) Tumblr was a trans technology: the meaning, importance, history, and future of trans technologies. Feminist Media Studies 21(3): 345–361.

37.

Haimson

Delmonaco

Nie

, et al. (2021b) Disproportionate removals and differing content moderation experiences for conservative, transgender, and black social media users: marginalization and moderation gray areas. Proceedings of the ACM on Human-Computer Interaction 5: 466.

38.

Hamilton

Garretson

Kerne

(2014) Streaming on twitch: fostering participatory communities of play within live mixed media. In: Proceedings of the SIGCHI conference on human factors in computing systems, Toronto, ON, Canada, 26 April–1 May, pp. 1315–1324. New York: Association for Computing Machinery.

39.

Hine

(2000) Virtual Ethnography. London; Thousand Oaks, CA: SAGE.

40.

Jan

Dwoskin

(2017) A white man called her kids the n-word. Facebook stopped her from sharing it. The Washington Post, 31 July. Available at: https://www.washingtonpost.com/business/economy/for-facebook-erasing-hate-speech-proves-a-daunting-challenge/2017/07/31/922d9bc6-6e3b-11e7-9c15-177740635e83_story.html

41.

Jhaver

Appling

Gilbert

, et al. (2019a) “Did you suspect the post would be removed?” Understanding user reactions to content removals on Reddit. Proceedings of the ACM on Human-Computer Interaction 3: 192.

42.

Jhaver

Bruckman

Gilbert

(2019b) Does transparency in moderation really matter? User behavior after content removal explanations on Reddit. Proceedings of the ACM on Human-Computer Interaction 3: 150.

43.

Köver

(2019) Discrimination: TikTok curbed reach for people with disabilities. Netzpolitik, 2 December. Available at: https://netzpolitik.org/2019/discrimination-tiktok-curbed-reach-for-people-with-disabilities/

44.

Lux

Lil Miss Hot Mess (2017) Facebook’s hate speech policies censor marginalized users. Wired, 14 August. Available at: https://www.wired.com/story/facebooks-hate-speech-policies-censor-marginalized-users/

45.

Markham

(2012) Fabrication as ethical practice. Information, Communication & Society 15(3): 334–353.

46.

Massanari

(2015) Participatory Culture, Community, and Play: Learning from Reddit. Bern: Peter Lang.

47.

Massanari

(2017) #Gamergate and the fappening: how Reddit’s algorithm, governance, and culture support toxic technocultures. new media & society 19(3): 329–346.

48.

Middlebrook

(2020) The Grey Area: Instagram, Shadowbanning, and the Erasure of Marginalized Communities. Rochester, NY: Social Science Research Network.

49.

Murthy

(2008) Digital ethnography: an examination of the use of new technologies for social research. Sociology 42(5): 837–855.

50.

Murthy

(2011) Emergent digital ethnographic methods for social research. In: Hesse-Biber

(ed.) Handbook of Emergent Technologies in Social Research. Oxford: Oxford University Press, pp. 158–179.

51.

Nakamura

(2015) The unwanted labour of social media: women of colour call out culture as venture community management. New Formations 86: 106–112.

52.

Nightingale

(2021) Hot tub Twitch streamer banned after transphobic pile-on slams platform for failing creators. PinkNews, 13 July. Available at: https://www.pinknews.co.uk/2021/07/13/twitch-trans-streamer-anne-atomic-2/

53.

Oliva

Antonialli

Gomes

(2021) Fighting hate speech, silencing drag queens? Artificial intelligence in content moderation and risks to LGBTQ voices online. Sexuality & Culture 25(2): 700–732.

54.

Reddit (2021a) R/changelog safety update on Reddit’s follow feature. Reddit, 14 July. Available at: https://www.reddit.com/r/changelog/comments/ok8xzb/safety_update_on_reddits_follow_feature/ (accessed 22 April 2022).

55.

Reddit (2021b) R/changelog you can now opt-out of being followed. Reddit, 25 August. Available at: https://www.reddit.com/r/changelog/comments/pbjshi/you_can_now_optout_of_being_followed/ (accessed 22 April 2022).

56.

Reddit (2022a) AutoModerator. Reddit. Available at: https://mods.reddithelp.com/hc/en-us/articles/360002561632-AutoModerator (accessed 22 April 2022).

57.

Reddit (2022b) General moderation info. Reddit. Available at: https://mods.reddithelp.com/hc/en-us/categories/360000255232-General-Moderation-Info (accessed 22 April 2022).

58.

Reddit (2022c) R/blog – announcing blocking updates. Reddit, 18 January. Available at: https://www.reddit.com/r/blog/comments/s71g03/announcing_blocking_updates/ (accessed 22 April 2022).

59.

Roberts

(2016) Commercial content moderation: digital laborers’ dirty work. In: Noble

Tynes

(eds) The Intersectional Internet: Race, Sex, Class and Culture Online. Bern: Peter Lang, p. 12.

60.

Roberts

(2019) Behind the Screen: Content Moderation in the Shadows of Social Media. New Haven, CT: Yale University Press.

61.

Rodriguez

(2019) Instagram apologizes to pole dancers after hiding their posts. CTVnews, 6 August. Available at: https://www.ctvnews.ca/sci-tech/instagram-apologizes-to-pole-dancers-after-hiding-their-posts-1.4537820?cache=

62.

Rosen

(2019) Remove, reduce, inform: new steps to manage problematic content. Meta, 10 April. Available at: https://about.fb.com/news/2019/04/remove-reduce-inform-new-steps/

63.

Ruberg

(2021) “Obscene, pornographic, or otherwise objectionable”: biased definitions of sexual content in video game live streaming. new media & society 23(6): 1681–1699.

64.

Salty (2019) Exclusive: an investigation into algorithmic bias in content policing on Instagram. Salty, 23 October. Available at: https://twitter.com/saltyworldbabes/status/1313579751546728449

65.

Seering

Kraut

Dabbish

(2017) Shaping pro and anti-social behavior on Twitch through moderation and example-setting. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (CSCW’17), Portland, OR, 25 February–1 March, pp. 111–125. New York: ACM.

66.

Seering

Wang

Yoon

, et al. (2019) Moderator engagement and community development in the age of algorithms. new media & society 21(7): 1417–1443.

67.

Smith

Haimson

Fitzsimmons

, et al. (2021) Censorship of marginalized communities on Instagram. Salty, 29 September. Available at: https://twitter.com/Saltyworldbabes/status/1443068587170803712

68.

Sullivan

Dale

(2022) One of Marjorie Taylor Greene’s verified Twitter accounts permanently suspended from Twitter. CNN, 2 January. Available at: https://edition.cnn.com/2022/01/02/politics/marjorie-taylor-greene-twitter-suspension/index.html

69.

Suzor

West

Quodling

, et al. (2019) What do we mean when we talk about transparency? Toward meaningful transparency in commercial content moderation. International Journal of Communication 13: 18.

70.

Sybert

(2021) The demise of #NSFW: contested platform governance and Tumblr’s 2018 adult content ban. new media & society. Epub ahead of print 26 February. DOI: 10.1177/1461444821996715.

71.

TeamYouTube (2020) @Herclueless we don’t shadowban channels. . .https://t.co/f25cOgmwRV [Tweet]. @Teamyoutube, 22 October. Available at: https://twitter.com/TeamYouTube/status/1319372516398452737?ref_src=twsrc%5Etfw

72.

Twitch (2021) Let’s Talk About Hot Tub Streams. Twitch Blog. Available at: https://blog.twitch.tv/en/2021/05/21/lets-talk-about-hot-tub-streams

73.

Twitter Inc (2021) Permanent suspension of @realDonaldTrump. Twitter Blog, 8 January. Available at: https://blog.twitter.com/en_us/topics/company/2020/suspension

74.

Uttarapong

Cai

Wohn

(2021) Harassment experiences of women and LGBTQ live streamers and how they handled negativity. In: ACM international conference on interactive media experiences, 2123 June, pp. 7–19. New York: Association for Computing Machinery.

75.

Vaccaro

Sandvig

Karahalios

(2020) “At the end of the day Facebook does what it wants”: how users experience contesting algorithmic content moderation. Proceedings of the ACM on Human-Computer Interaction 4: 167.

76.

West

(2018) Censored, suspended, shadowbanned: user interpretations of content moderation on social media platforms. new media & society 20(11): 4366–4383.

77.

Winter

Lavis

(2020) Looking, but not listening? Theorizing the practice and ethics of online ethnography. Journal of Empirical Research on Human Research Ethics 15(1–2): 4366–4383.

78.

Zolides

(2020) Gender moderation and moderating gender: sexual content policies in Twitch’s community guidelines. new media & society 23(10): 2999–3015.