Content moderators (CMs) analyze and remove offensive or harmful user generated content that has been uploaded to the internet. Jobs which involve exposure to other people's suffering are associated with raised rates of secondary traumatic stress and mental health problems. However, research establishing psychological baseline symptoms in CMs is lacking. This study used an online survey to explore rates of psychological distress, secondary trauma, and well-being in a sample of CMs. Regression analysis explored how various features of the work affected mental health. There was a dose–response effect between frequency of exposure to distressing content and psychological distress and secondary trauma, but not well-being. The results suggested supportive colleagues and feedback about the importance of their role ameliorated this relationship. Implications for CM working conditions are discussed.
Get full access to this article
View all access options for this article.
References
1.
AhmadS, KrzywdzinskiM.Moderating in obscurity: How indian content moderators work in global content moderation value chains. In: Digital Work in the Planetary Market. (GrahamM, FerrariF. eds.) The MIT Press: Cambridge, MA, USA; 2022; pp. 77–95.
2.
BarrettPM.Who Moderates the Social Media Giants? A Call to End Outsourcing. New York University Stern Center for Business and Human Rights: New York, NY, USA; 2020.
3.
GillespieT.Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. Yale University Press: New Haven, CT, USA; 2018.
4.
BurnsCM, MorleyJ, BradshawR, et al.The emotional impact on and coping strategies employed by police teams investigating internet child exploitation. Traumatology, 2008; 14(2):20–31; doi: 10.1177/1534765608319082.
5.
FoleyJ, MasseyKLD. The ‘cost’ of caring in policing: From burnout to PTSD in police officers in England and Wales. Police J, 2021; 94(3):298–315; doi: 10.1177/0032258X20917442.
6.
PerezLM, JonesJ, EnglertDR, et al.Secondary traumatic stress and burnout among law enforcement investigators exposed to disturbing media images. J Police Crim Psychol, 2010; 25:113–124; doi: 10.1007/s11896-010-9066-7.
7.
ShojiK, LesnierowskaM, SmoktunowiczE, et al.What comes first, job burnout or secondary traumatic stress? Findings from two longitudinal studies from the U.S. and Poland. PLoS One, 2015; 10(8):e0136730; doi: 10.1371/journal.pone.0136730.
8.
KrauseM.Identifying and managing stress in child pornography and child exploitation investigators. J Police Crim Psychol, 2009; 24:22–29; doi: 10.1007/s11896-008-9033-8.
9.
LevinAP, PutneyH, CrimminsD, et al.Secondary traumatic stress, burnout, compassion satisfaction and perceived organizational trauma readiness in forensic science professionals. J Forensic Sci, 2021; 66(5):1758–1769; doi: 10.1111/1556-4029.14747.
10.
GreinacherA, Derezza-GreevenC, HerzogW, et al.Secondary traumatization in first responders: A systematic review. Eur J Psychotraumatol, 2019; 10(1):1562840; doi: 10.1080/20008198.2018.1562840.
11.
SollieH, KopN, EuwemaMC. Mental resilience of crime scene investigators: How police officers perceive and cope with the impact of demanding work situations. Crim Justice Behav, 2017; 44(12):1580–1603; doi: 10.1177/0093854817716959.
12.
BourkeML, CraunSW. Coping with secondary traumatic stress: Differences between U.K. and U.S. child exploitation personnel. Traumatology, 2014; 20(1):57–64; doi: 10.1037/h0099381.
13.
RalphN.Dealing with the personal impact of crimes against children. College of Policing; 2020. Available from: https://www.college.police.uk/article/dealing-personal-impact-crimes-against-children [Last accessed: January14, 2023].
14.
StanleyIH, HomMA, JoinerTE. A systematic review of suicidal thoughts and behaviors among police officers, firefighters, EMTs, and paramedics. Clin Psychol Rev, 2016; 44:25–44; doi: 10.1016/j.cpr.2015.12.002.
15.
DwoskinE.Inside Facebook, the second-class workers who do the hardest job are waging a quiet battle. The Washington Post; 2019. Available from: https://www.washingtonpost.com/technology/2019/05/08/inside-facebook-second-class-workers-who-do-hardestjob-are-waging-quiet-battle [Last accessed: January15, 2023].
16.
SteigerM, BharuchaTJ, VenkatagiriS, et al.The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In: CHI Conference on Human Factors in Computing Systems (CHI’21), May 8–13, 2021; Yokohama, Japan; doi: 10.1145/3411764.3445092.
17.
DosonoB, SemaanB. Moderation practices as emotional labor in sustaining online communities: The case of AAPI identity work on Reddit. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019; pp. 1–13; doi: 10.1145/3290605.3300372.
18.
LoC.When All You Have is a Banhammer: The Social and Communicative Work of Volunteer Moderators. PhD Thesis, Massachusetts Institute of Technology: Cambridge, MA, USA; 2018.
19.
SpenceR, BifulcoA, BradburyP, et al.The psychological impacts of content moderation on content moderators: A qualitative study. Cyberpsychology, 2023; 17(4):Article 8; doi: 10.5817/CP2023-4-8.
20.
WohnDY.Volunteer moderators in twitch micro communities: How they get involved, the roles they play, and the emotional labor they experience. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland, UK) (CHI’19). Association for Computing Machinery: New York, NY, USA; 2019; Article 160; doi: 10.1145/3290605.3300390.
21.
CookCL, CaiJ, WohnDY. Awe versus aww: The effectiveness of two kinds of positive emotional stimulation on stress reduction for online content moderators. Proc ACM Hum Comput Interact, 2022; 6:1–19; doi: 10.1145/3555168.
22.
RiedlMJ, MasulloGM, WhippleKN. The downsides of digital labor: Exploring the toll incivility takes on online comment moderators. Comput Human Behav, 2020; 107:106262; doi: 10.1016/j.chb.2020.106262.
23.
RobertsST.Commercial content moderation and worker wellness: Challenges & opportunities. Techdirt; 2018. Available from: https://www.techdirt.com/2018/02/08/commercial-content-moderation-worker-wellness-challenges-opportunities [Last accessed: January12, 2023].
24.
DwoskinE, WhalenJ, CabatoR. Content moderators at YouTube, Facebook and Twitter see the worst of the web—And suffer silently. The Washington Post; 2019. Available from: https://www.washingtonpost.com/technology/2019/07/25/social-media-companies-are-outsourcing-their-dirty-work-philippines-generation-workers-is-paying-price [Last accessed: January12, 2023].
25.
NewtonC.The Terror Queue. The Verge; 2019. Available from: https://www.theverge.com/2019/12/16/21021005/google-youtube-moderators-ptsd-accenture-violent-disturbing-content-interviews-video [Last accessed: January12, 2023].
26.
SpenceR, HarrisonA, BradburyP, et al.Content moderators' strategies for coping with the stress of moderating content online. J Online Trust Saf, 2023; 1(5); doi: 10.54501/jots.v1i5.91.
27.
BradyPQ, FansherAK, ZedakerSB. Are parents at a higher risk for secondary traumatic stress?: How interviewing child victims impacts relationships with forensic interviewer's friends and family. Child Abuse Negl, 2019; 88:275–287; doi: 10.1016/j.chiabu.2018.11.017.
28.
FoleyJ, HassettA, WilliamsE. ‘Getting on with the job’: A systematised literature review of secondary trauma and post-traumatic stress disorder (PTSD) in policing within the United Kingdom (UK). Police J, 2022; 95(1):224–252; doi: 10.1177/0032258X21990412.
29.
BradyPQ.Crimes against caring: Exploring the risk of secondary traumatic stress, burnout, and compassion satisfaction among child exploitation investigators. J Police Crim Psychol, 2017; 32:305–318; doi: 10.1007/s11896-016-9223-8.
30.
GrayML, SuriS.Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt: Boston, MA, USA; 2019.
31.
PowellM, CassematisP, BensonM, et al.Police officers' perceptions of the challenges involved in internet child exploitation investigation. Policing Int J, 2014; 37(3):543–557; doi: 10.1108/PIJPSM-08-2013-0080.
32.
SpenceR, DeMarcoJ, MartellozzoE.Invisible Workers, Hidden Dangers. The Psychologist; 2023.
33.
DoyleM, TapsonK, KaragiannopoulosV, et al.Impacts of organisational role and environmental factors on moral injury and trauma amongst police investigators in Internet Child Abuse Teams. Police J, 2023; 96(1):153–171; doi: 10.1177/0032258X211043331.
34.
TennantR, HillerL, FishwickR, et al.The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): Development and UK validation. Health Qual Life Outcomes, 2007; 5:63; doi: 10.1186/1477-7525-5-63.
35.
BarkhamM, BewickB, MullinT, et al.The CORE-10: A short measure of psychological distress for routine use in the psychological therapies. Couns Psychotherapy Res, 2013; 13(1):3–13; doi: 10.1080/14733145.2012.729069.
36.
ConwayP, LundriganS, RedmondT, et al.Protecting the Protectors: Moral Injury, Coping Styles, and Mental Health of UK Police Officers and Staff Investigating Child Sexual Abuse Material. 2023. Available from: https://www.stah.org/assets/Uploads/04.-Dr-Paul-Conway.pdf [Last accessed: December17, 2023].
37.
Ng FatL, ScholesS, BonifaceS, et al.Evaluating and establishing national norms for mental wellbeing using the short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS): Findings from the Health Survey for England. Qual Life Res, 2017; 26(5):1129–1144; doi: 10.1007/s11136-016-1454-8.
38.
AltemusM, SarvaiyaN, EppersonCN. Sex differences in anxiety and depression clinical perspectives. Front Neuroendocrinol, 2014; 35(3):320–330; doi: 10.1016/j.yfrne.2014.05.004.
39.
BoydA, Van de VeldeS, VilagutG, et al.Gender differences in mental disorders and suicidality in Europe: Results from a large cross-sectional population-based study. J Affect Disord, 2015; 173:245–254; doi: 10.1016/j.jad.2014.11.002.
40.
AdamsSA, RiggsSA. An exploratory study of vicarious trauma among therapist trainees. Train Educ Prof Psychol, 2008; 2(1):26–34; doi: 10.1037/1931-3918.2.1.26.
41.
GrayC, Rydon-GrangeM. Individual characteristics, secondary trauma and burnout in police sexual and violent offending teams. Police J, 2020; 93(2):146–161; doi: 10.1177/0032258X19847499.
42.
HurrellAK, DraycottS, AndrewsL. Secondary traumatic stress in police officers investigating childhood sexual abuse. Policing Int J, 2018; 41(5):636–650; doi: 10.1108/PIJPSM-08-2016-0131.
43.
BoscarinoJA, FigleyCR, AdamsRE. Compassion fatigue following the September 11 terrorist attacks: A study of secondary trauma among New York City social workers. Int J Emerg Ment Health, 2004; 6(2):57.
44.
ColesJ, AstburyJ, DartnallE, et al.A qualitative exploration of researcher trauma and researchers' responses to investigating sexual violence. Violence Against Women, 2014; 20(1):95–117; doi: 10.1177/1077801213520578.
45.
RobertsST.Behind the Screen: Content Moderation in the Shadows of Social Media. Yale University Press: New Haven, CT, USA; 2019.
