Anti-discrimination is a crucial focus for researchers, policymakers and designers of technical systems. Leveraging insights from AI fairness and inclusion studies, we examine algorithmic bias and its tendency to perpetuate discrimination (Selbst et al., 2019, Fairness and abstraction in sociotechnical systems, pp. 59–68) through two case studies: one in a gaming environment and another on a dating platform. Our research on gaming explores the manifestation of toxicity, particularly its impact on gender identities. We examine the normalisation of toxic behaviours within gaming spaces, negatively influencing gendered perceptions and experiences. In the second study, we conduct an experiment on the Bumble dating platform to uncover and address discrimination within the matching algorithms, which influence the profiles shown to users as potential dates. These interactions in user discovery pose significant risks, including sexual violence and other harms, especially for diverse gender communities on dating platforms. By documenting the experiences of non-binary individuals on dating platforms, we contribute to a largely underexplored area of research. This study not only highlights the discrepancies and biases in dating algorithms but also aims to inspire more inclusive design principles that recognise and accommodate the diverse needs of genders and communities.
Adelphi University. (2023, March17). Examining how gender bias is built into AI. https://www.adelphi.edu/news/examining-how-gender-bias-is-built-into-ai/
2.
AhmadiM., EilertR., WeibertA., WulfV., & MarsdenN. (2019). Hacking masculine cultures: Career ambitions of female young professionals in a video game company. Proceedings of the Annual Symposium on Computer–Human Interaction in Play (CHI PLAY), New York. Association for Computing Machinery, pp. 413–426.
3.
BadjatiyaP., GuptaS., GuptaM., & VarmaV. (2017). Deep learning for hate speech detection in tweets. Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, pp. 759–760
4.
BartlettR., MorseA., StantonR., & WallaceN. (2022). Consumer-lending discrimination in the fintech era. Journal of Financial Economics, 143(1), 30–56.
5.
BeresN., FrommelJ., ReidE., MandrykR. L., & KlarkowskiM. (2021, May8–13). Don’t you know that you’re toxic: Normalization of toxicity in online gaming. CHI Conference on Human Factors in Computing Systems (CHI), Yokohama, Japan. ACM, p. 15.
6.
BuolamwiniJ., & GebruT. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency. PMLR, pp. 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
7.
BuonoF. D., PaulE., SprongM. E., SmithE. C., GarakaniA., & GriffithsM. D. (2020). Gaming and gaming disorder: A mediation model gender, salience, age of gaming onset, and time spent gaming. Cyberpsychology, Behavior, and Social Networking, 23(9), 647–651. https://doi.org/10.1089/cyber.2019.0445
8.
ButtM.-A. (2015, October). Shut up and play: Vivian James and the presence of women in gaming cultures. The University of Sydney..
9.
ButtM.-A. R. (2017). Lifeworlds for player 2: Affective networks, affordances, and gender dynamics in player-computer interaction. In Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’17 (pp. 687–690). Association for Computing Machinery.
10.
ButtM.-A. R. (2019, October2–5). ‘I never use headsets’: Women, wariness, and hypervigilance for the inevitability of online harassment in gaming culture. The 20th Annual Conference of the Association of Internet Researchers, Brisbane, Australia, p. 4.
11.
ChatzakouD., KourtellisN., BlackburnJ., De ChristofaroE., StringhiniG., & VakaliA. (2017). Measuring GamerGate: A tale of hate, sexism, and bullying. ResearchGate. https://doi.org/10.1145/3041021.3053890
12.
DahlJ. (2014, October17). ‘Gamergate’, guns and threats against women collide in Utah. CBS News. https://www.cbsnews.com/news/gamergate-guns-and-threats-against-women-collide-in-utah/
13.
DastinJ. (2018). Insight: Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. https://www.reuters.com/article/idUSKCN1MK0AG/
14.
DattaA., TschantzM. C., & DattaA. (2015). Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. Cornell University. http://arxiv.org/abs/1408.6491
15.
DavidsonT., BhattacharyaD., & WeberI. (2019). Racial bias in hate speech and abusive language detection datasets. Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy. Association for Computational Linguistics, pp. 25–35. https://aclanthology.org/W19-3504
16.
DeshbandhuA. (2020). Gaming culture(s) in India: Digital play in everyday life. Taylor & Francis.
17.
DeshbandhuA. (2023). Game on: Gender equality, gaming industry have miles to go. Telangana Today. https://telanganatoday.com/game-on-gender-equality-gaming-industry-have-miles-to-go
18.
DeslauriersP., Lafrance St-MartinL. I., & BonenfantM. (2020). Assessing toxic behaviour in Dead by Daylight perceptions and factors of toxicity according to the game’s official subreddit contributors. Game Studies.
19.
Díaz RedondoR. P., Fernández VilasA., Ramos MerinoM., Valladares RodríguezS. M., Torres GuijarroS., & HafezM. M. (2023). Anti-sexism alert system: Identification of sexist comments on social media using AI techniques. Applied Sciences, 13(7). https://doi.org/10.3390/app13074341
20.
DiPlacidoD. (2023). The ‘Hogwarts Legacy’ debate explained. Forbes. https://www.forbes.com/sites/danidiplacido/2023/01/12/the-hogwarts-legacy-debate-explained/
21.
DocktermanE. (2014, October16). What is #GamerGate and why are women being threatened about video games?TIME. https://time.com/3510381/gamergate-faq/
22.
DombrowskiL., HarmonE., & FoxS. (2016). Social justice-oriented interaction design: Outlining key design strategies and commitments. Proceedings of the 2016 ACM Conference on Designing Interactive Systems, Brisbane, Queensland, Australia. ACM, pp. 656–671. https://dl.acm.org/doi/10.1145/2901790.2901861
23.
FranzoniV. (2023). Gender differences and bias in artificial intelligence. In VallverdúJ. (Ed.), Gender in AI and robotics (Vol. 235). Intelligent Systems Reference Library. Springer. https://doi.org/10.1007/978-3-031-21606-0_2
24.
GaoG., MinA., & ShihP. C. (2017). Gendered design bias: Gender differences of in-game character choice and playing style in League of Legends. Proceedings of the 29th Australian Conference on Computer-Human Interaction, Brisbane, Queensland, Australia. ACM, pp. 307–317.
25.
Gematsu. (2023). Hogwarts Legacy sales top 1.5 million. https://www.gematsu.com/2023/05/hogwarts-legacy-sales-top-15-million
26.
HutsonJ., TaftJ., BarocasS., & LevyK. (2018). Debiasing desire: Addressing bias and discrimination on intimate platforms. SSRN. https://papers.ssrn.com/abstract=3244459
27.
HutsonM., ResnickP., & KrautR. E. (2012). Building successful online communities: Evidence-based social design. The MIT Press. https://direct.mit.edu/books/book/2912/Building-Successful-Online-CommunitiesEvidence
28.
JhaA., & MamidiR. (2017, August). When does a compliment become sexist? Analysis and classification of ambivalent sexism using twitter data. Proceedings of the Second Workshop on NLP and Computational Social Science. pp. 7–16. https://aclanthology.org/W17-2902
29.
KalraR. A., GuptaP., VargheseB., & RangaswamyN. (2023, December10). Exploring gender disparities in Bumble’s match recommendations. Proceedings of the 15th Annual Association of Information Systems, AIS Special Interest Group for ICT and Global, Development (SIG-GlobDev) Workshop, Hyderabad, India. https://aisel.aisnet.org/globdev2023/6/
30.
Kantar & IAMAI. (2023). Internet in India 2023. India Digital Summit. https://uat.indiadigitalsummit.in/sites/default/files/thoughtleadership/pdf/Kantar_iamai_Report_20_Page_V3_FINAL_web_0.pdf
31.
KarlekarS., & BansalM. (2018). Safecity: Understanding diverse forms of sexual harassment personal stories. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark, pp. 2805–2811.
32.
KayM., MatuszekC., & MunsonS. A. (2015). Unequal representation and gender stereotypes in image search results for occupations. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, New York. ACM, pp. 3819–3828. https://dl.acm.org/doi/10.1145/2702123.2702520
33.
KlareB. F., BurgeM. J., KlontzJ. C., BrueggeR. W. V., & JainA. K. (2012). Face recognition performance: Role of demographic information. IEEE Transactions on Information Forensics and Security, 7(6), 1789–1801.
34.
KouY. (2020). Toxic behaviours in team-based competitive gaming. Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY’20), New York. ACM, p. 12.
35.
KouY., & GuiX. (2021). Flag and flaggability in automated moderation: The case of reporting toxic behavior in an online game community. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York. Association for Computing Machinery, pp. 1–12.
36.
KowertR., MartelA., & SwannW. B. (2023). You are what you play: The risks of identity fusion in toxic gamer cultures. Games: Research and Practice, 1(2), 17:1–17:3.
37.
KrautR. E., & ResnickP. (2012, March). Building successful online communities: Evidence-based social design. The MIT Press. https://direct.mit.edu/books/book/2912/Building-Successful-Online-CommunitiesEvidence
38.
LambrechtA., & TuckerC. (2019). Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management Science, 65(7), 2966–2981.
39.
LahotiP., GummadiK. P., & WeikumG. (2019). ifair: Learning individually fair data representations for algorithmic decision making. IEEE 35th International Conference on Data Engineering (ICDE), pp. 1334–1345.
40.
Lopez-FernandezO., WilliamsA. J., & KussD. J. (2019). Measuring female gaming: Gamer profile, predictors, prevalence, and characteristics from psychological and gender perspectives. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.00898
MacLeodC., & McArthurV. (2019). The construction of gender in dating apps: An interface analysis of Tinder and Bumble. Feminist Media Studies, 19(6), 822–840.
43.
MaddenD., LiuY., YuH., SonbudakM. F., TroianoG. M., & HarteveldC. (2021). ‘Why are you playing games? You are a girl!’: Exploring gender biases in esports. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York. Association for Computing Machinery, pp. 1–15.
44.
MallR., NagpalM., SalminenJ., AlmerekhiH., JungS.-G., & JansenB. J. (2020). Four types of toxic people: Characterizing online users’ toxicity over time. Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pp. 1–11. https://doi.org/10.1145/3419249.3420142
MassanariA. (2017). #Gamergate and the fappening: How Reddit’s algorithm, governance, and culture support toxic technocultures. New Media & Society, 19(3), 329–346.
47.
MattuJ., LarsonJ., AngwinJ., KirchnerL., & SuryaM. (2016). How we analyzed the COMPAS recidivism algorithm. ProPublica. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
48.
MittelstadtB. D., AlloP., TaddeoM., WachterS., & FloridiL. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2). https://doi.org/10.1177/2053951716679679
49.
MohammadiH., GiachanouA., & BagheriA. (2023). Towards robust online sexism detection: A multi-model approach with BERT, XLM-RoBERTa, and DistilBERT for EXIST 2023 tasks. CLEF.
50.
ObermeyerZ., PowersB., VogeliC., & MullainathanS. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
51.
O’KeeffeE., RiordanE., LoudounF., & BoyleB. (2022). Understanding the social dividends and risks for female gamers in online spaces. In Extended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’22 (pp. 215–220). Association for Computing Machinery.
52.
ParikhP., AbburiH., BadjatiyaB., KrishnanR., ChhayaN., GuptaM., & VarmaV. (2019, November). Multi-label categorization of accounts of sexism using a neural framework. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China, pp. 1642–1652.
53.
ParikhP.AbburiH., ChhayaN., GuptaM., & VarmaV. (2021). Categorizing sexism and misogyny through neural approaches. ACM Transactions on the Web, 15(4), 2021.
54.
PoellerS., DechantM. J., KlarkowskiM., & MandrykR. L. (2023). Suspecting sarcasm: How League of Legends players dismiss positive communication in toxic environments. Proceedings of the ACM on Human-Computer Interaction, 7(CHI PLAY), Canada, pp. 1–26.
55.
PoyaneR. (2018). Toxic communication during streams on Twitch.tv. The case of Dota 2. Proceedings of the 22nd International Academic Mindtrek Conference, Mindtrek ’18, New York. Association for Computing Machinery, pp. 262–265.
56.
RaghavanM., BarocasS., KleinbergJ., & LevyK. (2019). Mitigating bias in algorithmic hiring: Evaluating claims and practices. SSRN. https://papers.ssrn.com/abstract=3408010
57.
RatanR., ChenV. H. H., De GroveF., BreuerJ., QuandtT., & WilliamsJ. P. (2022). Gender, gaming motives, and genre: Comparing Singaporean, German, and American players. IEEE Transactions on Games, 14(3), 456–465. https://doi.org/10.1109/TG.2021.3116077
58.
RomanoA. (2020, January20). What we still haven’t learned from Gamergate. Vox. https://www.vox.com/culture/2020/1/20/20808875/gamergate-lessons-cultural-impact-changes-harassment-laws
59.
RoyD., & DeshbandhuA. (2022). Anxious postcolonial masculinity in online video games: Race, gender and colonialism in Indian digital spaces. Gender, Place & Culture, 29(1), 104–129. https://doi.org/10.1080/0966369X.2020.1858030
60.
RuotsalainenM., & FrimanU. (2018). ‘There are no women and they all play mercy’: Understanding and explaining (the lack of) women’s presence in esports and competitive gaming. Conference of Digital Games Research Association, p. 14. https://jyx.jyu.fi/handle/123456789/82815
61.
SelbstA. D., BoydD., FriedlerS. A., VenkatasubramanianS., & VertesiJ. (2019). Fairness and abstraction in sociotechnical systems. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA. ACM, pp. 59–68. https://dl.acm.org/doi/10.1145/3287560.3287598
62.
ShaerO., WestendorfL., KnoufN. A., & PedersonC. (2017). Understanding gaming perceptions and experiences in a women’s college community. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, New York. Association for Computing Machinery, pp. 1544–1557.
63.
SpielK., KeyesO., & BarlasP. (2019). Patching gender: Non-binary utopias in HCI. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19 (pp. 1–11). Association for Computing Machinery.
64.
StuartK. (2014, December3). Zoe Quinn: ‘All Gamergate has done is ruin people’s lives’. The Observer. https://www.theguardian.com/technology/2014/dec/03/zoe-quinn-gamergate-interview
65.
TriceM. (2015). Putting gamergate in context: How group documentation informs social media activity. ACM.
66.
WohnD. Y., RatanR., & CherchigliaL. (2020). Gender and genre differences in multiplayer gaming motivations. In FangX. (Ed.), HCI in Games (pp. 233–248). Springer International Publishing. https://doi.org/10.1007/978-3-030-50164-8_16
67.
ZytkoD., FurloN., AljasimN., & Hanan (2021). Human-AI interaction for user safety in social matching apps: Involving marginalized users in design. Cornell University. arXiv:2204.00691
