Artificial Intelligence (AI) is widely used in decision-making systems, including the criminal justice system. Automated decision-making systems can speed up the handling of cases and improve consistency and efficiency. These systems were expected to enhance transparency and equip judges with data-driven insights. AI in criminal justice also pointed out concerns about bias and fairness. The idea is to build more inclusive legal systems. This paper explored different sources of potential biases in the judiciary. We compared different approaches to identify and measure the biases. We also reviewed techniques for bias mitigation, such as in-processing, pre-processing, and post-processing approaches. This work aims to comprehensively understand how to build fair AI models. We examined the widely used datasets and the fairness metrics used for evaluation. Most of the work on addressing biases is done in the Western context, leaving a notable gap in the Indian context. India, a country with rich diversity and a complex legal structure, needs AI models that are accurate and also equitable across different demographics to ensure justice and equity for all citizens. To achieve that, India needs bias detection and mitigation approaches that suit the Indian context, as well as evaluation metrics to measure fairness in decisions influenced by gender, caste, religion, etc. The approaches discussed in this paper were supported by case studies that explain the historical and cultural dimensions of the Indian judiciary.
JassalN. Does victim gender matter for justice delivery? police and judicial responses to women’s cases in India. Am Political Sci Rev2023; 118(3): 1278–1304.
9.
SanjuthaTHarshaA. Casteism in Indian criminal justice system. Int J Legal Sci Innov2021; 3: 12–19.
10.
SurendranathA. Death penalty India report - volume 1.
HuqAZ. Racial equity in algorithmic criminal justice. Duke Law J2019; 68: 1043–1134.
14.
PillaiVGChandranLR. Verdict prediction for Indian courts using bag of words and convolutional neural network. In: 2020 Third international conference on smart systems and inventive technology (ICSSIT) pp.676–683. DOI: 10.1109/ICSSIT48917.2020.9214278.
15.
ShaikhRASahuTPAnandV. Predicting outcomes of legal cases based on legal factors using classifiers. Procedia Comput Sci2020; 167: 2393–2402. International Conference on Computational Intelligence and Data Science.
16.
SivaranjaniNJayabharathyJTejaPC. Predicting the supreme court decision on appeal cases using hierarchical convolutional neural network. Int J Speech Technol2021; 24: 643–650.
17.
BhatnagarMHuchhanavarS. Predicting delays in indian lower courts using automl and decision forests. In: García Márquez FP, Jamil A, Ramirez IS, et al. (eds) Computing, internet of things and data analytics. Cham: Springer Nature Switzerland, pp.166–180. ISBN 978-3-031-53717-2.
18.
SharmaSShandilyaRSharmaS. Predicting Indian supreme court judgments, decisions, or appeals. ArXiv, abs/2110.09251, 2021.
19.
BhattacharyaPPaulSGhoshK, et al. Identification of rhetorical roles of sentences in Indian legal judgments. CoRR, abs/1911.05405, 2019. http://arxiv.org/abs/1911.05405.
20.
MalikVSanjayRNigamSK, et al. ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation. In: Zong C, Xia F, Li W, et al. (eds) Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (Volume 1: Long Papers). Online: Association for computational linguistics, pp.4046–4062. DOI: 10.18653/v1/2021.acl-long.313. https://aclanthology.org/2021.acl-long.313.
21.
BarmanARoyDPaulD, et al. Convolutional neural networks can achieve binary bail judgement classification. ArXiv, abs/2401.14135, 2024.
22.
KapoorADhawanMGoelA, et al. HLDC: Hindi legal documents corpus. In: Muresan S, Nakov P and Villavicencio A (eds) Findings of the association for computational linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics, pp.3521–3536. DOI: 10.18653/v1/2022.findings-acl.278. https://aclanthology.org/2022.findings-acl.278.
23.
SelbstADBoydDFriedlerSA, et al. Fairness and abstraction in sociotechnical systems. Proc Conf Fair Accountability Transparency2019: 59–68.
24.
EubanksVE. Automating inequality: How high-tech tools profile, police, and punish the poor.
25.
NobleSU. Algorithms of oppression: How search engines reinforce racism.
26.
O’NeilC. Weapons of math destruction: How big data increases inequality and threatens democracy. United States: Crown Publishing Group, 2016.
27.
ZhouNZZhangZNairVN, et al. Bias, fairness, and accountability with AI and ML algorithms. ArXiv, abs/2105.06558, 2021.
28.
ŽliobaitėI. Measuring discrimination in algorithmic decision making. Data Min Knowl Discov2017; 31: 1060–1089.
29.
OlteanuACastilloCDiazF, et al. Social data: Biases, methodological pitfalls, and ethical boundaries. Front Big Data2019; 2.
30.
RehaviMMStarrSB. Racial disparity in federal criminal charging and its sentencing consequences. Empir Stud eJ2012: 1320–1354.
31.
KirchnerJAngwinSMattuJ, et al. Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks. New York, NY, USA: Pro Publica, 2016.
32.
KangJBennettMWCarbadoDW, et al. Implicit bias in the courtroom. Soc Sci Res Netw2012; 59(5): 1124–1186.
33.
DeshpandeASSpearsD. Who is the identifiable victim? caste and charitable giving in modern India. Econ Dev Cult Change2016; 64: 299–321.
34.
RehaviMMStarrSB. Racial disparity in federal criminal sentences. J Political Econ2014; 122: 1320–1354.
35.
SommersSR. Interpersonal relations and group processes on racial diversity and group decision making: Identifying multiple effects of racial composition on jury deliberations.
BareinboimETianJPearlJ. Recovering from selection bias in causal and statistical inference. Probabilisti Causal Infer2014: 2410–2416.
38.
HinnefeldJHCoomanPMammoN, et al. Evaluating fairness metrics in the presence of dataset bias. ArXiv, 2018.
39.
ArnoldDDobbieWHullP. Measuring racial discrimination in algorithms. PSN: Politics of Race (Topic), 2020.
40.
WangZZhouYQiuM, et al. Towards fair machine learning software: Understanding and addressing model bias through counterfactual thinking. ArXiv, abs/2302.08018, 2023.
41.
HeFWuXZhangW, et al. A novel bias-alleviated hybrid ensemble model based on over-sampling and post-processing for fair classification. Conn Sci2023; 35.
42.
BohrenAHullPBrownU, et al. Systemic discrimination: Theory and measurement. SSRN Electron J2022. https://doi.org/10.3386/w29820
43.
SharmaMDeshpandeAShahRR. On comparing fair classifiers under data bias.
44.
BlumAStanglK. Recovering from biased data: Can fairness constraints improve accuracy? In: Symposium on foundations of responsible computing.
45.
KamiranFCaldersT. Data preprocessing techniques for classification without discrimination. Knowl Inf Syst2011; 33: 1–33.
46.
AgarwalABeygelzimerADudíkM, et al. A reductions approach to fair classification. ArXiv, abs/1803.02453, 2018.
47.
KamiranFCaldersT. Classifying without discriminating. In: 2009 2nd International conference on computer, control and communication pp.1–6. DOI: 10.1109/IC4.2009.4909197.
48.
LuongBTRuggieriSTuriniF. K-nn as an implementation of situation testing for discrimination discovery and prevention. KDD ’11, New York, NY, USA: Association for Computing Machinery, pp.502–510. ISBN 9781450308137. DOI: 10.1145/2020408.2020488.
49.
LiYWangYLiT, et al. Sp-smote: A novel space partitioning based synthetic minority oversampling technique. Knowl Based Syst2021; 228: 107269.
50.
YuanXChenSZhouH, et al. Chsmote: Convex hull-based synthetic minority oversampling technique for alleviating the class imbalance problem. Inf Sci2023; 623: 324–341.
51.
SáezJALuengoJStefanowskiJ, et al. SMOTE-IPF: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering. Inf Sci2015; 291: 184–203.
52.
LiJZhuQWuQ, et al. A novel oversampling technique for class-imbalanced learning based on smote and natural neighbors. Inf Sci2021; 565: 438–455.
53.
SusanSKumarA. SSOMaj-SMOTE-SSOMin: Three-step intelligent pruning of majority and minority samples for learning from imbalanced datasets. Appl Soft Comput2019; 78: 141–149.
54.
ZafarMBValeraIGomez-RodriguezM, et al. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. Proceedings of the 26th international conference on World Wide Web, 2016.
55.
KrasanakisEXioufisESPapadopoulosS, et al. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. Proceedings of the 2018 World Wide Web Conference, 2018.
56.
JiangHNachumO. Identifying and correcting label bias in machine learning. In: International conference on artificial intelligence and statistics.
57.
ShihJYChinZH. A fairness approach to mitigating racial bias of credit scoring models by decision tree and the reweighing fairness algorithm. In: 2023 IEEE 3rd international conference on electronic communications, internet of things and big data (ICEIB) pp.100–105. DOI: 10.1109/ICEIB57887.2023.10170339.
CalmonFPWeiDVinzamuriB, et al. Optimized pre-processing for discrimination prevention. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17, Red Hook, NY, USA: Curran Associates Inc., pp.3995–4004. ISBN 9781510860964.
60.
KamishimaTAkahoSSakumaJ. Fairness-aware learning through regularization approach. In: 2011 IEEE 11th international conference on data mining workshops pp.643–650. DOI: 10.1109/ICDMW.2011.83.
61.
JiangRPacchianoAStepletonT, et al. Wasserstein fair classification. In: Adams RP and Gogate V (eds) Proceedings of the 35th uncertainty in artificial intelligence conference, proceedings of machine learning research, volume 115. PMLR, pp.862–872. https://proceedings.mlr.press/v115/jiang20a.html.
62.
BeutelAChenJDoshiT, et al. Putting fairness principles into practice: Challenges, metrics, and improvements. In: Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society. AIES ’19, New York, NY, USA: Association for Computing Machinery, pp.453–459. ISBN 9781450363242, DOI: 10.1145/3306618.3314234.
LahotiPBeutelAChenJ, et al. Fairness without demographics through adversarially reweighted learning. In: Proceedings of the 34th international conference on neural information processing systems. NIPS ’20, Red Hook, NY, USA: Curran Associates Inc., ISBN 9781713829546.
65.
SikderMFRamachandranpillaiRde LengD, et al. Fairx: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability. 2024, https://arxiv.org/abs/2406.14281.
66.
HanXChiJChenY, et al. Ffb: A fair fairness benchmark for in-processing group fairness methods. 2024, https://arxiv.org/abs/2306.09468.
67.
HardtMPriceESrebroN. Equality of opportunity in supervised learning. ArXiv, abs/1610.02413, 2016.
68.
SargentJRWeberM. Identifying biases in legal data: An algorithmic fairness perspective. ArXiv, abs/2109.09946, 2021.
69.
PleissGRaghavanMWuF, et al. On fairness and calibration. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17, Red Hook, NY, USA: Curran Associates Inc., pp.5684–5693. ISBN 9781510860964.
70.
Noriega-CamperoABakkerMAGarcia-BulleB, et al. Active fairness in algorithmic decision making. In: Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society. AIES ’19, New York, NY, USA: Association for Computing Machinery, pp.77–83. ISBN 9781450363242, DOI: 10.1145/3306618.3314277.
71.
HöltgenBWilliamsonRC. On the richness of calibration. In: Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, FAccT ’23, Association for Computing Machinery. 2023, pp.1124–1138. New York, NY, USA. isbn:9798400701924. DOI: 10.1145/3593013.3594068.
72.
IosifidisVFetahuBNtoutsiE. FAE: A fairness-aware ensemble framework. 2019 IEEE International Conference on Big Data (Big Data), 2019, pp.1375–1380.
73.
Corbett-DaviesSGaeblerJDNilforoshanH, et al. The measure and mismeasure of fairness. J Mach Learn Res2023; 24: 1–117.
74.
RezaeiAFathonyRMemarrastO, et al. Fairness for robust log loss classification. Proc AAAI Conf Artif Intell2020; 34: 5511–5518.
75.
WangJLiuYLevyC. Fair classification with group-dependent label noise. In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. FAccT ’21, ACM. DOI: 10.1145/3442188.3445915.
76.
LiYMengLChenL, et al. Training data debugging for the fairness of machine learning software. In: 2022 IEEE/ACM 44th international conference on software engineering (ICSE). pp.2215–2227. DOI: 10.1145/3510003.3510091.
77.
ChakrabortyJMajumderSYuZ, et al. Fairway: a way to build fair ML software. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. ESEC/FSE 2020, New York, NY, USA: Association for Computing Machinery, pp.654–665. ISBN 9781450370431, DOI: 10.1145/3368089.3409697.
78.
NguyenDGuptaSRanaS, et al. Fairness improvement for black-box classifiers with gaussian process. Inf Sci (Ny)2021; 576: 542–556.
79.
MohammadiKSivaramanAFarnadiG. Feta: Fairness enforced verifying, training, and predicting algorithms for neural networks. In: Proceedings of the 3rd ACM conference on equity and access in algorithms, mechanisms, and optimization. EAAMO ’23, New York, NY, USA: Association for Computing Machinery, ISBN 9798400703812. DOI: 10.1145/3617694.3623243.
80.
WangYSinghL. Analyzing the impact of missing values and selection bias on fairness. Int J Data Sci Anal2021; 12: 101–119.
81.
SharmaSZhangYRíos AliagaJM, et al. Data augmentation for discrimination prevention and bias disambiguation. In: AIES ’20: proceedings of the AAAI/ACM conference on AI, ethics, and society. pp.358–364. DOI: 10.1145/3375627.3375865.
82.
SalazarTSantosMSAraújoH, et al. Fawos: Fairness-aware oversampling algorithm based on distributions of sensitive attributes. IEEE Access2021; 9: 81370–81379.
83.
BiswasSRajanH. Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. ESEC/FSE 2021, New York, NY, USA: Association for Computing Machinery, pp.981–993. ISBN 9781450385626, DOI: 10.1145/3468264.3468536.
84.
ZhangYZhouFLiZ, et al. Bias-tolerant fair classification. In: Asian conference on machine learning.
85.
MishlerAKennedyEH. Fade: Fair double ensemble learning for observable and counterfactual outcomes. In: 2022 ACM conference on fairness, accountability, and transparency (FAccT ’22). New York, NY, USA: ACM, p.1. DOI: 10.1145/3531146.3533167.
RachlinskiJJJohnsonSLWistrichAJ, et al. Does unconscious racial bias affect trial judges. Notre Dame Law Rev2009; 84: 1195.
92.
GreenwaldAGMcGheeDESchwartzJLK. Measuring individual differences in implicit cognition: The implicit association test. J Pers Soc Psychol1998; 74: 1464–1480.
93.
GreenwaldAGPoehlmanTAUhlmannEL, et al. Understanding and using the implicit association test: III meta-analysis of predictive validity. J Pers Soc Psychol2009; 97: 17–41.
94.
BoydCLEpsteinLMartinAD. Untangling the causal effects of sex on judging. In: 2nd Annual Conference on Empirical Legal Studies Paper n.d. DOI: https://doi.org/10.2139/ssrn.1001748.
95.
AssumpcaoATrecentiJAZ. Judicial favoritism of politicians: Evidence from small claims court. arXiv: General Economics, 2020.
96.
DepewBErenOMocanNH. Judges, juveniles, and in-group bias. J Law Econ2016; 60: 209–239.
97.
BielenSGrajzlP. Gender-based judicial ingroup bias in sex crime sentencing: Evidence from belgium. Int J Law Crime Justice2020; 62: 100394.
98.
AshEAsherSBhowmickA, et al. In-group bias in the indian judiciary: Evidence from 5.5 million criminal cases. Proceedings of the 4th ACM SIGCAS conference on computing and sustainable societies, 2021.
99.
ShayoMZussmanA. Judicial ingroup bias in the shadow of terrorism. Q J Econ2011; 126: 1447–1484.
100.
GirhepujeSGoelAKrishnanGS, et al. Are models trained on Indian legal data fair? ArXiv, abs/2303.07247, 2023.
101.
TripathiYDonakantiRGirhepujeS, et al. InSaAF: Incorporating safety through accuracy and fairness — are LLMs ready for the indian legal domain? ArXiv, abs/2402.10567, 2024.
102.
BielenSMarneffeWMocanNH. Racial bias and in-group bias in virtual reality courtrooms. J Law Econ2021; 64: 269–300.
103.
SambasivanNArnesenEHutchinsonB, et al. Re-imagining algorithmic fairness in india and beyond. In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. FAccT ’21, New York, NY, USA: Association for Computing Machinery, pp.315–328. ISBN 9781450383097, DOI: 10.1145/3442188.3445896.
104.
Rodrigo-GinésFJCarrillo-de AlbornozJPlazaL. A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect it. Expert Syst Appl2024; 237. DOI: https://doi.org/10.1016/j.eswa.2023.121641.
SahuJK. Media trials and the erosion of presumption of innocence: A critical examination of the Indian criminal justice system. Int J Hum Rights Law Rev2023; 2: 145–182.
