Testing Predictive Biases at the Intersection of Race-Ethnicity and Sex: A Multi-Site Evaluation of a Pretrial Risk Assessment Tool

Abstract

Pretrial assessment instruments provide courtroom actors with information about the likelihood that someone who has been charged with a criminal offense would engage in criminal behavior if released prior to trial. Although prior research supports the ability of pretrial instruments to predict pretrial outcomes, there are concerns that pretrial instruments may inadvertently exacerbate racial-ethnic and sex disparities found in the larger criminal legal system. In the current study, we conduct multi-site predictive bias tests of a widely used pretrial instrument—the Public Safety Assessment (PSA)—across six racial-ethnic and sex groups (i.e., White males, White females, Black males, Black females, Hispanic males, and Hispanic females). Study results support the PSA as a valid and consistent predictor of failure to appear, new criminal activity, and new violent criminal activity across these six racial-ethnic and sex groups. These findings support the use of the PSA and identify several areas for future research.

Keywords

jail sex pretrial PSA race pretrial assessment

Get full access to this article

View all access options for this article.

References

Albonetti

C. A.

(1991). An integration of theories to explain judicial discretion. Social Problems, 38(2), 247–266. https://psycnet.apa.org/doi/10.1525/sp.1991.38.2.03a00090

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Psychological Association. https://www.apa.org/science/programs/testing/standards

Angwin

Larson

Mattu

Kirchner

(2016, May 23). Machine bias: There’s software used across the country to predict future criminals, and its biased against Blacks. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

Ares

C. E.

Rankin

Sturtz

(1963). The Manhattan bail project: An interim report on the use of pretrial parole. New York University Law Review, 38, 67–95.

Arnold Ventures. (2022). About the public safety assessment. https://advancingpretrial.org/psa/about

Ashford

Spivak

Ogloff

Shepherd

(2023). Statistical learning methods and cross-cultural fairness: Trade-offs and implications for risk assessment instruments. Psychological Assessment, 35(6), 484–496. https://doi.org/10.1037/pas0001228

Biddle

(2006). Adverse impact and test validation: A practitioner’s guide to valid and defensible employment testing. Ashgate.

Brennan

P. K.

Spohn

(2009). The joint effects of offender race/ethnicity and sex on sentence length decisions in federal courts. Race and Social Problems, 1(4), 200–217. https://psycnet.apa.org/doi/10.1007/s12552-009-9016-0

Brittain

B. J.

Georges

Martin

(2021). Examining the predictive validity of the Public Safety Assessment. Criminal Justice and Behavior, 48(10), 1431–1449. https://doi.org/10.1177/00938548211005836

10.

Campbell

C. M.

Labrecque

R. M.

Weinerman

Sanchagrin

(2020). Gauging detention dosage: Assessing the impact of pretrial detention on sentencing outcomes using propensity score modeling. Journal of Criminal Justice, 70, 101719. https://doi.org/10.1016/j.jcrimjus.2020.101719

11.

Chouldechova

(2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2), 153–163. https://arxiv.org/pdf/1703.00056.pdf

12.

Cohen

(1988). Statistical power analysis for the behavioural sciences. Lawrence Erlbaum Associates.

13.

Cohen

Lowenkamp

(2019). Revalidation of the federal PTRA: Testing the PTRA for predictive bias. Criminal Justice and Behavior, 46(2), 234–260. https://psycnet.apa.org/doi/10.1177/0093854818810315

14.

Collier

C. S.

(2019). Black males and courts. In Williams

J. M.

Kniffley

(Eds.), Black males and the criminal justice system (pp. 49–58). Routledge.

15.

Crenshaw

(1989). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics. The University of Chicago Legal Forum, 1, 139–167. http://chicagounbound.uchicago.edu/uclf/vol1989/iss1/8?utm_source=chicagounbound.uchicago.edu%2Fuclf%2Fvol1989%2Fiss1%2F8&utm_medium=PDF&utm_campaign=PDFCoverPages

16.

DeLong

E. R.

DeLong

D. M.

Clarke-Pearson

D. L.

(1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845.

17.

DeMichele

Baumgartner

Wenger

Barrick

Comfort

(2020). Public safety assessment: Predictive utility and differential prediction by race in Kentucky. Criminology & Public Policy, 19(2), 409–431. https://doi.org/10.1111/1745-9133.12481

18.

DeMichele

Comfort

Barrick

Baumgartner

(2021). The intuitive-override model: Nudging judges toward pretrial risk assessment instruments. Federal Probation, 85(2), 22–39.

19.

Demuth

(2003). Racial and ethnic differences in pretrial release decisions and outcomes: A comparison of Hispanic, Black, and White felony arrestees. Criminology, 41(3), 873–908. https://doi.org/10.1111/j.1745-9125.2003.tb01007.x

20.

Desmarais

S. L.

Monahan

Austin

(2022). The empirical case for pretrial risk assessment instruments. Criminal Justice and Behavior, 49(6), 807–816. https://psycnet.apa.org/doi/10.1177/00938548211041651

21.

Desmarais

S. L.

Zottola

S. A.

Duhart Clarke

S. E.

Lowder

E. M.

(2021). Predictive validity of pretrial risk assessments: A systematic review of the literature. Criminal Justice and Behavior, 48(4), 398–420. https://doi.org/10.1177/0093854820932959

22.

Eckhouse

Lum

Conti-Cook

Ciccolini

(2019). Layers of bias: A unified approach for understanding problems with risk assessment. Criminal Justice and Behavior, 46(2), 185–209. https://doi.org/10.1177/0093854818811379

23.

Feltz

C. J.

Miller

G. E.

(1996). An asymptotic test for the equality of coefficients of variation from k populations. Statistics in Medicine, 15(6), 647–658. https://doi.org/10.1002/(sici)1097-0258(19960330)15:6%3C647::aid-sim184%3E3.0.co;2-p

24.

Frank

(1930). Law and the modern mind. Coward-Mccann.

25.

Freiburger

T. L.

Burke

A. S.

(2011). Status offenders in the juvenile court: The effects of gender, race, and ethnicity on the adjudication decision. Youth Violence and Juvenile Justice, 9(4), 352–365. https://psycnet.apa.org/doi/10.1177/1541204011399933

26.

Goulette

Wooldredge

(2018). Collateral consequences of pretrial detention. In Huebner

B. M.

Frost

N. A.

(Eds.), Handbook on the consequences of sentencing and punishment decisions (pp. 271–289). Routledge.

27.

Hochschild

J. L.

Weaver

(2007). The skin color paradox and the American racial order. Social Forces, 86(2), 643–670. https://doi.org/10.1093/sf/86.2.643

28.

Holmes

Feldmeyer

Kulig

T. C.

(2020). Sentencing departures and focal concerns: The joint effect of race and gender on departures in United States district courts, 2014–2016. Journal of Crime and Justice, 43(5), 598–622. https://doi.org/10.1080/0735648X.2020.1730933

29.

Hood

Schneider

(2019). Bail and pretrial detention: Contours and causes of temporal and county variation. RSF: The Russell Sage Foundation Journal of the Social Sciences, 5(1), 126–149.

30.

Kahneman

(2011). Thinking, fast and slow. Macmillan.

31.

Kang

Bennett

Carbado

Casey

Levinson

(2011). Implicit bias in the courtroom. University of California Los Angeles Law Review, 59, 1124–1186.

32.

Kim

H. Y.

(2017). Statistical notes for clinical researchers: Chi-squared test and Fisher's exact test. Restorative Dentistry and Endodontics, 42(2), 152–155. https://doi.org/10.5395%2Frde.2017.42.2.152

33.

Kleinberg

Ludwig

Mullainathan

Sustein

C. R.

(2019). Discrimination in the age of algorithms. Pre-Print. https://arxiv.org/abs/1902.03731

34.

Lattimore

P. K.

Tueller

Levin-Rector

Witwer

(2020). The prevalence of local criminal justice practices. Federal Probation, 84, 28–37.

35.

Lowder

E. M.

Diaz

C. L.

Grommon

Ray

B. R.

(2022). Differential prediction and disparate impact of pretrial risk assessments in practice: A multi-site evaluation. Journal of Experimental Criminology, 19, 561–594. https://doi.org/10.1007/s11292-021-09492-9

36.

Lowder

E. M.

Lawson

S. G.

Grommon

Ray

B. R.

(2020). Five-county validation of the Indiana Risk Assessment System–Pretrial Assessment Tool (IRAS-PAT) using a local validation approach. Justice Quarterly, 37(7), 1241–1260. https://doi.org/10.1080/07418825.2020.1829006

37.

Lowenkamp

C. T.

(2009). The development of an actuarial risk assessment instrument for US Pretrial Services. Fed. Probation, 73, 33. https://heinonline.org/HOL/Page?handle=hein.journals/fedpro73&div=18&g_sent=1&casa_token=tpuF02fp0PYAAAAA:Ky8K4hqCsQ3T4LNfguUkixBvolyEVYfDExjWCXAPMe3qBeGii8nK8a6aJ88W8YGZ_teAlmRZMxY&collection=journals

38.

Mayson

S. G.

(2019). Detention by any other name. Duke Law Journal, 69, 1643–1680.

39.

McShane

B. B.

Gal

Gelman

Robert

Tackett

J. L.

(2019). Abandon statistical significance. The American Statistician, 73(suppl. 1), 235–245. https://doi.org/10.1080/00031305.2018.1527253

40.

Petersen

(2020). Do detainees plead guilty faster? A survival analysis of pretrial detention and the timing of guilty pleas. Criminal Justice Policy Review, 31(7), 1015–1035. https://doi.org/10.1177/0887403419838020

41.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law and human behavior, 29, 615–620. https://doi.org/10.1007/s10979-005-6832-7

42.

Sackett

P. R.

Borneman

M. J.

Connelly

B. S.

(2008). High stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63(4), 215–227. https://doi.org/10.1037/0003-066x.63.4.215

43.

Sackett

P. R.

Laczo

R. M.

Lippe

Z. P.

(2003). Differential prediction and the use of multiple predictors: The omitted variables problem. Journal of Applied Psychology, 88(6), 1046–1056. https://doi.org/10.1037/0021-9010.88.6.1046

44.

Sacks

Sainato

V. A.

Ackerman

A. R.

(2015). Sentenced to pretrial detention: A study of bail decisions and outcomes. American Journal of Criminal Justice, 40(3), 661–681. https://doi.org/10.1007/s12103-014-9268-0

45.

Schuck

A. M.

(2004). The masking of racial and ethnic disparity in police use of physical force: The effects of gender and custody status. Journal of Criminal Justice, 32(6), 557–564.

46.

Schlesinger

(2005). Racial and ethnic disparity in pretrial criminal processing. Justice Quarterly, 22(2), 170–192. https://doi.org/10.1080/07418820500088929

47.

Skeem

J. L.

Lowenkamp

C. T.

(2016). Risk, race, and recidivism: Predictive bias and disparate impact. Criminology, 54, 680–712. https://doi.org/10.1111/1745-9125.12123

48.

Skeem

J. L.

Monahan

Lowenkamp

(2016). Gender, risk assessment, and sanctioning: The cost of treating women like men. Law and Human Behavior, 40(5), 580–593. https://doi.org/10.1037/lhb0000206

49.

Spohn

(2008). Race, sex, and pretrial detention in federal court: Indirect effects and cumulative disadvantage. University of Kansas Law Review, 57, 879–901.

50.

Stanovich

K. E.

West

R. F.

(2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645–665. https://doi.org/10.1017/s0140525x00003435

51.

Stevenson

(2018). Assessing risk assessment in action. Minnesota Law Review, 103, 303–384.

52.

Sutton

J. R.

(2013). Structural bias in the sentencing of felony defendants. Social Science Research, 42(5), 1207–1221. https://doi.org/10.1016/j.ssresearch.2013.04.003

53.

Tueller

S. J.

Chew

Godwin

A. K.

Williams

J. A.

Weitzel

(2022). psa2013: Validation tools for the Public Safety Assessment (PSA) (R package version 0.3.9). https://osf.io/3gsmf/

54.

Van Brunt

Bowman

L. E

. (2018). Toward a just model of pretrial release: A history of bail reform and a prescription for what’s next. Journal of Criminal Law & Criminology, 108, 701–774.

55.

Vincent

G. M.

Viljoen

J. L.

(2020). Racist algorithms or systemic problems? Risk assessments and racial disparities. Criminal Justice and Behavior, 47(12), 1576–1584. https://doi.org/10.1177/0093854820954501

56.

Woldgabreal

Day

Tamatea

(2020). Do risk assessments play a role in the enduring “color line..” Advancing Corrections, 10, 18–28.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB