Where’s the Bias: No Evidence of Bias by Sex When Testing the Public Safety Assessment

Abstract

Pretrial assessments are criticized for inherent biases. We conduct research in Kentucky to assess the predictive validity and differential prediction by sex of one pretrial assessment, the Public Safety Assessment (PSA). Our research is unique because we find equal base rates by sex for missing a court date, which allows us to assess for error rate balance by sex. We find the PSA to have predictive validity within acceptable ranges for the criminal legal field. The analyses show a lack of evidence of predictive bias for any arrest or missing a court date, and we find equal error rates for five different error measures. The analyses contribute to methodological debates about how to measure predictive bias with assessments.

Keywords

courts risk assessment predictive bias pretrial

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). The standards for educational and psychological testing. AERA Publications.

Angwin

Larson

Mattu

Kirchner

(2016). Machine bias. There is software that is used across the country to predict future criminals. And it is biased against blacks. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

Belknap

(2006). The Invisible Woman: Sex, crime, and Justice (3rd ed.). Cengage Learning.

Berk

Heidari

Jabbari

Kearns

Roth

(2021). Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50, 3–44.

Brittain

B. J.

Georges

Martin

(2021). Examining the predictive validity of the Public Safety Assessment. Criminal Justice and Behavior, 48(10), 1431–1449.

Burgess

E. W.

(1928). Factors determining success or failure on parole. In Bruce

A. A.

Burgess

E. W.

Harno

A. J.

(Eds.), The workings of the indeterminate sentence law and the parole system in Illinois (pp. 221–234). State Board of Parole.

Chouldechova

(2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163.

Cohen

T. H.

Lowenkamp

C. T.

(2019). Revalidation of the federal PTRA: Testing the PTRA for predictive biases. Criminal Justice and Behavior, 46(2), 234–260.

Corbett-Davies

Goel

(2018). The measure and mismeasure of fairness: a critical review of fair machine learning.https://arxiv.org/pdf/1808.00023.pdf

10.

DeMichele

Baumgartner

(2021). Bias testing of the public safety assessment: Error rate balance between Whites and Blacks for new arrests. Crime and Delinquency, 67, 2088–2113.

11.

DeMichele

Baumgartner

Wenger

Barrick

Comfort

Misra

(2020). The Public Safety Assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in Kentucky. Criminology & Public Policy, 19(2), 409–431.

12.

Denno

D. W.

(1997). Sex differences in biological and sociological predictors of crime. Vermont Law Review, 22, 305.

13.

Desmarais

S. L.

Johnson

K. L.

Singh

J. P.

(2016). Performance of recidivism risk assessment instruments in U.S. correctional settings. Psychological Services, 13(3), 206–222.

14.

Desmarais

S. L.

Zottola

S. A.

Duhart Clarke

S. E.

Lowder

E. M.

(2021). Predictive validity of pretrial risk assessments: A systematic review of the literature. Criminal Justice and Behavior, 48(4), 398–420.

15.

Dobbie

Goldin

Yang

C. S.

(2018). The effects of pre-trial detention on conviction, future crime, and employment: Evidence from randomly assigned judges. American Economic Review, 108(2), 201–240. https://doi.org/10.1257/aer.20161503

16.

Flores

Bechtel

Lowenkamp

(2016). False positives, false negatives, and false analyses: A rejoinders to “Machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks”. Federal Probation, 80(2), 38–46.

17.

Fox

J. A.

Fridel

E. E.

(2017). Gender differences in patterns and trends in U.S. Homicide, 1976–2015. Violence and Sex, 4(2), 37–43. https://doi.org/10.1089/vio.2017.0016

18.

Greene

J. T.

(2019). Categorical exclusions: How racialized gender regulation reproduces reentry hardship. Social Problems, 66(4), 548–563.

19.

Guthrie

Rachlinski

Wistrich

(2007). Blinking on the bench: how judges decide. Cornell Law Faculty Publications, Paper 917.

20.

Harcourt

B. E.

(2015). Risk as a proxy for race: The dangers of risk assessment. Federal Sentencing Reporter, 27, 237–243.

21.

Kalluri

(2020). Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583, 169. https://www.nature.com/articles/d41586-020-02003-2

22.

King

Zeng

(2001). Logistic regression in rare events data. Political Analysis, 9, 137–163.

23.

Kleinberg

Lakkaraju

Leskovec

Ludwig

Mullainathan

(2018). Human decisions and machine predictions. The Quarterly Journal of Economics, 133, 237–293. https://doi.org/10.1093/qje/qjx032

24.

Koepke

J. L.

Robinson

D. G.

(2018). Danger ahead: Risk assessment and the future of bail reform. Washington Law Review, 93, 1725–1795. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3041622

25.

Kohler-Hausmann

(2020). Nudging people to court. Science, 370(6517), 658–659.

26.

Lauritsen

J. L.

Heimer

Lynch

J. P.

(2009). Trends in the gender gap in violent offending: New evidence from the National Crime Victimization Survey. Criminology, 47(2), 361–399.

27.

Maxim

L. D.

Niebo

Utell

M. J.

(2014). Screening tests: A review with examples. Inhalation Toxicology, 26(13), 811–828.

28.

Mayson

(2019). Bias in, bias out. The Yale Law Journal, 128, 2218–2300.

29.

Monahan

Skeem

Lowenkamp

(2017). Age, risk assessment, and sanctioning: Overestimating the old, underestimating the young. Law and Human Behavior, 41(2), 191–201.

30.

National Association of Pretrial Services Agencies (2020). Standards on Pretrial Release: Revised 2020. NAPSA.

31.

Powers

(2011). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness, and correlation. Journal of Machine Learning Technologies, 2(1), 37–63.

32.

Rennison

C. M.

(2009). A new look at the gender gap in offending. Women & Criminal Justice, 19(3), 171–190. https://doi.org/10.1080/08974450903001461

33.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law and Human Behavior, 29, 615–620. https://doi.org/10.1007/s10979-005-6832-7

34.

Richie

B. E.

Eife

(2021). Black bodies at the dangerous intersection of gender violence and mass criminalization. Journal of Aggression Maltreatment & Trauma, 30(7), 877–888.

35.

Sackett

P. R.

Borneman

M. J.

Connelly

B. S.

(2008). High stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63, 215–227. https://doi.org/10.1037/0003-066X.63.4.215

36.

Salisbury

Van Voorhis

(2009). Gendered pathways: An empirical investigation of women probationers’ paths to incarceration. Criminal Justice & Behavior, 36, 541–566.

37.

Scott-Hayward

C. S.

Fradella

H. F.

(2019). Punishing poverty: How bail and pretrial detention fuel inequalities in the criminal justice system. University of California Press.

38.

Skeem

Monahan

Lowenkamp

(2016). Gender, risk assessment, and sanctioning: The cost of treating women like men. Law and Human Behavior, 40, 580–593.

39.

Skeem

J. L.

Lowenkamp

C. T.

(2016). Risk, race, and recidivism: Predictive bias and disparate impact. Criminology, 54, 680–712.

40.

Smith

Cullen

F. T.

Latessa

E. J.

(2009). Can 14,737 women be wrong? A meta-analysis of the LSI-R and recidivism for female offenders. Criminology & Public Policy, 8(1), 183–208.

41.

Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures, 5th ed. https://www.google.com/search?client=firefox-b-1-d&q=Principles+for+the+validation+and+use+of+personnel+selection+procedures%2C+5th+ed

42.

Starr

(2014). Evidence-based sentencing and the scientific rationalization of discrimination. Stanford Law Review, 66, 803–872.

43.

Stevenson

(2018). Assessing risk assessment in action. University of Minnesota Law Review, 103, 303–384.

44.

Thaler

R. H.

(2016). Behavioral economics: Past, present, and future. American Economic Review, 106(7), 1577–1600.

45.

United States v. Salerno: 481 US 739, 1987.

46.

Viljoen

J. L.

Jonnson

M. R.

Cochrane

D. M.

Vargen

L. M.

Vincent

G. M.

(2019). Impact of risk assessment instruments on rates of pretrial detention, postconviction placements, and release: A systematic review and meta-analysis. Law and Human Behavior, 43(5), 397–420.

47.

Webb

(2017). An exploration of sex differences in higher risk young offenders: implications for assessment and service delivery [Electronic Thesis and Dissertation Repository]. https://ir.lib.uwo.ca/etd/4376