Reliability and Validity of START and LSI-R Assessments in Mental Health Jail Diversion Clients

Abstract

Risk assessment instruments are increasingly used in mental health jail diversion programs. This study examined the reliability and validity of Short-Term Assessment of Risk and Treatability (START) and Level of Service Inventory–Revised (LSI-R) assessments overall and by client race. Research assistants completed START and LSI-R assessments for 95 diversion clients. Arrests and jail days were collected via official records and self-report 3, 6, 9, 12, and 18 months after baseline. Assessments demonstrated good interrater reliability and convergent validity. START strength total scores and LSI-R risk estimates were the strongest predictors of recidivism. Total scores and risk estimates did not differ as a function of client race, but there were some differences in accuracy of START vulnerability and LSI-R total scores and risk estimates in predicting jail days (but not arrests), over shorter follow-ups. No such differences were found for START strength total scores across any follow-up period or recidivism measure.

Keywords

mental health jail diversion risk assessment protective factors recidivism

Get full access to this article

View all access options for this article.

References

American Educational Research Association. (2014). Standards for educational and psychological testing. Washington, DC: Author.

Andrews

D. A.

Bonta

(1998). Level of Service Inventory–Revised: Screening Version (LSI-R: SV). Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Bonta

(2001). Level of Service Inventory–Revised (LSI-R): User’s manual. Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Bonta

. (2010). Rehabilitating criminal justice policy and practice. Psychology, Public Policy, and Law, 16, 39-55. doi:10.1037/a0018362

Andrews

D. A.

Bonta

Hoge

R. D.

(1990). Classification for effective rehabilitation: Rediscovering psychology. Criminal Justice and Behavior, 17, 19-52. doi:10.1177/0093854890017001004

Andrews

D. A.

Bonta

Wormith

J. S.

(2004). Level of Service/Case Management Inventory (LS/CMI). Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Bonta

Wormith

J. S.

(2008). The Level of Service/Risk-Need-Responsivity (LS/RNR). Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Bonta

Wormith

J. S.

(2010). The Level of Service (LS) assessment of adults and older adolescents. In Otto

R. K.

Douglas

K. S.

(Eds.), Handbook of violence risk assessment (1st ed., pp. 199-225). New York, NY: Routledge.

Baillargeon

Binswanger

I. A.

Penn

J. V.

Williams

B. A.

Murray

O. J.

(2009). Psychiatric disorders and repeat incarcerations: The revolving prison door. American Journal of Psychiatry, 166, 103-109. doi:10.1176/appi.ajp.2008.08030416

10.

Barber-Rioja

Dewey

Kopelovich

Kucharski

L. T.

(2012). The utility of the HCR-20 and PCL:SV in the prediction of diversion noncompliance and reincarceration in diversion programs. Criminal Justice and Behavior, 39, 475-492. doi:10.1177/0093854811432609

11.

Braithwaite

Charette

Crocker

A. G.

Reyes

(2010). The predictive validity of clinical ratings of the Short-Term Assessment of Risk and Treatability (START). International Journal of Forensic Mental Health, 9, 271-281. doi:10.1080/14999013.2010.534378

12.

Callis

R. R.

Kresin

(2016). Residential vacancies and homeownership in the second quarter. Washington, DC: U.S. Census Bureau, U.S. Department of Commerce.

13.

Campbell

M. A.

Canales

D. D.

Wei

Totten

A. E.

Alex

Wershler

J. L.

(2015). Multidimensional evaluation of a mental health court: Adherence to the risk-need-responsivity model. Law and Human Behavior, 39, 489-502. doi:10.1037/lhb0000135

14.

Canales

D. D.

Campbell

M. A.

Wei

Totten

A. E.

(2014). Prediction of general and violent recidivism among mentally disordered adult offenders: Test of the Level of Service/Risk–Need–Responsivity (LS/RNR) instrument. Criminal Justice and Behavior, 41, 971-991. doi:10.1177/0093854814523003

15.

Case

Steadman

H. J.

Dupuis

S. A.

Morris

L. S.

(2009). Who succeeds in jail diversion programs for persons with mental illness? A multi-site study. Behavioral Sciences & the Law, 27, 661-674. doi:10.1002/bsl.883

16.

Chenane

J. L.

Brennan

P. K.

Steiner

Ellison

J. M.

(2015). Racial and ethnic differences in the predictive validity of the Level of Service Inventory–Revised among prison inmates. Criminal Justice and Behavior, 42, 286-303. doi:10.1177/0093854814548195

17.

Chu

C. M.

Thomas

S. D. M.

Ogloff

J. R. P.

Daffern

(2011a). The predictive validity of the Short-Term Assessment of Risk and Treatability (START) in a secure forensic hospital: Risk factors and strengths. International Journal of Forensic Mental Health, 10, 337-345. doi:10.1080/14999013.2011.629715

18.

Chu

C. M.

Thomas

S. D. M.

Ogloff

J. R. P.

Daffern

(2011b). The short- to medium-term predictive accuracy of static and dynamic risk assessment measures in a secure forensic hospital. Assessment, 20, 230-241. doi:10.1177/1073191111418298

19.

Cicchetti

Bronen

Spencer

Haut

Berg

Oliver

Tyrer

(2006). Rating scales, scales of measurement, issues of reliability: Resolving some critical issues for clinicians and researchers. Journal of Nervous and Mental Disease, 194, 557-564. doi:10.1097/01.nmd.0000230392.83607.c5

20.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York, NY: Routledge.

21.

de Ruiter

Nicholls

T. L

. (2011). Protective factors in forensic mental health: A new frontier. International Journal of Forensic Mental Health, 10, 160-170. doi:10.1080/14999013.2011.600602

22.

de Vries Robbé

de Vogel

Douglas

K. S

. (2013). Risk factors and protective factors: A two-sided dynamic approach to violence risk assessment. Journal of Forensic Psychiatry & Psychology, 24, 440-457. doi:10.1080/14789949.2013.818162

23.

Desmarais

S. L.

(2009). START research summary. In Webster

C. D.

Martin

M-L.

Brink

Nicholls

T. L.

Desmarais

S. L.

(Eds.), Manual for the Short-Term Assessment of Risk and Treatability (START) (Version 1.1, pp. 89-104). Coquitlam, British Columbia, Canada: British Columbia Mental Health & Addiction Services.

24.

Desmarais

S. L.

Johnson

K. L.

Singh

J. P.

(2016). Performance of recidivism risk assessment instruments completed in U.S. correctional settings. Psychological Services, 13, 206-222.

25.

Desmarais

S. L.

Nicholls

T. L.

Wilson

C. M.

Brink

(2012). Using dynamic risk and protective factors to predict inpatient aggression: Reliability and validity of START assessments. Psychological Assessment, 24, 685-700. doi:10.1037/a0026668

26.

Desmarais

S. L.

Van Dorn

R. A.

Telford

R. P.

Petrila

Coffey

(2012). Characteristics of START assessments completed in mental health jail diversion programs. Behavioral Sciences & the Law, 30, 448-469. doi:10.1002/bsl.2022

27.

Douglas

K. S.

Ogloff

J. R. P.

Hart

S. D.

(2003). Evaluation of a model of violence risk assessment among forensic psychiatric patients. Psychiatric Services, 54, 1372-1379. doi:10.1176/appi.ps.54.10.1372

28.

Douglas

K. S.

Yeomans

Boer

D. P.

(2005). Comparative validity analysis of multiple measures of violence risk in a sample of criminal offenders. Criminal Justice and Behavior, 32, 479-510. doi:10.1177/0093854805278411

29.

Doyle

Lewis

Brisbane

(2008). Implementing the Short-Term Assessment of Risk and Treatability (START) in a forensic mental health service. BJPsych Bulletin, 32, 406-408. doi:10.1192/pb.bp.108.019794

30.

Fass

T. L.

Heilbrun

DeMatteo

Fretz

(2008). The LSI-R and the Compas: Validation data on two risk-needs tools. Criminal Justice and Behavior, 35, 1095-1108. doi:10.1177/0093854808320497

31.

Ferguson

A. M.

Ogloff

J. R. P.

Thomson

(2009). Predicting recidivism by mentally disordered offenders using the LSI-R:SV. Criminal Justice and Behavior, 36, 5-20. doi:10.1177/0093854808326525

32.

Gelman

Fagan

Kiss

(2007). An analysis of the New York City police department’s “stop-and-frisk” policy in the context of claims of racial bias. Journal of the American Statistical Association, 102, 813-823. doi:10.1198/016214506000001040

33.

Grove

W. M.

Meehl

P. E.

(1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical–statistical controversy. Psychology, Public Policy, and Law, 2, 293-323. doi:10.1037/1076-8971.2.2.293

34.

Harcourt

B. E

. (2015). Risk as a proxy for race: The dangers of risk assessment. Federal Sentencing Reporter, 27, 237-243. doi:10.1525/fsr.2015.27.4.237

35.

Holder

(2014). Attorney General Eric Holder speaks at the National Association of Criminal Defense Lawyers 57th Annual Meeting. Retrieved from https://www.justice.gov/opa/speech/attorney-general-eric-holder-speaks-national-association-criminal-defense-lawyers-57th

36.

Inett

Wright

Roberts

Sheeran

(2014). Predictive validity of the START with intellectually disabled offenders. Journal of Forensic Practice, 16, 78-88. doi:10.1108/JFP-12-2012-0029

37.

Kelly

C. E.

Welsh

W. N.

(2008). The predictive validity of the Level of Service Inventory—Revised for drug-involved offenders. Criminal Justice and Behavior, 35, 819-831. doi:10.1177/0093854808316642

38.

Labrecque

R. M.

Smith

Lovins

B. K.

Latessa

E. J.

(2014). The importance of reassessment: How changes in the LSI-R risk score can improve the prediction of recidivism. Journal of Offender Rehabilitation, 53, 116-128. doi:10.1080/10509674.2013.868389

39.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. doi:10.2307/2529310

40.

Lodewijks

H. P. B.

de Ruiter

Doreleijers

T. A. H.

(2010). The impact of protective factors in desistance from violent reoffending: A study in three samples of adolescent offenders. Journal of Interpersonal Violence, 25, 568-587. doi:10.1177/0886260509334403

41.

Lowenkamp

C. T.

Holsinger

A. M.

Brusman-Lovins

Latessa

E.J.

(2004). Assessing the inter-rater agreement of the Level of Service Inventory Revised. Federal Probation, 68, 34-38.

42.

Lowenkamp

C. T.

Holsinger

A. M.

Cohen

T. H.

(2015). PCRA revisited: Testing the validity of the Federal Post Conviction Risk Assessment (PCRA). Psychological Services, 12, 149-157. doi:10.1037/ser0000024

43.

Lowenkamp

C. T.

Lovins

Latessa

E. J.

(2009). Validating the Level of Service Inventory—Revised and the Level of Service Inventory: Screening Version with a sample of probationers. Prison Journal, 89, 192-204. doi:10.1177/0032885509334755

44.

Manchak

S. M.

Skeem

J. L.

Douglas

K. S.

(2008). Utility of the Revised Level of Service Inventory (LSI-R) in predicting recidivism after long-term incarceration. Law and Human Behavior, 32, 477-488. doi:10.2307/30219000

45.

Mauer

(2006). The crisis of the young African American male and the criminal justice system. In Harris

Miller

R. R.

(Eds.), Impacts of incarceration on the African American family (pp. 199-218). New Brunswick, NJ: Transaction.

46.

McGraw

K. O.

Wong

S. P.

(1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46. doi:10.1037/1082-989X.1.1.30

47.

Nicholls

T. L.

Desmarais

S. L.

Martin

M.-L.

Brink

Webster

C. D.

(2006). Short-Term Assessment of Risk and Treatability (START). In Morgan

R. D.

(Ed.), The SAGE encyclopedia of criminal psychology. Thousand Oaks, CA: Sage.

48.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2014). Thirty years of research on the Level of Service scales: A meta-analytic examination of predictive accuracy and sources of variability. Psychological Assessment, 26, 156-176. doi:10.1037/a0035080

49.

O’Shea

L. E.

Dickens

G. L.

(2014). Short-Term Assessment of Risk and Treatability (START): Systematic review and meta-analysis. Psychological Assessment, 26, 990-1002. doi:10.1037/a0036794

50.

Ostermann

Salerno

L. M.

(2016). The validity of the Level of Service Inventory–Revised at the intersection of race and gender. Prison Journal, 96, 554-575. doi:10.1177/0032885516650878

51.

Pettit

Western

(2004). Mass imprisonment and the life course: Race and class inequality in U.S. incarceration. American Sociological Review, 69, 151-169. doi:10.1177/000312240406900201

52.

Proctor

B. D.

Semega

J. L.

Kollar

M. A.

(2016). Income and poverty in the United States: 2015. Washington, DC: U.S. Census Bureau, U.S. Department of Commerce. Retrieved from https://www.census.gov/content/dam/Census/library/publications/2016/demo/p60-256.pdf

53.

Richardson

McSherry

(2010). Diversion down under: Programs for offenders with mental illnesses in Australia. International Journal of Law and Psychiatry, 33, 249-257. doi:10.1016/j.ijlp.2010.06.007

54.

Rocque

Plummer-Beale

(2014). In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency. Journal of Criminal Justice, 42, 568-578. doi:10.1016/j.jcrimjus.2014.09.011

55.

Schlager

M. D.

Simourd

D. J.

(2007). Validity of the Level of Service Inventory—Revised (LSI-R) among African American and Hispanic male offenders. Criminal Justice and Behavior, 34, 545-554. doi:10.1177/0093854806296039

56.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.

57.

Singh

J. P.

Grann

Fazel

(2011). A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clinical Psychology Review, 31, 499-513. doi:10.1016/j.cpr.2010.11.009

58.

Sirotich

(2009). The criminal justice outcomes of jail diversion programs for persons with mental illness: A review of the evidence. Journal of the American Academy of Psychiatry and the Law, 37, 461-472.

59.

Skeem

J. L.

Lowenkamp

C. T.

(2015). Risk, race, & recidivism: Predictive bias and disparate impact (SSRN Scholarly Paper No. ID 2687339). Rochester, NY: Social Science Research Network. Retrieved from http://papers.ssrn.com/abstract=2687339

60.

Skeem

J. L.

Monahan

(2011). Current directions in violence risk assessment. Current Directions in Psychological Science, 20, 38-42. doi:10.1177/0963721410397271

61.

Skeem

J. L.

Winter

Kennealy

P. J.

Louden

J. E.

Tatar

J. R.

(2014). Offenders with mental illness have criminogenic needs, too: Toward recidivism reduction. Law and Human Behavior, 38, 212-224.

62.

Slinger

Roesch

(2010). Problem-solving courts in Canada: A review and a call for empirically-based evaluation methods. International Journal of Law and Psychiatry, 33, 258-264. doi:10.1016/j.ijlp.2010.06.008

63.

Starr

S. B.

(2014). Evidence-based sentencing and the scientific rationalization of discrimination. Stanford Law Review, 66, 803. Retrieved from https://www.stanfordlawreview.org/print/article/evidence-based-sentencing-and-the-scientific-rationalization-of-discrimination/

64.

Steadman

H. J.

Barbera

S. S.

Dennis

D. L.

(1994). A national survey of jail diversion programs for mentally ill detainees. Hospital & Community Psychiatry, 45, 1109-1113.

65.

Steadman

H. J.

Naples

(2005). Assessing the effectiveness of jail diversion programs for persons with serious mental illness and co-occurring substance use disorders. Behavioral Sciences & the Law, 23, 163-170. doi:10.1002/bsl.640

66.

Steadman

H. J.

Osher

Robbins

P. C.

Case

Samuels

(2009). Prevalence of serious mental illness among jail inmates. Psychiatric Services, 60, 761-765. doi:10.1176/appi.ps.60.6.761

67.

Steadman

H. J.

Redlich

Callahan

Robbins

P. C.

Vesselinov

(2011). Effect of mental health courts on arrests and jail days: A multisite study. Archives of General Psychiatry, 68, 167-172. doi:10.1001/archgenpsychiatry.2010.134

68.

Sweeney

L. T.

Haney

(1992). The influence of race on sentencing: A meta-analytic review of experimental studies. Behavioral Sciences & the Law, 10, 179-195. doi:10.1002/bsl.2370100204

69.

Torrey

E. F.

Stieber

Ezekiel

(1998). Criminalizing the seriously mentally ill: The abuse of jails as mental hospitals. Collingdale, PA: Diane.

70.

Urban Institute. (2016). Measuring recidivism at the local level: A quick guide. Retrieved from http://www.urban.org/sites/default/files/recidivism-measures_final-for-website.pdf

71.

U.S. Census Bureau. (2014). Wealth and asset ownership. Washington, DC: Author.

72.

U.S. Census Bureau. (2016). Educational attainment in the United States: 2015. Washington, DC: Author.

73.

Varela

J. G.

Boccaccini

M. T.

Murrie

D. C.

Caperton

J. D.

Gonzalez

E. J.

(2013). Do the Static-99 and Static-99R perform similarly for White, Black, and Latino sexual offenders? International Journal of Forensic Mental Health, 12, 231-243. doi:10.1080/14999013.2013.846950

74.

Vornovitsky

Gottschalck

Smith

(2011). Distribution of household wealth in the U.S.: 2000 to 2011. Washington, DC: U.S. Census Bureau.

75.

Vose

Cullen

F. T.

Smith

(2008). The empirical status of the Level of Service Inventory. Federal Probation, 72, 22-29.

76.

Vose

Smith

Cullen

F. T.

(2013). Predictive validity and the impact of change in total LSI-R score on recidivism. Criminal Justice and Behavior, 40, 1383-1396. doi:10.1177/0093854813508916

77.

Walters

G. D.

(2007). Using Poisson class regression to analyze count data in correctional and forensic psychology: A relatively old solution to a relatively new problem. Criminal Justice and Behavior, 34, 1659-1674. doi:10.1177/0093854807307030

78.

Webster

C. D.

Martin

M. L.

Brink

Nicholls

T. L.

Desmarais

S. L.

(2009). Short-Term Assessment of Risk and Treatability (START) (Version. 1.1). Coquitlam, British Columbia, Canada: British Columbia Mental Health & Addiction Services.

79.

Webster

C. D.

Nicholls

T. L.

Martin

M.-L.

Desmarais

S. L.

Brink

(2006). Short-Term Assessment of Risk and Treatability (START): The case for a new structured professional judgment scheme. Behavioral Sciences & the Law, 24, 747-766. doi:10.1002/bsl.737

80.

Whiteacre

K. W.

(2006). Testing the Level of Service Inventory–Revised (LSI-R) for racial/ethnic bias. Criminal Justice Policy Review, 17, 330-342. doi:10.1177/0887403405284766

81.

Wilper

A. P.

Woolhandler

Boyd

J. W.

Lasser

K. E.

McCormick

Bor

D. H.

Himmelstein

D. U.

(2009). The health and health care of US prisoners: Results of a nationwide survey. American Journal of Public Health, 99, 666-672. doi:10.2105/AJPH.2008.144279

82.

Wilson

C. M.

Desmarais

S. L.

Nicholls

T. L.

Brink

(2010). The role of client strengths in assessments of violence risk using the Short-Term Assessment of Risk and Treatability (START). International Journal of Forensic Mental Health, 9, 282-293. doi:10.1080/14999013.2010.534694

83.

Woldgabreal

Day

Ward

(2014). The community-based supervision of offenders from a positive psychology perspective. Aggression and Violent Behavior, 19, 32-41. doi:10.1016/j.avb.2013.12.001