Criminal justice researchers and agencies often look to empirical reviews and meta-analyses when evaluating offender risk instruments. However, variations in meta-analytical methods can influence the interpretation of statistical findings, resulting in potentially misleading conclusions. Through our re-examination of a meta-analysis by Singh, Grann, and Fazel (2011), we outline common methodological problems that may occur and demonstrate how alternative interpretations might be derived. We conclude by providing recommendations for conducting meta-analyses. Suggestions are made for researchers to consider alternative strategies when examining the validity of risk instruments, and for correctional agencies to consider, but go beyond, predictive validity when selecting offender risk instruments.
Get full access to this article
View all access options for this article.
References
1.
ÆgisdóttirS.WhiteM. J.SpenglerP. M.MaughermanA. S.AndersonL. A.CookR. S.…RushJ. D. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34, 341–382. doi: 10.1177/0011000005285875
2.
American Educational Research AssociationAmerican Psychological AssociationNational Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3.
American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: Author.
4.
American Psychological Association (2013). Specialty guidelines for forensic psychology. American Psychologist, 68, 7–19. doi: 10.1037/a0029889
5.
AndrewsD. A.BontaJ. (1995). Level of service inventory – Revised: User's manual. Toronto, Canada: Multi-Health Systems.
6.
AndrewsD. A.BontaJ.WormithJ. S. (2004). User's manual for the Level of Service/Case Management Inventory (LS/CMI): An offender management system. Toronto, Canada: Multi-Health Systems.
7.
AndrewsD. A.BontaJ.WormithJ. S. (2006). The recent past and near future of risk and/or need assessment. Crime and Delinquency, 52, 7–27. doi: 10.1177/0011128705281756
8.
AndrewsD. A.BontaJ.WormithJ. S.GuzzoL.BrewsA.RettingerJ.RoweR. (2011). Sources of variability in estimates of predictive validity: A specification with Level of Service general risk and need. Criminal Justice and Behavior, 38, 413–432. doi: 10.1177/0093854811401990
9.
AndrewsD. A.DowdenC. (2006). Risk principle of case classification in correctional treatment: A meta-analytic investigation. International Journal of Offender Treatment and Comparative Criminology, 50, 88–100. doi: 10.1177/0306624X05282556
10.
AndrewsD. A.GuzzoL.RaynorP.RoweR. C.RettingerL. J.BrewsA.WormithJ. S. (2012). Are the major risk/need factors predictive of both female and male reoffending? A test with the eight domains of the Level of Service/Case Management Inventory. International Journal of Offender Therapy and Comparative Criminology, 56, 113–133. doi: 10.1177/0306624X10395716
11.
ArcherR. P.Buffington-VollumJ. K.StrednyR. V.HandelR. W. (2006). A survey of psychological test use patterns among forensic psychologists. Journal of Personality Assessment, 87, 84–94. doi: 10.1207/s15327752jpa8701_07
12.
AustinJ. (2004). The proper and improper use of risk assessment in corrections. Federal Sentencing Reporter, 16, 1–6. doi: 10.1525/fsr.2004.16.3.194
13.
BarbareeH. E.SetoM. C.LangtonC. M.PeacockE. J. (2001). Evaluating the predictive accuracy of six risk assessment instruments for adult sex offenders. Criminal Justice and Behavior, 28, 490–521. doi: 10.1177/009385480102800406
14.
BergM. T.HuebnerB. M. (2011). Reentry and the ties that bind: An examination of social ties, employment, and recidivism. Justice Quarterly, 28, 382–410. doi: 10.1080/07418825.2010.498383
15.
BoccacciniM. T.MurrieD. C.ClarkJ. W.CornellD. G. (2008). Describing, diagnosing, and naming psychopathy: How do youth psychopathy labels influence jurors?Behavioral Sciences and the Law, 26, 487–510. doi: 10.1002/bsl.821
16.
BoerD. P.HartS. D.KroppP. R.WebsterC. D. (1997). Manual for the Sexual Violence Risk-20: Professional guidelines for assessing risk of sexual violence. Burnaby, Canada: Simon Fraser University.
17.
BontaJ. (2002). Offender risk assessment: Guidelines for selection and use. Criminal Justice and Behavior, 29, 355–379. doi: 10.1177/0093854802029004002
18.
BontaJ. (1996). Risk-needs assessment and treatment. In HarlandA. T. (Ed.), Choosing correctional options that work: Defining the demand and evaluating the supply (pp. 18–32). Thousand Oaks, CA: SAGE.
19.
BontaJ.AndrewsD. A. (2017). The psychology of criminal conduct (6th ed.). New York, NY: Routledge.
20.
BontaJ.WormithS. J. (2007). Risk and need assessment. In McIvorG.RaynorP. (Eds.), Developments in social work with offenders (pp. 131–152). Philadelphia, PA: Jessica Kingsley.
21.
BorumR.BartelP.ForthA. (2006). Structured Assessment of Violence Risk in Youth: Professional manual. Lutz, FL: Psychological Assessment Resources.
22.
BrennanT.OliverW. L. (2000). Evaluation of reliability and validity of COMPAS scales: National aggregate sample. Traverse City, MI: Northpointe Institute for Public Management.
23.
ByrneJ. M.PattavinaA. (2006). Assessing the role of clinical and actuarial risk assessment in an evidence-based community system: Issues to consider. Federal Probation, 70, 64–68.
24.
CampbellM. A.FrenchS.GendreauP. (2009). The prediction of violence in adult offenders: A meta-analytic comparison of instruments and methods of assessment. Criminal Justice and Behavior, 36, 567–590. doi: 10.1177/0093854809333610
25.
CardN. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford.
26.
CatchpoleR. E. H.GrettonH. M. (2003). The predictive validity of risk assessment with violent young offenders: A 1-year examination of criminal outcome. Criminal Justice and Behavior, 30, 688–708. doi: 10.1177/0093854803256455
27.
CraigL. A.BrowneK. D.StringerI.BeechA. (2005). Sexual recidivism: A review of static, dynamic, and actuarial predictors. Journal of Sexual Aggression, 11, 65–84. doi: 10.1080/13552600410001667733
28.
CummingG.FinchS. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170–180. doi: 10.1037/0003-066X.60.2.170
29.
DaffernM. (2007). The predictive validity and practical utility of structured schemes used to assess risk for aggression in psychiatric inpatient settings. Aggression and Violent Behavior, 12, 116–130. doi: 10.1016/j.avb.2006.03.005
30.
DahleK.-P. (2006). Strengths and limitations of actuarial prediction of criminal reoffence in a German prison sample: A comparative study of LSI-R, HCR-20, and PCL-R. International Journal of Law and Psychiatry, 29, 431–442. doi: 10.1016/j.ijlp.2006.03.001
31.
DelaneyA.BagshawS. M.FerlandA.MannsB.LauplandK. B.DoigC. J. (2005). A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care, 9, 575–582. doi: 10.1186/cc3803
32.
DouglasK.S.HartS. D.WebsterC. D.BelfrageH.GuyL. S.WilsonC. M. (2014). Historical-Clinical-Risk Management-20, Version 3 (HCR-20 v3): Development and overview. International Journal of Forensic Mental Health, 13, 93–98. doi: 10.1080/14999013.2014.906519
33.
EdensJ. F.MarcusD. K.LilienfeldS. O.PoythressN. G.Jr. (2006). Psychopathic, not psychopath: Taxometric evidence for the dimensional structure of psychopathy. Journal of Abnormal Psychology, 115, 131–144. doi: 10.1037/0021-843X.115.1.131
34.
EdensJ. F.MarcusD. K.VaughnM. G. (2011). Exploring the taxometric status of psychopathy among youthful offenders: Is there a juvenile psychopathy taxon?Law and Human Behavior, 35, 13–24. doi: 10.1007/s10979-010-9230-8
35.
FazelS.SinghJ. P.DollH.GrannM. (2012). Use of risk assessment instruments to predict violence and antisocial behaviour in 73 samples involving 24,827 people: Systematic review and meta-analysis. BMJ, 345, e4692. doi: 10.1136/bmj.e4692
36.
GendreauP.GogginC. E.LawM. A. (1997). Predicting prison misconducts. Criminal Justice and Behavior, 24, 414–431. doi: 10.1177/0093854897024004002
37.
GendreauP.GogginC.SmithP. (2002). Is the PCL-R really the “unparalleled” measure of offender risk? A lesson in knowledge cumulation. Criminal Justice and Behavior, 29, 397–426. doi: 10.1177/0093854802029004004
38.
GendreauP.LittleT.GogginC. (1996). A meta-analysis of the predictors of adult offender recidivism: What works!Criminology, 34, 575–607. doi: 10.1111/j.1745-9125.1996.tb01220.x
39.
GendreauP.SmithP. (2007). Influencing the “people who count”: Some perspectives on the reporting of meta-analytic results for prediction and treatment outcomes with offenders. Criminal Justice and Behavior, 34, 1536–1559. doi: 10.1177/0093854807307025
40.
GenestJ.FrohlichJ.FodorG.McPhersonR. (2003). Recommendations for the management of dyslipidemia and the prevention of cardiovascular disease: Summary of the 2003 update. Canadian Medical Association Journal, 169, 921–924.
41.
GlancyG. D.SainiM. (2009). The confluence of evidence-based practice and Daubert within the fields of forensic psychiatry and the law. The Journal of the American Academy of Psychiatry and the Law, 37, 438–441.
42.
GoldsteinH.HealyM. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 175–177.
43.
GreenhalghT. (1997). How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). British Medical Journal, 315, 672–675. doi: 10.1136/bmj.315.7109.672
44.
GroveW. M.ZaldD. H.LebowB. S.SnitzB. E.NelsonC. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19–30. doi: 10.1037/1040-3590.12.1.19
45.
GutierrezL.WilsonH. A.RuggeT.BontaJ. (2013). The prediction of recidivism with Aboriginal offenders: A theoretically informed meta-analysis. Canadian Journal of Criminology and Criminal Justice, 55, 55–99. doi: 10.3138/cjccj.2011.E.51
46.
GuyL. S.DouglasK. S.HendryM. C. (2010). The role of psychopathic personality disorder in violence risk assessments using the HCR-20. Journal of Personality Disorders, 24, 551–580. doi: 10.1521/pedi.2010.24.5.551
47.
HansonR. K. (2009). The psychological assessment of risk for crime and violence. Canadian Psychology, 50, 172–182. doi: 10.1037/a0015726
48.
HansonR. K.BourgonG.HelmusL.HodgsonS. (2009). The principles of effective correctional treatment also apply to sexual offenders: A meta-analysis. Criminal Justice and Behavior, 36, 865–891. doi: 10.1177/0093854809338545
49.
HansonR. K.HowardP. D. (2010). Individual confidence intervals do not inform decision-makers about the accuracy of risk assessment evaluations. Law and Human Behavior, 34, 275–281. doi: 10.1007/s10979-010-9227-3
50.
HansonR. K.Morton-BourgonK. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118-prediction studies. Psychological Assessment, 21, 1–21. doi: 10.1037/a0014421
51.
HansonR. K.ThorntonD. (2000). Improving risk assessments for sex offenders: A comparison of three actuarial scales. Law and Human Behavior, 24, 119–136. doi: 10.1023/A:1005482921333
52.
HareR. D. (2003). Hare PCL-R: Technical manual (2nd ed.). Toronto, Canada: Multi-Health Systems.
53.
HarrisG. T.RiceM. E. (2007). Characterizing the value of actuarial violence risk assessments. Criminal Justice and Behavior, 34, 1638–1658. doi: 10.1177/0093854807307029
54.
HarrisG. T.RiceM. E.QuinseyV. L. (1993). Violent recidivism of mentally disordered offenders: The development of a statistical prediction instrument. Criminal Justice and Behavior, 20, 315–335.
HartS. D.CookeD. J. (2013). Another look at the (im)-precision of individual risk estimates made using actuarial risk assessment instruments. Behavioral Sciences and the Law, 31, 81–102. doi: 10.1002/bsl.2049
57.
HartS. P.KroppP. R.HareR. D. (1988). Performance of psychopaths following conditional release from prison. Journal of Consulting and Clinical Psychology, 56, 227–232. doi: org/10.1037/0022-006X.56.2.227
58.
HartS. D.MichieC.CookeD. J. (2007). Precision of actuarial risk assessment instruments: Evaluating the margins of error' of group v. individual predictions of violence. British Journal of Psychiatry, 190, 60–65. doi: 10.1192/bjp.190.5.s60
59.
HaslamN.HollandE.KuppensP. (2012). Categories versus dimensions in personality and psychopathology: A quantitiative review of taxometric research. Psychological Medicine, 42, 903–920. doi: 10.1017/S0033291711001966
60.
HumphreysL. G.SwetsJ. A. (1991). Comparison of predictive validities measured with biserial correlations and ROCs of signal detection theory. Journal of Applied Psychology, 76, 316–321. doi: 10.1037/0021-9010.76.2.316
61.
JacksonR. L.HessD. T. (2007). Evaluation for civil commitment of sex offenders: A survey of experts. Sexual Abuse: A Journal of Research and Treatment, 19, 425–448. doi: 10.1177/107906320701900407
62.
JanusE. S.PrentkyR. A. (2003). Forensic use of actuarial risk assessment with sex offenders: Accuracy, admissibility, and accountability. American Criminal Law Review, 40, 1443–1499.
63.
KronerD. G.MillsJ. F. (2001). The accuracy of five risk appraisal instruments in predicting institutional misconduct and new convictions. Criminal Justice and Behavior, 28, 471–489. doi: 10.1177/009385480102800405
KruegerR. F.WatsonD.BarlowD. H. (2005). Introduction to the special section: Toward a dimensionally based taxonomy of psychopathology. Journal of Abnormal Psychology, 114, 491–493. doi: 10.1037/0021-843X.114.4.491
66.
LallyS. J. (2003). What tests are acceptable for use in forensic evaluations? A survey of experts. Professional Psychology: Research and Practice, 34, 491–498. doi: 10.1037/0735-7028.34.5.491
67.
LangtonC. M.BarbareeH. E.SetoM. C.PeacockE. J.HarkinsL.HansenK. T. (2007). Actuarial assessment of risk for reoffense among adult sex offenders: Evaluating the predictive accuracy of the Static-2002 and five other instruments. Criminal Justice and Behavior, 34, 37–59. doi: 10.1177/0093854806291157
68.
LipseyM. W. (2003). Those confounded moderators in meta-analysis: Good, bad and ugly. Annals of the American Academy of Political and Social Sciences, 587, 69–81. doi: 10.1177/0002716202250791
69.
LoucksA. D.ZambleE. (2000). Predictors of criminal behavior and prison misconduct in serious female offenders. Empirical and Applied Criminal Justice Review, 1, 1–47.
70.
LozaW.GreenK. (2003). The Self-Appraisal Questionnaire – A self-report measure for predicting recidivism versus clinician-administered measures: A 5-year follow-up study. Journal of Interpersonal Violence, 18, 781–797. doi: 10.1177/0886260503253240
71.
LozaW.VilleneuveD. B.Loza-FanousA. (2002). Predictive validity of the Violence Risk Appraisal Guide: A tool for assessing violent offender's recidivism. International Journal of Law and Psychiatry, 25, 85–92. doi: 10.1016/S0160-2527(01)00092-9
72.
MacCallumR. C.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi: 10.1037/1082-989X.7.1.19
73.
MarcusD. K.LilienfeldS. O.EdensJ. F.PoythressN. G. (2006). Is antisocial personality disorder continuous or categorical? A taxometric analysis. Psychological Medicine, 36, 1571–1581. doi: 10.1017/S0033291706008245
74.
McGillR.TukeyJ. W.LarsenW. A. (1978). Variations of box plots. The American Statistician, 32, 12–16. doi: 10.1080/00031305.1978.10479236
75.
MonahanJSteadmanH. J. (1996). Violent storms and violent people: How meteorology can inform risk communication in mental health law. American Psychologist, 51, 931–938. doi: 10.1037/0003-066X.51.9.931
76.
MurrieD. C.MarcusD. K.DouglasK. S.LeeZ.SalekinR. T.VincentG. (2007). Youth with psychopathy features are not a discrete class: A taxometric analysis. Journal of Child Psychology and Psychiatry, 48, 714–723. doi: 10.1111/j.1469-7610.2007.01734.x
77.
OlverM.E.StockdaleK.C.WormithJ.S. (2009). Risk assessment with young offenders: A meta-analysis of three assessment measures. Criminal Justice and Behavior, 36, 329–353. doi: 10.1177/0093854809331457
78.
OlverM. E.StockdaleK. C.WormithJ. S. (2014). Thirty years of research on the Level of Service Scales: A meta-analytic examination of predictive accuracy and sources of variability. Psychological Assessment, 26, 156–176. doi: 10.1037/a0035080
79.
QuinseyV. L.HarrisG. T.RiceM. E.CormierC. A. (2006). Violent offenders: Appraising and managing risk (2nd ed.). Washington, DC: American Psychological Association.
80.
RegierD. A.NarrowW. E.KuhlE. A.KupferD. J. (2011). The conceptual evolution of DSM-5. Arlington, VA: American Psychiatric Publishing.
81.
ReimannB. J.NussbaumD. (2011). Predicting seclusion in a medium secure forensic inpatient setting. International Journal of Forensic Mental Health, 10, 150–158. doi: 10.1080/14999013.2011.578299
82.
RighthandS.PrentkyR.KnightR.CarpenterE.HeckerJ. E.NangleD. (2005). Factor structure and validation of the Juvenile Sex Offender Assessment Protocol (J-SOAP). Sexual Abuse: A Journal of Research and Treatment, 17, 13–30. doi: 10.1007/s11194-005-1207-7
83.
SackettP. R.YangH. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118. doi: 10.1037/0021-9010.85.1.112
84.
SchenkerN.GentlemanJ. F. (2001). On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician, 55, 182–186. doi: 10.1198/000313001317097960
85.
SerinR. C.AmosN. L. (1995). The role of psychopathy in the assessment of dangerousness. International Journal of Law and Psychiatry, 18, 231–238. doi: 10.1016/0160-2527(95)00008-6
86.
SinghJ. P.FazelS. (2010). Forensic risk assessment: A metareview. Criminal Justice and Behavior, 37, 965–988. doi: 10.1177/0093854810374274
87.
SinghJ. P.GrannM.FazelS. (2011). A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clinical Psychology Review, 31, 499–513. doi: 10.1016/j.cpr.2010.11.009
88.
SinghJ. P.DesmaraisS. L.HurducusC.Arbach-LucioniK.CondemarinC. (2014). International perspectives on the practical application of violence risk assessment: A global survey of 44 countries. International Journal of Forensic Mental Health, 13, 193–206. doi: 10.1080/14999013.2014.922141
89.
SkeemJ. L.MonahanJ. (2011). Current directions in violence risk assessment. Current Directions in Psychological Science, 20, 38–42. doi: 10.1177/0963721410397271
90.
SmithP.CullenF. T.LatessaE. J. (2009). Can 14,737 women be wrong? A meta-analysis of the LSI-R and recidivism for female offenders. Criminology and Public Policy, 8, 183–208. doi: 10.1111/j.1745-9133.2009.00551.x
91.
TolmanA. O.MullendoreK. B. (2003). Risk evaluations for the courts: Is service quality a function of specialization?Professional Psychology: Research and Practice, 34, 225–232. doi: 10.1037/0735-7028.34.3.225
ViljoenJ. L.McLachlanK.VincentG. M. (2010). Assessing violence risk and psychopathy in juvenile and adult offenders: A survey of clinical practices. Assessment17, 377–395. doi: 10.1177/1073191109359587
94.
ViljoenJ. L.MordellS.BeneteauJ. L. (2012). Prediction of adolescent sexual reoffending: A meta-analysis of the J-SOAP-II, ERASOR, J-SORRAT-II, and Static-99. Law and Human Behavior, 36, 423–438. doi: 10.1037/h0093938
95.
VincentG. M.GuyL. S.GrissoT. (2012). Risk assessment in juvenile justice: A guidebook for implementation. Systems and Psychosocial Advances Research Center Publications and Presentations. Paper 573. Retrieved from http://escholarship.umassmed.edu/psych_cmhsr/573
96.
VincentG. M.Paiva-SalisburyM. L.CookN. E.GuyL. S.PerraultR. T. (2012). Impact of risk/needs assessment on juvenile probation officers' decision making: Importance of implementation. Psychology, Public Policy, and Law, 18, 549–576. doi: 10.1037/a0027186
97.
VoseB.CullenF. T.SmithP. (2008). The empirical status of the Level of Service Inventory. Federal Probation, 72, 22–29.
98.
WaltersG. D. (2006). Risk-appraisal versus self-report in the prediction of criminal justice outcomes. Criminal Justice and Behavior, 33, 279–304. doi: 10.1177/0093854805284409
99.
WaltersG. D. (2012). Taxometrics and criminal justice: Assessing the latent structure of crime-related constructs. Journal of Criminal Justice, 40, 10–20. doi: 10.1016/j.jcrimjus.2011.11.003
100.
WaltersG. D.KnightR. A.ThorntonD. (2009). The latent structure of sexual violence risk: A taxometric analysis of widely used sex offender actuarial risk measures. Criminal Justice and Behavior, 36, 290–306. doi: 10.1177/0093854808330341
101.
WaltersG. D.RonenT.RosenbaumM. (2010). The latent structure of childhood aggression: A taxometric analysis of self-reported and teacher-rated aggression in Israeli schoolchildren. Psychological Assessment, 22, 628–637. doi: 10.1037/a0019779
102.
WebsterC. D.DouglasK. S.EavesD.HartS. D. (1997). HCR-20: Assessing risk for violence (version 2). Burnaby, Canada: Simon Fraser University.
103.
WidigerT. A.SamuelD. B. (2005). Diagnostic categories or dimensions? A question for the Diagnostic and Statistical Manual of Mental Disorders–Fifth Edition. Journal of Abnormal Psychology, 4, 494–504. doi: 10.1037/0021-843X.114.4.494
104.
WolfeR.HanleyJ. (2002). If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2. Canadian Medical Association Journal, 166, 65–66.
105.
WongS.HareR. D. (2005). Guidelines for a psychopathy treatment program. Toronto, Canada: Multi-Health Systems.
106.
WormithJ.S. (2014). The risks of communicating sexual offender risk. Journal of Threat Assessment and Management, 1, 162–178.
107.
WormithJ. S.OlverM. E.StevensonH. E.GirardL. (2007). The long-term prediction of offender recidivism using diagnostic, personality, and risk/need approaches to offender assessment. Psychological Services, 4, 287–305. doi: 10.1037/1541-1559.4.4.287
108.
YangM.WongS. C. P.CoidJ. (2010). The efficacy of violence prediction: A meta-analysis comparison of nine risk assessment tools. Psychological Bulletin, 136, 740–767. doi: 10.1037/a0020473
