This commentary on Williams et al. (2017) focuses on an additional and equally important issue not addressed in their critique: inter-rater reliability --particularly reliability in field settings. A growing body of evidence indicates that risk assessment instruments administered in applied (and especially adversarial) contexts may be considerably less stable across examiners than what typically is reported in well-controlled, peer-reviewed journal publications. Because reliability constrains validity, effect sizes from such published research may overestimate predictive validity in real-world contexts. Although validity evidence is important, field reliability remains “the boss” when considering how well an assessment procedure will perform in applied settings.
Get full access to this article
View all access options for this article.
References
1.
AcklinM. W.FugerK. (2016). Assessing field reliability of forensic decision making in criminal court. Journal of Forensic Psychology Practice, 16(2), 74–93. doi:10.1080/15228932.2016.1148452
2.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3.
BoccacciniM. T.MurrieD. C.CapertonJ. D.HawesS. W. (2009). Field validity of the Static-99 and MnSOST-R among sex offenders evaluated for civil commitment as sexually violent predators. Psychology, Public Policy, & Law, 15(4), 278–314. doi:10.1037/a0017232
4.
BoccacciniM. T.MurrieD. C.MercadoC.QuesadaS.HawesS.RiceA. K.JeglicE. L. (2012). Implications of Static-99 field reliability findings for score use and reporting. Criminal Justice and Behavior, 39(1), 42–58. doi:10.1177/0093854811427131
5.
BoccacciniM. T.MurrieD. C.RufinoK. A.GardnerB. O. (2014). Evaluator differences in Psychopathy Checklist-Revised factor and facet scores. Law and Human Behavior, 38(4), 337–345. doi:10.1037/lhb0000069
6.
BoccacciniM. T.TurnerD. B.MurrieD. C.RufinoK. A. (2012). Do PCL-R scores from state or defense experts best predict future misconduct among civilly committed sex offenders?Law and Human Behavior, 36(3), 159–169. doi:10.1037/h0093949
7.
BoccacciniM. T.TurnerD. B.MurrieD. C. (2008). Do some evaluators report consistently higher or lower scores on the PCL-R? Findings from a state-wide sample of sexually violent predator evaluations. Psychology, Public Policy, and Law, 14(4), 262–283. doi:10.1037/a0014523
8.
BoerD. P.HartS. D.KroppP. R.WebsterC. D. (1997). Manual for the Sexual Violence Risk-20: Professional guidelines for assessing risk of sexual violence. Burnaby, Canada: Simon Fraser University.
9.
BorumR.BartelP.ForthA. (2006). Structured Assessment of Violence Risk in Youth: Professional manual. Lutz, FL: Psychological Assessment Resources.
10.
ChevalierC. S.BoccacciniM. T.MurrieD. C.VarelaJ. G. (2015). Static-99R reporting practices in sexually violent predator cases: Does norm selection reflect adversarial allegiance?Law and Human Behavior, 39(3), 209–218. doi:10.1037/lhb0000114
11.
DeMatteoD.EdensJ. F.GallowayM.CoxJ.SmithS. T.FormonD. (2014). The role and reliability of the Psychopathy Checklist-Revised in U.S. sexually violent predator evaluations: A case law survey. Law and Human Behavior, 38(3), 248–255. doi:10.1037/lhb0000059
12.
DrorI. E.KassinS. M.KukuckaJ. (2013). New application of psychology to law: improving forensic evidence and expert witness contributions. Journal of Applied Research in Memory and Cognition, 2(1), 78–81. doi:10.1016/j.jarmac.2013.02.003
13.
DouglasK. S.HartS. D.WebsterC. D.BelfrageH.GuyL. S.WilsonC. M. (2014). Historical-Clinical-Risk Management-20, Version 3 (HCR-20 v3): Development and overview. International Journal of Forensic Mental Health, 13, 93–98. doi:10.1080/14999013.2014.906519
14.
EdensJ.F.BoccacciniM.T. (press). Taking forensic mental health assessment “out of the lab” and into “the real world:” Introduction to the special issue on the field utility of forensic assessment instruments and procedures. Psychological Assessment.
15.
EdensJ. F.CoxJ.SmithS. T.DeMatteoD.SörmanK. (2015). How reliable are Psychopathy Checklist-Revised scores in Canadian criminal trials? A case law review. Psychological Assessment, 27(2), 447–456. doi:10.1037/pas0000048
16.
EdensJ. F.MagyarM. S.CoxJ. (2013). Taking psychopathy measures “out of the lab” and into the legal system: Some practical concerns. In KiehlK.Sinnott-ArmstrongW. (Eds.), Handbook on psychopathy and law (pp. 250–272). New York, NY: Oxford University Press.
17.
EdensJ. F.PensonB. N.RuchenskyJ. R.CoxJ.SmithS. T. (2016). Inter-rater reliability of Violence Risk Appraisal Guide scores provided in Canadian criminal proceedings. Psychological Assessment, 28(12), 1543–1549. doi:10.1037/pas0000278
18.
EdensJ. F.PetrilaJ.KelleyS. E. (press). Legal and ethical issues in the assessment and treatment of psychopathy. In PatrickC. (Ed.), Handbook of psychopathy (2nd ed.). New York, NY: Guilford.
19.
HansonR. K.LunettaA.PhenixA.NeeleyJ.EppersonD. (2014). The field validity of Static-99/R sex offender risk assessment tool in California. Journal of Threat Assessment and Management, 1(2), 102–117. doi:10.1037/tam0000014
20.
HareR. D. (2003). Hare Psychopathy Checklist—Revised manual (2nd ed.). Toronto, Canada: Multi-Health Systems.
21.
HeilbrunK. (1992). The role of psychological testing in forensic assessment. Law and Human Behavior, 16(3), 257–272. doi:10.1007/BF01044769
22.
HelmusL.ThorntonD.HansonR. K.BabchishinK. M. (2012). Improving the predictive accuracy of Static-99 and Static-2002 with older sex offenders: Revised age weights. Sexual Abuse: Journal of Research and Treatment, 24(1), 64–101. doi:10.1177/1079063211409951
23.
IsmailG.LoomanJ. (press). Field inter-rater reliability of the Psychopathy Checklist-Revised. International Journal of Offender Therapy and Comparative Criminology. Advance online publication. doi:10.1177/0306624x16652452
24.
JeandarmeI.EdensJ. F.HabetsP.BruckersL.OeiK.BogaertsS. (press). Psychopathy Checklist-Revised field validity in prison and hospital settings. Law and Human Behavior. Advance online publication. doi:10.1037/lhb0000222
25.
JeandarmeI.PoulsC.LaenderJ. D.OeiT. I.BogaertsS. (press). Field validity of the HCR-20 in forensic medium security units in Flanders. Psychology, Crime, & Law. Advance online publication. doi:10.1080/1068316X.2016.1258467
26.
KassinS. M.DrorI. E.KukuckaJ. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), 42–52. doi:10.1016/j.jarmac.2013.01.001
27.
KennealyP. J.SkeemJ. L.HernandezI. R. (press). Does staff see what experts see? Accuracy of front line staff in scoring juveniles' risk factors. Psychological Assessment. Advance online publication. doi:10.1037/pas0000316
28.
LawingK.ChildsK.FrickP.VincentG. (press). Use of structured professional judgment by probation officers to assess risk for recidivism in adolescent offenders. Psychological Assessment.
29.
MillerC. S.KimonisE. R.OttoR. K.KlineS. M.WassermanA. L. (2012). Reliability of risk assessment measures used in sexually violent predator proceedings. Psychological Assessment, 24(4), 944–953. doi:10.1037/a0028411
30.
MurrieD. C.BoccacciniM. T.CapertonJ.RufinoK. (2012). Field validity of the Psychopathy Checklist-Revised in sex offender risk assessment. Psychological Assessment, 24(2), 524–529. doi:10.1037/a0026015
31.
MurrieD. C.BoccacciniM. T.GuarneraL. A.RufinoK. (2013). Are forensic experts biased by the side that retained them?Psychological Science, 24(10), 1889–1897. doi:10.1177/0956797613481812
32.
MurrieD. C.BoccacciniM. T.JohnsonJ.JankeC. (2008). Does interrater (dis)agreement on Psychopathy Checklist scores in sexually violent predator trials suggest partisan allegiance in forensic evaluations?Law and Human Behavior, 32(4), 353–362. doi:10.1007/s10979-007-9097-5
33.
MurrieD.C.BoccacciniM.TurnerD.MeeksM.WoodsC.TusseyC. (2009). Rater (dis)agreement on risk assessment measures in sexually violent predator proceedings: Evidence of adversarial allegiance in forensic evaluation?Psychology, Public Policy, and Law, 15(1), 19–53. doi;10.1037/a0014897
34.
NealT. M. S.MillerS. L.ShealyR. C. (2015). A field study of a comprehensive violence risk assessment battery. Criminal Justice and Behavior, 42(9), 952–968. doi:10.1177/0093854815572252
35.
PenneyS. R.McMasterR.WilkieT. (2014). Multirater reliability of the Historical, Clinical, and Risk Management-20. Assessment, 21(1), 15–27. doi:10.1177/1073191113514107
36.
QuesadaS. P.CalkinsC.JeglicE. L. (2014). An examination of the interrater reliability between practitioners and researchers on the Static-99. International Journal of Offender Therapy and Comparative Criminology, 58(11), 1364–1375. doi:10.1177/0306624x13495504
37.
QuinseyV. L.HarrisG. T.RiceM. E.CormierC. A. (2006). Violent offenders: Appraising and managing risk (2nd ed.). Washington, DC: American Psychological Association.
38.
RettenbergerM.RiceM.HarrisG.EherR. (press). Actuarial risk assessment of sexual offenders: The psychometric properties of the Sex Offender Risk Appraisal Guide (SORAG). Psychological Assessment.
39.
RiceA. K.BoccacciniM. T.HarrisP. B.HawesS. W. (2014). Does field reliability for Static-99 scores decrease as scores increase?Psychological Assessment, 26(4), 1085–1094. doi:10.1037/pas0000009
40.
RiceM. E.HarrisG. T.LangC. (2013). Validation of and revision to the VRAG and SORAG: The Violence Risk Appraisal Guide – Revised (VRAG-R). Psychological Assessment, 25(3), 951–965. doi:10.1037/a0032878
41.
RosenthalR.RosnowR. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York, NY: McGraw-Hill.
42.
RufinoK. A.BoccacciniM. T.GuyL. S. (2011). Scoring subjectivity and item performance on measures used to assess violence risk: The PCL-R and HCR-20 as exemplars. Assessment, 18(4), 453–463. doi:10.1177/1073191110378482
43.
SinghJ. P.GrannM.FazelS. (2011). A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clinical Psychology Review, 31(3), 499–513. doi:10.1016/j.cpr.2010.11.009
44.
SturupJ.EdensJ. F.SörmanK.KarlbergD.FredrikssonB.KristianssonM. (2014). Field reliability of the Psychopathy Checklist-Revised among life sentenced prisoners in Sweden. Law and Human Behavior, 38(4), 315–324. doi:10.1037/lhb0000063
45.
TownsendP. (1971). Won't get fooled again [recorded by The Who]. On Who's Next. [Album]. Universal City, CA: MCA Records.
46.
VincentG. M.GuyL. S.FuscoS. L.GershensonB. G. (2012). Field reliability of the SAVRY with juvenile probation officers: Implications for training. Law and Human Behavior, 36(3), 225–236. doi:10.1037/h0093974
47.
WilliamsK.WormithS.BontaJ.SitareniosG. (2017). The use of meta-analysis to compare and select offender risk instruments: A commentary on Singh, Grann, and Fazel (2011). International Journal of Forensic Mental Health16(1), 1-15.
