Although reliability is essential to validity, most research on violence risk assessment tools has paid little attention to strategies for improving rater agreement. The authors evaluated the degree to which perceived subjectivity in scoring guidelines for items from two measures—the Psychopathy Checklist–Revised (PCL-R) and the Historical, Clinical, Risk Management-20 (HCR-20)—were related to indices of interrater agreement reported in the research literature. Nine doctoral students with experience scoring the PCL-R and HCR-20 showed a high level of agreement with respect to the amount of subjectivity required to score items. Decreased subjectivity was associated with increased rater agreement for both tools. For the PCL-R, items on aggregate measures that have the strongest effects in predictive validity research (i.e., Factor 2, Facet 4) were rated as the least subjective. For the HCR-20, items on the Historical scale were rated as the least subjective. Implications for measure development and refinement are discussed.
ArcherR. P.Buffington-VollumJ. K.StrednyR. V.HandelR. W. (2006). A survey of psychological test use patterns among forensic psychologists. Journal of Personality Assessment, 87, 84-94.
2.
BelfrageH. (1998). Implementing the HCR-20 scheme for risk assessment in a forensic psychiatric hospital: Integrating research and clinical practice. Journal of Forensic Psychiatry, 9, 328-338.
3.
BoccacciniM. T.TurnerD. B.MurrieD. C. (2008). Do some evaluators report consistently higher or lower PCL-R scores than others? Findings from a statewide sample of sexually violent predator evaluations. Psychology, Public Policy, and Law, 14, 262-283.
4.
BrodskyS. L. (1990). Professional ethics and professional morality in the assessment of competence for execution. Law and Human Behavior, 14, 91-97.
5.
CicchettiD. V.SparrowS. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127-137.
6.
ComptonW. M.CottlerL. B.DorseyK. B.SpitznagelE. L.MagerD. E. (1996). Structured and semistructured assessment of ICD-10 substance dependence disorders: CIDI-SAM vs. SCAN. International Journal of Methods in Psychiatric Research, 6, 285-293.
7.
CronbachL. J. (1970). Essentials of psychological testing (3rd ed.). New York: Harper & Row.
8.
DouglasK. S. (2008, July). Principles and research guiding the revision of the HCR-20. In DvoskinJ. (Chair), Clarifying the principles of the structured professional judgment approach to violence risk assessment: Current status and future directions. Symposium presented at the annual conference of the International Association of Forensic Mental Health Services, Vienna, Austria.
9.
DouglasK. S.HartS. D.WebsterC. D.BelfrageH.EavesD. (2008). Historical Clinical Risk Management (Version 3, Draft Version 1.0): Assessing Risk for Violence. Burnaby, British Columbia, Canada: Mental Health, Law, and Policy Institute, Simon Fraser University.
10.
DouglasK. S.OgloffJ. R. P.HartS. D. (2003). Evaluation of a model of violence risk assessment among forensic psychiatric patients. Psychiatric Services, 54, 1372-1379.
11.
DouglasK. S.ReevesK. (2010). The HCR-20 violence risk assessment scheme: Overview and re-view of the research. In OttoR.DouglasK. S. (Eds.), Handbook of violence risk assessment (pp. 147-185). Oxford, England: Routledge/Taylor & Francis.
12.
DouglasK. S.SkeemJ. L. (2005). Violence risk assessment: Getting specific about being dynamic. Psychology, Public Policy, and Law, 11, 347-383.
13.
FunderD. C.ColvinC. R. (1988). Friends and strangers: Acquaintanceship, agreement, and the accuracy of personality judgment. Journal of Personality and Social Psychology, 55, 149-158.
14.
FunderD. C.DobrothK. M. (1987). Differences between traits: Properties associated with interjudge agreement. Journal of Personality and Social Psychology, 52, 409-418.
15.
GrannM. (2000). The PCL-R and gender. European Journal of Psychological Assessment, 16, 147-149.
16.
GuyL. S. (2008). Performance indicators of the structured professional judgment approach for assessing risk for violence to others: A meta-analytic survey (Unpublished doctoral dissertation). Simon Fraser University, Burnaby, British Columbia, Canada.
17.
GuyL. S.EdensJ. F.AnthonyC.DouglasK. S. (2005). Does psychopathy predict institutional misconduct among adults? A meta-analytic investigation. Journal of Consulting and Clinical Psychology, 73, 1056-1064.
18.
HansonR. K.Morton-BourgonK. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 21, 1-21.
HildebrandM.de RuiterC.de VogelV.van der WolfP. (2002). Reliability and factor structure of the Dutch language version of Hare’s Psychopathy Checklist–Revised. International Journal of Forensic Mental Health, 1, 139-154.
21.
HowardK. I. (1963). Ratings of projective test protocols as a function of degree of inference. Educational and Psychological Measurement, 23, 267-275.
22.
JohnO. P.RobinsR. W. (1993). Determinants of interjudge agreement on personality traits: The big five domains, observability, evaluativeness, and the unique perspectives of the self. Journal of Personality, 61, 521-551.
23.
LaurellJ.DadermanA. M. (2007). Psychopathy (PCL-R) in a forensic psychiatric sample of homicide offenders: Some reliability issues. International Journal of Law and Psychiatry, 30, 127-135.
24.
LeisticoA. R.SalekinR. T.DeCosterJ.RogersR. (2008). A large-scale meta-analysis relating the Hare measures of psychopathy to antisocial conduct. Law and Human Behavior, 32, 28-45.
25.
LidzC. W.MulveyE. P.GardnerW. (1993). The accuracy of predictions of violence to others. Journal of the American Medical Association, 269, 1007-1111.
26.
LloydC. D.ClarkH. J.ForthA. H. (in press). Psychopathy, expert testimony and indeterminate sentences: Exploring the relationship between Psychopathy Checklist–Revised testimony and trial outcomes in Canada. Legal and Criminological Psychology. doi:10.1348/135532509X46843210.1348/135532509X468432
27.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
28.
McNielD. E.BinderR. L. (1995). Correlates of accuracy in the assessment of psychiatric inpatients’ risk of violence. American Journal of Psychiatry, 152, 901-906.
29.
McNielD. E.SandbergD. A.BinderR. L. (1998). The relationship between confidence and accuracy in the assessment of psychiatric inpatients’ risk of violence. American Journal of Psychiatry, 152, 901-906.
30.
MillsJ. F.KronerD. G.HemmatiT. (2003). Predicting violent behavior through a static-stable variable lens. Journal of Interpersonal Violence, 18, 891-904.
31.
MonahanJ. (1981). Predicting violent behavior: An assessment of clinical techniques. Beverley Hills, CA: SAGE.
32.
MurrieD. C.BoccacciniM. T.TurnerD.MeeksM.WoodsC.TusseyC. (2009). Rater (dis)agreement on risk assessment measures in sexually violent predator proceedings: Evidence of adversarial allegiance in forensic evaluation?Psychology, Public Policy, and Law, 15, 19-53.
33.
NearyA. M. (1990). DSM-III and Psychopathy Checklist assessment of antisocial personality disorder in black and white female felons. Dissertation Abstracts International, 51, 7B, 3605.
34.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
35.
SalekinR. T.RogersR.SewellK. W. (1997). Construct validity of psychopathy in a female offender sample: A multitrait–multimethod evaluation. Journal of Abnormal Psychology, 106, 576-585.
36.
TolmanA. O.MullendoreK. B. (2003). Risk evaluations for the courts: Is service quality a function of specialization?Professional Psychology: Research and Practice, 34, 225-232.
37.
TubbV. A. (2001). The factor structure and psychometric properties of Psychopathy Checklist Revised: Data from an Hispanic Federal inmate population. Dissertation Abstracts International, 62, 5426.
38.
WaltersG. D. (2003). Predicting institutional adjustment and recidivism with the Psychopathy Checklist factor scores: A meta-analysis. Law and Human Behavior, 27, 541-558.
39.
WaltersG. D.KnightR. A.GrannM.DahleK. P. (2008). Incremental validity of the Psychopathy Checklist facet scores: Predicting release outcome in six samples. Journal of Abnormal Psychology, 117, 396-405.
40.
WarrenJ. I.BurnetteM. L.SouthS. C.ChauhanP.BaleR.FriendR.Van PattenI. (2003). Psychopathy in women: Structural modeling and comorbidity. International Journal of Law and Psychiatry, 26, 223-242.
41.
WebsterC. D.DouglasK. S.EavesD.HartS. (1997). HCR-20 (version 2): Historical/Clinical/Risk–20. Burnaby, British Columbia, Canada: Mental Health Law and Policy Institute, Simon Fraser University.
