“Meet the New Boss. Same as the Old Boss:” A Commentary on Williams,Wormith,Bonta,and Sitarenios (2017)

Abstract

This commentary on Williams et al. (2017) focuses on an additional and equally important issue not addressed in their critique: inter-rater reliability --particularly reliability in field settings. A growing body of evidence indicates that risk assessment instruments administered in applied (and especially adversarial) contexts may be considerably less stable across examiners than what typically is reported in well-controlled, peer-reviewed journal publications. Because reliability constrains validity, effect sizes from such published research may overestimate predictive validity in real-world contexts. Although validity evidence is important, field reliability remains “the boss” when considering how well an assessment procedure will perform in applied settings.