Formulae for Estimating Rater Reliability from the Significance of Treatment Effects

Abstract

The ability to discriminate true treatment effects in clinical research depends on the reliability of measurements. Rating scales are often used to assess treatment effects, but raters differ in their individual reliabilities. Rather than considering a rating instrument to have inherent reliability, it is important to focus on the reliability of the instrument in the hands of particular raters. Formulae for estimating individual rater reliabilities from analysis of treatment effects are presented and evaluated. However, results indicate that large sample sizes, large true treatment effects, and large differences in the actual reliabilities of the raters are required for the approach to prove useful.