Criterion-Referenced Reliability Estimated by Anova

Abstract

For criterion-referenced (C-R) tests, variance was defined as a measure of the variability of scores from the criterion of minimal, acceptable performance. By analogy to norm-referenced test theory C-R reliability was defined as the proportion of observed variance attributable to true variance. The definition of C-R reliability was then extended to the mean of a number of parallel measures. A typical test situation was described as a randomized, complete block design. For this design a score model was formulated which incorporated the criterion of minimal, acceptable performance. Expected values for the mean squares, error and person, were derived and shown to be equal to observed and error variance for C-R tests. C-R reliability was then redefined in terms of the expected values, error and person. Using the methods developed, C-R was calculated for a hypothetical example.