Further Comments on Reliability and Power of Significance Tests

Abstract

The controversy about the relationship between reliability and the power of significance tests exists because statisticians obtain numerical solutions by varying independently the parameters of the power of statistical tests. In contrast, researchers have empirical limitations placed on them in varying the same parameters. Reliability and power can legiti mately be decoupled by selection of the population from which to sample (Zimmerman & Williams, 1986), but this is an undependable way to increase power (Humphreys, 1991). Reducing population var iance by selection of the sample can be considered a special case of (and a crude approximation to) the analysis of covariance, which is also a more effective way of controlling individual differences in true scores than the use of difference scores. Both the regressed differences and the raw differences are less reliable within treatments than their components, but can have more power in statistical tests. As the reliability of derived scores increases, however, power increases.