Unreliability of Difference Scores: A Clarification of the Issues

Abstract

This paper is an attempt to clarify several recent issues that have been raised concerning the unreliability of difference scores. Articles by Overall and Woodward (1975, 1976) and Fleiss (1976) are considered. The conclusions of these articles are shown to be incomplete and to some extent misleading. We begin by noting that within the context of a repeated-measures design, when the reliability of difference scores is other than zero, there exists an interaction of subject by treatment. This interaction will imply the existence of at least one additional systematic variable which contributes to the experimental results. We argue that without specific knowledge about the mean and variance of this systematic variable the interpretation of mean changes across pretest and posttest is ambiguous. With complete knowledge of all variables affecting the experimental situation, the power of the t test on difference scores is directly related to both the individual reliabilities of the separate pretest and posttests and the reliability of the difference scores as long as interaction of subject by treatment is constant across experiments. If, on the other hand, that interaction is not constant across experiments, then the power of the test on difference scores is inversely related to the reliability of the difference scores as long as the individual reliabilities of the pretests and posttests are constant across experiments. Since it is much more reasonable to make the assumption that the interaction is constant across experiments rather than individual test reliabilities, it is more reasonable to conclude that, contrary to Overall and Woodward, power of the t test for difference scores is directly related to the reliability of the difference scores.