Reliability of Measurement and Power of Significance Tests Based on Differences

Abstract

The power of significance tests based on differ ence scores is indirectly influenced by the reliability of the measures from which differences are obtained. Reliability depends on the relative magnitude of true score and error score variance, but statistical power is a function of the absolute magnitude of these components. Explicit power calculations reaffirm the paradox put forward by Overall & Woodward (1975, 1976)—that significance tests of differences can be powerful even if the reliability of the difference scores is 0. This anomaly arises because power is a function of observed score variance but is not a function of reliability unless either true score variance or error score variance is constant. Provided that sample size, significance level, directionality, and the alternative hypothesis associated with a significance test remain the same, power always increases when population variance decreases, independently of reliability.

Keywords

Index terms: difference scores error of measurement power,significance tests t test test reliability true scores.

Get full access to this article

View all access options for this article.

References

Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press .

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (3rd ed.). Hillsdale NJ: Erlbaum.

Cohen, J. (1990). Things I have learned (so far. American Psychologist, 45, 1304-1312.

Feller, W. (1966). An introduction to probability theory and its applications (Vol. II). New York: Wiley.

Fleiss, J.J. (1976). Comment on Overall & Woodward's asserted paradox concerning the measurement of change. Psychological Bulletin , 83, 774-775.

Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.

Howell, D.C. (1987). Statistical methods for psychology (2nd ed.). Boston: Duxbury Press.

Humphreys, L.G. , & Drasgow, F. (1989a). Some comments on the relation between reliability and statistical power. Applied Psychological Measurement, 13, 429-431.

Humphreys, L.G. , & Drasgow, F. (1989b). Paradoxes, contradictions, and illusions. Applied Psychological Measurement, 13, 429-431.

10.

Lord, F.M. , & Novick, M.R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.

11.

Nicewander, W.A. , & Price, J.M. (1983). Reliability of measurement and the power of significance tests. Psychological Bulletin, 94, 524-533.

12.

Overall, J.E. (1989a). Contradictions can never a paradox resolve. Applied Psychological Measurement, 13, 426-428.

13.

Overall, J.E. (1989b). Distinguishing between measurements and dependent variables. Applied Psychological Measurement, 13, 432-433.

14.

Overall, J.E. , & Ashby, B. (1991). Baseline corrections in experimental and quasi-experimental clinical trials. Neuropsychopharmacology, 4, 273-281.

15.

Overall, J.E. , & Woodward, J.A. (1975). Unreliability of difference scores: A paradox for the measurement of change. Psychological Bulletin, 82, 85-86.

16.

Overall, J.E. , & Woodward, J.A. (1976). Reassertion of the paradoxical power of tests of significance based on unreliable difference scores. Psychological Bulletin, 83, 776-777.

17.

Welkowitz, J. , Ewen, R.B. , & Cohen, J. (1976). Introductory statistics for the behavioral sciences (2nd ed.). New York: Academic Press.

18.

Williams, R.H. , & Zimmerman, D.W. (1989). Statistical power analysis and reliability of measurement. Journal of General Psychology , 116, 359-369.

19.

Zimmerman, D.W. , & Williams, R.H. (1986). Note on the reliability of experimental measures and the power of significance tests. Psychological Bulletin, 100, 123-124.