The Relationship between Two Methods of Evaluating an Examinee's Difference Scores

Abstract

The “nonoverlapping intervals” and “reliable difference” approaches for assessing difference scores are compared and shown to be consistent alternatives when the proper z is used to construct confidence intervals. Formulas for computing the probabilities of correct interpretation (power), overinterpretation, misinterpretation, and underinterpretation for four popular confidence interval approaches and the reliable difference approach are presented. The probability formulas show that the intuitive inference concerning the statistical significance level of nonoverlapping intervals is incorrect. The limitations of the nonoverlapping intervals approach in applied situations are discussed. It appears that in most situations the reliable difference is the easiest to apply.

Get full access to this article

View all access options for this article.

References

Allen, M. J. , & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole.

Anastasi, A. (1988). Psychological testing (6th ed). New York: Macmillan.

Cahan, S. (1986). Significance testing of subtest score differences: The rules of the game. Journal of Psychoeducational Assessment, 4, 273-280.

Cahan, S. , & Cohen, N. (1988). Significance testing of subtest score differences: The case of nonsignificant results. Journal of Psychoeducational Assessment, 6, 107-117.

Cronbach, L. J. , & Furby, L. (1970). How we should measure “change"-or should we? Psychological Bulletin, 74, 68-80.

Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement. Psychological Bulletin, 86, 335-337.

Educational Testing Service (1956). Sequential Tests of Educational Progress. Princeton, NJ: Author.

Feldt, L. S. (1967). A note on the use of confidence bands to evaluate the reliability of a difference between two scores. American Educational Research Journal, 4, 139-145.

Feldt, L. S. , & Brennan, R. L. (1989). Reliability. In R. H. Linn (Ed.), Educational measurement (3rd ed.). American Counsel of Education, NY: Macmillan.

10.

Glutting, J. J. , McDermott, P. A. , & Konold, T. R. (1997). Ontology, structure, and diagnostic benefits of a normative subtest taxonomy from the WAIS-III standardization sample. In D. P. Flanagan , J. L. Genshaft , & P. L. Harrision (Eds.), Contemporary intellectual assessment: Theories, tests, and issues. New York: Guilford.

11.

Glutting, J. J. , McDermott, P. A. , & Stanley, J. C. (1987). Resolving differences among methods of establishing confidence limits of test scores. Educational and Psychological Measurement, 47, 607-614.

12.

Gulliksen, H. (1950). Theory of mental tests. New York: John Wiley.

13.

Lord, F. M. (1958). Further problems in the measurement of growth. Educational and Psychological Measurement, 18, 437-451.

14.

Mehrens, W. A. , & Lehmann, I. J. (1991). Measurement and evaluation in education and psychology (4th ed.). New York: Harcourt Brace.

15.

Nunnally, J. C. , & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

16.

Payne, R. W. , & Jones, H. G. (1957). Statistics for the investigation of individual cases. Journal of Clinical Psychology, 18, 115-121.

17.

Silverstein, A. B. (1981). Reliability and abnormality of test score differences. Journal of Clinical Psychology, 37, 392-394.

18.

Silverstein, A. B. (1984a). New formulas for evaluating the abnormality of test score differences. Journal of Psychoeducational Assessment, 2, 79-82.

19.

Silverstein, A. B. (1984b). Pattern analysis: The question of abnormality. Journal of Consulting and Clinical Psychology, 52, 936-939.

20.

Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356-442). Wash-ington DC: American Council on Education.

21.

Thorndike, R. L. , & Hagen, E. P. (1994). Cognitive Abilities Test (CogAT). Itasca, IL: Riverside.

22.

Wechsler, D. (1997). Manual for the WAIS-III. San Antonio: The Psychological Corporation.

23.

Zimmerman, D. W. , & Williams, R. H. (1982). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154.