Examinee Characteristics and Score Reliability: An Empirical Investigation

Abstract

The literature on measurement reliability shows the general consensus that examinee group heterogeneity with regard to the trait being measured affects the score reliability. Potentially, the performance level of the examinee group may also affect score reliability because of its potential effect on the relative magnitude of error variance. This article empirically examines the effects of these two examinee sample characteristics on score reliability of optimal-performance measurement. Two large extant data sets (criterionand norm-referenced, respectively) were used in the investigation. The results suggest that both performance variability and group performance level affect score reliability, and measurement error tends to be smaller for high-performance groups than for low-performance groups.

Get full access to this article

View all access options for this article.

References

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Holt, Rinehart and Winston.

Gulliksen, H. (1987). Theory of mental tests. Hillsdale, NJ: Lawrence Erlbaum.

Holzinger, K. J. (1921). On the assumption that errors of estimate are equal in narrow and wide ranges. Journal of Educational Research, 4, 237-239.

Magnusson, D. (1967). Test theory. Boston: Addison-Wesley.

Thompson, B. (Ed.). (2002). Score reliability: Contemporary thinking on reliability issues. Newbury Park, CA: Sage.

Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.

Vacha-Haase, T., Henson, R. K., & Caruso, J. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.

Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.