Sage Journals: Discover world-class research

Abstract

This article studies the ordinal reliability of (total) test scores. This study is based on a classical-type linear model of observed score (X), true score (T), and random error (E). Based on the idea of Kendall's tau-a coefficient, a measure of ordinal reliability for small-examinee populations is developed. This measure is extended to large (theoretically, infinite) examinee populations. This extended measure is identified as the population version of Kendall's tau-a. Appropriateness of the traditional product moment correlation-based measures,. _TX and .XX0, is reviewed. It is observed that. ^TX and .XX0 are appropriate within the family of normal distributions of (T, E) but are inappropriate in the family of nonnormal distributions. A simulated data example is considered to show that the traditional product moment correlation-based measures may be misleading. Furthermore, practical procedures are suggested for estimating ordinal reliability. Real test score data are used for demonstration. Index terms: ordinal measures, linear model in classical test theory, test score reliability, Kendall's tau-a

Get full access to this article

View all access options for this article.

References

Barton, D. E. , & Dennis, K. E. R. (1952). The conditions under which Gram-Charlier and Edgeworth curves are positive definite and unimodal. Biometrika, 39, 425-427.

Cliff, N. (1989). Ordinal consistency and ordinal true scores. Psychometrika, 54, 75-91.

Cliff, N. (1993). Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114, 494-509.

Cliff, N. , & Keats, J. A. (2003). Ordinal measurement in the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.

Elderton, W. P. (1953). Frequency curves and correlation. Cambridge,UK:Cambridge University Press.

Gibbons, J. D. (1971). Nonparametric statistical inference. New York: John Wiley.

Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.

Gulliksen, H. (1967). Theory of mental tests. New York: John Wiley.

Guttman, L. (1950). The basis of scalogram analysis. In S. A. Stouffer (Ed.), Measurement and prediction. Princeton, NJ: Princeton University Press.

10.

Hoeffding, W. (1947). On the distribution of the rank correlation coefficient t when the variables are not independent. Biometrika, 34, 183-196.

11.

Keats, J. A. (1951). A statistical theory of mental test scores. Melbourne: Australian Council for Educational Research.

12.

Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30, 81-93.

13.

Kendall, M. G. (1948). Rank correlation methods. London: Charles Griffins.

14.

Kruskal, W. H. (1958). Ordinal measures of association. Journal of the American Statistical Association, 53, 814-861.

15.

Loevinger, J. A. (1947). A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 61(4, Whole No. 285).

16.

Loevinger, J. A. (1948). A technique of homogeneous test evaluation compared with some aspects of “scale analysis.” Psychological Bulletin, 45, 507-529.

17.

Long, J. D. , & Cliff, N. (1997). Confidence intervals for Kendall's tau. British Journal of Mathematical and Statistical Psychology, 50, 31-41.

18.

Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

19.

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

20.

Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.

21.

Mokken, R. J. (1971). A theory and procedure of scale analysis. Hawthorne, NY: Mouton & Co.

22.

Mokken, R. J. , & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6, 417-430.

23.

Mokken, R. J. , Lewis, C. , & Sytsma, K. (1986). Rejoinder to “The Mokken Scale: A critical discussion”. Applied Psychological Measurement, 10, 229-285.

24.

Schulman, R. S. (1976). Correlation and prediction in ordinal test theory. Psychometrika, 41, 329-340.

25.

Schulman, R. S. , & Haden, R. L. (1975). A test theory model for ordinal measurements. Psychometrika, 40, 455-472.

26.

Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.

27.

Wilcox, R. R. (1990). Comparing the means of two independent groups. Biometrical Journal, 32, 771-780.

Reliability of Total Test Scores When Considered as Ordinal Measurements

Abstract

Get full access to this article

References