Estimation of Reliability Coefficients Using the Test Information Function and Its Modifications

Abstract

The reliability coefficient and the standard error of measurement in classical test theory are not properties of a specific test, but are attributed to both a specific test and a specific trait distribution. In latent trait mod els, or item response theory, the test information func tion (TIF) provides more precise local measures of accuracy in trait estimation than are available from the reliability coefficient. The reliability coefficient is still widely used, however, and is popular because of its simplicity. Thus, it is worthwhile to relate it to the TIF. In this paper, the reliability coefficient is predicted from the TIF, or two modified TIF formulas, and a spe cific trait distribution. Examples demonstrate the vari ability of the reliability coefficient across different trait distributions, and the results are compared with empiri cal reliability coefficients. Practical suggestions are given as to how to make better use of the reliability coefficient. Index terms: adaptive testing, bias, clas sical test theory, item information function, latent trait models, maximum likelihood estimation, reliability co efficieno, standard error of measurement, test informa tion function, trait estimation.

Get full access to this article

View all access options for this article.

References

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick, Statistical theories of mental test scores (pp. 397-479). Reading MA: Addison-Wesley.

Elderton, W.P. , & Johnson, N.L. (1969). Systems of frequency curves. London: Cambridge University Press.

Lawley, D.N. (1943). On problems connected with item selection and test construction. Proceedings of the Royal Society of Edinburgh, 61A, 273-287.

Lord, F.M. (1952). A theory of test scores. Psychometric Monograph, No. 7.

Lord, F.M. (1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-fonns reliability. Psychometrika, 48, 233-245.

Lord, F.M. (1984, April). Technical problems arising in parameter estimation. Paper presented at the annual meeting of the American Educational Research Association, New Orleans .

Lord, F.M. , & Novick, M.R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.

Samejima, F. (1969). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph, No. 17.

Samejima, F. (1972). A general model for free-response data. Psychometrika Monograph, No. 18.

10.

Samejima, F. (1977a). Effects of individual optimization in setting boundaries of dichotomous items on accuracy of estimation. Applied Psychological Measurement, 1, 77-94.

11.

Samejima, F. (1977b). A use of the information function in tailored testing. Applied Psychological Measurement, 1, 233-247.

12.

Samejima, F. (1979). Convergence of the conditional distribiatton of the maximum likelihood estimate, given latent trait, to the asymptotic normality: Observations made through the constcant information model (Office of Naval Research Rep. No. 79-3). Knoxville: Department of Psychology, University of Tennessee.

13.

Samejima, F. (1981). Efficient methods of estimating the operating characteristics of item response categories and challenge to a new modelfor the multiple-choice item (Office of Naval Research Final Report of N00014-77-C-0360) . Knoxville: Department of Psychology , University of Tennessee.

14.

Samejima, F. (1987). Bias function ofthe maximum likelihood estimate of ability for discrete item responses (Office of Naval Research Report No. 87-1). Knoxville: Department of Psychology, University of Tennessee.

15.

Samejima, F. (1990). Modifications of the test information function (Office of Naval Research Report No. 90-1). Knoxville: Department of Psychology, University of Tennessee.

16.

Samejima, F. (1993a). An approximation for the bias function of the maximum likelihood estimate of a latent variable for the general case where the item responses are discrete. Psychonietrika, 58, 119-138.

17.

Samejima, F. (1993b). The bias function of the maximum likelihood estimate of ability for the dichotomous response level. Psychometrika , 58, 195-209.

18.

Samejima, F. (1994). Roles of Fisher type information in latent trait models. In H. Bozdogan (Ed.), Proceedings of the first US/Japan conference on the frontiers of statistical modeling: An informational approach (pp. 347-378). The Netherlands: Kluwer.