Evaluation of the Efficiency of Item Calibration

Abstract

This study compared several IRT calibration proce dures to determine which procedure, if any, consis tently produced the most accurate item parameter esti mates. A new criterion of calibration efficiency was used for evaluating the calibration procedures; this cri terion considers the joint effects of individual item pa rameter errors as they relate to the accuracy of θ esti mation. Four methods of item calibration were evaluated: (1) heuristic estimates obtained from trans formations of traditional item statistics; (2) ANCILLES, a program that first fits the c parameter and then trans forms traditional item statistics to IRT a and b parame ters ; (3) LOGIST, a joint maximum likelihood proce dure ; and (4) ASCAL, a modification of LOGIST'S algorithm which applies Bayesian priors to the abilities and item parameters. These were compared with each other and with a constant item parameter baseline con dition. ASCAL and LOGIST produced estimates of essen tially equivalent accuracy, although ASCAL's estimates of the c parameters were slightly superior. The heuris tic estimates and those from ANCILLES were generally poor in comparison, particularly for smaller sample sizes. Index terms: Calibration efficiency, Item calibration, Item parameter estimation, Item response theory, Latent trait models.

Get full access to this article

View all access options for this article.

References

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick, Statistical theories of mental test scores (pp. 397-479). Reading MA: Addison-Wesley.

Bock, R.D. , & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika , 46, 443-459.

Croll, P.R. , & Urry, V.W. (1978). ANCILLES: A program for estimation of the item parameters of normal ogive and logistic mental test models—Version 78.5 . Washington DC: US Civil Service Commission, Personnel Research and Development Center.

Gugel, J.F. , Schmidt, F.L. , & Urry, V.W. (1976). Effectiveness of the ancillary estimation procedure . In C. L. Clark (Ed.), Proceedings of the First Conference on Computerized Adaptive Testing (PS-75-6) . Washington DC: US Civil Service Commission, Personnel Research and Development Center.

Jensema, C.J. (1976). A simple technique for estimating latent trait mental test parameters. Educational and Psychological Measurement , 36, 705-715.

Lord, F.M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247-264.

Lord, F.M. (1975). Evaluation with artificial data of a procedure for estimating ability and item characteristic curve parameters (Research Bulletin 75-33). Princeton NJ: Educational Testing Service.

Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale NJ: Erlbaum.

Lord, F.M. , & Novick, M.R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.

10.

Ree, M.J. (1979). Estimating item characteristic curves. Applied Psychological Measurement, 3, 371-385.

11.

Schmidt, F.L. , & Gugel, J.F. (1976). The Urry item parameter estimation technique: How effective? In W. A. Gorham (Chair), Computers and testing: Steps toward the inevitable conquest (PS-76-1). Washington DC: US Civil Service Commission, Personnel Research and Development Center.

12.

Swaminathan, H. , & Gifford, J. (1983). Estimation of parameters in the three-parameter latent trait model. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 13-30). New York: Academic Press.

13.

Swaminathan, H. , & Gifford, J. (1985). Bayesian estimation in the two-parameter logistic model. Psychometrika, 50, 349-364.

14.

Thissen, D. , & Wainer, H. (1982). Some standard errors in item response theory. Psychometrika, 47, 397-412.

15.

Urry, V. (1977). OGIVIA: Item parameter estimation program with normal ogive and logistic three-parameter model options. Washington DC: US Civil Service Commission, Personnel Research and Development Center.

16.

Urry, V. (1978). ANCILLES: Item parameter estimation program with normal ogive and logistic three-parameter model options. Washington DC: US Civil Service Commission, Personnel Research and Development Center.

17.

Vale, C.D. , & Gialluca, K.A. (1985). ASCAL: A microcomputer program for estimating logistic IRT item parameters (ONR-85-4). St. Paul: Assessment Systems Corporation.

18.

Vale, C.D. , Maurelli, V.A. , Gialluca, K.A. , Weiss, D.J. , & Ree, M.J. (1981). Methods for linking item parameters (AFHRL-TR-81-10) . Brooks Air Force Base TX: U.S. Air Force Human Resources Laboratory, Manpower and Personnel Division.

19.

Vale, C.D. , & Weiss, D.J. (1975). A simulation study of stradaptive ability testing (Research Report 75-6). Minneapolis: University of Minnesota, Department of Psychology, Psychometric Methods Program .

20.

Wingersky, M.S. , Barton, M.A. , & Lord, F.M. (1982). LOGIST user's guide. Princeton NJ: Educational Testing Service.

21.

Wood, R.L. , Wingersky, M.S. , & Lord, F.M. (1976). LOGIST: A computer program for estimating examinee ability and item characteristic curve parameters (Research Memorandum RM-76-6) . Princeton NJ: Educational Testing Service.

22.

Wright, B.D. (1977). Solving measurement problems with the Rasch model . Journal of Educational Measurement, 14, 97-116.