The early stage of computerized adaptive testing (CAT) refers to the phase of the trait estimation during the administration of only a few items. This phase can be characterized by bias and instability of estimation. In this study, an item selection criterion is introduced in an attempt to lessen this instability: the D-optimality criterion. A polytomous unconstrained CAT simulation is carried out to evaluate this criterion's performance under different test premises. The simulation shows that the extent of early stage instability depends primarily on the quality of the item pool information and its size and secondarily on the item selection criteria. The efficiency of the D-optimality criterion is similar to the efficiency of other known item selection criteria. Yet, it often yields estimates that, at the beginning of CAT, display a more robust performance against instability.
Berger, M.P.F., & Mathijssen, E. (1997). Optimal test designs for polytomously scored items. British Journal of Mathematical and Statistical Psychology, 50, 127-141.
4.
Berger, M.P.F., & van der Linden, W.J. (1995). Ontwerpen van tests met verschillende optimaliteitscriteria [Designing educational tests with different criteria of optimality]. Tijdschrift voor Onderwijsresearch , 20, 79-92.
5.
Berger, M.P.F., & Veerkamp, W.J.J. (1996). A review of selection methods for optimal tests design. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (Vol. 3, pp. 437-455). Norwood, NJ: Ablex.
6.
Bock, R.D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
7.
Chang, H., & van der Linden, W.J. (2003). Optimal stratification of item pools in a-stratified computerized adaptive testing. Applied Psychological Measurement, 27, 262-274.
8.
Chang, H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
Chen, S.Y., & Ankenmann, R.D. (2004). Effects of practical constraints on item selection rules at the early stage of computerized adaptive testing . Journal ofEducationalMeasurement, 41, 149-174.
11.
Chen, S.Y., Ankenmann, R.D., & Chang, H.H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24, 241-255.
12.
Cheng, P.E., & Liou, M. (2000). Estimation of trait level in computerized adaptive testing. Applied Psychological Measurement, 24, 257-265.
13.
Cheng, P.E., & Liou, M. (2003). Computerized adaptive testing using the nearest-neighbors criterion. Applied Psychological Measurement, 27,204-216.
14.
Davis, L.L. (2004). Item exposure strategies with the general partial credit model. Applied Psychological Measurement, 28, 165-185.
15.
De Ayala, R.J. (1993). An introduction to polytomous item response theory models. Measurement and Evaluation in Counseling and Development , 25, 172-189.
16.
Dodd, B.G., De Ayala, R.J., & Koch, W.R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19, 5-22.
17.
Eggen, T.J.H.M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement , 23, 249-261.
18.
Fan, M., & Hsu, Y. (1996, April). Utility of Fisher information, global information and different starting abilities in mini CAT. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
Heinen, A.G.J.J. (1996). Latent class and discrete latent trait models: Similarities and differences. London: Sage.
21.
Lima Passos, V. (2005). Optimal test and sampling designs for polytomous item response theory models. Unpublished doctoral dissertation, Maastricht University, the Netherlands.
22.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
23.
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American StatisticalAssociation, 70, 351-356.
Samejima, F. (1977). A use of the information function in tailored testing. Applied Psychological Measurement,1, 233-247.
26.
Thissen, D., & Steinberg, L. (1986). Taxonomy of item response models. Psychometrika, 51, 567-577.
27.
Urry, V.W. (1977). Tailored testing: A successful application of latent trait theory. Journal of Educational Measurement, 14, 181-196.
28.
van der Linden, W.J. (1998). Bayesian item-selection criteria for adaptive testing. Psychometrika, 62, 201-216.
29.
van der Linden, W.J. (2000). Constrained adaptive testing with shadow tests . In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 27-52). Dordrecht, the Netherlands: Kluwer Academic.
30.
van der Linden, W.J., & Pashley, P.J. (2000). Item selection and θ estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 1-25). Dordrecht , the Netherlands: Kluwer Academic.
31.
van Rijn, P.W., Eggen, T.J.H.M., Hemker, B.T., & Sanders, P.F. (2002). Evaluation of selection procedures for computerized adaptive testing with polytomous items. Applied Psychological Measurement, 26, 393-411.
32.
Veerkamp, W.J.J., & Berger, M.P.F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203-226.
33.
Veldkamp, B.P. (2001). Principles and methods of constrained test assembly . Unpublished doctoral dissertation, University of Twente , the Netherlands.
34.
Veldkamp, B.P. (2003). Item selection in polytomous CAT. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, & J. J. Meulman (Eds.), New developments in psychometrics (pp. 207-214). Tokyo, Japan: Springer-Verlag.
35.
Wang, T., Hanson, B.A., & Che-Ming, A.L. (1999). Reducing bias in CAT trait estimation: A comparison of approaches. Applied Psychological Measurement, 23, 263-278.
36.
Weissman, A. (2003, July). Information theoretic approaches to item selection. Paper presented at the International Meeting of the Psychometric Society (IMPS), ChiaLaguna , Italy.
37.
Xing, D., & Hambleton, R.K. (2004). Impact of test design, item quality and item bank size on the psychometric properties of computer-based credentialing Examinations. Educational and Psychological Measurement, 64, 5-21.
38.
Yi, Q., & Chang, H.H. (2003). a-Stratified CAT design with content blocking . British Journal of Mathematical and Statistical Psychology , 56, 359-378.
