This study examined the performance of the maximum Fisher’s information, the maximum posterior weighted information, and the minimum expected posterior variance methods for selecting items in a computerized adaptive testing system when the items were grouped in testlets. A simulation study compared the efficiency of ability estimation among the item selection techniques under varying conditions of local-item dependency when the response model was either the three-parameter-logistic item response theory or the three-parameter-logistic testlet response theory. The item selection techniques performed similarly within any particular condition, the practical implications of which are discussed within the article.
Birnbaum, A. ( 1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (chaps. 17-20). Reading, MA: Addison-Wesley.
2.
Bock, R.D. ( 1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika , 37, 29-51.
3.
Boyd, A.M. ( 2003). Strategies for controlling testlet exposure rates in computerized adaptive testing systems. Unpublished doctoral dissertation, University of Texas at Austin. (UMI No. 3110732)
4.
Bradlow, E.T., Wainer, H., & Wang, X. ( 1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
5.
Chang, H.-H., & Ying, Z. ( 1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
6.
Chen, S., Hou, L., & Dodd, B.G. ( 1998). A comparison of maximum likelihood estimation and expected a posteriori estimation on CAT using the partial credit model. Educational and Psychological Measurement, 53, 61-77.
7.
Chen, S.-Y., Ankenmann, R.D., & Chang, H.-H. ( 2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement , 24, 241-255.
Cohen, J. ( 1988). Statistical power analysis of the behavioral sciences (2nd ed.). New York: Academic Press.
10.
Davis, L.L., & Dodd, B.G. ( 2001). An examination of testlet scoring and item exposure constraints in the verbal reasoning section of the MCAT. MCAT Monograph Series.
11.
Hauck, W.W., & Anderson, S. ( 1984). A survey regarding the reporting of simulation studies . American Statistician, 38, 214-216.
12.
Kingsbury, G.G., & Zara, A.R. ( 1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375.
Lima Passos, V., Berger, M.P.F., & Tan, F.E. ( 2007). Test design optimization in CAT early stage with the Nominal Response Model. Applied Psychological Measurement , 31, 213-232.
15.
Lima Passos, V., Berger, M.P.F., & Tan, F.E. ( 2008). The D-optimality item selection criterion in the early stage of CAT: A study with the graded response model. Journal of Educational and Behavioral Statistics, 33, 88-110.
16.
Masters, G.N. ( 1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
17.
Muraki, E. ( 1993). Information functions of the generalized partial credit model. Applied Psychological Measurement, 17, 351-363.
18.
Olejnik, S., & Algina, J. ( 2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.
19.
Penfield, R.D. ( 2006). Applying Bayesian item selection approaches to adaptive tests using polytomous items. Applied Measurement in Education , 19, 1-20.
20.
Samejima, F. ( 1969). Estimation of latent ability using response patterns of graded scores. Psychometrika Monograph, 17.
21.
Shih, C.-L., Wang, W.-C., & Yi, C. ( 2008, April). A comparison of item selection strategies in computerized adaptive testing for testlet-based items. Paper presented at the 2008 Annual Meeting of the National Council on Measurement in Education, New York.
22.
Sireci, S.G., Wainer, H., Thissen, D. ( 1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
23.
van der Linden, W.J. ( 1998). Bayesian item selection criteria for adaptive testing . Psychometrika, 63, 201-216.
24.
van der Linden, W.J., & Pashley, P.J. ( 2000). Item selection and ability estimation in adaptive testing . In W. J. van der Linden & C. A. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 1-25). Boston: Kluwer Academic.
25.
van Rijn, P.W., Eggen, T.M., Hemker, B.T., & Sanders, P.F. ( 2002). Evaluation of selection procedures for computerized adaptive testing with polytomous items. Applied Psychological Measurement , 26, 393-411.
26.
Veerkamp, W.J.J., & Berger, M.P.F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203-226.
27.
Wainer H., Bradlow, E.T., & Du, Z. ( 2000). Testlet response theory: An analog for the 3PL model useful in testlet-based adaptive testing. In W. J. van der Linden & C. A. W. Glass (Eds.), Computerized Adaptive Testing: Theory and Practice (pp. 245-269). Dordrecht, Netherlands: Kluwer Academic.
28.
Wainer, H., Bradlow, E.T., & Wang, X. ( 2007). Testlet response theory and its applications. New York: Cambridge University Press.
29.
Wainer, H., & Kiely, G.L. ( 1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-201.
30.
Wainer, H., & Lewis, C. ( 1990). Toward a psychometrics for testlets. Journal of Educational Measurement, 27, 1-14.
31.
Wainer, H., & Thissen, D. ( 1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability? Educational Measurement: Issues and Practice, 15, 22-29.
32.
Wang, X., Bradlow, E.T., & Wainer, H. ( 2002). A general Bayesian model for testlets: Theory and applications . Applied Psychological Measurement, 26, 109-128.
33.
Yen, W. ( 1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement , 30, 187-213.
