The primary purpose of this research is to examine the impact of estimation methods, actual latent trait distributions, and item pool characteristics on the performance of a simulated computerized adaptive testing (CAT) system. In this study, three estimation procedures are compared for accuracy of estimation: maximum likelihood estimation (MLE), expected a priori (EAP), and Warm's weighted likelihood estimation (WLE). Some research has shown that MLE and EAP perform equally well under certain conditions in polytomous CAT systems, such that they match the actual latent trait distribution. However, little research has compared these methods when prior estimates of. distributions are extremely poor. In general, it appears that MLE, EAP, and WLE procedures perform equally well when using an optimal item pool. However, the use of EAP procedures may be advantageous under nonoptimal testing conditions when the item pool is not appropriately matched to the examinees.
Get full access to this article
View all access options for this article.
References
1.
Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
2.
Boyd, A. M., Dodd, B. G., & Fitzpatrick, S. J. (2002). A comparison of exposure control procedures in CAT systems based on different measurement models for testlets. Washington, DC: Association of American Medical Colleges.
3.
Chen, S. (1997). A comparison of maximum likelihood estimation and expected a posteriori estimation in computerized adaptive testing using the generalized partial credit model. Dissertation Abstracts International, 58, 453-453.
4.
Chen, S., Hou, L., & Dodd, B. G. (1998). A comparison of maximum likelihood estimation and expected a posteriori estimation in CAT using the partial credit model. Educational and Psychological Measurement, 58, 569-595.
5.
Chen, S., Hou, L., Fitzpatrick, S. J., & Dodd, B. G. (1997). The effect of population distribution and methods of theta estimation on CAT using the rating scale model. Educational and Psychological Measurement, 57, 422-439.
6.
Davis, L. L. (2004). Strategies for controlling item exposure in computerized adaptive testing with the generalized partial credit model. Applied Psychological Measurement, 28, 165-185.
7.
Davis, L. L., & Dodd, B. G. (2003). Item exposure constraints for testlets in the Verbal Reasoning section of the MCAT. Applied Psychological Measurement, 27, 335-356.
8.
Davis, L. L., Pastor, D. A., Dodd, B. G., Chiang, C.,& Fitzpatrick, S. (2003). An examination of exposure control and content balancing restrictions on item selection in CATs using the partial credit model. Journal of Applied Measurement, 4, 24-42.
9.
Dodd, B. G., De Ayala, R. J., & Koch, W. R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19, 5-22.
10.
Dodd, B. G., & Koch, W. R. (1987). Effects of variations in step values on item and test information for the partial credit model. Applied Psychological Measurement, 11, 371-384.
11.
Dodd, B. G., Koch, W. R., & De Ayala, R. J. (1993). Computerized adaptive testing using the partial credit model: Effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53, 61-77.
12.
Hou, L., Chen, S., Dodd, B. G., & Fitzpatrick, S. J. (1996, April). The effect of methods of theta estimation, prior distribution, and number of quadrature points on CAT using the graded response model. Paper presented at the annualmeeting of the American Educational Research Association, New York.
13.
Koch, W. R., & Dodd, B. G. (1985, April). Computerized adaptive attitude measurement. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
14.
Koch, W. R.,& Dodd, B. G. (1989). An investigation of procedures for computerized adaptive testing using the partial credit model. Applied Measurement in Education, 2, 335-357.
15.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
16.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
17.
Muraki, E., & Bock, R. D. (1993). PARSCALE: IRT based test scoring and item analysis for graded open-ended exercises and performance tasks. Mooresville, IN: Scientific Software.
18.
Pastor, D. A., Dodd, B. G., & Chang, H. (2002). A comparison of item selection techniques and exposure control mechanisms in CATs using the generalized partial credit model. Applied Psychological Measurement, 26, 147-163.
19.
Samejima, F. (1998, April). Expansion of Warm's weighted likelihood estimator of ability for threeparameter logistic model to general discrete responses. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
20.
Wang, S., & Wang, T. (2001). Precision of Warm's weighted likelihood estimates for a polytomous model in computerized adaptive testing. Applied Psychological Measurement, 25, 317-332.
21.
Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450.
22.
Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.
