A difficult result to interpret in Computerized Adaptive Tests (CATs) occurs when an ability estimate initially drops and then ascends continuously until the test ends, suggesting that the true ability may be higher than implied by the final estimate. This study explains why this asymmetry occurs and shows that early mistakes by high-ability students can lead to considerable underestimation, even in tests with 45 items. The opposite response pattern, where low-ability students start with lucky guesses, leads to much less bias. The authors show that using Barton and Lord's four-parameter model (4PM) and a less informative prior can lower bias and root mean square error (RMSE) for high-ability students with a poor start, as the CAT algorithm ascends more quickly after initial underperformance. Results also show that the 4PM slightly outperforms a CAT in which less discriminating items are initially used. The practical implications and relevance for psychological measurement more generally are discussed.
Baker, F.B., & Kim, S. (2004). Item Response Theory: Parameter estimation techniques (2nd ed.). New York: Marcel Dekker.
2.
Barton, M. .A., & Lord, F.M. (1981). An upper asymptote for the three-parameter logistic item-response model (Tech. Rep. RR-81-20). Prince-ton, NJ: Educational Testing Service.
3.
Bock, R., & Mislevy, R.J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
4.
Bradlow, E.T. (1996). Negative information and the Three-Parameter Logistic Model. Journal of Educational and Behavioral Statistics, 21, 179-185.
5.
Carlson, S. (2000). ETS finds flaws in the way online GRE rates some students. Chronicle of Higher Education, 47, a47.
6.
Chang, H.H. (2004). Understanding computerized adaptive testing: From Robbins-Munro to Lord and beyond. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 117-133). New York: Sage.
7.
Chang, H.H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
Chang, H.H., & Ying, Z. (2002). To weight or not to weight-Balancing influence of initial and later items in adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
10.
Chen, S.Y., Ankenmann, R.D., & Chang, H.H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24, 241-255.
11.
Cheng, P.E., & Liou, M. (2000). Estimation of trait level in computerized adaptive testing. Applied Psychological Measurement, 24, 257-265.
12.
Kaplan Publishing. (2004). GRE exam, 2005edition. New York: Simon & Schuster.
13.
Lurie, K., Pecsenye, M., Robinson, A., & Ragsdale, D. (2005). Cracking the GRE. New York : Prince-ton Review.
Mislevy, R.J., & Bock, R. (1982). Biweight estimates of latent ability. Educational and Psychological Measurement, 42, 725-737.
17.
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of American Statistical Association, 70, 351-356.
18.
Passos, V.L., Berger, M.P.F., & Tan, F.E. (2007). Test design optimization in CAT early stage with the nominal response model. Applied Psychological Measurement , 31, 213-232.
19.
Reise, S.P., & Waller, N.G. (2003). How many IRT parameters does it take to model psychopathology items? Psychological Methods, 8, 164-184.
20.
Rouse, S.V., Finger, M.S., & Butcher, J.N. (1999). Advances in clinical personality measurement: An item response theory analysis of the MMPI-2 PSY-5 scales. Journal of Personality Assessment, 72, 282-307.
21.
van der Linden, W.J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201-216.
22.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2000). Detecting person-misfit in adaptive testing using statistical process control techniques. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 201-219). Boston: Kluwer Academic.
23.
Wang, T., & Vispoel, W.P. (1998). Properties of ability estimation methods in computerized adaptive testing. Journal of Educational Measurement, 35, 109-135.
24.
Wainer, H., Dorans, N.J., Eignor, D., Flaugher, R., Green, B.F., Mislevy, R.J., et al. (2000). Computerized adaptive testing: A primer . Hillsdale, NJ: Lawrence Earlbaum .
25.
Waller, N.G., & Reise, S.P. (in press). Measuring psychopathology with nonstandard IRT models: Fitting the four parameter model to the MMPI. In S. Embretson & J. S. Roberts (Eds.), New directions in psychological measurement with model-based approaches.
26.
Weiss, D.J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473-492.
