This monte carlo study evaluated the relative accuracy of Warm’s (1989) weighted likelihood estimate (WLE) compared to the maximum likelihood estimate (MLE), expected a posteriori (EAP) estimate, and maximum a posteriori (MAP) estimate. The generalized partial-credit model was used under a variety of computerized adaptive testing (CAT) conditions. The results indicated that WLE was more accurate than MLE with a fixed-length CAT, consistent with previous findings. WLE and MLE had smaller bias and larger standard errors than EAP and MAP. EAP was more accurate than MAP in a variety of CAT conditions. Although root mean squared errors were different among the four estimation methods, no statistically significant mean differences were found. EAP and MAP had advantages over WLE and MLE in terms of test efficiency. These results suggest that the test termination rule has more impact on the accuracy of θ estimation methods than does the item bank size.
Get full access to this article
View all access options for this article.
References
1.
Allen, N. L. , Carlson, J. E., & Zelenak, C. A. (1999). The NAEP 1996 technical report, NCES 1999-452. Washington DC: National Center for Educational Statistics.
2.
Bennett, R. E. , Steffen, M., Singley, M. K., Morley, M., & Jacquemin, D. (1997). Evaluating an auto-matically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement, 34, 162–176.
3.
Bock, R. D. , & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431–444.
4.
Copas, J. B. (1988). Binary regression models for contaminated data. Journal of the Royal Statistical Society B, 50, 225–265.
5.
De Ayala, R. J. (1992). The nominal response model in computerized adaptive testing. Applied Psychological Measurement, 16, 327–343.
6.
Dodd, B. G. , & De Ayala, R. J. (1994). Item information as a function of threshold values in the rating scale model. In M. Wilson (Ed.), Objective measurement: Theory into practice (pp. 301–317). Norwood NJ: Ablex.
7.
Dodd, B. G. , De Ayala, R. J., & Koch, W. R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19, 5–22.
8.
Dodd, B. G. , & Koch, W. R. (1987). Effects of variations in item step values on item and test information in the partial credit model. Applied Psychological Measurement, 11, 371–384.
9.
Dodd, B. G. , Koch, W. R., & De Ayala, R. J. (1989). Operational characteristics of adaptive testing procedures using the graded response model. Applied Psychological Measurement, 13, 129–143.
10.
Dodd, B. G. , Koch, W. R., & De Ayala, R. J. (1993). Computerized adaptive testing using the partial credit model: Effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53, 61–77.
11.
Eignor, D. R. , & Schaeffer, G. A. (1995, April). Comparability studies for GRE General CAT and the NCLEX using CAT. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco CA.
12.
Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80, 27–38.
13.
Gall, M. D. , Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. New York: Longman.
14.
Harwell, M. R. (1997). Analyzing the results of Monte Carlo studies in item response theory. Educational and Psychological Measurement, 57, 266–279.
15.
Kingsbury, G. G. , & Weiss, D. J. (1983). A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 257–238). New York: Academic Press.
16.
Koch, W. R. , & Dodd, B. G. (1985, April). Computerized adaptive attitude measurement. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
17.
Koch, W. R. , & Dodd, B. G. (1989). An investigation of procedures for computerized adaptive testing using partial credit scoring. Applied Measurement in Education, 2, 335–357.
18.
Lord, F. M. (1977). A broad-range tailored test of verbal ability. Applied Psychological Measurement, 1, 95–100.
19.
Lord, F. M. (1983). Unbiased estimation of ability parameters, of their variance, and of their parallelforms reliability. Psychometrika, 48, 233–245.
20.
Lord, F. M. (1983, August). Memorandum for: Ms. Stocking, Ms. M. Wang, Ms. Wingersky. Subject: Sampling variance and bias for MLE and Bayesian estimation of θ [Internal Memorandum]. Princeton NJ: Educational Testing Service.
21.
Lord, F. M. (1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal of Educational Measurement, 23, 157–162.
22.
McBride, J. R. , & Martin, J. T. (1983). Reliability and validity of adaptive ability tests in a military setting. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 224–236). New York: Academic Press.
23.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.
24.
Muraki, E. & Bock, D. (1999). PARSCALE: Parameter scaling of rating data (Version 3.5) [Computer software]. Lincolnwood IL: Scientific Software International.
25.
Quenouille, M. H. (1949). Approximate tests of correlation in time-series. Journal of the Royal Statistical Society, B, 11, 68–84.
26.
Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika, 43, 353–360.
27.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika MonographSupplement No. 17, 34.
28.
Samejima, F. (1993a). An approximation for the bias function of the maximum likelihood estimate of a latent variable for the general case where the item responses are discrete. Psychometrika, 58, 119–138.
29.
Samejima, F. (1993b). The bias function of the maximum likelihood estimate of ability for the dichotomous response level. Psychometrika, 58, 195–209.
30.
Samejima, F. (1998, April). Expansion of Warm’s weighted likelihood estimator of ability for the three-parameter logistic model to general discrete responses. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego CA.
31.
Segall, D. O. (1995, April). Equating the CAT-ASVAB: Experiences and lessons learned. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
32.
Segall, D. O. , & Carter, G. (1995, April). Equating the CAT-GATB: Issues and approach. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
33.
Spence, I. (1983). Monte carlo simulation studies. Applied Psychological Measurement, 7, 405–425.
34.
Stone, C. A. (1993, July). The use of multiple replications in IRT based Monte Carlo research. Paper presented at the European meeting of the Psychometric Society, Barcelona, Spain.
35.
Urry, V. W. (1977). Tailored testing: A successful application of latent trait theory. Journal of Educational Measurement, 14, 181–196.
36.
van der Linden, W. J. , & Glas, C. A. W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13, 35–53.
37.
Wainer, H. , Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L., & Thissen, D. (1990). Computerized adaptive testing: A primer. Hillsdale NJ: Erlbaum.
38.
Wang, S. (1999). The accuracy of ability estimation methods for computerized adaptive testing using the generalized partial credit model. Unpublished doctoral dissertation, University of Pittsburgh.
39.
Wang, T. (1995). The precision of ability estimation methods in computerized adaptive testing. Unpublished doctoral dissertation, University of Iowa, Iowa City (UM No. 9945102).
40.
Wang, T. , Hanson, B. A., & Lau, C. M. (1999). Reducing bias in computerized adaptive testing trait estimation: A comparison of approaches. Applied Psychological Measurement, 23, 263–278.
41.
Wang, T. , & Kolen, M. J. (2001). Evaluating comparability in computerized adaptive testing: Issues, criteria, and an example. Journal of Educational Measurement, 38, 19–49.
42.
Wang, T. , & Vispoel, W. P. (1998). Properties of ability estimation methods in computerized adaptive testing. Journal of Educational Measurement, 35, 109–135.
43.
Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450.
