The Effect of Population Distribution and Method of Theta Estimation on Computerized Adaptive Testing (CAT) Using the Rating Scale Model

Abstract

A simulation study was conducted to investigate the effect of population distribution on maximum likelihood estimation (MLE) and expected a posteriori estimation (EAP) in computerized adaptive testing (CAT) based on Andrich's rating scale model. Comparisons were made among MLE and EAP with a normal prior distribution and EAP with a uniform prior distribution within two data sets: one generated using a normal trait distribution and the other using a negatively skewed trait distribution. Descriptive statistics, correlations, scattergrams, and accuracy indices were used to compare the different methods of trait estimation. EAP estimation with a normal prior or uniformprior yielded results similar to those obtained with MLE, even though the prior did not match the underlying trait distribution. An additional simulation study based on real data suggested that more work is needed to determine the optimal number of quadrature points for EAP in CAT based on the rating scale model. The choice between MLE and EAP for particular measurement situations is discussed.

Get full access to this article

View all access options for this article.

References

Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.

Baker, F. B. (1992). Item response theory: Parameter estimation techniques. New York: Marcel Dekker.

Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Bock, R. D. , & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.

De Ayala, R. J. (1989). A comparison of the nominal response model and the three-parameter logistic model in computerized adaptive testing. Educational and Psychological Measurement, 49, 789-805.

De Ayala, R. J. (1992). The nominal response model in computerized adaptive testing. Applied Psychological Measurement, 16, 327-343. CHEN et al.

De Ayala, R. J. (1993). YeomanDG'": A data generation program for the Macintosh. Applied Psychological Measurement, 17, 393.

Dodd, B. G. (1987, April). Computerized adaptive testing with the rating scale model. Paper presented at the Fourth International Objective Measurement Workshop, Chicago.

Dodd, B. G. (1990). The effect of item selection procedure and stepsize on computerized adaptive attitude measurement using the rating scale model. Applied Psychological Measurement, 14, 355-366.

10.

Dodd, B. G. , & De Ayala, R. J. (1994). Item information as a function of threshold values in the rating scale model. In M. Wilson (Ed.), Objective measurement: Theory into practice (Vol. 2, pp. 301-317). Norwood, NJ: Ablex.

11.

Dodd, B. G. , Koch, W. R. , & De Ayala, R. J. (1988, April). Computerized adaptive attitude measurement: A comparison of the graded response and rating scale models. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.

12.

Dodd, B. G. , Koch, W. R. , & De Ayala, R. J. (1989). Operational characteristics of adaptive testing procedures using the graded response model. Applied Psychological Measurement, 13, 129-143.

13.

Dodd, B. G. , Koch, W. R. , & De Ayala, R. J. (1993). Computerized adaptive testing using the partial credit model: Effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53, 61-77.

14.

Hambleton, R. K. , & Swaminata H. (1985). Item response theory: Principles and applications. Norwell, MA: Kluwer Academic.

15.

Koch, W. R. , & Dodd, B. G. (1995). An investigation of procedures for computerized adaptive testing using the successive intervals Rasch model. Educational and Psychological Measurement, 55, 976-990.

16.

Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

17.

Mislevy, R. J. , & Stocking, M. L. (1989). A consumer's guide to LOGIST and BILOG. Applied Psychological Measurement, 13, 57-75.

18.

Rost, J. (1988). Measuring attitudes with a dtreshold model drawing on a traditional scaling concept Applied Psychological Measurement, 12, 397-409.

19.

Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika (Monograph Suppl. 17).

20.

Samejima, F. (1973). A comment on Birnbaum's three-parameter logistic model in the latent trait theory. Psychometrika, 38, 221-233.

21.

Seong, T. (1990). Sensitivity of marginal maximum likelihood estimation of item and ability parameters to the characteristics of the prior ability distributions. Applied Psychological Measurement, 14, 299-311.

22.

Wainer, H. , & Thissen, D. (1987). Esimating ability with the wrong model. Journal of Educational Statistics, 12, 339-368.