Recovery of Item Parameters in the Nominal Response Model: A Comparison of Marginal Maximum Likelihood Estimation and Markov Chain Monte Carlo Estimation

Abstract

Markov chain Monte Carlo (MCMC) methods, such as Gibbs sampling, present an alternative to marginal maximum likelihood (MML) estimation, which offers some promise for parameter estimation particularly with complex models, in small sample situations, and for other applications where MML algorithms have not been established. MCMC circumvents the problems associated with implementing an estimation algorithm for complex, multidimensional probability distributions by sampling the parameters from each of the one-dimensional conditional posterior distributions at each stage of the Markov chain. In this article, the authors compared the quality of item parameter estimates for MML and MCMC with one type of complex item response theory model, the nominal response model. The quality of item parameter recovery was nearly identical for both MML and MCMC. Both methods tended to produce good estimates, even for short tests and relatively small sample sizes. Parameter recovery was best for items of moderate dif.culty (i.e., items matched to the latent trait distribution); recovery was worst for items that were extremely easy or dif.cult. The quality of item parameter recovery improved as test length increased from 10 to 30 items, but did not change as test length increased from 20 to 30 items. MCMC estimation takes substantially longer but appears to be a good surrogate for MML for those situations for which an MML algorithm has not been developed.

Get full access to this article

View all access options for this article.

References

Albert, J. H. (1992). Bayesian estimation of normal ogive item response curves using Gibbs sampling. Journal of Educational Statistics, 17, 251-269.

Baker, F. B. (1988). GENIRV: A program to generate item response vectors [Computer program]. Madison: University of Wisconsin, Department of Educational Psychology, Laboratory of Experimental Design.

Baker, F. B. (1992). Equating tests under the graded response model. Applied Psychological Measurement, 16, 87-96.

Baker, F. B. (1993a). EQUATE 2.0: A computer program for the characteristic curve method of IRT equating. Applied Psychological Measurement, 17, 20-20.

Baker, F. B. (1993b). Equating tests under the nominal response model. Applied Psychological Measurement, 17, 239-251.

Baker, F. B. (1998). An investigation of the item parameter recovery characteristics of a Gibbs sampling approach. Applied Psychological Measurement, 22, 153-169.

Baker, F. B. , & Al-Karni, A. (1991). A comparison of two procedures for computing IRT equating coefficients. Journal of Educational Measurement, 28, 147-162.

Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 46, 443-459.

Bock, R. D. , & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of an EM algorithm. Psychometrika, 35, 179-197.

10.

Bock, R. D. , & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.

11.

Chib, S. , & Greenberg, E. (1995). Understanding the Metropolis-Hastings algorithm. The American Statistician, 49, 327-335.

12.

Cohen, A. S. , Kim, S.-H. , & Baker, F. B. (1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement, 17, 335-350.

13.

De Ayala, R. J. , & Sava-Bolesta, M. (1999). Item parameter recovery for the nominal response model. Applied Psychological Measurement, 23, 3-19.

14.

Gelfand, A. E. , & Smith, A. F. M. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association, 85, 398-409.

15.

Geman, S. , & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721-741.

16.

Geyer, C. J. (1992). Practical Markov chain Monte Carlo. Statistical Science, 7, 473-483.

17.

Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57, 97-109.

18.

Johnson, V. E. , & Albert, J. H. (1999). Ordinal data modeling. New York: Springer-Verlag.

19.

Kim, S.-H. , & Cohen, A. S. (1991). A comparison of two area measures for detecting differential item functioning. Applied Psychological Measurement, 15, 269-278.

20.

Kim, S.-H. , & Cohen, A. S. (1999, April). Accuracy of parameter estimation in Gibbs sampling under the two-parameter logistic model. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.

21.

Lee, Y. S. (1997). A parameter recovery study for the nominal response model. Unpublished manuscript, Department of Educational Psychology, University of Wisconsin—Madison.

22.

Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

23.

Metropolis, N. , Rosenbluth, A. W. , Rosenbluth, M. N. , Teller, A. H. , & Teller, E. (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21, 1087-1092.

24.

Metropolis, N. , & Ulam, S. (1949). The Monte Carlo method. Journal of the American Statistical Association, 44, 335-341.

25.

Mislevy, R. J. , & Bock, R. D. (1984). BILOG Version 2.2: Item analysis and test scoring with binary logistic models. Mooresville, IN: Scientific Software.

26.

Patz, R. J. , & Junker, B. W. (1999a). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24(4), 342-366.

27.

Patz, R. J. , & Junker, B. W. (1999b). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146-178.

28.

Raftery, A. E. , & Lewis, S. M. (1996). Implementing MCMC. In W. R. Gilks , S. Richardson , & D. J. Spiegelhalter (Eds.), Markov chain Monte Carlo in practice (pp. 115-130). London: Chapman & Hall.

29.

Raju, N. S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.

30.

Reise, S. P. , & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.

31.

Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, No. 17.

32.

Spiegelhalter, D. J. , Thomas, A. , Best, N. G. , & Gilks, W. R. (1997). BUGS: Bayesian inference using Gibbs sampling (Version 0.6) [Computer program]. Cambridge, UK: University of Cambridge, Institute of Public Health, Medical Research Council Biostatistics Unit.

33.

Stocking, M. L. , & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.

34.

Thissen, D. (1991). MULTILOG: Multiple, categorical item analysis and test scoring using item response theory [Computer program]. Chicago, IL: Scientific Software.

35.

Tsutakawa, R. K. , & Soltys, M. J. (1988). Approximation for Bayesian ability estimation. Journal of Educational Statistics, 13, 117-130.

36.

Wollack, J. A. , & Cohen, A. S. (1997, March). Detection of answer copying with unknown item and ability parameters. Paper presented at the annual meeting of the American Educational Research Association.

37.

Yen, M. W. (1987) A comparison of the efficiency and accuracy of BILOG and LOGIST. Psychometrika, 52, 275-291.