Comparative Analyses of MIRT Models and Software (BMIRT and flexMIRT)

Abstract

Application of MIRT modeling procedures is dependent on the quality of parameter estimates provided by the estimation software and techniques used. This study investigated model parameter recovery of two popular MIRT packages, BMIRT and flexMIRT, under some common measurement conditions. These packages were specifically selected to investigate the model parameter recovery of three item parameter estimation techniques, namely, Bock–Aitkin EM (BA-EM), Markov chain Monte Carlo (MCMC), and Metropolis–Hastings Robbins–Monro (MH-RM) algorithms. The results demonstrated that all estimation techniques had similar root mean square error values when larger sample size and higher test length were used. Depending on the number of dimensions, sample size, and test length, each estimation technique exhibited some strengths and weaknesses. Overall, the BA-EM technique was found to have shorter estimation time with all test specifications.

Keywords

multidimensional item response theory BMIRT flexMIRT item parameter recovery MCMC MH-RM BA-EM

Get full access to this article

View all access options for this article.

References

Bahry

L. M.

(2012). Polytomous item response theory parameter recovery: An investigation of non-normal distributions and small sample size (Master’s thesis). University of Alberta, Edmonton, Canada. Retrieved from http://hdl.handle.net/10402/era.25356

Beguin

A. A.

Glas

C. A. W.

(2001). MCMC estimation and some model fit analysis of multidimensional IRT models. Psychometrika, 66, 541-562.

Bock

R. D.

Aitkin

(1981). Marginal maximum likelihood estimation of item parameters: Applications of an EM algorithm. Psychometrika, 46, 443-459.

Bolt

D. M.

Lall

V. F.

(2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 29, 395-414.

Bulut

(2013). Between-person and within-person subscore reliability: Comparison of unidimensional and multidimensional IRT models (Unpublished doctoral dissertation). University of Minnesota, Minneapolis. Retrieved from http://conservancy.umn.edu/bitstream/handle/11299/155592/Bulut_umn_0130E_13879.pdf?sequence=1

Cai

(2010a). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581-612.

Cai

(2010b). High-dimensional exploratory item factor analysis by a Metropolis-Hastings Robbins-Monro algorithm. Psychometrika, 75, 33-57.

Cai

(2010c). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307-335.

Cai

du Toit

S. H. C.

Thissen

(2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling. Lincoln, IL: Scientific Software International.

10.

Cai

Monroe

(2013). IRT model fit evaluation from theory to practice: Progress and some unanswered questions. Measurement: Interdisciplinary Research and Perspectives, 11(3), 102-106.

11.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1-29.

12.

De la Torre

(2009). Improving the quality of ability estimates through multidimensional scoring and incorporation of ancillary variables. Applied Psychological Measurement, 33, 465-485.

13.

De la Torre

Song

Hong

(2011). A comparison of four methods of IRT subscoring. Applied Psychological Measurement, 35, 296-316.

14.

DeMars

(2002). Incomplete data and item parameter estimates under JMLE and MML. Applied Measurement in Education, 15, 15-31.

15.

Finch

W. H.

(2011). Multidimensional item response theory parameter estimation with non-simple structure items. Applied Psychological Measurement, 35, 67-82.

16.

Finch

W. H.

French

B. F.

(2012). A comparison of methods for estimating confidence intervals for omega squared effect size. Educational and Psychological Measurement, 72, 68-77.

17.

Guyer

Thompson

N. A.

(2011). User’s manual for Xcalibre item response theory calibration software (Version 4.1.3). St. Paul, MN: Assessment Systems Corporation.

18.

Han

K. T.

(2007). WinGen: Windows software that generates IRT parameters and item responses. Applied Psychological Measurement, 31, 457-459.

19.

Hsieh

Proctor

T. P.

Hou

Teo

K. S.

(2010). A comparison of Bayesian MCMC and marginal maximum likelihood methods in estimating the item parameters of the 2PL IRT model. International Journal of Innovative Management, Information & Production, 1(1), 81-89.

20.

Hulin

C. L.

Lissak

R. L.

Drasgow

(1982). Recovery of two- and there-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249-260.

21.

Jurich

Goodman

(2009, October). A comparison of IRT parameter recovery in mixed format examinations using PARSCALE and ICL. Paper presented at the annual meeting of the Northeastern Educational Research Association, Rocky Hill, CT.

22.

Kieftenbeld

Natesan

(2012). Recovery of graded response model parameters: A comparison of marginal maximum likelihood and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 36, 399-419.

23.

Lautenschlager

G. J.

Meade

A. W.

Kim

S.-H.

(2006). Cautions regarding sample characteristics when using the graded response model. Paper presented at the conference of the Society for Industrial and Organizational Psychology, Dallas, TX.

24.

Lee

(2012). Multidimensional item response theory: An investigation of interaction effects between factors on item parameter recovery using Markov chain Monte Carlo estimation (Unpublished doctoral dissertation). Michigan State University, East Lansing, MI. Retrieved from http://etd.lib.msu.edu/islandora/object/etd%3A1577

25.

Lee

S. H.

(2007). Multidimensional item response theory: A SAS MDIRT MACRO and empirical study of PIAT MATH Test (Unpublished doctoral dissertation). The University of Oklahoma, Norman.

26.

Montgomery

Skorupski

, (2012). Investigation of IRT parameter recovery and classification accuracy in mixed format. Paper presented at the meeting of the National Council of Measurement in Education, Vancouver, Canada.

27.

Reise

S. P.

(1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-144.

28.

Svetina

(2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM methods (Unpublished doctoral dissertation). Arizona State University, Tempe, AZ.

29.

Wang

W. C.

Chen

C. T.

(2005). Item parameter recovery, standard error estimates, and fit statistics of the Winsteps program for the family of Rasch models. Educational and Psychological Measurement, 65, 376-404.

30.

Weiss

J. D.

Minden

S. V.

(2012). A comparison of item parameter estimates from Xcalibre 4.1 and Bilog-MG (Technical Report). St. Paul, MN: Assessment Systems Corporation.

31.

Yao

(2003). BMIRT: Bayesian multivariate item response theory. Monterey, CA: CTB/McGraw-Hill.

32.

Yao

Boughton

K. A.

(2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83-105.

33.

Yao

Schwarz

(2006). A multidimensional partial credit model with associated item and test statistics: An application to mixed format tests. Applied Psychological Measurement, 30, 469-492.

34.

Zhang

Stone

C. A.

(2004, April). Direct and indirect estimation of three-parameter multidimensional item response models. Paper presented at the meeting of the American Educational Research Association, San Diego, CA.

35.

Zhang

Stone

C. A.

(2007). Evaluating item fit for multidimensional item response models. Educational and Psychological Measurement, 68, 181-196.

36.

Zhang

(2012). Calibration of response data using MIRT models with simple and mixed structures. Applied Psychological Measurement, 36, 375-398.