Parameter Recovery in Multidimensional Item Response Theory Models Under Complexity and Nonnormality

Abstract

Information about the psychometric properties of items can be highly useful in assessment development, for example, in item response theory (IRT) applications and computerized adaptive testing. Although literature on parameter recovery in unidimensional IRT abounds, less is known about parameter recovery in multidimensional IRT (MIRT), notably when tests exhibit complex structures or when latent traits are nonnormal. The current simulation study focuses on investigation of the effects of complex item structures and the shape of examinees’ latent trait distributions on item parameter recovery in compensatory MIRT models for dichotomous items. Outcome variables included bias and root mean square error. Results indicated that when latent traits were skewed, item parameter recovery was generally adversely impacted. In addition, the presence of complexity contributed to decreases in the precision of parameter recovery, particularly for discrimination parameters along one dimension when at least one latent trait was generated as skewed.

Keywords

multidimensional IRT complex structure parameter estimation recovery nonnormality

Get full access to this article

View all access options for this article.

References

Allen

N. L.

Donoghue

J. R.

Schoeps

T. L.

(2001). The NAEP 1998 technical report No. NCES 2001-509. Washington, DC: National Center for Educational Statistics. Retrieved from https://eric.ed.gov/?q=the+NAEP+1998+Technical+report&id=EJ646496

Andreis

Ferrari

P. A.

(2012). Missing data and parameters estimates in multidimensional item response models. Electronic Journal of Applied Statistical Analysis, 5, 431-437.

Babcock

(2009). Estimating a noncompensatory IRT model using a modified Metropolis algorithm (Doctoral dissertation). Retrieved from http://purl.umn.edu/58172

Bolt

M. D.

Lall

V. F.

(2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov Chain Monte Carlo. Applied Psychological Measurement, 27, 395-414.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1-29.

Chalmers

R. P.

Flora

D. B.

(2014). Maximum-likelihood estimation of noncompensatory IRT models with the MH-RM algorithm. Applied Psychological Measurement, 38, 339-358.

De Ayala

R. J.

Sava-Bolesta

(1999). Item parameter recovery for the Nominal Response Model. Applied Psychological Measurement, 23, 3-19.

DeMars

C. E.

(2015). Partially compensatory multidimensional item response theory models: Two alternate model forms. Educational and Psychological Measurement, 76, 231-257.

Ercikan

Oliveri

M. O.

(2016). In search of validity evidence in support of the interpretation and use of assessments of complex constructs: Discussion of research on assessing 21st century skills. Applied Measurement in Education, 29, 310-318.

10.

Finch

(2010). Item parameter estimation for the MIRT model: Bias and precision of confirmatory factor analysis-based models. Applied Psychological Measurement, 34, 10-26.

11.

Finch

(2011). Multidimensional item response theory parameter estimation with nonsimple structure items. Applied Psychological Measurement, 35, 67-82.

12.

Finch

Edwards

J. M.

(2016). Rasch model parameter estimation in the presence of a nonnormal latent trait using a nonparametric Bayesian approach. Educational and Psychological Measurement, 76, 662-684. doi:10.1177/001316441560418

13.

Kirisci

Hsu

T. C.

(2001). Robustness of item parameter estimation programs to assumptions of unidimensionality and normality. Applied Psychological Measurement, 25, 146-162.

14.

Knol

D. L.

Berger

(1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26, 457-477.

15.

Monroe

Cai

(2014). Estimation of a Ramsay-curve item response theory model by the Metropolis–Hastings Robbins–Monro algorithm. Educational and Psychological Measurement, 74, 343-369.

16.

Preston

K. S. J.

Reise

S. P.

(2014). Estimating the Nominal Response Model under nonnormal conditions. Educational and Psychological Measurement, 74, 377-399.

17.

R Core Development Team. (2015). R: A language and environment for statistical computing (Version 3.2.1) [Computer software]. R Foundation for Statistical Computing, Vienna, Austria. Available from https://www.R-project.org/

18.

San Martin

Jaran

Rolin

J.-M.

Mouchart

(2011). On the Bayesian nonparametric generalization of IRT-type models. Psychometrika, 76, 385-409.

19.

Sass

D. A.

Schmitt

T. A.

Walker

C. M.

(2008). Estimating non-normal latent trait distributions within item response theory using true and estimated item parameters. Applied Measurement in Education, 21, 65-88. doi:10.1080/08957340701796415

20.

Seong

T. J.

(1990). Sensitivity of marginal maximum likelihood estimation of item and ability parameters to the characteristics of the prior ability distributions. Applied Psychological Measurement, 14, 299-311.

21.

Sheng

Wikle

C. K.

(2007). Comparing multiunidimensional and unidimensional item response theory models. Educational and Psychological Measurement, 33, 620-639.

22.

Stone

C. A.

(1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1-16.

23.

Wainer

(1990). Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum.

24.

Way

W. D.

Ansley

T. N.

Forsyth

R. A.

(1988). The comparative effects of compensatory and noncompensatory two-dimensional data on unidimensional IRT estimates. Applied Psychological Measurement, 12, 239-252. doi:10.1177/014662168801200303

25.

Wetzel

Hell

(2014). Multidimensional item response theory models in vocational interest measurement: An illustration using the AIST-R. Journal of Psychoeducational Assessment, 32, 342-355. doi:10.1177/0734282913508244

26.

Wiberg

(2012). Can a multidimensional test be evaluated with unidimensional item response theory?Educational Research and Evaluation, 18, 307-320.

27.

Woods

C. M.

(2006). Ramsay-curve item response theory (RC-IRT) to detect and correct for nonnormal latent variables. Psychological Methods, 11, 253.

28.

Woods

C. M.

(2008). Ramsay-curve item response theory for the three-parameter logistic item response model. Applied Psychological Measurement, 32, 447-465.

29.

Zhang

(2012). Calibration of response data using MIRT models with simple and mixed structures. Applied Psychological Measurement, 36, 375-398.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB