Methods to detect item response theory (IRT) item-level misfit are typically derived assuming fixed test forms. However, IRT is also employed with more complicated test designs, such as the balanced incomplete block (BIB) design used in large-scale educational assessments. This study investigates two modifications of Douglas and Cohen’s 2001 nonparametric method of assessing item misfit, based on A) using block total score and B) pooling booklet level scores for analyzing BIB data. Block-level scores showed extreme inflation of Type I error for short blocks containing 5 or 10 items. The pooled booklet method yielded Type I error rates close to nominal in most conditions and had power to detect misfitting items. The study also found that the Douglas and Cohen procedure is only slightly affected by the presence of other misfitting items in the block. The pooled booklet method is recommended for practical applications of Douglas and Cohen’s method with BIB data.
BartholomewD. J.LeungS. O. (2002). A goodness of fit test for sparse 2p contingency tables. The British Journal of Mathematical and Statistical Psychology, 55(Pt 1), 1–15. https://doi.org/10.1348/000711002159617.
2.
BishopY. M.FeinbergS. E.HollandP. W. (1975). Discrete multivariate analysis: Theory and practice. MIT Press.
3.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/bf02291411.
4.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/bf02293801
5.
BoltD. M. (2002). A monte carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15(2), 113–141. https://doi.org/10.1207/s15324818ame1502_01
6.
CaiL. (2013). flexMIRT version 2: Flexible multilevel multidimensional item analysis and test scoring [computer program]. Vector Psychometric Group.
7.
CohenJ. (1973). Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educational and Psychological Measurement, 33(1), 107–112. https://doi.org/10.1177/001316447303300111
8.
Darrell BockR.LiebermanM. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35(2), 179–197. https://doi.org/10.1007/bf02291262
9.
DonoghueJ. R.IshamS. P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22(1), 33–51. https://doi.org/10.1177/01466216980221002
10.
DouglasJ. (1997). Joint consistency of nonparametric item characteristic curve and ability estimation. Psychometrika, 62(1), 7–28. https://doi.org/10.1007/bf02294778
DouglasJ.CohenA. (2001). Nonparametric item response function estimation for assessing parametric model fit. Applied Psychological Measurement, 25(3), 234–243. https://doi.org/10.1177/01466210122032046
13.
GlasC. A. WFalcónJ. C. S. (2003). A comparison of item fit statistics for the three-parameter logistic model. Applied Psychological Measurement, 27(2), 87–106. https://doi.org/10.1177/0146621602250530
14.
HabermanS. J.SinharayS.ChonK. H. (2013). Assessing item fit for unidimensional item response theory models using residuals from estimated item response functions. Psychometrika, 78(3), 417–440. https://doi.org/10.1007/s11336-012-9305-1
15.
LiangT.WellsC. S. (2009). A model fit statistic for generalized partial credit model. Educational and Psychological Measurement, 69(6), 913–928. https://doi.org/10.1177/0013164409332222
16.
LiangT.WellsC. S.HambletonR. K. (2014). An assessment of the nonparametric approach for evaluating the fit of item response models. Journal of Educational Measurement, 51(1), 1–17. https://doi.org/10.1111/jedm.12031
17.
LittleR. J. A.RubinD. B. (2002). Statistical analysis with missing data (2nd ed). John Wiley & Sons.
18.
LordF. M. (1980). Applications of item response theory to practical testing problems. Laurence Erlbaum Associates.
19.
LordF. M.WingerskyM. S. (1984). Comparison of IRT true-score and equipercentile observed-score equatings. Applied Psychological Measurement, 8(4), 453–461. https://doi.org/10.1177/014662168400800409.
20.
MatsumotoM.NishimuraT. (1998). Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Simulation, 8(1), 3–30. https://doi.org/10.1145/272991.272995
21.
Maydeu-OlivaresA. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research & Perspective, 11(3), 71–101. https://doi.org/10.1080/15366367.2013.831680
22.
MurakiE.BockR. D. (1997). PARSCALE: IRT item analysis and test scoring for rating- scale data [computer program]. Scientific Software International.
23.
OrlandoM.ThissenD. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003
24.
PressW. H.TeukolskyS. A.VetterlingW. T.FlanneryB. P. (1992). Numerical recipes in C: The art of scientific computing (2nd ed). Cambridge.
25.
RamsayJ. O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika, 56(4), 611–630. https://doi.org/10.1007/bf02294494
ReiserM. (1996). Analysis of residuals for the multionmial item response model. Psychometrika, 61(3), 509–528. https://doi.org/10.1007/bf02294552
28.
SgammatoA.DonoghueJ. R. (2016). Evaluation of Item Response Theory (IRT) fit indices. Paper presented at the annual meeting of the National Council on Measurement in Education.
29.
SgammatoA.DonoghueJ. R. (2018). Evaluating a modified nonparametric procedure to assess parametric IRT model fit. Paper presented at the annual meeting of the National Council on Measurement in Education.
30.
SinharayS.LuY. (2008). A further look at the correlation between item parameters and item fit statistics. Journal of Educational Measurement, 45, 1–15. https://doi.org/10.1111/j.1745-3984.2007.00049.x
31.
StoneC. A. (2000). Monte Carlo based null distribution for an alternative goodness of fit test statistic in IRT models. Journal of Educational Measurement, 37(1), 58–75. https://doi.org/10.1111/j.1745-3984.2000.tb01076.x
32.
StoneC. A. (2003). Empirical power and Type I error rates for an IRT fit statistic that considers the precision of ability estimates. Educational and Psychological Measurement, 63(4), 566–583. https://doi.org/10.1177/0013164402251034
33.
StoneC. A.HansenM. A. (2000). The effect of errors in estimating ability on goodness-of-fit tests for irt models. Educational and Psychological Measurement, 60(6), 974–991. https://doi.org/10.1177/00131640021970907
34.
StoneC. A.MislevyR. J.MazzeoJ. (1994). Classification error and goodness-of-fit in IRT models. Paper presented at the annual meeting of theAmerican Educational Research Association.
35.
StoneC. A.ZhangB. (2003). Assessing goodness of fit of item response theory models: A comparison of traditional and alternative procedures. Journal of Educational Measurement, 40(4), 331–352. https://doi.org/10.1111/j.1745-3984.2003.tb01150.x
36.
van RijnP.W.SinharayS.HabermanS.J.JohnsonM. S. (2016). Assessment of fit of item response theory models used in large-scale educational survey assessments. Large-scale Assessments in Education, 4(1), 10. https://doi.org/10.1186/s40536-016-0025-3
37.
WellsC. S.BoltD. M. (2008). Investigation of a nonparametric procedure for assessing goodness-of-fit in item response theory. Applied Measurement in Education, 21(1), 22–40. https://doi.org/10.1080/08957340701796464
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
