Testing item-level fit is important in scale development to guide item revision/deletion. Many item-level fit indices have been proposed in literature, yet none of them were directly applicable to an important family of models, namely, the higher order item response theory (HO-IRT) models. In this study, chi-square-based fit indices (i.e., Yen’s Q1, McKinley and Mill’s G2, Orlando and Thissen’s S-X2, and S-G2) were extended to HO-IRT models. Their performances are evaluated via simulation studies in terms of false positive rates and correct detection rates. The manipulated factors include test structure (i.e., test length and number of dimensions), sample size, level of correlations among dimensions, and the proportion of misfitting items. For misfitting items, the sources of misfit, including the misfitting item response functions, and misspecifying factor structures were also manipulated. The results from simulation studies demonstrate that the S-G2 is promising for higher order items.
BakerF. B.KimS.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed., Revised and expanded). New York, NY: Marcel Dekker.
2.
BartholomewD. J.LeungS. O. (2002). A goodness of fit test for sparse 2p contingency tables. British Journal of Mathematical and Statistical Psychology, 55, 1-15.
3.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
4.
BockR. D.HabermanS. J. (2009, July). Confidence bands for examining goodness-of-fit of estimated item response functions. Paper presented at Annual Meeting of the Psychometric Society, Cambridge, UK.
5.
CaiL. (2015). Lord-Wingersky algorithm version 2.0 for hierarchical item factor models with applications in test scoring, scale alignment, and model fit testing. Psychometrika, 80, 535-559.
6.
CaiL.Maydeu-OlivaresA.CoffmanD. L.ThissenD. (2006). Limited-information goodness-of-fit testing of item response theory models for sparse 2P tables. British Journal of Mathematical and Statistical Psychology, 59, 173-194.
7.
ChonK. H.LeeW. C.DunbarS. B. (2010). A comparison of item fit statistics for mixed IRT models. Journal of Educational Measurement, 47, 318-338.
8.
de la TorreJ.HongY. (2010). Parameter estimation with small sample size: A higher-order IRT model approach. Applied Psychological Measurement, 34, 267-285.
9.
de la TorreJ.SongH. (2009). Simultaneous estimation of overall and domain abilities: A higher-order IRT model approach. Applied Psychological Measurement, 33, 620-639.
10.
DemarsC. E. (2005). Type I error rates for Parscale’s fit index. Educational and Psychological Measurement, 65, 42-50.
11.
DouglasJ.CohenA. S. (2001). Nonparametric item response function estimation for assessing parametric model fit. Applied Psychological Measurement, 25, 234-243.
12.
FienbergS. E. (1979). The use of chi-squared statistics for categorical data problems. Journal of the Royal Statistical Society: Series B (Statistical Methodological), 41, 54-64.
13.
FoxJ. P.GlasC. A. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 271-288.
14.
GlasC. A. W.Suárez-FalcónJ. C. (2003). A comparison of item-fit statistics for the three-parameter logistic model. Applied Psychological Measurement, 27, 87-106.
15.
HabermanS. J. (2009). Use of generalized residuals to examine goodness of fit of item response models. ETS Research Report Series, 2009(1), 1-17.
16.
HabermanS. J.SinharayS.ChonK. H. (2013). Assessing item fit for unidimensional item response theory models using residuals from estimated item response functions. Psychometrika, 78, 417-440.
17.
HambletonR. K.SwaminathanH.RogersH. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
18.
HuangH. Y. (2015). A multilevel higher order item response theory model for measuring latent growth in longitudinal data. Applied Psychological Measurement, 39, 362-372.
19.
HuangH. Y. (2017). Mixture IRT model with a higher-order structure for latent traits. Educational and Psychological Measurement, 77, 275-304.
20.
HuangH. Y.ChenP. H.WangW. C. (2012). Computerized adaptive testing using a class of high-order item response theory models. Applied Psychological Measurement, 36, 689-706.
21.
HuangH. Y.WangW. C. (2013). Higher order testlet response models for hierarchical latent traits and testlet-based items. Educational and Psychological Measurement, 73, 491-511.
22.
HuangH. Y.WangW. C.ChenP. H.SuC. M. (2013). Higher-order item response models for hierarchical latent traits. Applied Psychological Measurement, 37, 619-637.
23.
HuoY.de la TorreJ.MunE. Y.KimS. Y.RayA. E.JiaoY.WhiteH. R. (2015). A hierarchical multi-unidimensional IRT approach for analyzing sparse, multi-group data for integrative data analysis. Psychometrika, 80, 834-855.
24.
LaHuisD. M.ClarkP.O’BrienE. (2011). An examination of item response theory item fit indices for the graded response model. Organizational Research Methods, 14, 10-23.
25.
LeeM. (2014). Application of higher-order IRT models and hierarchical IRT models to computerized adaptive testing (Electronic Theses and Dissertations). University of California, Los Angeles.
26.
LiY.RuppA. A. (2011). Performance of the S-X2 statistic for full-information bifactor models. Educational and Psychological Measurement, 71, 986-1005.
27.
LiangT.WellsC. S. (2009). A model fit statistic for generalized partial credit model. Educational and Psychological Measurement, 69, 913-928.
28.
LiangT.WellsC. S. (2015). A nonparametric approach for assessing goodness-of-fit of IRT models in a mixed format test. Applied Measurement in Education, 28, 115-129.
29.
LiangT.WellsC. S.HambletonR. K. (2014). An assessment of the nonparametric approach for evaluating the fit of item response models. Journal of Educational Measurement, 51, 1-17.
30.
LiuY.Maydeu-OlivaresA. (2013). Local dependence diagnostics in IRT modeling of binary data. Educational and Psychological Measurement, 73, 254-274.
31.
LiuY.Maydeu-OlivaresA. (2014). Identifying the source of misfit in item response theory models. Multivariate Behavioral Research, 49, 354-371.
32.
LordF. M.WingerskyM. S. (1984). Comparison of IRT true-score and equipercentile observed-score “equatings.”Applied Psychological Measurement, 8, 453-461.
33.
KangT.ChenT. T. (2008). Performance of the generalized S-X2 item fit index for polytomous IRT models. Journal of Educational Measurement, 45, 391-406.
34.
Maydeu-OlivaresA.JoeH. (2005). Limited- and full-information estimation and goodness-of-fit testing in 2n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009-1020.
35.
McKinleyR. L.MillsC. N. (1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9, 49-57.
36.
MicceriT. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.
37.
MurakiE.BockR. D. (1997). PARSCALE: IRT item analysis and test scoring for rating-scale data. [Computer software]. Chicago, IL: Scientific Software International.
38.
OrlandoM.ThissenD. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64.
39.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298.
40.
RangerJ.KuhnJ. T. (2012). Assessing fit of item response models using the information matrix test. Journal of Educational Measurement, 49, 247-268.
41.
ReckaseM. (2009). Multidimensional item response theory (Vol. 150). New York, NY: Springer.
42.
ReiserM. (2008). Goodness-of-fit testing using components based on marginal frequencies of multinomial data. British Journal of Mathematical and Statistical Psychology, 61, 331-360.
43.
RijmenF.JeonM.von DavierM.Rabe-HeskethS. (2014). A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 39, 235-256.
44.
RobertsJ. S. (2008). Modified likelihood-based item fit statistics for the generalized graded unfolding model. Applied Psychological Measurement, 32, 407-423.
45.
SinharayS. (2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42, 375-394.
46.
SinharayS. (2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429-449.
47.
StoneC. A. (2000). Monte Carlo based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of Educational Measurement, 37, 158-175.
48.
StoneC. A.ZhangB. (2003). Assessing goodness of fit of item response theory models: A comparison of traditional and alternative procedures. Journal of Educational Measurement, 40, 331-352.
49.
StroudA. H. (1974). Numerical quadrature and solution of ordinary differential equations. New York, NY: Springer.
50.
Suárez-FalcónJ. C.GlasC. A. W. (2003). Evaluation of global testing procedures for item fit to the Rasch model. British Journal of Mathematical and Statistical Psychology, 56, 127-143.
51.
van der LindenW. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247-272.
52.
WangC. (2014). Improving measurement precision of hierarchical latent traits using adaptive testing. Journal of Educational and Behavioral Statistics, 39, 452-477.
53.
WangC.KohliN.HennL. (2016). A second-order longitudinal model for binary outcomes: Item response theory versus structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23, 455-465.
54.
WangC.ShuZ.ShangZ.XuG. (2015). Assessing item-level fit for the DINA model. Applied Psychological Measurement, 39, 525-538.
55.
WellsC. S.BoltD. M. (2008). Investigation of a nonparametric procedure for assessing goodness-of-fit in item response theory. Applied Measurement in Education, 21, 22-40.
56.
YenW. M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245-262.
57.
ZhangB.StoneC. A. (2008). Evaluating item fit for multidimensional item response models. Educational and Psychological Measurement, 68, 181-196.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
