Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies. Previous research showed that the RMSD was unable to detect departures from MI when the latent trait distribution was far from item difficulty. In this study, we developed three alternative approaches to the original RMSD: equal, item information, and b-norm weighted RMSDs. Specifically, we considered the item-centered normalized weight distributions to compute the item characteristic curve difference in the RMSD procedure more efficiently. We further compared all methods’ performance via a simulation study and the item information and b-norm weighted RMSDs showed the most promising results. An empirical example is demonstrated, and implications for researchers are discussed.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores. Addison-Wesley.
2.
BuchholzJ.HartigJ. (2019). Comparing attitudes across groups: An IRT-based item-fit statistic for the analysis of measurement invariance. Applied Psychological Measurement, 43, 241–250.
3.
DonoghueJ. R. (1994). An empirical examination of the IRT information of polytomously scored reading items under the generalized partial credit model. Journal of Educational Measurement, 31, 295–311.
4.
FletcherR.ReevesC. M. (1964). Function minimization by conjugate gradients. The Computer Journal, 7, 149–154.
5.
JooS.AliU.RobinF.ShinH. J. (2022). Impact of differential item functioning on group score reporting in the context of large-scale assessments. Large-Scale Assessments in Education, 10, 1–21.
6.
JooS.KhorramdelL.YamamotoK.ShinH. J.RobinF. (2021). Evaluating item fit statistic thresholds in PISA: Analysis of cross-country comparability of cognitive items. Educational Measurement: Issues and Practice, 40, 37–48.
7.
JooS.LeeP. (2022). Detecting differential item functioning using posterior predictive model checking: A comparison of discrepancy statistics. Journal of Educational Measurement, 59, 442–469.
8.
JooS.LeeP.StarkS. (2022). Bayesian approaches for detecting differential item functioning using the generalized graded unfolding model. Applied Psychological Measurement, 46, 98–115.
9.
KöhlerC.RobitzschA.FährmannK.von DavierM.HartigJ. (2021). A semiparametric approach for item response function estimation to detect item misfit. British Journal of Mathematical and Statistical Psychology, 74, 157–175.
10.
KöhlerC.RobitzschA.HartigJ. (2020). A bias-corrected RMSD item fit statistic: An evaluation and comparison to alternatives. Journal of Educational and Behavioral Statistics, 45, 251–273.
11.
KönigC.KhorramdelL.YamamotoK.FreyA. (2021). The benefits of fixed item parameter calibration for parameter accuracy in small sample situations in large-scale assessments. Educational Measurement: Issues and Practice, 40, 17–27.
12.
Lopez-RivasG.StarkS.ChernyshenkoO. (2008). The effects of referent item parameters on differential item functioning detection using the free baseline likelihood ratio test. Applied Psychological Measurement, 33, 251–265.
13.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.
R Core Team. (2020). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing.
17.
RobitzschA. (2022). Statistical properties of estimators of the RMSD item fit statistic. Foundations, 2, 488–503.
18.
RobitzschA.LüdtkeO. (2022). Mean comparisons of many groups in the presence of DIF: An evaluation of linking and concurrent scaling approaches. Journal of Educational and Behavioral Statistics, 47, 36–68.
19.
RutkowskiD.RutkowskiL. (2021). Running the wrong race? The case of PISA for development. Comparative Education Review, 65, 147–165.
20.
RutkowskiL.SvetinaD. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74, 31–57.
21.
StarkS.ChernyshenkoO. S.DrasgowF. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292.
22.
SvetinaD.RutkowskiL. (2014). Detecting differential item functioning using generalized logistic regression in the context of large-scale assessments. Large-Scale Assessments in Education, 2, 1–17.
23.
TijmstraJ.BolsinovaM.LiawY. L.RutkowskiL.RutkowskiD. (2020). Sensitivity of the RMSD for detecting item-level misfit in low-performing countries. Journal of Educational Measurement, 57, 566–583.
24.
von DavierM. (2005). mdltm: Software for the general diagnostic model and for estimating mixtures of multidimensional discrete latent traits models [Computer software]. ETS.
von DavierM.YamamotoK.ShinH. J.ChenH.KhorramdelL.WeeksJ.DavisS.KongN.KandathilM. (2019). Evaluating item response theory linking and model fit for data from PISA 2000–2012. Assessment in Education: Principles, Policy & Practice, 26, 466–488.
27.
WangM.WoodsC. M. (2017). Anchor selection using the Wald test anchor-all-test-all procedure. Applied Psychological Measurement, 41, 17–29.
28.
WangW. C.YehY. L. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479–498.
