Applied and simulation studies document model convergence and accuracy issues in differential item functioning detection with multilevel models, hindering detection. This study aimed to evaluate the effectiveness of various estimation techniques in addressing these issues and ensure robust DIF detection. We conducted a simulation study to investigate the performance of multilevel logistic regression models with predictors at level 2 across different estimation procedures, including maximum likelihood estimation (MLE), Bayesian estimation, and generalized estimating equations (GEE). The simulation results demonstrated that all maintained control over the Type I error rate across conditions. In most cases, GEE had comparable or higher power compared to MLE for identifying DIF, with Bayes having the lowest power. When potentially important covariates at levels-1 and 2 were included in the model, power for all methods was higher. These results suggest that in many cases where multilevel logistic regression is used for DIF detection, GEE offers a viable option for researchers and that including important contextual variables at all levels of the data is desirable. Implications for practice are discussed.
AlbanoA. D.RodriguezM. C. (2013). Examining differential math performance by gender and opportunity to learn. Educational and Psychological Measurement, 73(5), 836–856. https://doi.org/10.1177/0013164413487375
2.
AnastasiA. (1993). A century of psychological testing: Origins, problems, and progress. In FaganT. K.VandenBosG. R. (Eds.), Exploring applied psychology: Origins and critical analyses (pp. 9–36): American Psychological Association. https://doi.org/10.1037/11104-001
3.
BerríoÁ. I.Gomez-BenitoJ.Arias-PatiñoE. M. (2020). Developments and trends in research on methods of detecting differential item functioning. Educational Research Review, 31(1), Article 100340. https://doi.org/10.1016/j.edurev.2020.100340
4.
BronfenbrennerU. (1994). Ecological models of human development. In GauvinM.ColeM. (Eds.), Readings on the development of children (4th ed., Vol. 2, pp. 37–43). Worth Publishers.
5.
BrowneW. J.DraperD. (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1(3), 473–514.
6.
ChaimongkolS. (2005). Modeling differential item functioning (DIF) using multilevel logistic regression models: A bayesian perspective. The Florida State University.
7.
ChenM. Y.ZumboB. D. (2017). Ecological framework of item responding as validity evidence: An application of multilevel DIF modeling using PISA data. In ZumboB. D.HubleyA. M. (Eds.), Understanding and investigating response processes in validation research (69, pp. 53–68). Springer. https://doi.org/10.1007/978-3-319-56129-5_4
8.
Elena OliveriM.LawlessR.RobinF.BridgemanB. (2018). An exploratory analysis of differential item functioning and its possible sources in a higher education admissions context. Applied Measurement in Education, 31(1), 1–16. https://doi.org/10.1080/08957347.2017.1391258
9.
EstajiM.ZhalehK. (2020). Does field of study matter in academic performance: Differential item functioning analysis of a high-stakes test using one-parameter and two-parameter item response theory models. Iranian Journal of English for Academic Purposes, 9(3), 14–31.
10.
FerneT.RuppA. A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4(2), 113–148. https://doi.org/10.1080/15434300701375923
11.
FrenchB. F.FinchW. H. (2010). Hierarchical logistic regression: Accounting for multilevel data in DIF detection. Journal of Educational Measurement, 47(3), 299–317. https://doi.org/10.1111/j.1745-3984.2010.00115.x
12.
FrenchB. F.FinchW. H. (2013). Extensions of mantel–haenszel for multilevel DIF detection. Educational and Psychological Measurement, 73(4), 648–671. https://doi.org/10.1177/0013164412472341
13.
FrenchB. F.FinchW. H.ImmekusJ. C. (2019). Multilevel generalized Mantel-Haenszel for differential item functioning detection. In Frontiers in Education (4, p. 47). Frontiers Media SA. https://doi.org/10.3389/feduc.2019.00047
14.
HalekohU.HøjsgaardS.YanJ. (2006). The R package geepack for generalized estimating equations. Journal of Statistical Software, 15(2), 1–11. https://doi.org/10.18637/jss.v015.i02
15.
HoxJ. J.MaasC. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8(2), 157–174. https://doi.org/10.1207/s15328007sem0802_1
16.
HuelmannT.DebelakR.StroblC. (2020). A comparison of aggregation rules for selecting anchor items in multigroup DIF analysis. Journal of Educational Measurement, 57(2), 185–215. https://doi.org/10.1111/jedm.12246
17.
JinY.MyersN. D.AhnS. (2014). Complex versus simple modeling for DIF detection: When the intraclass correlation coefficient (ρ) of the studied item is less than the ρ of the total score. Educational and Psychological Measurement, 74(1), 163–190. https://doi.org/10.1177/0013164413497572
18.
KanA.BulutO. (2014). Examining the relationship between gender DIF and language complexity in mathematics assessments. International Journal of Testing, 14(3), 245–264. https://doi.org/10.1080/15305058.2013.877911
19.
LiH.HunterC. V.BialoJ. A. (2022). A revisit of Zumbo’s third generation DIF: How are we doing in language testing?Language Assessment Quarterly, 19(1), 27–53. https://doi.org/10.1080/15434303.2021.1963253
MúthenB.KaplanD.HollisM. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52(3), 431–462. https://doi.org/10.1007/BF02294365
22.
MuthénB. O.SatorraA. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25(1), 267–316. https://doi.org/10.2307/271070
23.
MuthénL. K.MuthénB. O. (2021). Mplus: Statistical Analysis with Latent Variables: User ’s Guide (Version 8). Los Angeles, CA: Authors.
24.
NortonE. C.BielerG. S.EnnettS. T.ZarkinG. A. (1996). Analysis of prevention program effectiveness with clustered data using generalized estimating equations. Journal of Consulting and Clinical Psychology, 64(5), 919–926. https://doi.org/10.1037//0022-006x.64.5.919
25.
OECD. (2018). PISA 2015 Results in Focus. OECD Publishing.
26.
OliveriM. E.ErcikanK.ZumboB. (2013). Analysis of sources of latent class differential item functioning in international assessments. International Journal of Testing, 13(3), 272–293. https://doi.org/10.1080/15305058.2012.738266
27.
PrietoG.NietoE. (2014). Influence of DIF on differences in performance of Italian and Asian individuals on a reading comprehension test of Spanish as a foreign language (negative emotionality) in Hong Kong. Journal of Applied Measurement, 15(2), 176–188.
28.
RajuN. S. (1988). The area between two item characteristic curves. Psychometrika, 54, 495–502.
29.
Rashvand SemiyariS.AhangariS. (2022). Examining Differential Item Functioning (DIF) for Iranian EFL test takers with different fields of study. Research in English Language Pedagogy, 10(1), 169–190.
30.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. https://doi.org/10.1007/bf02294572
31.
ShearB. R. (2018). Using hierarchical logistic regression to study DIF and DIF variance in multilevel data: DIF and DIF variance in multilevel data. Journal of Educational Measurement, 55(4), 513–542. https://doi.org/10.1111/jedm.12190
32.
VaughnB. K. (2006). Hierarchical generalized linear model of random differential item functioning for polytomous items: A bayesian multilevel approach.
33.
VoT. T.FrenchB. F. (2021). An ecological framework for item responding within the context of a youth risk and needs assessment. Educational measurement: Issues and practice, 40(2), 64–72.
34.
WoitschachP.ZumboB. D.Fernández-AlonsoR. (2019). An ecological view of measurement: Focus on multilevel model explanation of differential item functioning. Psicothema, 31(2), 194–203. https://doi.org/10.7334/psicothema2018.303
35.
ZumboB. D. (2003). Does item-level DIF manifest itself in scale-level analyses? Implications for translating language tests. Language Testing, 20(2), 136–147. https://doi.org/10.1191/0265532203lt248oa
36.
ZumboB. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
37.
ZumboB. D.LiuY.WuA. D.ShearB. R.Olvera AstiviaO. L.ArkT. K. (2015). A methodology for Zumbo’s third generation DIF analyses and the ecology of item responding. Language Assessment Quarterly, 12(1), 136–151. https://doi.org/10.1080/15434303.2014.972559
