Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord’s Wald test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.
BallasD.TranmerM. (2012). Happy people or happy places? A multilevel modeling approach to the analysis of happiness and well-being. International Regional Science Review, 35, 70–102.
2.
BashkovB. M.DeMarsC. E. (2017). Examining the performance of the Metropolis–Hastings Robbins–Monro algorithm in the estimation of multilevel multidimensional IRT models. Applied Psychological Measurement, 41, 323–337.
3.
BeggM. D. (1999). Analyzing k (2×2) tables under cluster sampling. Biometrics, 55(1), 302–307.
4.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57, 289–300.
5.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.
6.
BrykA. S.RaudenbushS. W. (1992). Hierarchical linear models: Applications and data analysis methods. Sage.
7.
CaiL. (2008a). A Metropolis-Hastings Robbins-Monro algorithm for maximum likelihood nonlinear latent structure analysis with a comprehensive measurement model. The University of North Carolina at Chapel Hill.
8.
CaiL. (2008b). SEM of another flavour: Two new applications of the supplemented EM algorithm. British Journal of Mathematical and Statistical Psychology, 61, 309–329.
9.
CaiL. (2010a). High-dimensional exploratory item factor analysis by a Metropolis-Hastings Robbins-Monro Algorithm. Psychometrika, 75, 33–57.
10.
CaiL. (2010b). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307–335.
11.
CaiL. (2017). flexMIRT®: Flexible multilevel multidimensional item analysis and test scoring. (version 3.51) [Computer software]. Vector Psychometric Group.
12.
CaiL.Du ToitS. H. C.ThissenD. (2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling. Scientific Software International Inc.
13.
CaoM.TayL.LiuY. (2017). A Monte Carlo study of an iterative Wald test procedure for DIF analysis. Educational and Psychological Measurement, 77(1), 104–118.
14.
CheongY. F.KamataA. (2013). Centering, scale indeterminacy, and differential item functioning detection in hierarchical generalized linear and generalized linear mixed models. Applied Measurement in Education, 26(4), 233–252.
15.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39, 1–22.
16.
DonoghueJ. R.IshamS. P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, 33–51.
17.
FinchH.FrenchB. (2010). Detecting differential item functioning of a course satisfaction instrument in the presence of multilevel data. Journal of the First-Year Experience & Students in Transition, 22, 27–47.
18.
FisherR. A. (1925). Theory of statistical estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 22(5), 700–725.
19.
FrenchB. F.FinchW. H. (2013). Extensions of Mantel–Haenszel for multilevel DIF detection. Educational and Psychological Measurement, 73, 648–671.
20.
FrenchB. F.FinchW. H.ImmekusJ. C. (2019). Multilevel generalized Mantel-Haenszel for differential item functioning detection. Frontiers in Education, 47.
21.
HanleyC. (2020). Assessing the Performance of Two Procedures for Detecting Differential Item Functioning within the Multilevel Partial Credit Model [Doctoral dissertation, University of Kentucky].
22.
HansenM.CaiL.StuckyB. D.TuckerJ. S.ShadelW. G.EdelenM. O. (2014). Methodology for developing and evaluating the PROMIS® smoking item banks. Nicotine & Tobacco Research, 16, S175–S189.
23.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.
24.
HollandP. W.ThayerD. T. (1988). Differential item performance and the Mantel–Haenszel procedure. In WainerH.BrauneH. I. (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.
HuangF. L.WiedermannW.ZhangB. (2022). Accounting for heteroskedasticity resulting from between-group differences in multilevel models. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2022.2077290
27.
HuangS. (2021). Estimation of cross-classified multilevel item response theory models with Metropolis-Hastings Robbins-Monro Algorithm [Doctoral dissertation, University of California, Los Angeles].
28.
JinY.MyersN. D.AhnS. (2014). Complex versus simple modeling for DIF detection: When the Intraclass correlation coefficient (ρ) of the studied item is less than the ρ of the Total score. Educational and Psychological Measurement, 74, 163–190.
29.
KamataA. (2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79–93.
30.
KimS.-H.CohenA. S.KimH.-O. (1994). An investigation of Lord’s procedure for the detection of differential item functioning. Applied Psychological Measurement, 18, 217–228.
31.
KimY. K.SaxL. J. (2011). Are the effects of student–faculty interaction dependent on academic major? An examination using multilevel modeling. Research in Higher Education, 52, 589–615.
32.
LangerM. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation [Unpublished doctoral dissertation]. The University of North Carolina at Chapel Hill.
33.
LimR. G.DrasgowF. (1990). Evaluation of two methods for estimating item response theory parameters when assessing differential item functioning. Journal of Applied Psychology, 75, 164–174.
34.
LorahJ. (2018). Effect size measures for multilevel models: Definition, interpretation, and TIMSS example. Large-Scale Assessments in Education, 6, 1–11.
35.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.
36.
LouisT. A. (1982). Finding the observed information matrix when using the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 44, 226–233.
37.
MantelN.HaenszelW. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719–748.
38.
McLaughlinM. E.DrasgowF. (1987). Lord’s chi-square test of item bias with estimated and with known person parameters. Applied Psychological Measurement, 11, 161–173.
39.
MengX.-L.RubinD. B. (1991). Using EM to obtain asymptotic variance-covariance matrices: The SEM algorithm. Journal of the American Statistical Association, 86, 899–909.
MetropolisN.RosenbluthA. W.RosenbluthM. N.TellerA. H.TellerE. (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21, 1087–1092.
NarayanonP.SwaminathanH. (1996). Identification of items that show nonuniform DIF. Applied Psychological Measurement, 20, 257–274.
44.
PenfieldR. D. (2001). Assessing differential item functioning among multiple groups: A comparison of three Mantel-Haenszel procedures. Applied Measurement in Education, 14(3), 235–259.
45.
RobbinsH.MonroS. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3) 400–407.
46.
RutkowskiL.SvetinaD. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74, 31–57.
47.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97.
48.
SomesG. W. (1986). The generalized Mantel–Haenszel statistic. The American Statistician, 40(2), 106–108.
49.
SturmA.HuangS.KuhfeldM. (2022). Advancing methodologies to improve RRB outcome measures in autism research: Evaluation of the RBS-R. Psychological Assessment, 34, 30–42. https://doi.org/https://doi.org/10.1037/pas0001062
50.
SturmA.KuhfeldM.KasariC.McCrackenJ. T. (2017). Development and validation of an item response theory-based social responsiveness scale short form. Journal of Child Psychology and Psychiatry, 58, 1053–1061.
51.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.
52.
TayL.MeadeA. W.CaoM. (2015). An overview and practical guide to IRT measurement equivalence analysis. Organizational Research Methods, 18(1), 3–46.
53.
ThissenD.SteinbergL.GerrardM. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99, 118–128.
54.
ThissenD.SteinbergL.KuangD. (2002). Quick and easy implementation of the Benjamini-Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27, 77–83.
55.
ThissenD.SteinbergL.WainerH. (1993). Detection of differential item functioning using the parameters of item response models. In WainerH.HollandP. W. (Eds.), Differential item functioning (pp. 67–113). Lawrence Erlbaum.
56.
WoodsC. M.CaiL.WangM. (2013). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73, 532–547.
57.
YangJ. S.CaiL. (2014). Estimation of contextual effects through nonlinear multilevel latent variable modeling with a Metropolis–Hastings Robbins–Monro algorithm. Journal of Educational and Behavioral Statistics, 39, 550–582.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
