The Mantel–Haenszel estimator is one of the most popular techniques for measuring differential item functioning (DIF). A generalization of this estimator is applied to the context of DIF to compare items by taking the covariance of odds ratio estimators between dependent items into account. Unlike the Item Response Theory, the method does not rely on the local item independence assumption which is likely to be violated when one item provides clues about the answer of another item. Furthermore, we use these (co)variance estimators to construct a hypothesis test to assess DIF for multiple items simultaneously. A simulation study is presented to assess the performance of several tests. Finally, the use of these DIF tests is illustrated via application to two real data sets.
AgrestiA.LiuI.-M. (1999). Modeling a categorical variable allowing arbitrarily many category choices. Biometrics, 55(3), 936–943.
2.
AgrestiA.LiuI.-M. (2001). Strategies for modeling a categorical variable allowing multiple category choices. Sociological Methods & Research, 29(4), 403–434.
3.
AndersenE. B. (1980). Discrete statistical models with social science applications. North-Holland Publishing Company.
4.
BaghaeiP. (2008). Local dependency and Rasch measures. Rasch Measurement Transactions, 21(3), 1105–1106.
5.
BarthélemyJ.SuesseT. (2018). mipfp: An R package for multidimensional array fitting and simulating multivariate Bernoulli distributions. Journal of Statistical Software, 86, 1–20.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300.
8.
BenjaminiY.KriegerA. M.YekutieliD. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika, 93(3), 491–507.
9.
BradlowE. T.WainerH.WangX. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153–168.
10.
BreslowN. (1981). Odds ratio estimators when the data are sparse. Biometrika, 68(1), 73–84.
11.
CamilliG.ShepardL. A.ShepardL. (1994). Methods for identifying biased test items (Vol. 4). SAGE.
12.
ChenY.-F.JiaoH. (2014). Exploring the utility of background and cognitive variables in explaining latent differential item functioning: An example of the PISA 2009 reading assessment. Educational Assessment, 19(2), 77–96.
13.
ChristensenK. B.MakranskyG.HortonM. (2017). Critical values for Yen’s Q 3: Identification of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178–194.
14.
ClauserB.MazorK.HambletonR. K. (1993). The effects of purification of matching criterion on the identification of DIF using the Mantel-Haenszel procedure. Applied Measurement in Education, 6(4), 269–279.
15.
DoransN. J. (1989). Two new approaches to assessing differential item functioning: Standardization and the Mantel–Haenszel method. Applied Measurement in Education, 2(3), 217–233.
16.
DoransN. J.HollandP. W. (1992). DIF detection and description: Mantel-Haenszel and Standardization. ETS Research Report Series, 1992(1), 1–40.
17.
DouglasJ. A.RoussosL. A.StoutW. (1996). Item-bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33(4), 465–484.
18.
EdwardsM. C.HoutsC. R.CaiL. (2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23(1), 138–149.
19.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. Chapman & Hall.
20.
FrenchB. F.MallerS. J. (2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67(3), 373–393.
21.
GreenlandS. (1989). Generalized Mantel-Haenszel estimators for tables. Biometrics, 45(1), 183–191.
22.
HemmelmannC.HornM.SüsseT.VollandtR.WeissS. (2005). New concepts of multiple tests and their use for evaluating high-dimensional EEG data. Journal of Neuroscience Methods, 142(2), 209–217.
23.
HollandP. W.ThayerD. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Erlbaum.
24.
HolmS. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
25.
IpE. H.-s. (2001). Testing for local dependency in dichotomous and polytomous item response models. Psychometrika, 66(1), 109–132.
26.
KhorramdelL.PokropekA.JooS.-H.KirschI.HaldermanL. (2020). Examining gender DIF and gender differences in the PISA 2018 reading literacy scale: A partial invariance approach. Psychological Test and Assessment Modeling, 62(2), 179–231.
27.
KimJ.OshimaT. C. (2013). Effect of multiple testing adjustment in differential item functioning detection. Educational and Psychological Measurement, 73, 458–470.
28.
KopfJ.ZeileisA.StroblC. (2015). Anchor selection strategies for DIF analysis: Review, assessment, and new approaches. Educational and Psychological Measurement, 75, 22–56.
29.
LeL. T. (2009). Investigating gender differential item functioning across countries and test languages for PISA science items. International Journal of Testing, 9(2), 122–133.
30.
LeeY.-W. (2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21(1), 74–100.
31.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30(1), 3–21.
32.
LiuI.AgrestiA. (1996). Mantel-Haenszel-type inference for cumulative odds ratios with a stratified ordinal response. Biometrics, 52(4), 1223–1234.
33.
LiuI.SuesseT. (2008). The analysis of stratified multiple responses. Biometrical Journal, 50(1), 135–149.
34.
LordF. M. (1980). Applications of item response theory to practical testing problems (1st ed.). Routledge, Taylor and Francis Group.
35.
LoughinT. M.SchererP. N. (1998). Testing for association in contingency tables with multiple column responses. Biometrics, 54(2), 630–637.
MagisD.BélandS.TuerlinckxF.De BoeckP. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862.
38.
MagisD.TuerlinckxF.BoeckP. D. (2015). Detection of differential item functioning using the Lasso approach. Journal of Educational and Behavioral Statistics, 40(2), 111–135.
39.
MantelN.HaenszelW. (1959). Statistical aspects of the analysis of data from retrospective studies. Journal of the National Cancer Institute, 22(4), 719–748.
40.
MillsapR. E.EversonH. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17(4), 297–334.
41.
Organisation for Economic Co-operation and Development. (2012a). Data base-PISA 2012.
OsterlindS. J.EversonH. T. (2009). Differential item functioning (2nd ed.). SAGE.
45.
PaekI. (2012). A note on three statistical tests in the logistic regression DIF procedure. Journal of Educational Measurement, 49(2), 121–126.
46.
PaekI.HollandP. (2015). A note on statistical hypothesis testing based on log transformation of the Mantel-Haenszel common odds ratio for differential item functioning classification. Psychometrika, 80(2), 406–411.
47.
ParkS. E.AhnS.ZopluogluC. (2021). Differential item functioning effect size from the multigroup confirmatory factor analysis for a meta-analysis: A simulation study. Educational and Psychological Measurement, 81(1), 182–199.
48.
PenfieldR. D. (2001). Assessing differential item functioning among multiple groups: A comparison of three Mantel-Haenszel procedures. Applied Measurement in Education, 14, 235–259.
49.
PenfieldR. D.AlginaJ. (2003). Applying the Liu-Agresti estimator of the cumulative common odds ratio to DIF detection in polytomous items. Journal of Educational Measurement, 40(4), 353–370.
50.
RaschG. (1961). On general laws and the meaning of measurement in psychology. In: NeymanJ (ed) Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability (Vol. 4, pp. 321–333), Berkeley, CA: University of California Press.
51.
RijmenF. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47(3), 361–372.
52.
RogersH. J.SwaminathanH. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17, 105–116.
53.
RothmanK. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology, 1(1), 43–46.
54.
RoussosL. A.StoutW. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel type I error performance. Journal of Educational Measurement, 33(2), 215–230.
55.
SauderD.DeMarsC. (2020). Applying a multiple comparison control to IRT item-fit testing. Applied Measurement in Education, 33(4), 362–377.
56.
SchervishM. J. (1996). P values: What they are and what they are not. The American Statistician, 50(3), 203–206.
57.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194.
58.
SochaA.DeMarsC. E.ZilberbergA.PhanH. (2015). Differential item functioning detection with the Mantel-Haenszel procedure: The effects of matching types and other factors. International Journal of Testing, 15(3), 193–215.
59.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370.
60.
ThissenD.SteinbergL.KuangD. (2002). Quick and easy implementation of the Benjamini-Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27, 77–83.
61.
WainerH.BradlowE. T.WangX. (2007). Testlet response theory and its applications. Cambridge University Press.
62.
WangT.MerkleE. C.ZeileisA. (2014). Score-based tests of measurement invariance: Use in practice. Frontiers in Psychology, 5, Article 438.
63.
WangW.-C. (2004). Effects of anchor item methods on the detection of differential item functioning within the family of Rasch models. The Journal of Experimental Education, 72(3), 221–261.
64.
WangW.-C.ShihC.-L.SunG.-W. (2012). The DIF-free-then-DIF strategy for the assessment of differential item functioning. Educational and Psychological Measurement, 72(4), 687–708.
65.
WellekS. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). CRC Press.
66.
ZumboB. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Directorate of Human Resources Research and Evaluation, Department of National Defense.
67.
ZwickR.YeL.IshamS. (2012). Improving Mantel-Haenszel DIF estimation through Bayesian updating. Journal of Educational and Behavioral Statistics, 37(5), 601–629.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
