Differential item functioning (DIF) is often used to examine validity evidence of alternate form test accommodations. Unfortunately, traditional approaches for evaluating DIF are prone to selection bias. This article proposes a novel DIF framework that capitalizes on regression discontinuity design analysis to control for selection bias. A simulation study was performed to compare the new framework with traditional logistic regression, with respect to Type I error and power rates of the uniform DIF test statistics and bias and root mean square error of the corresponding effect size estimators. The new framework better controlled the Type I error rate and demonstrated minimal bias but suffered from low power and lack of precision. Implications for practice are discussed.
AikensN.WestJ.McKeeK.MoiduddinE.Atkins-BurnettS.XueY. (2020). Screening approaches for determining the language of assessment for dual language learners: Evidence from Head Start and a universal preschool initiative. Early Childhood Research Quarterly, 51, 39–54. https://doi.org/10.1016/j.ecresq.2019.07.008
2.
BandelE.Atkins-BurnettS.CastroD.Smither WulsinC.PutnamM. (2012, June). Examining the use of language and literacy assessments with young dual language learners [Report submitted to the University of North Carolina, FPG Child Development Institute, Center for Early Care and Education—Dual Language Learners]. Mathematica Policy Research.
CalonicoS.CattaneoM. D.FarrellM. H.TitiunikR. (2021). rdrobust: Robust data-driven statistical inference in regression-discontinuity designs [Computer software] (R Package Version 1.0.7). https://CRAN.R-project.org/package=rdrobust
5.
CattaneoM. D.IdroboN.TitiunikR. (2020a). A practical introduction to regression discontinuity designs: Extensions. Cambridge University Press.
6.
CattaneoM. D.IdroboN.TitiunikR. (2020b). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press.
7.
ChenM. Y.LiuY.ZumboB. D. (2020). A propensity score method for investigating differential item functioning in performance assessment. Educational and Psychological Measurement, 80(3), 476–498. https://doi.org/10.1177/0013164419878861
8.
ClauserB. E.NungesterR. J.SwaminathanH. (1996). Improving the matching for DIF analysis by conditioning on both test score and an educational background variable. Journal of Educational Measurement, 33(4), 453–464. https://doi.org/10.1111/j.1745-3984.1996.tb00501.x
9.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
10.
DeMarsC. E. (2009). Modification of the Mantel-Haenszel and logistic regression DIF procedures to incorporate the SIBTEST regression correction. Journal of Educational and Behavioral Statistics, 34(2), 149–170. https://doi.org/10.3102/1076998607313923
11.
DeMarsC. E. (2010). Type I error inflation for detecting DIF in the presence of impact. Educational and Psychological Measurement, 70(6), 961–972. https://doi.org/10.1177/0013164410366691
12.
FidalgoA. M.AlaviS. M.AmirianS. M. R. (2014). Strategies for testing statistical and practical significance in detecting DIF with logistic regression models. Language Testing, 31(4), 433–451. https://doi.org/10.1177/0265532214526748
GierlM. J.GotzmannA.BoughtonK. A. (2004). Performance of SIBTEST when the percentage of DIF items is large. Applied Measurement in Education, 17(3), 241–264. https://doi.org/10.1207/s15324818ame1703_2
15.
GoodrichJ. M.KoziolN. A.YoonH. (2021). Are translated mathematics items a valid accommodation for dual language learners? Evidence from ECLS-K. Early Childhood Research Quarterly, 57, 89–101. https://doi.org/10.1016/j.ecresq.2021.06.001
HidalgoM. D.Gomez-BenitoJ.ZumboB. D. (2014). Binary logistic regression analysis for detecting differential item functioning: Effectiveness of R2 and delta log odds ratio effect size measures. Educational and Psychological Measurement, 74(6), 927–949. https://doi.org/10.1177/0013164414523618
18.
HollandP. W.ThayerD. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.
19.
HollandP. W.WainerH. (1993). Differential item functioning. Lawrence Erlbaum.
20.
JodoinM. G.GierlM. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349. https://doi.org/10.1207/S15324818AME1404_2
21.
LeeD. S.LemieuxT. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. https://doi.org/10.1257/jel.48.2.281
22.
LeeS. (2017). Detecting differential item functioning using the logistic regression procedure in small samples. Applied Psychological Measurement, 41(1), 30–43. https://doi.org/10.1177/0146621616668015
23.
LhamonC. E.GuptaV. (2015, January7). Dear Colleague Letter: English learner students and limited English proficient parents. Office for Civil Rights, U.S. Department of Education, Civil Rights Division, U.S. Department of Justice.
24.
LiY.BrooksG. P.JohansonG. A. (2012). Item discrimination and Type I error in the detection of differential item functioning. Educational and Psychological Measurement, 72(5), 847–861. https://doi.org/10.1177/0013164411432333
25.
LiZ. (2014). Power and sample size calculations for logistic regression tests for differential item functioning. Journal of Educational Measurement, 51(4), 441–462. https://doi.org/10.1111/jedm.12058
26.
LiuY.KimC.WuA. D.GustafsonP.KrocE. (2019). Investigating the performance of propensity score approaches for differential item functioning analysis. Journal of Modern Applied Statistical Methods, 18(1), eP2744. https://doi.org/10.22237/jmasm/1556669280
27.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.
28.
MagisD.De BoeckP. (2012). A robust outlier approach to prevent Type I error inflation in differential item functioning. Educational and Psychological Measurement, 72(2), 291–311. https://doi.org/10.1177/0013164411416975
29.
MazorK. M.ClauserB. E.HambletonR. K. (1994). Identification of non-uniform differential item functioning using a variation of the Mantel-Haenszel procedure. Educational and Psychological Measurement, 54(2), 284–291. https://doi.org/10.1177/0013164494054002003
30.
MellenbergG. J. (1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7(2), 105–108. https://doi.org/10.2307/1164960
MonahanP. O.McHorneyC. A.StumpT. E.PerkinsA. J. (2007). Odds ratio, delta, ETS classification, and standardization measures of DIF magnitude for binary logistic regression. Journal of Educational and Behavioral Statistics, 32(1), 92–109. https://doi.org/10.3102/1076998606298035
33.
MuthénL. K.MuthénB. O. (1998–2020). Mplus user’s guide (Version 8.5).
34.
NajarianM.SnowK.LennonJ.KinseyS. (2010). Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), preschool-kindergarten 2007 psychometric report (NCES 2010-009). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
35.
NajarianM.TourangeauK.NordC.Wallner-AllenK. (2018). Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011), kindergarten psychometric report (NCES 2018-182). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
36.
NarayananP.SwaminathanH. (1996). Identification of items that show nonuniform DIF. Applied Psychological Measurement, 20(3), 257–274. https://doi.org/10.1177/014662169602000306
PetersenM. A.GroenvoldM.BjornerJ. B.AaronsonN.ConroyT.CullA.FayersP.HjermstadM.SprangersM.SullivanM. (2003). Use of differential item functioning analysis to assess the equivalence of translations of a questionnaire. Quality of Life Research, 12(4), 373–385. https://doi.org/10.1023/a:1023488915557
39.
PitoniakM. J.YoungJ. W.MartinielloM.KingT. C.ButeuxA.GinsburghM. (2009). Guidelines for the assessment of English-Language Learners (ETS Office of Professional Standards Compliance’s Fairness Series). Educational Testing Service. https://www.ets.org/s/about/pdf/ell_guidelines.pdf
40.
R Core Team. (2019). R: A language and environment for statistical computing [Computer software] (Version 3.6.1). R Foundation for Statistical Computing. https://www.R-project.org/
41.
RockD. A.PollackJ. M. (2002). Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), psychometric report for kindergarten through first grade (NCES 2002-05). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
42.
RogersH. J.SwaminathanH. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105–116. https://doi.org/10.1177/014662169301700201
43.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
44.
SalviaJ.YsseldykeJ.WitmerS. (2017). Assessment in special and inclusive education. Cengage Learning.
ShealyR. T.StoutW. F. (1993). A model-biased standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. https://doi.org/10.1007/BF02294572
47.
ShihC.-L.LiuT.-H.WangW.-C. (2014). Controlling type I error rates in assessing DIF for logistic regression method combined with SIBTEST regression correction procedure and DIF-free-then-DIF strategy. Educational and Psychological Measurement, 74(6), 1018–1048. https://doi.org/10.1177/0013164413520545
48.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
49.
ThistlethwaiteD. L.CampbellD. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319
50.
U.S. Department of Education. (2007). Title I: Improving the academic achievement of the disadvantaged: Individuals with Disabilities Act (IDEA); Final rule. Federal Register, 72(67), 17747–17781.
WuA. D.LiuY.StoneJ. E.ZouD.ZumboB. D. (2017). Is difference in measurement outcome between groups differential responding, bias or disparity? A methodology for detecting bias and impact from an attributional stance. Frontiers in Education, 2, Article 39. https://doi.org/10.3389/feduc.2017.00039
ZiekyM. (1993). Practical questions in the use of DIF statistics in test development. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 337–347). Lawrence Erlbaum.
55.
ZumboB. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Directorate of Human Resources Research and Evaluation, Department of National Defense.
56.
ZwickR. (2012). A review of ETS differential item functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement (Research Report ETS RR-12-08; pp. 1–30). Educational Testing Service.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
