On the Complex Sources of Differential Item Functioning: A Comparison of Three Methods

Abstract

Differential item functioning (DIF) has been a long-standing problem in educational and psychological measurement. In practice, the source from which DIF originates can be complex in the sense that an item can show DIF on multiple background variables of different types simultaneously. Although a variety of non-item response theory-(IRT)-based and IRT-based DIF detection methods have been introduced, they do not sufficiently address the issue of DIF evaluation when its source is complex. The recently proposed least absolute shrinkage and selection operator (LASSO) regularization method has shown promising results of detecting DIF on multiple background variables. To provide more insight, in this study, we compared three DIF detection methods, including the non-IRT-based logistic regression (LR), the IRT-based likelihood ratio test (LRT), and LASSO regularization, through a comprehensive simulation and an empirical data analysis. We found that when multiple background variables were considered, the Type I error and Power rates of the three methods for identifying DIF items on one of the variables depended on not only the sample size and its DIF magnitude but also on the DIF magnitude of the other background variable and the correlation between them. We presented other findings and discussed the limitations and future research directions in this paper.

Keywords

differential item functioning logistic regression likelihood ratio test regularization complex DIF source

Get full access to this article

View all access options for this article.

References

Bauer

D. J.

(2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507. https://doi.org/10.1037/met0000077

Bauer

D. J.

(2023). Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning. British Journal of Mathematical and Statistical Psychology, 76(3), 435–461. https://doi.org/10.1111/bmsp.12316

Bauer

D. J.

Belzak

W. C. M.

Cole

V. T.

(2019). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754

Belzak

W. C. M.

(2023). The regDIF R package: Evaluating complex sources of measurement bias using regularized differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 30(6), 974–984. https://doi.org/10.1080/10705511.2023.2170235

Belzak

W. C. M.

Bauer

D. J.

(2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25(6), 673. https://doi.org/10.1037/met0000253

Belzak

W. C. M.

Bauer

D. J.

(2024). Using regularization to identify measurement bias across multiple background characteristics: A penalized expectation–maximization algorithm. Journal of Educational and Behavioral Statistics, 49(6), 976–1012 . https://doi-org/10.3102/10769986231226439

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.

Curran

P. J.

Cole

V. T.

Bauer

D. J.

Hussong

A. M.

Gottfredson

(2016). Improving factor score estimation through the use of observed background characteristics. Structural Equation Modeling, 23, 827–844. https://doi.org/10.1080/10705511.2016.1220839

10.

Curran

P. J.

Cole

V. T.

Bauer

D. J.

Rothenberg

W. A.

Hussong

A. M.

(2018). Recovering predictor–criterion relations using covariate-informed factor score estimates. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 860–875. https://doi.org/10.1080/10705511.2018.1473773

11.

Finch

W. H.

Hernández Finch

M. E.

(2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973–993. https://doi.org/10.1177/0013164413494776

12.

Goldman

S. H.

Raju

N. S.

(1986). Recovery of one- and two-parameter logistic item parameters: An empirical study. Educational and Psychological Measurement, 46(1), 11–21 . https://doi-org.proxy180.nclive.org/10.1177/0013164486461002

13.

Hastie

Tibshirani

Friedman

(2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

14.

Holland

P. W.

(1985). On the study of differential item performance without IRT. Proceedings of the Military Testing Association, 1, 282–287.

15.

Holland

P. W.

Wainer

(2012). Differential item functioning. Routledge. https://doi.org/10.4324/9780203357811.

16.

Holm

(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 65–70. https://doi.org/10.2307/4615733

17.

Huang

Svetina Valdivia

(2024). Wald χ2 test for differential item functioning detection with polytomous items in multilevel data. Educational and Psychological Measurement, 84(3), 530–548. https://doi.org/10.1177/00131644231181688

18.

Kim

E. S.

Yoon

(2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling: A Multidisciplinary Journal, 18(2), 212–228. https://doi.org/10.1080/10705511.2011.557337

19.

Lee

Kim

K. Y.

(2025). A comparison of anchor selection strategies for DIF analysis. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12429

20.

Lim

Choe

E. M.

Han

K. T.

(2022). A residual-based differential item functioning detection framework in item response theory. Journal of Educational Measurement, 59(1), 80–104. https://doi.org/10.1111/jedm.12313

21.

Maenner

M. J.

Shaw

K. A.

Bakian

A. V.

Bilder

D. A.

Durkin

M. S.

Esler

Furnier

S. M.

Hall-Lande

Hallas-Muchow

Hudson

Hughes

M. M.

Patrick

Pierce

Poynter

J. N.

Salinas

Shenouda

Vehorn

Warren

Cogswell

M. E.

(2023). Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2020. Morbidity and Mortality Weekly Report. Surveillance Summaries, 72(2), 1–14. https://doi.org/10.15585/mmwr.ss7202a1)

22.

Magis

Béland

Tuerlinckx

De Boeck

(2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847

23.

Magis

Tuerlinckx

De Boeck

(2015). Detection of differential item functioning using the lasso approach. Journal of Educational and Behavioral Statistics, 40(2), 111–135. https://doi.org/10.3102/1076998614559747

24.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543. https://doi.org/10.1007/BF02294825

25.

Nandakumar

Roussos

(2004). Evaluation of the CATSIB DIF procedure in a pretest setting. Journal of Educational and Behavioral Statistics, 29(2), 177–199. https://doi.org/10.3102/10769986029002177

26.

Paek

Wilson

(2011). Formulating the Rasch differential item functioning model under the marginal maximum likelihood estimation context and its comparison with Mantel–Haenszel procedure in short test and small sample conditions. Educational and Psychological Measurement, 71(6), 1023–1046. https://doi.org/10.1177/0013164411400734

27.

Penfield

R. D.

Camilli

(2006). 5 Differential item functioning and item bias. Handbook of Statistics, 26, 125–167. https://doi.org/10.1016/S0169-7161(06)26005-X

28.

Puhan

Moses

T. P.

Dorans

N. J.

(2009). Using log-linear smoothing to improve small-sample DIF Estimation. Journal of Educational Measurement, 46(1), 59–83. https://doi.org/10.1111/j.1745-3984.2009.01069.x

29.

R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

30.

Ritvo

R. A.

Ritvo

E. R.

Guthrie

Ritvo

M. J.

Hufnagel

D. H.

McMahon

Eloff

(2011). The Ritvo Autism Asperger Diagnostic Scale-Revised (RAADS-R): A scale to assist the diagnosis of autism spectrum disorder in adults: An international validation study. Journal of Autism and Developmental Disorders, 41, 1076–1089. https://doi.org/10.1007/s10803-010-1133-5

31.

Schwartzman

B. C.

Wood

J. J.

Kapp

S. K.

(2016). Can the five factor model of personality account for the variability of autism symptom expression? Multivariate approaches to behavioral phenotyping in adult autism spectrum disorder. Journal of Autism and Developmental Disorders, 46(1), 253–272 . https://doi.org/10.1007/s10803-015-2571-x

32.

Shealy

Stout

(1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. https://doi.org/10.1007/BF02294572

33.

Sturm

Huang

Bal

Schwartzman

(2024). Psychometric exploration of the RAADS-R with autistic adults: Implications for research and clinical practice. Autism, 28(9), 2334–2345. https://doi.org/10.1177/13623613241228329

34.

Svetina Valdivia

Huang

Botter

. (2024, April). Detecting differential item functioning in presence of multilevel data: Do methods accounting for multilevel data structure make a DIFference? In Frontiers in Education (Vol. 9, p. 1389165). Frontiers Media SA. https://doi.org/10.3389/feduc.2024.1389165

35.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x

36.

Tibshirani

(1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

37.

Tutz

Schauberger

(2015). A penalty approach to differential item functioning in Rasch models. Psychometrika, 80(1), 21–43. https://doi.org/10.1007/s11336-013-9377-6

38.

Wang

Zhu

(2024). Detecting uniform differential item functioning for continuous response computerized adaptive testing. Applied Psychological Measurement, 48(1-2), 18–37. https://doi.org/10.1177/01466216241227544

39.

Wang

Zhu

(2023). Using LASSO and adaptive LASSO to identify DIF in multidimensional 2PL models. Multivariate Behavioral Research, 58(2), 387–407. https://doi.org/10.1080/00273171.2021.1985950

40.

Wang

Woods

C. M.

(2017). Anchor selection using the Wald test anchor-all-test-all procedure. Applied Psychological Measurement, 41(1), 17–29. https://doi.org/10.1177/0146621616668014

41.

Wells

C. S.

Hambleton

R. K.

Kirkpatrick

Meng

(2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27(3), 214–231. https://doi.org/10.1080/08957347.2014.905786

42.

Wilson

De Boeck

Carstensen

C. H.

(2006). Explanatory item response models: A brief introduction. In Hartig

Klieme

Leutner

(Eds.), Assessment of competencies in educational contexts (pp. 91–120). Hogrefe & Huber Publishers.

43.

Woods

C. M.

(2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44(1), 1–27. https://doi.org/10.1080/00273170802620121

44.

Woods

C. M.

Cai

Wang

(2013). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532–547. https://doi.org/10.1177/0013164412464875

45.

Moses

Puhan

Dorans

(2008). DIF detection with small samples: Applying smoothing techniques to frequency distributions in the Mantel-Haenszel procedure. ETS Research Report Series, 2008(2), i–33. https://doi.org/10.1002/j.2333-8504.2008.tb02130.x

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.04 MB

0.00 MB