Sage Journals: Discover world-class research

Abstract

Many studies on differential item functioning (DIF) detection rely on single detection methods (SDMs), each of which necessitates specific assumptions that may not always be validated. Using an inappropriate SDM can lead to diminished accuracy in DIF detection. To address this limitation, a novel multi-detector combination (MDC) approach is proposed. Unlike SDMs, MDC effectively evaluates the relevance of different SDMs under various test conditions and integrates them using supervised learning, thereby mitigating the risk associated with selecting a suboptimal SDM for DIF detection. This study aimed to validate the accuracy of the MDC approach by applying five types of SDMs and four distinct supervised learning methods in MDC modeling. Model performance was assessed using the area under the curve (AUC), which provided a comprehensive measure of the ability of the model to distinguish between classes across all threshold levels, with higher AUC values indicating higher accuracy. The MDC methods consistently achieved higher average AUC values compared to SDMs in both matched test sets (where test conditions align with the training set) and unmatched test sets. Furthermore, MDC outperformed all SDMs under each test condition. These findings indicated that MDC is highly accurate and robust across diverse test conditions, establishing it as a viable method for practical DIF detection.

Keywords

differential item functioning multi-detector combination area under the curve supervised learning test fairness

Get full access to this article

View all access options for this article.

References

Ackerman

T. A.

(1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67–91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x

Bauer

D. J.

(2023). Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning. British Journal of Mathematical and Statistical Psychology, 76(3), 435–461. https://doi.org/10.1111/bmsp.12316

Berger

Tutz

(2016). Detection of uniform and nonuniform differential item functioning by item-focused trees. Journal of Educational and Behavioral Statistics, 41(6), 559–592. https://doi.org/10.3102/1076998616659371

Blanca

M. J.

Arnau

García-Castro

F. J.

Alarcón

Bono

(2023). Repeated measures ANOVA and adjusted F-tests when sphericity is violated: Which procedure is best? Frontiers in Psychology, 14(1), Article 1192453. https://doi.org/10.3389/fpsyg.2023.1192453

Candell

G. L.

Drasgow

(1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12(3), 253–260. https://doi.org/10.1177/014662168801200304

Cervantes

Garcia-Lamont

Rodríguez-Mazahua

Lopez

(2020). A comprehensive survey on support vector machine classification: Applications, challenges and trends. Neurocomputing, 408(1), 189–215. https://doi.org/10.1016/j.neucom.2019.10.118

Chow

Liu

(1968). Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory, 14(3), 462–467. https://doi.org/10.1109/tit.1968.1054142

Cook

L. L.

Eignor

D. R.

(1991). IRT equating methods. Educational Measurement: Issues and Practice, 10(3), 37–45. https://doi.org/10.1111/j.1745-3992.1991.tb00207.x

Dimitriadou

Hornik

Leisch

Meyer

(2008). Misc functions of the Department of Statistics (e1071), TU Wien. R package, 1(1), 5–24.

10.

Dorans

N. J.

Schmitt

A. P.

Bleistein

C. A.

(1992). The standardization approach to assessing comprehensive differential item functioning. Journal of Educational Measurement, 29(4), 309–319. https://doi.org/10.1111/j.1745-3984.1992.tb00379.x

11.

Frick

Strobl

Zeileis

(2015). Rasch mixture models for DIF detection: A comparison of old and new score specifications. Educational and Psychological Measurement, 75(2), 208–234. https://doi.org/10.1177/0013164414536183

12.

Friedman

Geiger

Goldszmidt

(1997). Bayesian network classifiers. Machine Learning, 29(2/3), 131–163. https://doi.org/10.1023/a:1007465528199

13.

Gierl

M. J.

Gotzmann

Boughton

K. A.

(2004). Performance of SIBTEST when the percentage of DIF items is large. Applied Measurement in Education, 17(3), 241–264. https://doi.org/10.1207/s15324818ame1703_2

14.

Goretzko

Bühner

(2020). One model to rule them all? Using machine learning algorithms to determine the number of factors in exploratory factor analysis. Psychological Methods, 25(6), 776–786. https://doi.org/10.1037/met0000262

15.

Hladká

Martinková

Magis

(2023). Combining item purification and multiple comparison adjustment methods in detection of differential item functioning. Multivariate Behavioral Research, 59(1), 46–61. https://doi.org/10.1080/00273171.2023.2205393

16.

Holland

P. W.

Thayer

D. T.

(1986). Differential item functioning and the Mantel‐Haenszel procedure. ETS Research Report Series, 1986(2), i–24. https://doi.org/10.1002/j.2330-8516.1986.tb00186.x

17.

Hutchinson

Mitchell

(2019). 50 years of test (un) fairness: Lessons for machine learning. In Proceedings of the conference on fairness, accountability, and transparency (pp. 49–58). ACM.

18.

Jiang

(2019). Regularization methods for detecting differential item functioning. Boston College. Lynch School of Education.

19.

Jin

Ling

(2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3), 299–310. https://doi.org/10.1109/tkde.2005.50

20.

Jin

Chen

Wang

W. C.

(2018). Using odds ratios to detect differential item functioning. Applied Psychological Measurement, 42(8), 613–629. https://doi.org/10.1177/0146621618762738

21.

Karami

Salmani Nodoushan

M. A.

(2011). Differential item functioning (DIF): Current problems and future directions. Online Submission, 5(3), 133–142.

22.

Lee

(2017). Detecting differential item functioning using the logistic regression procedure in small samples. Applied Psychological Measurement, 41(1), 30–43. https://doi.org/10.1177/0146621616668015

23.

Lim

Choe

Han

K. T.

(2022). A residual‐based differential item functioning detection framework in item response theory. Journal of Educational Measurement, 59(1), 80–104. https://doi.org/10.1111/jedm.12313

24.

Liu

Jane Rogers

(2022). Treatments of differential item functioning: A comparison of four methods. Educational and Psychological Measurement, 82(2), 225–253. https://doi.org/10.1177/00131644211012050

25.

Terzi

de la Torre

(2021). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37–53. https://doi.org/10.1177/0146621620965745

26.

Magis

Béland

Tuerlinckx

De Boeck

(2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847

27.

Magis

Tuerlinckx

(2016). On the use of ROC curves in DIF simulation studies.

28.

Martinková

Drabinová

Liaw

Y. L.

Sanders

E. A.

McFarland

J. L.

Price

R. M.

(2017). Checking equity: Why differential item functioning analysis should be a routine part of developing conceptual assessments. CBE-Life Sciences Education, 16(2), rm2. https://doi.org/10.1187/cbe.16-10-0307

29.

Martinkova

Hladka

(2018). ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(1), 503–515.

30.

McLaughlin

M. E.

Drasgow

(1987). Lord's chi-square test of item bias with estimated and with known person parameters. Applied Psychological Measurement, 11(2), 161–173. https://doi.org/10.1177/014662168701100205

31.

Partchev

Maris

Hattori

(2022). Package “irtoys”:A collection of functions related to item response theory (IRT).

32.

Raju

N. S.

(1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197–207. https://doi.org/10.1177/014662169001400208

33.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC area, Cohen's d, and r. Law and Human Behavior, 29(5), 615–620. https://doi.org/10.1007/s10979-005-6832-7

34.

Rizopoulos

(2007). “ltm: An R package for latent variable modeling and item response analysis”. Journal of Statistical Software, 17(1), 1–25.

35.

Robin

Turck

Hainard

(2014). Package ‘pROC’, technical report. Available online. https://cran._r-project._org/web

36.

Rogers

H. J.

Swaminathan

(1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105–116. https://doi.org/10.1177/014662169301700201

37.

Scutari

Denis

(2021). Bayesian networks: With examples in R.

38.

Youden

W. J.

(1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.28 MB

A Generalized Multi-Detector Combination Approach for Differential Item Functioning Detection

Abstract

Keywords

Get full access to this article

References

Supplementary Material