Since the 1960s, the testing community has striven to ensure fairness in tests and test items. Differential item functioning (DIF) is a widely used statistical notion for items that may unfairly disadvantage specific subgroups of test-takers. However, traditional DIF analyses focus only on statistical relationships in observed data and cannot explain why such unfairness occurs. To fill this gap, we introduce a novel causal framework for defining and detecting unfair items using single-world intervention graphs (SWIGs). By leveraging SWIGs and potential outcomes, we define causal DIF (CDIF) as the difference in item functioning between two hypothetical worlds: one where individuals were assigned to one group and another where they were assigned to a different group. We also connect CDIF to related fairness concepts, including group versus individual fairness and item impact. In particular, we use SWIGs to graphically distinguish between item fairness at the individual level and the population level. Additionally, we discuss causal identification strategies using SWIGs and demonstrate how our approach differs from traditional DIF methods through a simulation study. We further illustrate its application to a controversial item from New York’s Regents math exam and a real dataset from the Trends in International Mathematics and Science Study, concluding with broader implications of promoting causal fairness in testing practices.
AckermanT. A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67–91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x
2.
BarocasS.HardtM.NarayananA. (2023). Fairness and machine learning: Limitations and opportunities. MIT Press. http://www.fairmlbook.org
3.
BelzakW. C. M. (2022). The multidimensionality of measurement bias in high-stakes testing: Using machine learning to evaluate complex sources of differential item functioning. Educational Measurement: Issues and Practice, 42(1), 24–33. https://doi.org/10.1111/emip.12486
4.
BoltD. M.CohenA. S.WollackJ. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x
5.
CareyA. N.WuX. (2022). The causal fairness field guide: Perspectives from social and formal sciences. Frontiers in Big Data, 5, 892837. https://doi.org/10.3389/fdata.2022.892837
6.
ChaoH.-Y.ChenJ.-H.ChenC.-C. (2024). Mitigating omitted variable bias in exploratory differential item functioning assessment: A propensity score adjustment approach. Journal of Educational and Behavioral Statistics, 50, 927–961. https://doi.org/10.3102/10769986241280308
7.
ChiappaS. (2019). Path-specific counterfactual fairness. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 7801–7808. https://doi.org/10.1609/aaai.v33i01.33017801
8.
ChouldechovaA. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. https://doi.org/10.1089/big.2016.0047
9.
ClearyT. A. (1968). Test bias: Prediction of grades of Negro and white students in integrated colleges. Journal of Educational Measurement, 5(2), 115–124. https://doi.org/10.1111/j.1745-3984.1968.tb00613.x
10.
Corbett-DaviesS.PiersonE.FellerA.GoelS.HuqA. (2017). Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 797–806). Association for Computing Machinery. https://doi.org/10.1145/3097983.3098095
11.
CrenshawK. (1989). Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum, 1989(1), 139–167.
DoransN. J.KulickE. (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the scholastic aptitude test. Journal of Educational Measurement, 23(4), 355–368. https://doi.org/10.1111/j.1745-3984.1986.tb00255.x
14.
DoransN. J.SchmittA. P.BleisteinC. A. (1992). The standardization approach to assessing comprehensive differential item functioning. Journal of Educational Measurement, 29(4), 309–319. https://doi.org/10.1111/j.1745-3984.1992.tb00379.x
15.
DworkC.HardtM.PitassiT.ReingoldO.ZemelR. (2012). Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference (pp. 214–226). Association for Computing Machinery. https://doi.org/10.1145/2090236.2090255
KilbertusN.Rojas CarullaM.ParascandoloG.HardtM.JanzingD.SchölkopfB. (2017). Avoiding discrimination through causal reasoning. In GuyonI.von LuxburgU.BengioS.WallachH.FergusR.VishwanathanS.GarnettR. (Eds.), Advances in Neural Information Processing Systems (Vol. 30, pp. 656–666).
21.
KopfJ.ZeileisA.StroblC. (2014). Anchor selection strategies for DIF analysis: Review, assessment, and new approaches. Educational and Psychological Measurement, 75(1), 22–56. https://doi.org/10.1177/0013164414529792
22.
KusnerM. J.LoftusJ.RussellC.SilvaR. (2017). Counterfactual fairness. Advances in Neural Information Processing Systems, 30 (NIPS 2017).
23.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates.
24.
LyuW.WangC.XuG. (2025). Detecting differential item functioning across multiple groups using group pairwise penalty. Psychometrika, 90(5), 1594–1621. https://doi.org/10.1017/psy.2025.10034
25.
MagisD.BélandS.TuerlinckxF.De BoeckP. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/brm.42.3.847
26.
MorganS. L.WinshipC. (2015). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). Cambridge University Press.
NeymanJ. S. (1923). On the application of probability theory to agricultural experiments: Essay on principles. Statistical Science, 4, 465–480.
29.
PearlJ. (2001). Direct and indirect effects. In BreeseJ. S.KollerD. (Eds.), Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (pp. 411–420). Morgan Kaufmann.
PineS. M. (1977). Applications of item characteristic curve theory to the problem of test bias. In WeissD. J. (Ed.), Applications of computerized adaptive testing: Proceedings of a symposium presented at the 18th annual convention of military testing association (pp. 37–43). University of Minnesota, Department of Psychology, Psychometric Methods Program.
33.
PlečkoD.BareinboimE. (2024). Causal fairness analysis: A causal toolkit for fair machine learning. Foundations and Trends® in Machine Learning, 17(3), 304–589. https://doi.org/10.1561/2200000106
34.
RichardsonT. S.RobinsJ. M. (2013). Single world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality (Working Paper No. 128). Center for the Statistics and the Social Sciences, University of Washington Series.
35.
RobinsJ. M.RichardsonT. S.ShpitserI. (2022, February). An interventionist approach to mediation analysis. In GeffnerH.DechterR.HalpernJ. Y. (Eds.) Probabilistic and causal inference (pp. 713–764). ACM. https://doi.org/10.1145/3501714.3501754
36.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
37.
SuC.YuG.WangJ.YanZ.CuiL. (2022). A review of causality-based fairness machine learning. Intelligence & Robotics, 2(3), 244–274. https://doi.org/10.20517/ir.2022.17
38.
SukY.HanK. T. (2024). A psychometric framework for evaluating fairness in algorithmic decision making: Differential algorithmic functioning. Journal of Educational and Behavioral Statistics, 49(2), 151–172. https://doi.org/10.3102/10769986231171711
39.
SukY.HanK. T. (2025). Evaluating intersectional fairness in algorithmic decision making using intersectional differential algorithmic functioning. Journal of Educational and Behavioral Statistics, 50(5), 833–862. https://doi.org/10.3102/10769986241269820
40.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
41.
ThissenD.SteinbergL.WainerH. (1988). Use of item response theory in the study of group differences in trace lines. In WainerH.BraunH. I. (Eds.), Test validity (pp. 147–169). Lawrence Erlbaum.
42.
ThomassonG. L. (1997). The goal of equity within and between computerized adaptive tests and paper and pencil forms. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
43.
VanderWeeleT. J. (2010). Controlled direct and mediated effects: Definition, identification and bounds. Scandinavian Journal of Statistics, 38(3), 551–563. https://doi.org/10.1111/j.1467-9469.2010.00722.x
44.
YangS.LorenziE.PapadogeorgouG.WojdylaD. M.LiF.ThomasL. E. (2021). Propensity score weighting for causal subgroup analysis. Statistics in Medicine, 40(19), 4294–4309. https://doi.org/10.1002/sim.9029
45.
ZumboB. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
