The growing diversity among test takers in second or foreign language (L2) assessments makes the importance of fairness front and center. This systematic review aimed to examine how fairness in L2 assessments was evaluated through differential item functioning (DIF) analysis. A total of 83 articles from 27 journals were included in a systematic review. The findings suggested that classical DIF techniques were dominant in use, particularly Rasch-based methods, the Mantel–Haenszel procedure, item response theory (IRT) approaches, logistic regression, and SIBTEST, but emerging methods such as DIF analysis based on cognitive diagnostic models were also identified. Most DIF studies examined manifest grouping variables such as gender and language background and were based on assessments of receptive language skills such as reading and listening comprehension. DIF analyses were mostly conducted in an exploratory fashion and causes of DIF were often justified on speculative rather than empirical grounds. In addition, the quality of DIF analyses was undermined by suboptimal reporting practices. Our results suggest the need to improve current DIF practices, to consider alternative DIF detection methods aligning with emerging views of measurement bias, and to adequately account for the heterogeneity of L2 test takers. The findings have implications for test design and use, fairness, and validity in L2 assessments.
References marked with an asterisk indicate studies included in the systematic review.
2.
*AbbottM. L. (2007). A confirmatory approach to differential item functioning on an ESL reading assessment. Language Testing, 24(1), 7–36. https://doi.org/10.1177/0265532207071510
3.
AllaloufA.AbramzonA. (2008). Constructing better second language assessments based on differential item functioning analysis. Language Assessment Quarterly, 5(2), 120–141. https://doi.org/10.1080/15434300801934710
4.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
5.
*AryadoustV. (2012). Differential item functioning in while-listening performance tests: The case of the International English Language Testing System (IELTS) listening module. International Journal of Listening, 26(1), 40–60. https://doi.org/10.1080/10904018.2012.639649
6.
*AryadoustV. (2015). Fitting a mixture Rasch model to English as a foreign language listening tests: The role of cognitive and background variables in explaining latent differential item functioning. International Journal of Testing, 15(3), 216–238. https://doi.org/10.1080/15305058.2015.1004409
7.
*AryadoustV. (2018). Using recursive partitioning Rasch trees to investigate differential item functioning in second language reading tests. Studies in Educational Evaluation, 56, 197–204. https://doi.org/10.1016/j.stueduc.2018.01.003
*AryadoustV.GohC. C.KimL. O. (2011). An investigation of differential item functioning in the MELAB listening test. Language Assessment Quarterly, 8(4), 361–385. https://doi.org/10.1080/15434303.2011.628632
10.
AryadoustV.MinS.ChenX. (2024). Investigating differential item functioning across interaction variables in listening comprehension assessment. Studies in Educational Evaluation, 80, Article 101322. https://doi.org/10.1016/j.stueduc.2024.101322
11.
AryadoustV.NgL. Y.SayamaH. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487
12.
BauerD. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077
13.
BauerD. J.BelzakW. C.ColeV. T. (2020). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754
14.
BelzakW. C. (2023). The multidimensionality of measurement bias in high-stakes testing: Using machine learning to evaluate complex sources of differential item functioning. Educational Measurement: Issues and Practice, 42(1), 24–33. https://doi.org/10.1111/emip.12486
15.
*BerendesK.WagnerW.MeurersD.TrautweinU. (2019). When a silent reading fluency test measures more than reading fluency: Academic language features predict the test performance of students with a non-German home language. Reading and Writing, 32, 561–583. https://doi.org/10.1007/s11145-018-9878-x
16.
BerríoÁ. I.Gomez-BenitoJ.Arias-PatiñoE. M. (2020). Developments and trends in research on methods of detecting differential item functioning. Educational Research Review, 31, 100340. https://doi.org/10.1016/j.edurev.2020.100340
17.
*Bourdeaud’HuiH.AesaertK.van BraakJ. (2021). Exploring the validity of a comprehensive listening test to identify differences in primary school students’ listening skills. Language Assessment Quarterly, 18(3), 228–252. https://doi.org/10.1080/15434303.2020.1860059
18.
*BowlesM. A. (2022). Using instructor judgment, learner corpora, and DIF to develop a placement test for Spanish L2 and heritage learners. Language Testing, 39(3), 355–376. https://doi.org/10.1177/02655322221076033
19.
*BrelandH.LeeY. W. (2007). Investigating uniform and non-uniform gender DIF in computer-based ESL writing assessment. Applied Measurement in Education, 20(4), 377–403. https://doi.org/10.1080/08957340701429652
20.
BuonoS.JangE. E. (2021). The effect of linguistic factors on assessment of English language learners’ mathematical ability: A differential item functioning analysis. Educational Assessment, 26(2), 125–144. https://doi.org/10.1080/10627197.2020.1858783
21.
*ChenY. F.JiaoH. (2014). Exploring the utility of background and cognitive variables in explaining latent differential item functioning: An example of the PISA 2009 reading assessment. Educational Assessment, 19(2), 77–96. https://doi.org/10.1080/10627197.2014.903650
CrenshawK. W. (1989). Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics. University of Chicago Legal Forum, 1989(1), 139–167. https://scholarship.law.columbia.edu/faculty_scholarship/3007
24.
De BoeckP.WilsonM. (2004). Explanatory item response models: A generalized linear and nonlinear approach. Springer-Verlag.
25.
DesjardinsC. D.BulutO. (2018). Handbook of educational measurement and psychometrics using R. CRC Press.
26.
*DeygersB.VanbuelM. (2021). Gauging the impact of literacy and educational background on receptive vocabulary test scores. Language Testing, 39(2), 191–211. https://doi.org/10.1177/02655322211049097
27.
DrackertA.TimukovaA. (2020). What does the analysis of C-test gaps tell us about the construct of a C-test? A comparison of foreign and heritage language learners’ performance. Language Testing, 37(1), 107–132. https://doi.org/10.1177/0265532219861042
28.
EbrahimiV.BagheriZ.ShayanZ.JafariP. (2021). A machine learning approach to assess differential item functioning in psychometric questionnaires using the elastic net regularized ordinal logistic regression in small sample size groups. Biomed Research International, 2021, Article 6854477. https://doi.org/10.1155/2021/6854477
29.
*ElderC. (1996). The effect of language background on “foreign” language test performance: The case of Chinese, Italian, and Modern Greek. Language Learning, 46(2), 233–282. https://doi.org/10.1111/j.1467-1770.1996.tb01236.x
30.
FerneT.RuppA. A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4(2), 113–148. https://doi.org/10.1080/15434300701375923
31.
FulcherG.DavidsonF. (2013). The Routledge handbook of language testing. Routledge.
32.
GohC. C.AryadoustV. (2015). Examining the notion of listening subskill divisibility and its implications for second language listening. International Journal of Listening, 29(3), 109–133. https://doi.org/10.1080/10904018.2014.936119
33.
Gómez-BenitoJ.SireciS.PadillaJ. L.HidalgoM. D.BenítezI. (2018). Differential item functioning: Beyond validity evidence based on internal structure. Psicothema, 30(1), 104–109.
34.
GunnerudH. L.ten BraakD.ReikeråsE. K. L.DonolatoE.Melby-LervågM. (2020). Is bilingualism related to a cognitive advantage in children? A systematic review and meta-analysis. Psychological Bulletin, 146(12), 1059–1083. https://doi.org/10.1037/bul0000301
35.
*HardingL. (2012). Accent, listening assessment and the potential for a shared-L1 advantage: A DIF perspective. Language Testing, 29(2), 163–180. https://doi.org/10.1177/0265532211421161
36.
HollandP. W.WainerH. (Eds.). (1993). Differential item functioning. Lawrence Erlbaum.
37.
HollandT. W.ThayerD. T. (1988). Differential item performance and the Mantel–Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.
38.
KangO.YanX.KostromitinaM.ThomsonR.IsaacsT. (2024). Fairness of using different English accents: The effect of shared L1s in listening tasks of the Duolingo English test. Language Testing, 41(2), 263–289. https://doi.org/10.1177/02655322231179134
39.
*KaramiM.PishghadamR.BaghaeiP. (2019). A probe into EFL learners’ emotioncy as a source of test bias: Insights from differential item functioning analysis. Studies in Educational Evaluation, 60, 170–178. https://doi.org/10.1016/j.stueduc.2019.01.003
40.
KatoK.MoenR. E.ThurlowM. L. (2009). Differentials of a state reading assessment: Item functioning, distractor functioning, and omission frequency for disability categories. Educational Measurement: Issues and Practice, 28(2), 28–40. https://doi.org/10.1111/j.1745-3992.2009.00145.x
41.
*KimY. H.JangE. E. (2009). Differential functioning of reading subskills on the OSSLT for L1 and ELL students: A multidimensionality model-based DBF/DIF approach. Language Learning, 59(4), 825–865. https://doi.org/10.1111/j.1467-9922.2009.00527.x
42.
*KooJ.BeckerB. J.KimY. S. (2014). Examining differential item functioning trends for English language learners in a reading test: A meta-analytical approach. Language Testing, 31(1), 89–109. https://doi.org/10.1177/0265532213496097
43.
*KunnanA. J. (1990). DIF in native language and gender groups in an ESL placement test. TESOL Quarterly, 24(4), 741–746. https://doi.org/10.2307/3587128
44.
KunnanA. J. (2014). Fairness and justice in language assessment. In KunnanA. J. (Ed.), The companion to language assessment (pp. 1–17). John Wiley & Sons. https://doi.org/10.1002/9781118411360
45.
*LakinJ. M.ElliottD. C.LiuO. L. (2012). Investigating ESL students’ performance on outcomes assessments in higher education. Educational and Psychological Measurement, 72(5), 734–753. https://doi.org/10.1177/0013164412442376
46.
LiH.HunterC. V.BialoJ. A. (2022). A revisit of Zumbo’s third generation DIF: How are we doing in language testing?Language Assessment Quarterly, 19(1), 27–53. https://doi.org/10.1080/15434303.2021.1963253
47.
LiH.HunterC. V.OshimaT. C. (2013). Gender DIF in reading tests: A synthesis of research. In MillsapR. E.AndriesL.Van Der ArkD. M. B.WoodsC. M. (Eds.), Springer proceedings in mathematics & statistics: New developments in quantitative psychology (pp. 489–506). Springer.
48.
*LiuO. L. (2011). Do major field of study and cultural familiarity affect TOEFL® iBT reading performance? A confirmatory approach to differential item functioning. Applied Measurement in Education, 24(3), 235–255. https://doi.org/10.1080/08957347.2011.580645
49.
LiuY.YinH.XinT.ShaoL.YuanL. (2019). A comparison of differential item functioning detection methods in cognitive diagnostic models. Frontiers in Psychology, 10, Article 1137. https://doi.org/10.3389/fpsyg.2019.01137
50.
*MehrazmayR.GhonsoolyB.delaTorreJ. (2021). Detecting differential item functioning using cognitive diagnosis models: Applications of the Wald Test and Likelihood ratio test in a university entrance examination. Applied Measurement in Education, 34(4), 262–284. https://doi.org/10.1080/08957347.2021.1987906
51.
*MinS.HeL. (2020). Test fairness: Examining differential functioning of the reading comprehension section of the GSEEE in China. Studies in Educational Evaluation, 64, Article 100811. https://doi.org/10.1016/j.stueduc.2019.100811
52.
*MoradiE.GhabanchiZ.PishghadamR. (2022). Reading comprehension test fairness across gender and mode of learning: Insights from IRT-based differential item functioning analysis. Language Testing in Asia, 12(1), 1–18. https://doi.org/10.1186/s40468-022-00192-3
53.
*OliveriM. E.ErcikanK.Lyons-ThomasJ.HoltzmanS. (2016). Analyzing fairness among linguistic minority populations using a latent class differential item functioning approach. Applied Measurement in Education, 29(1), 17–29. https://doi.org/10.1080/08957347.2015.1102913
*PaeT. I. (2012). Causes of gender DIF on an EFL language test: A multiple–data analysis over nine years. Language Testing, 29(4), 533–554. https://doi.org/10.1177/0265532211434027
56.
PaeT. I.ParkG. P. (2006). Examining the relationship between differential item functioning and differential test functioning. Language Testing, 23(4), 475–496. https://doi.org/10.1191/0265532206lt338oa
57.
PageM. J.McKenzieJ. E.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.MoherD. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1), 1–11. https://doi.org/10.1186/s13643-021-01626-4
58.
*ParkG. P. (2008). Differential item functioning on an English listening test across gender. TESOL Quarterly, 42(1), 115–123.
PenfieldR. D.LamT. C. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19(3), 5–15. https://doi.org/10.1111/j.1745-3992.2000.tb00033.x
61.
PishghadamR. (2016). Emotioncy, extraversion, and anxiety in willingness to communicate in English. In LokmanW. A.FazidahF. M.SalahuddinI.MohdI. A. W. (Eds.), Proceedings of the 5th international conference on language, education, and innovation (pp. 1–5). Infobase Creation Sdn Bhd.
62.
PurpuraJ. E. (2016). Second and foreign language assessment. The Modern Language Journal (Boulder, Colo.), 100(S1), 190–208. https://doi.org/10.1111/modl.12308
*SandilandsD.OliveriM. E.ZumboB. D.ErcikanK. (2013). Investigating sources of differential item functioning in international large–scale assessments using a confirmatory approach. International Journal of Testing, 13(2), 152–174. https://doi.org/10.1080/15305058.2012.690140
65.
*SandilosL. E.LewisK.KomaroffE.HammerC. S.ScarpinoS. E.LopezL.GoldsteinB. (2015). Analysis of bilingual children’s performance on the English and Spanish versions of the Woodcock-Muñoz Language Survey-R (WMLS-R). Language Assessment Quarterly, 12(4), 386–408. https://doi.org/10.1080/15434303.2015.1100198
66.
*SariB. T.van de VijverF. J.ChasiotisA.BenderM. (2019). Contextualized bilingualism among adolescents from four different ethnic groups in Indonesia. International Journal of Bilingualism, 23(6), 1469–1482. https://doi.org/10.1177/1367006918803678
67.
SasakiM. (1991). A comparison of two methods for detecting differential item functioning in an ESL placement test. Language Testing, 8(2), 95–111. https://doi-org/10.1177/026553229100800201
68.
SchottenM.AisatiM.MeesterW. J. N.SteigningaS.RossC. A. (2018). A brief history of Scopus: The world’s largest abstract and citation database of scientific literature. In Cantu-OrtizF. J. (Ed.), Research analytics: Boosting university productivity and competitiveness through Scientometrics (pp. 33–57). Taylor & Francis.
69.
ScottN. W.FayersP. M.AaronsonN. K.BottomleyA.de GraeffA.GroenvoldM.SprangersM. A. G. (2010). Differential item functioning (DIF) analyses of health-related quality of life instruments using logistic regression. Health and Quality of Life Outcomes, 8(1), Article 81. https://doi.org/10.1186/1477-7525-8-81
70.
*SeoD.TaherbhaiH.FrantzR. (2016). Psychometric evaluation and discussions of English language learners’ listening comprehension. International Journal of Listening, 30(1–2), 47–66. https://doi.org/10.1080/10904018.2015.1065747
71.
ShieldsS. A. (2008). Gender: An intersectionality perspective. Sex Roles, 59(5–6), 301–311.
72.
*SireciS. G.AllaloufA. (2003). Appraising item equivalence across multiple languages and cultures. Language Testing, 20(2), 148–166. https://doi.org/10.1191/0265532203lt249oa
73.
SochaA.DeMarsC. E.ZilberbergA.PhanH. (2015). Differential item functioning detection with the Mantel-Haenszel procedure: The effects of matching types and other factors. International Journal of Testing, 15(3), 193–215. https://doi.org/10.1080/15305058.2014.984066
74.
*SongX.HeL. (2015). The effect of a national education policy on language test performance: A fairness perspective. Language Testing in Asia, 5(4), 1–14. https://doi.org/10.1186/s40468-014-0011-z
von DavierM.YamamotoK. (2004). Partially observed mixtures of IRT models: An extension of the generalized partial–credit model. Applied Psychological Measurement, 28, 389–406. https://doi.org/10.1177/0146621604268734
77.
von ZansenA.HildenR.LaihanenE. (2022). The multimodal listening test in a high-stakes context: Gender-neutral or not?International Journal of Listening, 36(2), 152–170. https://doi.org/10.1080/10904018.2021.1993446
YooH.MannaV. F.MonfilsL. F.OhH. J. (2019). Measuring English language proficiency across subgroups: Using score equity assessment to evaluate test fairness. Language Testing, 36(2), 289–309. https://doi.org/10.1177/0265532218776040
83.
*ZhuX.AryadoustV. (2022). An investigation of mother tongue differential item functioning in a high-stakes computerized academic reading test. Computer Assisted Language Learning, 35(3), 412–436. https://doi.org/10.1080/09588221.2019.1704788
84.
ZumboB. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
85.
ZumboB. D.LiuY.WuA. D.ShearB. R.Olvera AstiviaO. L.ArkT. K. (2015). A methodology for Zumbo’s third generation DIF analyses and the ecology of item responding. Language Assessment Quarterly, 12(1), 136–151. https://doi.org/10.1080/15434303.2014.972559
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
