Mitigating Omitted Variable Bias in Exploratory Differential Item Functioning Assessment: A Propensity Score Adjustment Approach

Abstract

Omitted variable bias (OVB) in differential item functioning (DIF) assessment can lead to biased estimates of DIF effects, increasing type I error rates and reducing statistical power. Methodologies for addressing OVB, particularly under exploratory conditions where DIF is assessed without prior information, remain underdeveloped. This study addresses these challenges by applying the propensity score (PS) method to enhance the parsimony of the DIF model. Furthermore, this study incorporated the DIF-free-then-DIF (DFTD) strategy into the PS method, establishing the PS.DFTD method for exploratory DIF assessment. The simulation results reveal that the PS.DFTD method appears promising for mitigating OVB in DIF assessment, demonstrating its potential for practical applications.

Keywords

differential item functioning omitted variable bias propensity score method DIF-free-then-DIF strategy

Get full access to this article

View all access options for this article.

References

Andrich

(1978). Rating formulation for ordered response category. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814

Arikan

van de Vijver

Yagmur

(2018). Propensity score matching helps to understand sources of DIF and mathematics performance differences of Indonesian, Turkish, Australian, and Dutch students in PISA. International Journal of Research in Education and Science, 4(1), 69–81. https://doi.org/10.21890/ijres.382936

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–424). Addison Wesley.

Bolt

D. M.

(2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15(2), 113–141. https://doi.org/10.1207/S15324818AME1502_01

Caliendo

Kopeinig

(2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31–72. https://doi.org/10.1111/j.1467-6419.2007.00527.x

Chao

H. Y.

Chen

C. C.

Cheng

C. P.

Chen

J. H.

(2018). Omitted variable bias in differential item functioning assessment. Chinese Journal of Psychology, 60(4), 233–250. https://doi.org/10.6129/CJP.201812_60(4).0001

Chen

J. H.

Chen

C. T.

Shih

C. L.

(2014). Improving the control of type I error rate in assessing differential item functioning for hierarchical generalized linear model when impact is presented. Applied Psychological Measurement, 38(1), 18–36. https://doi.org/10.1177/0146621613488643

Chen

C. T.

Hwu

B. S.

(2018). Improving the assessment of differential item functioning in large-scale programs with dual-scale purification of Rasch models: The PISA example. Applied Psychological Measurement, 42(3), 206–220. https://doi.org/10.1177/0146621617726786

Chen

M. Y.

Liu

Zumbo

B. D.

(2020). A propensity score method for investigating differential item functioning in performance assessment. Educational and Psychological Measurement, 80(3), 476–498. https://doi.org/10.1177/0013164419878861

10.

Cheng

Shao

Lathrop

Q. N.

(2016). The mediated MIMIC model for understanding the underlying mechanism of DIF. Educational and Psychological Measurement, 76(1), 43–63. https://doi.org/10.1177/0013164415576187

11.

Chun

Stark

Kim

E. S.

Chernyshenko

O. S.

(2016). MIMIC methods for detecting DIF among multiple groups: Exploring a new sequential-free baseline procedure. Applied Psychological Measurement, 40(7), 486–499.

12.

Clarke

K. A.

(2005). The phantom menace: Omitted variable bias in econometric research. Conflict Management and Peace Science, 22(4), 341–352. https://doi.org/10.1080/07388940500339183

13.

Clarke

K. A.

(2009). Return of the phantom menace: Omitted variable bias in political research. Conflict Management and Peace Science, 26(1), 46–66. https://doi.org/10.1177/0738894208097666

14.

D’Agostino

R. B.

Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17(19), 2265–2281. https://doi.org/10.1002/(SICI)1097-0258(19981015)17:19<2265::AID-SIM918>3.0.CO;2-B

15.

DeMars

C. E.

(2010). Type I error inflation for detecting DIF in the presence of impact. Educational and Psychological Measurement, 70(6), 961–972. https://doi.org/10.1177/0013164410366691

16.

Douglas

J. A.

Roussos

L. A.

Stout

(1996). Item-bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33(4), 465–484. https://doi.org/10.1111/j.1745-3984.1996.tb00502.x

17.

Finch

W. H.

(2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4), 278–295. https://doi.org/10.1177/0146621605275728

18.

Finch

W. H.

(2016). Detection of differential item functioning for more than two groups: A Monte Carlo comparison of methods. Applied Measurement in Education, 29(1), 30–45. https://doi.org/10.1080/08957347.2015.1102916

19.

French

B. F.

Finch

W. H.

(2013). Extensions of Mantel-Haenszel for multilevel DIF detection. Educational and Psychological Measurement, 73(4), 648–671. https://doi.org/10.1177/0013164412472341

20.

French

B. F.

Finch

W. H.

Immekus

J. C.

(2019). Multilevel generalized Mantel-Haenszel for differential item functioning detection. Frontiers in Quantitative Psychology, 4, 47–57. https://doi.org/10.3389/feduc.2019.00047

21.

French

B. F.

Maller

S. J.

(2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67(3), 373–393. https://doi.org/10.1177/0013164406294781

22.

Greene

W. H.

(2003). Econometric analysis. Pearson Education.

23.

Hauser

R. M.

Goldberger

A. S.

(1971). The treatment of unobservable variables in path analysis. Sociological Methodology, 3, 81–117. https://www.jstor.org/stable/270819

24.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Erlbaum.

25.

Jodoin

M. G.

Gierl

M. J.

(2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349. https://doi.org/10.1207/S15324818AME1404_2

26.

Kopf

Zeileis

Strobl

(2015a). Anchor selection strategies for DIF analysis: Review, assessment, and new approaches. Educational and Psychological Measurement, 75(1), 22–56. https://doi.org/10.1177/0013164414529792

27.

Kopf

Zeileis

Strobl

(2015b). A framework for anchor methods and an iterative forward approach for DIF detection. Applied Psychological Measurement, 39(2), 83–103. https://doi.org/10.1177/0146621614544195

28.

Lee

Geisinger

K. F.

(2014). The effect of propensity scores on DIF analysis: Inference on the potential cause of DIF. International Journal of Testing, 14(4), 313–338. https://doi.org/10.1080/15305058.2014.922567

29.

Liu

Kim

A. D.

Gustafson

Kroc

(2019). Investigating the performance of propensity score approaches for differential item functioning analysis. Journal of Modern Applied Statistical Methods, 18(1), eP2744. https://doi.org/10.22237/jmasm/1556669280

30.

Liu

Zumbo

B. D.

Gustafson

Huang

Kroc

A. D.

(2016). Investigating causal DIF via propensity score methods. Practical Assessment, Research, and Evaluation, 21(1), 13. https://doi.org/10.7275/ewqz-n963

31.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.

32.

MacKinnon

D. P.

Krull

J. L.

Lockwood

C. M.

(2000). Equivalence of the mediation, confounding and suppression effect. Prevention Science, 1, 173–181. https://doi.org/10.1023/A:1026595011371

33.

McCaffrey

D. F.

Griffin

B. A.

Almirall

Slaughter

M. E.

Ramchand

Burgette

L. F.

(2013). A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Statistics in Medicine, 32(19), 3388–3414. https://doi.org/10.1002/sim.5753

34.

Melguizo

Kienzl

G. S.

Alfonso

(2011). Comparing the educational attainment of community college transfer students and four-year college rising juniors using propensity score matching methods. The Journal of Higher Education, 82(3), 265–291. https://doi.org/10.1080/00221546.2011.11777202

35.

Oneal

J. R.

Russett

(2005). Rule of three, let it be? When more really is better. Conflict Management and Peace Science, 22(4), 293–310. https://doi.org/10.1080/07388940500339209

36.

Oort

F. J.

(1998). Simulation study of item bias detection with restricted factor analysis. Structural Equation Modeling, 5(2), 107–124. https://doi.org/10.1080/10705519809540095

37.

Pine

S. M.

(1977). Applications of item characteristic curve theory to the problem of test bias. In Weiss

D. J.

(Ed.), Applications of computerized adaptive testing: Proceedings of a symposium presented at the 18th annual convention of the Military Testing Association (Research Report No. 77-1, pp. 37–43). University of Minnesota, Department of Psychology, Psychometric Methods Program.

38.

Rasch

(1960). Probabilistic models for some intelligence and attainment test. University of Chicago Press.

39.

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

40.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

41.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

42.

Shih

C. L.

Liu

T. H.

Wang

W. C.

(2014). Controlling type I error rates in assessing DIF for logistic regression method combined with SIBTEST regression correction procedure and DIF-free-then-DIF strategy. Educational and Psychological Measurement, 74(6), 1018–1048. https://doi.org/10.1177/0013164413520545

43.

Shih

C. L.

Wang

W. C.

(2009). Differential item functioning detection using the multiple indicators, multiple causes method with a pure short anchor. Applied Psychological Measurement, 33(3), 184–199. https://doi.org/10.1177/0146621608321758

44.

Thibodeau

M. A.

Welch

P. G.

Sareen

Asmundson

G. J.

(2013). Anxiety disorders are independently associated with suicide ideation and attempts: Propensity score matching in two epidemiological samples. Depression and Anxiety, 30(10), 947–954. https://doi.org/10.1002/da.22203

45.

Tsutani

Miyata

Nakayama

Okumura

Adachi

Yoshimura

Okada

(2013). Oncologic outcomes of segmentectomy compared with lobectomy for clinical stage IA lung adenocarcinoma: Propensity score–matched analysis in a multicenter study. The Journal of Thoracic and Cardiovascular Surgery, 146(2), 358–364. https://doi.org/10.1016/j.jtcvs.2013.02.008

46.

Wang

W. C.

(2004). Effects of anchor item methods on the detection of differential item functioning within the family of Rasch models. Journal of Experimental Education, 72(3), 221–261. https://doi.org/10.3200/JEXE.72.3.221-261

47.

Wang

W. C.

(2008). Assessment of differential item functioning. Journal of Applied Measurement, 9(4), 387–408.

48.

Wang

W. C.

Shih

C. L.

(2010). MIMIC methods for assessing differential item functioning in polytomous items. Applied Psychological Measurement, 34(3), 166–180. https://doi.org/10.1177/0146621609355279

49.

Wang

W. C.

Shih

C. L.

Sun

G. W.

(2012). The DIF-Free-Then-DIF strategy for the assessment of differential item functioning. Educational and Psychological Measurement, 72(4), 687–708. https://doi.org/10.1177/0013164411426157

50.

Wang

W. C.

Yeh

Y. L.

(2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27(6), 479–498. https://doi.org/10.1177/0146621603259902

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.37 MB