Using a Generalized Logistic Regression Method to Detect Differential Item Functioning With Multiple Groups in Cognitive Diagnostic Tests

Abstract

Items with the presence of differential item functioning (DIF) will compromise the validity and fairness of a test. Studies have investigated the DIF effect in the context of cognitive diagnostic assessment (CDA), and some DIF detection methods have been proposed. Most of these methods are mainly designed to perform the presence of DIF between two groups; however, empirical situations may contain more than two groups. To date, only a handful of studies have detected the DIF effect with multiple groups in the CDA context. This study uses the generalized logistic regression (GLR) method to detect DIF items by using the estimated attribute profile as matching criteria. A simulation study is conducted to examine the performance of the two GLR methods, GLR-based Wald test (GLR-Wald) and GLR-based likelihood ratio test (GLR-LRT), in detecting the DIF items, the results based on the ordinary Wald test are also reported. Results show that (1) both GLR-Wald and GLR-LRT have more reasonable performance in controlling Type I error rates than the ordinary Wald test in most conditions; (2) the GLR method also produces higher empirical rejection rates than the ordinary Wald test in most conditions; and (3) using the estimated attribute profile as the matching criteria can produce similar Type I error rates and empirical rejection rates for GLR-Wald and GLR-LRT. A real data example is also analyzed to illustrate the application of these DIF detection methods in multiple groups.

Keywords

cognitive diagnostic assessment differential item functioning generalized logistic regression multiple groups

Get full access to this article

View all access options for this article.

References

Agresti

(2002). Categorical data analysis (2nd ed.). Wiley.

Bradley

J. V.

(1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31(2), 144–152. https://doi.org/10.1111/j.2044-8317.1978.tb00581.x

Chen

Woollacott

Pologe

Moore

G. P.

(2013). Stochastic aspects of motor behavior and their dependence on auditory feedback in experienced cellists. Frontiers in Human Neuroscience, 7(6), 419–437. https://doi.org/10.3389/fnhum.2013.00419

Chiu

C. Y.

Köhn

H. F.

Zheng

Henson

(2016). Joint maximum likelihood estimation for diagnostic classification models. Psychometrika, 81(4), 1069–1092. https://doi.org/10.1007/s11336-016-9534-9

Chiu

C. Y.

Sun

Bian

(2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355–375. https://doi.org/10.1007/s11336-017-9595-4

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

DeMars

C. E.

(2011). An analytic comparison of effect sizes for differential item functioning. Applied Measurement in Education, 24(3), 189–209. https://doi.org/10.1080/08957347.2011.580255

Dogan

Tatsuoka

(2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263–272. https://doi.org/10.1007/s10649-007-9099-8

10.

Feng

(2021). Effect size measures for differential item functioning in cognitive diagnostic models (Unpublished doctoral dissertation). Indiana University.

11.

Field

Miles

Field

(2012). Discovering statistics using R. Sage Publications.

12.

French

B. F.

Maller

S. J.

(2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67(3), 373–393. https://doi.org/10.1177/0013164406294781

13.

George

A. C.

Robitzsch

(2014). Multiple group cognitive diagnosis models, with an emphasis on differential item functioning. Psychological Test and Assessment Modeling, 56(4), 405–432.

14.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (unpublished doctoral dissertation). University of Illinois at Urbana.

15.

Holm

(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.

16.

Hou

de la Torre

(2015). Applying Wald test to detect multi-group DIF in CDM. Paper presented at the Annual Meeting of the National Council on Measurement in Education Conference.

17.

Hou

la Torre

J. d.

Nandakumar

(2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51(1), 98–125. https://doi.org/10.1111/jedm.12036

18.

Hou

Terzi

de la Torre

(2020). Wald test formulations in DIF detection of CDM data with the proportional reasoning test. International Journal of Assessment Tools in Education, 7(2), 145–158.

19.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

20.

Kassambara

(2021). rstatix: Pipe-friendly framework for basic statistical tests [Computer software manual] (R Package Version 0.7.0). https://CRAN.R-project.org/package=rstatix

21.

Kim

S.-H.

Cohen

A. S.

Park

T.-H.

(1995). Detection of differential item functioning in multiple groups. Journal of Educational Measurement, 32(3), 261–276. https://doi.org/10.1111/j.1745-3984.1995.tb00466.x

22.

F. M.

(2008). A modified higher-order DINA model for detecting differential item functioning and differential attribute functioning (Unpublished doctoral dissertation). University of Georgia.

23.

Wang

W. C.

(2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28–54. https://doi.org/10.1111/jedm.12061

24.

Liu

Xin

Tian

Liu

(2016). An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix. Acta Psychologica Sinica, 48(5), 588–598. https://doi.org/10.3724/sp.j.1041.2016.00588

25.

Liu

Yin

Xin

Shao

Yuan

(2019). A comparison of differential item functioning detection methods in cognitive diagnostic models. Frontiers in Psychology, 10, Article 1137. https://doi.org/10.3389/fpsyg.2019.01137

26.

de la Torre

Sorrel

Jiang

(2021a). GDINA: The generalized DINA model framework. R package version 2.8.0. https://CRAN.R-project.org/package=GDINA

27.

Terzi

de la Torre

(2021b). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37–53. https://doi.org/10.1177/0146621620965745

28.

Madison

M. J.

Bradshaw

L. P.

(2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75(3), 491–511. https://doi.org/10.1177/0013164414539162

29.

Magis

Béland

Tuerlinckx

De Boeck

(2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847

30.

Magis

Raîche

Béland

Gérard

(2011). A generalized logistic regression procedure to detect differential item functioning among multiple groups. International Journal of Testing, 11(4), 365–386. https://doi.org/10.1080/15305058.2011.602810

31.

Mantel

Haenszel

(1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748.

32.

Nye

C. D.

Bradburn

Olenick

Bialko

Drasgow

(2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678–709. https://doi.org/10.1177/1094428118761122

33.

Park

Y. S.

Lee

Y. S.

(2014). An extension of the DINA model using covariates: Examining factors affecting response probability and latent classification. Applied Psychological Measurement, 38(5), 376–390. https://doi.org/10.1177/0146621614523830

34.

Paulsen

Svetina

Feng

Valdivia

(2020). Examining the impact of differential item functioning on classification accuracy in cognitive diagnostic models. Applied Psychological Measurement, 44(4), 267–281. https://doi.org/10.1177/0146621619858675

35.

Penfield

R. D.

(2001). Assessing differential item functioning among multiple groups: A comparison of three mantel-haenszel procedures. Applied Measurement in Education, 14(3), 235–259. https://doi.org/10.1207/s15324818ame1403_3

36.

Philipp

Strobl

de la Torre

Zeileis

(2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43(1), 88–115. https://doi.org/10.3102/1076998617719728

37.

Qiu

X.-L.

Wang

W.-C.

(2019). Differential item functioning in diagnostic classification models. In von Davier

Lee

Y.-S.

(Eds) Handbook of diagnostic classification models (pp. 379–393). Springer International Publishing.

38.

R Development Core Team . (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

39.

Rosenblatt

(1956). A central limit theorem and a strong mixing condition. Proceedings of the National Academy of Sciences of the United States of America, 42(1), 43–47. https://doi.org/10.1073/pnas.42.1.43

40.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

41.

Shealy

Stout

(1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. https://doi.org/10.1007/bf02294572

42.

Svetina

Dai

Wang

(2017). Use of cognitive diagnostic model to study differential item functioning in accommodations. Behaviormetrika, 44(2), 313–349. https://doi.org/10.1007/s41237-017-0021-0

43.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x

44.

Wang

Guo

Bian

(2014). Comparison of DIF detecting methods in cognitive diagnostic test. Acta Psychologica Sinica, 46(12), 1923–1932. https://doi.org/10.3724/sp.j.1041.2014.01923

45.

Wilcox

(2005). Introduction to robust estimation and hypothesis testing (2nd ed.). Academic Press.

46.

Wilks

S. S.

(1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The Annals of Mathematical Statistics, 9(1), 60–62. https://doi.org/10.1214/aoms/1177732360

47.

Woods

C. M.

Cai

Wang

(2013). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532–547. https://doi.org/10.1177/0013164412464875

48.

Zhang

(2006). Detecting differential item functioning using the DINA model (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.61 MB