This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.
BakemanR. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379–384.
2.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.
3.
ClauserB.MazorK.HambletonR. K. (1993). The effects of purification of matching criterion on the identification of DIF using the Mantel-Haenszel procedure. Applied Measurement in Education, 6(4), 269–279.
4.
CohenA. S.KimS.-H.WollackJ. A. (1996). An investigation of the likelihood ratio test for detection of differential item functioning. Applied Psychological Measurement, 20, 15–26.
5.
CohenJ. (2013). Statistical power analysis for the behavioral sciences. Academic Press.
6.
de la TorreJ. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.
7.
de la TorreJ.DouglasJ. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.
8.
de la TorreJ.LeeY. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355–373.
9.
GeorgeA. C.RobitzschA. (2014). Multiple group cognitive diagnosis models, with an emphasis on differential item functioning. Psychological Test and Assessment Modeling, 56(4), 405–432.
10.
González-BetanzosF.AbadF. J. (2012). The effects of purification and the evaluation of differential item functioning with the likelihood ratio test. Methodology, 8, 134–145.
11.
HaertelE. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.
12.
HensonR. A.TemplinJ. L.WillseJ. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.
13.
HollandP. W.ThayerD. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.
14.
HolmS. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.
15.
HouL.de la TorreJ.NandakumarR. (2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51, 98–125.
16.
HouL.TerziR.de la TorreJ. (2020). Wald test formulations in DIF detection of CDM data with the proportional reasoning test. International Journal of Assessment Tools in Education, 7(2), 145–158.
17.
JohnsonM.LeeY.-S.SachdevaR. J.ZhangJ.WaldmanM.ParkJ. Y. (2013, April). Examination of gender differences using the multiple groups DINA model[Paper presentation]. Annual meeting of the National Council on Measurement in Education, San Francisco, CA, United States.
18.
KassambaraA. (2020). rstatix: Pipe-friendly framework for basic statistical tests [Computer software manual] (R Package Version 0.4.0). https://CRAN.R-project.org/package=rstatix
19.
LiF. (2008). A modified higher-order DINA model for detecting differential item functioning and differential attribute functioning (Unpublished doctoral dissertation). University of Georgia, Georgia.
20.
LiX.WangW.-C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52, 28–54.
21.
LiuY.YinH.XinT.ShaoL.YuanL. (2019). A comparison of differential item functioning detection methods in cognitive diagnostic models. Frontiers in Psychology, 10, Article 1137.
22.
MaW.de la TorreJ. (2019a). Category-level model selection for the sequential G-DINA model. Journal of Educational and Behavioral Statistics, 44, 45–77.
23.
MaW.de la TorreJ. (2019b). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73, 142–163.
24.
MaW.de la TorreJ. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93, 1–26.
25.
MaW.IaconangeloC.de la TorreJ. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200–217.
26.
MagisD.BélandS.TuerlinckxF.De BoeckP. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862.
27.
Maydeu-OlivaresA.CaiL. (2006). A cautionary note on using G2(dif) to assess relative model fit in categorical data analysis. Multivariate Behavioral Research, 41, 55–64.
NicholsP. D.ChipmanS. F.BrennanR. L. (Eds.). (1995). Cognitively diagnostic assessments. Lawrence Erlbaum.
30.
OlejnikS.AlginaJ. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447.
31.
PaulsenJ.SvetinaD.FengY.ValdiviaM. (2020). Examining the impact of differential item functioning on classification accuracy in cognitive diagnostic models. Applied Psychological Measurement, 44, 267–281. https://doi.org/10.1177/0146621619858675
32.
PhilippM.StroblC.de la TorreJ.ZeileisA. (2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43, 88–115.
33.
QiuX.-L.LiX.WangW.-C. (2019). Differential item functioning in diagnostic classification models. In von DavierM.LeeY.-S. (Eds.), Handbook of diagnostic classification models (pp. 379–393). Springer.
34.
RoseN.von DavierM.NagengastB. (2017). Modeling omitted and not-reached items in IR models. Psychometrika, 82(3), 795–819.
35.
RossiG.van der ArkL.SlooreH. (2007). Factor analysis of the Dutch-language version of the MCMI-III. Journal of Personality Assessment, 88, 144–157.
36.
SessomsJ.HensonR. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16(1), 1–17. https://doi.org/10.1080/15366367.2018.1435104
37.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194.
38.
SvetinaD.DaiS.WangX. (2017). Use of cognitive diagnostic model to study differential item functioning in accommodations. Behaviormetrika, 44(2), 313–349.
39.
TatsuokaK. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.
40.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.
41.
TerziR. (2017). New Q-matrix validation procedures (Unpublished doctoral dissertation). Rutgers, The State University of New Jersey
42.
TerziR.SenS. (2019). A nondiagnostic assessment for diagnostic purposes: Q-matrix validation and item-based model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9, 1–11.
43.
WaldA. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54, 426–482.
44.
WangW.-C.YehY.-L. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479–498.
45.
XuX.DavierM. (2008). Comparing multiple-group multinomial log-linear models for multidimensional skill distributions in the general diagnostic model (ETS Research Report Series, 2008(1)). https://files.eric.ed.gov/fulltext/EJ1111190.pdf
46.
ZhangW. (2006). Detecting differential item functioning using the DINA model (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
