Multidimensional forced choice (MFC) measures are gaining prominence in noncognitive assessment. Yet there has been little research on detecting differential item functioning (DIF) with models for forced choice measures. This research extended two well-known DIF detection methods to MFC measures. Specifically, the performance of Lord’s chi-square and item parameter replication (IPR) methods with MFC tests based on the Multi-Unidimensional Pairwise Preference (MUPP) model was investigated. The Type I error rate and power of the DIF detection methods were examined in a Monte Carlo simulation that manipulated sample size, impact, DIF source, and DIF magnitude. Both methods showed consistent power and were found to control Type I error well across study conditions, indicating that established approaches to DIF detection work well with the MUPP model. Lord’s chi-square outperformed the IPR method when DIF source was statement discrimination while the opposite was true when DIF source was statement threshold. Also, both methods performed similarly and showed better power when DIF source was statement location, in line with previous research. Study implications and practical recommendations for DIF detection with MFC tests, as well as limitations, are discussed.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–472). Addison-Wesley.
3.
BrooksS. P.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434–455. https://doi.org/10.1080/10618600.1998.10474787
4.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
5.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12(3), 253–260. https://doi.org/10.1177/014662168801200304
6.
CaoM.DrasgowF. (2019). Does forcing reduce faking? A meta-analytic review of forced choice personality measures in high-stakes situations. The Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/apl0000414
7.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
8.
ChunS.StarkS.KimE. S.ChernyshenkoO. S. (2016). MIMIC methods for detecting DIF among multiple groups: Exploring a new sequential-free baseline procedure. Applied Psychological Measurement, 40(7), 486–499. https://doi.org/10.1177/0146621616659738
9.
CohenA. S.KimS. H. (1993). A comparison of Lord’s χ2 and Raju’s area measures in detection of DIF. Applied Psychological Measurement, 17(1), 39–52. https://doi.org/10.1177/014662169301700109
10.
CohenA. S.KimS. H.BakerF. B. (1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement, 17(4), 335–350. https://doi.org/10.1177/014662169301700402
11.
DoornikJ. A. (2009). An object-oriented matrix language: Ox 6. Timberlake.
12.
du ToitM. (Ed.). (2003). IRT from SSI (pp. 34–35). Scientific Software International.
13.
FlowersC. P.OshimaT. C.RajuN. S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23(4), 309–326. https://doi.org/10.1177/01466219922031437
14.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
15.
HaebaraT. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144–149. https://doi.org/10.4992/psycholres1954.22.144
16.
HollandP. W.ThayerD. T. (1988). Differential item functioning and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp. 129–145). Erlbaum.
HulinC. L.DrasgowF.ParsonsC. K. (1983). Item response theory: Application to psychological measurement.
19.
JooS.LeeP.StarkS. (2022). Bayesian approaches for detecting differential item functioning using the Generalized Graded Unfolding Model. Applied Psychological Measurement, 46(2), 98–115. https://doi.org/10.1177/01466216211066606
20.
JooS. H.LeeP.StarkS. (2017). Evaluating anchor-item designs for concurrent calibration with the GGUM. Applied Psychological Measurement, 41(2), 83–96. https://doi.org/10.1177/0146621616673997
21.
JooS. H.LeeP.StarkS. (2018). Development of information functions and indices for the GGUM‐RANK multidimensional forced choice IRT model. Journal of Educational Measurement, 55(3), 357–372. https://doi.org/10.1111/jedm.12183
22.
KimS. H.CohenA. S. (1995). A comparison of Lord’s chi-square, Raju’s area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8(4), 291–312. https://doi.org/10.1207/s15324818ame0804_2
23.
LangerM. (2008). A reexamination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation. (Unpublished doctoral dissertation). University of North Carolina, Chapel Hill.
24.
LeeP.JooS.StarkS. (2021). Detecting DIF in multidimensional forced choice measures using the Thurstonian item response theory model. Organizational Research Methods, 24(4), 739–771. https://doi.org/10.1177/1094428120959822
25.
LeeP.JooS.StarkS.ChernyshenkoO. S. (2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
26.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.
27.
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
28.
NandaA. O.OshimaT.GagneP. (2006). DIFCUT: A SAS/IML program for conducting significance tests for differential functioning of items and tests (DFIT). Applied Psychological Measurement, 30(2), 150–151. https://doi.org/10.1177/0146621605280971
29.
OshimaT. C.RajuN. S.NandaA. O. (2006). A new method for assessing the statistical significance in the differential functioning of items and tests (DFIT) framework. Journal of Educational Measurement, 43(1), 1–17. https://doi.org/10.1111/j.1745-3984.2006.00001.x
30.
QiuX.-L. (2020). Assessing differential statement functioning in polytomous multidimensional pairwise comparison items. Journal of Applied Measurement, 21(3), 329–346.
31.
QiuX.-L.WangW.-C. (2021). Assessment of differential statement functioning in ipsative tests with multidimensional forced-choice items. Applied Psychological Measurement, 45(2), 79–94. https://doi.org/10.1177/0146621620965739
32.
RajuN. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197–207. https://doi.org/10.1177/014662169001400208
33.
RajuN. S.DrasgowF.SlindeJ. A. (1993). An empirical comparison of the area methods, Lords’ chi-square test, and the Mantel-Haenszel technique for assessing differential item functioning. Educational and Psychological Measurement, 53(2), 301–314. https://doi.org/10.1177/0013164493053002001
34.
RajuN. S.Fortmann-JohnsonK. A.KimW.MorrisS. B.NeringM. L.OshimaT. C. (2009). The item parameter replication method for detecting differential functioning in the polytomous DFIT framework. Applied Psychological Measurement, 33(2), 133–147. https://doi.org/10.1177/0146621608319514
35.
RajuN. S.van der LindenW. J.FleerP. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19(4), 353–368. https://doi.org/10.1177/014662169501900405
36.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests.
37.
RichardsonM.AbrahamC.BondR. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 353–387. https://doi.org/10.1037/a0026838
38.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001
39.
RobertsJ. S.FangH.CuiW.WangY. (2006). GGUM2004: A windows-based program to estimate parameters in the generalized graded unfolding model. Applied Psychological Measurement, 30(1), 64–65. https://doi.org/10.1177/0146621605280141
40.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 1–97. https://doi.org/10.1007/BF03372160
41.
SchmittN. (2012). Development of rationale and measures of noncognitive college student potential. Educational Psychologist, 47(1), 18–29. https://doi.org/10.1080/00461520.2011.610680
42.
SeybertJ.StarkS. (2012). Iterative linking with the differential functioning of items and tests (DFIT) method: Comparison of testwide and item parameter replication (IPR) critical values. Applied Psychological Measurement, 36(6), 494–515. https://doi.org/10.1177/0146621612445182
43.
SeybertJ.StarkS.ChernyshenkoO. S. (2014). Detecting DIF with ideal point models: A comparison of area and parameter difference methods. Applied Psychological Measurement, 38(2), 151–165. https://doi.org/10.1177/0146621613508306
44.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi- unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
45.
StarkS.ChernyshenkoO. S.DrasgowF. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. The Journal of Applied Psychology, 91(6), 1292–1306. https://doi.org/10.1037/0021-9010.91.6.1292
46.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?The Journal of Applied Psychology, 91(1), 25–39. https://doi.org/10.1037/0021-9010.91.1.25
47.
ThissenD.SteinbergL.WainerH. (1988). Use of item response theory in the study of group differences in trace lines. In WainerH.BraunH. I. (Eds.), Test validity (pp. 147–169). Erlbaum.
48.
TuN.JooS.LeeP.StarkS. (2022). Comparison of parameter estimation approaches for multi-unidimensional pairwise preference tests. Behavior Research Methods, 55(6), 1–23. https://doi.org/10.3758/s13428-022-01927-z
49.
TuN.KumarL. S.JooS.StarkS. (2024). Linking methods for multidimensional forced choice tests using the Multi-Unidimensional Pairwise Preference model. Applied Psychological Measurement, 48(3), 104–124. https://doi.org/10.1177/01466216241238741
50.
TuN.ZhangB.AngraveL.SunT. (2021). Bmggum: An R package for Bayesian estimation of the multidimensional generalized graded unfolding model with covariates. Applied Psychological Measurement, 45(7-8), 553–555. https://doi.org/10.1177/01466216211040488
51.
Van IddekingeC. H.PutkaD. J.CampbellJ. P. (2011). Reconsidering vocational interests for personnel selection: The validity of an interest-based selection test in relation to job knowledge, job performance, and continuance intentions. The Journal of Applied Psychology, 96(1), 13–33. https://doi.org/10.1037/a0021193
52.
WoodsC. M.CaiL.WangM. (2012). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532–547. https://doi.org/10.1177/0013164412464875
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
