In recent decades, multidimensional forced-choice (MFC) tests have gained widespread popularity in organizational settings due to their effectiveness in reducing response biases. Detecting differential item functioning (DIF) is crucial in developing MFC tests, as it relates to test fairness and validity. However, existing methods appear insufficient for detecting DIF induced by the interaction between multiple covariates. Furthermore, for multi-category, ordered or continuous covariates, existing approaches often dichotomize them using a-priori cutoffs, commonly using the median of the covariates. This may lead to information loss and reduced power in detecting MFC DIF. To address these limitations, we propose a method to identify both main effect DIF and interactive DIF. This method can automatically search for the optimal cutoffs for ordered or continuous covariates without pre-defined cutoffs. We introduce the rationale behind the proposed method and evaluate its performance through three Monte Carlo simulation studies. Results demonstrate that the proposed method effectively identifies various DIF forms in MFC tests, thereby increasing detection power. Finally, we provide an empirical application to illustrate the practical applicability of the proposed method.
American Education Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). The standards for educational and psychological testing. AERA Publications.
2.
AndrichD. (1989). A probabilistic IRT model for unfolding preference data. Applied Psychological Measurement, 13(2), 193–216. https://doi.org/10.1177/014662168901300211
3.
Anguiano-CarrascoC.MacCannC.GeigerM.SeybertJ. M.RobertsR. D. (2015). Development of a forced-choice measure of typical-performance emotional intelligence. Journal of Psychoeducational Assessment, 33(1), 83–97. https://doi.org/10.1177/0734282914550387
4.
BaerR. A.SmithG. T.HopkinsJ.KrietemeyerJ.ToneyL. (2006). Using self-report assessment methods to explore facets of mindfulness. Assessment, 13(1), 27–45. https://doi.org/10.1177/1073191105283504
BartramD. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272. https://doi.org/10.1111/j.1468-2389.2007.00386.x
7.
BauerD. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077
8.
BelzakW. C. (2023). The multidimensionality of measurement bias in high-stakes testing: Using machine learning to evaluate Complex sources of differential item functioning. Educational Measurement: Issues and Practice, 42(1), 24–33. https://doi.org/10.1111/emip.12486
9.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/J.2517-6161.1995.TB02031.X
10.
BergerM.TutzG. (2016). Detection of uniform and nonuniform differential item functioning by item-focused trees. Journal of Educational and Behavioral Statistics, 41(6), 559–592. https://doi.org/10.3102/1076998616659371
11.
BollmannS.BergerM.TutzG. (2018). Item-focused trees for the detection of differential item functioning in partial credit models. Educational and Psychological Measurement, 78(5), 781–804. https://doi.org/10.1177/0013164417722179
12.
Bourion-BédèsS.SchwanR.LaprevoteV.BédèsA.BonnetJ.-L.BaumannC. (2015). Differential item functioning (DIF) of SF-12 and Q-LES-Q-SF items among French substance users. Health and Quality of Life Outcomes, 13, 172. https://doi.org/10.1186/s12955-015-0365-7.
13.
BradleyR. A.TerryM. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029
14.
BreimanL.FriedmanJ. H.OlshenR. A.StoneJ. C. (1984). Classification and regression trees. Wadsworth.
BrownA.BartramD. (2009). Development and psychometric properties of OPQ32r (Supplement to the OPQ32 technical manual). SHL Group.
17.
BrownA.Maydeu-OlivaresA. (2010). Issues that should not be overlooked in the dominance versus ideal point controversy. Industrial and Organizational Psychology, 3(4), 489–493. https://doi.org/10.1111/j.1754-9434.2010.01277.x
18.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
19.
BrownA.Maydeu-OlivaresA. (2012). Fitting a Thurstonian IRT model to forced-choice data using mplus. Behavior Research Methods, 44(4), 1135–1147. https://doi.org/10.3758/s13428-012-0217-x
20.
BrownA.Maydeu-OlivaresA. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52. https://doi.org/10.1037/a0030641
21.
BunjiK.OkadaK. (2020). Joint modeling of the two-alternative multidimensional forced-choice personality measurement and its response time by a Thurstonian D-diffusion item response model. Behavior Research Methods, 52(3), 1091–1107. https://doi.org/10.3758/s13428-019-01302-5
22.
BunjiK.OkadaK. (2022). Linear ballistic accumulator item response theory model for multidimensional multiple-alternative forced-choice measurement of personality. Multivariate Behavioral Research, 57(4), 658–678. https://doi.org/10.1080/00273171.2021.1896351
23.
BürknerP.-C.SchulteN.HollingH. (2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063
24.
Calderón CarvajalC.Ximénez GómezC.Lay-LisboaS.BriceñoM. (2021). Reviewing the structure of Kolb’s learning style inventory from factor analysis and Thurstonian item response theory (IRT) model approaches. Journal of Psychoeducational Assessment, 39(5), 593–609. https://doi.org/10.1177/07342829211003739
ChenC.-W.WangW.-C. (2014, April). Detecting differential statement functioning in Ipsative tests using the logistic regression method. Paper presented at the annual meeting of National Council on Measurement in Education, Philadelphia, PA.
27.
ChoS.DrasgowF.CaoM. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241–1252. https://doi.org/10.1037/pas0000132
28.
ChristiansenN. D.BurnsG. N.MontgomeryG. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267–307. https://doi.org/10.1207/s15327043hup1803_4
29.
CollinsP. H. (1990). Black feminist thought: Knowledge, consciousness, and the politics of empowerment. UnwinHyman.
30.
CostaP. T.TerraccianoA.MccraeR. R. (2001). Gender differences in personality traits across cultures: robust and surprising findings. Journal of Personality and Social Psychology, 81(2), 322–331. https://doi.org/10.1037/0022-3514.81.2.322
CutlerD. R.EdwardsT. C.BeardK. H.CutlerA.HessK. T.GibsonJ.LawlerJ. J. (2007). Random forests for classification in ecology. Ecology, 88(11), 2783–2792. https://doi.org/10.1890/07-0539.1
33.
DannerD.BlasiusJ.BreyerB.EiflerS.MenoldN.PaulhusD. L.RammstedtB.RobertsR. D.SchmittM.ZieglerM. (2016). Current challenges, new developments, and future directions in scale construction. European Journal of Psychological Assessment, 32(3), 175–180. https://doi.org/10.1027/1015-5759/a000375
34.
DengY.-Q.LiuX.-H.RodriguezM. A.XiaC.-Y. (2011). The five facet mindfulness questionnaire: Psychometric properties of the Chinese version. Mindfulness, 2(2), 123–128. https://doi.org/10.1007/s12671-011-0050-9
35.
DenisP. L.MorinD.GuindonC. (2010). Exploring the capacity of NEO PI-R facets to predict job performance in two French-Canadian samples. International Journal of Selection and Assessment, 18(2), 201–207. https://doi.org/10.1111/j.1468-2389.2010.00501.x
36.
Díaz-UriarteR.Alvarez de AndrésS. (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics, 7(1), 3. https://doi.org/10.1186/1471-2105-7-3
37.
FinchW. H.Hernández FinchM. E.FrenchB. F. (2015). Recursive partitioning to identify potential causes of differential item functioning in cross-national data. International Journal of Testing, 16(1), 21–53. https://doi.org/10.1080/15305058.2015.1039644
38.
FrickS. (2022). Modeling faking in the multidimensional forced-choice format: The faking mixture model. Psychometrika, 87(2), 773–794. https://doi.org/10.1007/s11336-021-09818-6
39.
GriffithR. L.ChmielowskiT.YoshitaY. (2007). Do applicants fake? An examination of the frequency of applicant faking behavior. Personnel Review, 36(3), 341–355. https://doi.org/10.1108/00483480710731310
40.
GuenoleN.BrownA. A.CooperA. J. (2018). Forced-Choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526. https://doi.org/10.1177/1073191116641181
41.
GuoZ.WangD.CaiY.TuD. (2023). An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures. Educational and Psychological Measurement, Advanced online publication. https://doi.org/10.1177/00131644231171193
42.
HallquistM. N.WileyJ. F. (2018). Mplus automation: An R package for facilitating large-scale latent variable analyses in mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621–638. https://doi.org/10.1080/10705511.2017.1402334
43.
HayesT.UsamiS.JacobucciR.McArdleJ. J. (2015). Using classification and regression trees (CART) and random forests to analyze attrition: Results from two simulations. Psychology and Aging, 30(4), 911–929. https://doi.org/10.1037/pag0000046
44.
HeggestadE. D.MorrisonM.ReeveC. L.McCloyR. A. (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91(1), 9–24. https://doi.org/10.1037/0021-9010.91.1.9
45.
HontangasP. M.de la TorreJ.PonsodaV.LeenenI.MorilloD.AbadF. J. (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851
46.
HungS.-P.HuangH.-Y. (2022). Forced-Choice ranking models for Raters’ ranking data. Journal of Educational and Behavioral Statistics, 47(5), 603–634. https://doi.org/10.3102/10769986221104207
47.
JacksonD. N.WroblewskiV. R.AshtonM. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution?Human Performance, 13(4), 371–388. https://doi.org/10.1207/s15327043hup1304_3
48.
JooS. H.LeeP.StarkS. (2021). Modeling multidimensional forced choice measures with the Zinnes and Griggs pairwise preference item response theory model. Multivariate Behavioral Research, 58(2), 241–261. https://doi.org/10.1080/00273171.2021.1960142
49.
KhaliliaM.ChakrabortyS.PopescuM. (2011). Predicting disease risks from highly imbalanced data using random forest. BMC Medical Informatics and Decision Making, 11, 51. https://doi.org/10.1186/1472-6947-11-51.
50.
KimE. S.JooS.-H.LeeP.WangY.StarkS. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870–887. https://doi.org/10.1080/10705511.2016.1196108
51.
KombozB.StroblC.ZeileisA. (2016). Tree-Based global model tests for polytomous rasch models. Educational and Psychological Measurement, 78(1), 128–166. https://doi.org/10.1177/0013164416664394
52.
LeeH.SmithW. Z. (2020a). A Bayesian random block item response theory model for forced-choice formats. Educational and Psychological Measurement, 80(3), 578–603. https://doi.org/10.1177/0013164419871659
53.
LeeH.SmithW. Z. (2020b). Fit indices for measurement invariance tests in the Thurstonian IRT model. Applied Psychological Measurement, 44(4), 282–295. https://doi.org/10.1177/0146621619893785
54.
LeeP.JooS.-H.StarkS. (2021). Detecting DIF in multidimensional forced choice measures using the Thurstonian item response theory model. Organizational Research Methods, 24(4), 739–771. https://doi.org/10.1177/1094428120959822
55.
LeeP.JooS.-H.StarkS.ChernyshenkoO. S. (2019). GGUM-RANK Statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
56.
LooR. (1999). Issues in factor-analyzing ipsative measures: The learning style inventory (LSI-1985) example. Journal of Business & Psychology, 14(1), 149–154. https://doi.org/10.1023/A:1022918803653
MacCallumR. C.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1), 19–40. https://doi.org/10.1037/1082-989X.7.1.19
59.
MagisD.BélandS.TuerlinckxF.BoeckP. d. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847
60.
McCraeR. R.CostaP. T. (1992). Discriminant validity of NEO-PIR facet scales. Educational and Psychological Measurement, 52(1), 229–237. https://doi.org/10.1177/001316449205200128
61.
MerkJ.SchlotzW.FalterT. (2017). The motivational value systems questionnaire (MVSQ): Psychometric analysis using a forced choice Thurstonian IRT model. Frontiers in Psychology, 8, Article 1626. https://doi.org/10.3389/fpsyg.2017.01626
62.
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
NgV.LeeP.HoM.-H. R.KuykendallL.StarkS.TayL. (2020). The development and validation of a multidimensional forced-choice format character measure: Testing the Thurstonian IRT approach. Journal of Personality Assessment, 103(2), 224–237. https://doi.org/10.1080/00223891.2020.1739056
65.
NyeC. D. (2011). The development and validation of effect size measures for IRT and CFA studies of measurement equivalence (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.
66.
PaulhusD. L. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverR. P.WrightsmanL. S. (Eds.), Measures of social psychological attitudes (pp. 17–59). Academic.
67.
PengS.ManK.VeldkampB. P.CaiY.TuD. (2023). A mixture model for random responding behavior in forced-choice noncognitive assessment: Implication and application in organizational research. Organizational Research Methods, Advanced online publication. https://doi.org/10.1177/10944281231181642
68.
Principles for the validation and use of personnel selection procedures. (2018). Industrial and Organizational Psychology: Perspectives on Science and Practice, 11(S1), 1–97. https://doi.org/10.1017/iop.2018.195
69.
QiuX.-L.WangW.-C. (2021). Assessment of differential statement functioning in ipsative tests with multidimensional forced-choice items. Applied Psychological Measurement, 45(2), 79–94. https://doi.org/10.1177/0146621620965739
70.
RobertC.LeeW. C.ChanK.-Y. (2006). An empirical analysis of measurement equivalence with the INDCOL measure of individualism and collectivism: Implications for valid cross-cultural inference. Personnel Psychology, 59(1), 65–99. https://doi.org/10.1111/j.1744-6570.2006.00804.x
71.
SalgadoJ. F.AndersonN.TaurizG. (2015). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, 88(4), 797–834. https://doi.org/10.1111/joop.12098
72.
SchmittN.OswaldF. L. (2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. Journal of Applied Psychology, 91(3), 613–621. https://doi.org/10.1037/0021-9010.91.3.613
73.
ShafferJ. A.PostlethwaiteB. E. (2013). The validity of conscientiousness for predicting job performance: A meta-analytic test of two hypotheses. International Journal of Selection and Assessment, 21(2), 183–199. https://doi.org/10.1111/ijsa.12028
74.
SitserT.van der LindenD.BornM. P. (2013). Predicting sales performance criteria with personality measures: The use of the general factor of personality, the big five and narrow traits. Human Performance, 26(2), 126–149. https://doi.org/10.1080/08959285.2013.765877
75.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
76.
StarkS.ChernyshenkoO. S.DrasgowF. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292–1306. https://doi.org/10.1037/0021-9010.91.6.1292.
77.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items. Organizational Research Methods, 15(3), 463–487. https://doi.org/10.1177/1094428112444611
78.
StroblC.KopfJ.ZeileisA. (2013). Rasch trees: A new method for detecting differential item functioning in the Rasch model. Psychometrika, 80(2), 289–316. https://doi.org/10.1007/s11336-013-9388-3
79.
StroblC.MalleyJ.TutzG. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323–348. https://doi.org/10.1037/a0016973
80.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
81.
TayL.HuangQ.VermuntJ. K. (2015). Item response theory with covariates (IRT-C): Assessing item recovery and differential item functioning for the three-parameter logistic model. Educational and Psychological Measurement, 76(1), 22–42. https://doi.org/10.1177/0013164415579488
82.
TettR. P.ChristiansenN. D.RobieC.SimonetD. V. (2011). International survey of personality test use: An American baseline. In 15th Conference of the European Association of Work and Organizational Psychology, Maastricht, The Netherlands.
TutzG.BergerM. (2016). Item-focussed trees for the identification of items in differential item functioning. Psychometrika, 81(3), 727–750. https://doi.org/10.1007/s11336-015-9488-3
85.
WangW.-C.QiuX.-L.ChenC.-W.RoS.JinK.-Y. (2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183
86.
WetzelE.BöhnkeJ. R.BrownA. (2016). Response biases. In LeongF. T. L.BartramD.CheungF. M.GeisingerK. F.IliescuD. (Eds.), The ITC international handbook of testing and assessment (pp. 349–363). Oxford University Press.
87.
WinkelspechtC.LewisP.ThomasA. (2006). Potential effects of faking on the NEO-PI-R: Willingness and ability to fake changes who gets hired in simulated selection decisions. Journal of Business and Psychology, 21(2), 243–259. https://doi.org/10.1007/s10869-006-9027-4
88.
ZhangB.TuN.AngraveL.ZhangS.SunT.TayL.LiJ. (2023). The generalized Thurstonian unfolding model (GTUM): Advancing the modeling of forced-choice data. Organizational Research Methods, Advanced online publication. https://doi.org/10.1177/10944281231210481
89.
ZiekyM. J. (2015). Developing fair tests. In Handbook of test development (pp. 97–115). Routledge.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
