The field of psychometrics has made remarkable progress in developing item response theory (IRT) models for analyzing multidimensional forced choice (MFC) measures. This study introduces an innovative method that enhances the latent trait estimation of the Multi-Unidimensional Pairwise Preference (MUPP) model by incorporating latent regression modeling. To validate the efficacy of the new method, we conducted a comprehensive simulation study. The results of the study provide compelling evidence that the proposed latent regression MUPP (LR-MUPP) model significantly improves the accuracy of the latent trait estimation. This study opens new avenues for future research and encourages further development and refinement of MFC IRT models and their applications.
AdamsR. J.WilsonM.WuM. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76. https://doi.org/10.2307/1165238
2.
AndrichD. (1978). Relationships between the Thurstone and Rasch approaches to item scaling. Applied Psychological Measurement, 2(3), 451–462. https://doi.org/10.1177/014662167800200319
3.
AndrichD. (1989). A probabilistic IRT model for unfolding preference data. Applied Psychological Measurement, 13(2), 193–216. https://doi.org/10.1177/014662168901300211
4.
AndrichD. (1995). Hyperbolic cosine latent trait models for unfolding direct responses and pairwise preferences. Applied Psychological Measurement, 19(3), 269–290. https://doi.org/10.1177/014662169501900306
5.
AriasV. B.GarridoL. E.JenaroC.Martínez-MolinaA.AriasB. (2020). A little garbage in, lots of garbage out: Assessing the impact of careless responding in personality survey data. Behavior Research Methods, 52(6), 2489–2505. https://doi.org/10.3758/s13428-020-01401-8
BolsinovaM.TijmstraJ. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71(1), 13–38. https://doi.org/10.1111/bmsp.12104
8.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
9.
CaoM.DrasgowF. (2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/apl0000414
10.
De BoeckP.ChoS. J.WilsonM. (2011). Explanatory secondary dimension modeling of latent differential item functioning. Applied Psychological Measurement, 35(8), 583–603. https://doi.org/10.1177/0146621611428446
11.
de la TorreJ. (2009). Improving the quality of ability estimates through multidimensional scoring and incorporation of ancillary variables. Applied Psychological Measurement, 33(6), 465–485. https://doi.org/10.1177/0146621608329890
12.
de la TorreJ.SongH. (2009). Simultaneous estimation of overall and domain abilities: A higher-order IRT model approach. Applied Psychological Measurement, 33(8), 620–639. https://doi.org/10.1177/0146621608326423
13.
DoornikJ. A. (2021). An object-oriented matrix programming language Ox 9. Timberlake Consultants Press. [Computer software].
14.
FerrandoP. J.Lorenzo-SevaU. (2007). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31(6), 525–543. https://doi.org/10.1177/0146621606295197
GelmanA. (1995). Inference and monitoring convergence. In GilksW. R.SpiegelhalterD. J.RichardsonS. (Eds.), Practical Markov chain Monte Carlo (pp. 131–144). Chapman & Hall.
19.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
20.
GoldammerP.AnnenH.StöckliP. L.JonasK. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4), Article 101384. https://doi.org/10.1016/j.leaqua.2020.101384
21.
GuoZ.WangD.CaiY.TuD. (2023). An item response theory model for incorporating response times in forced-choice measures. Educational and Psychological Measurement, 84(3), Article 00131644231171193. https://doi.org/10.1177/00131644231171193
22.
HontangasP. M.De La TorreJ.PonsodaV.LeenenI.MorilloD.AbadF. J. (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851
23.
HuangH. Y. (2023). Diagnostic classification model for forced-choice items and noncognitive tests. Educational and Psychological Measurement, 83(1), 146–180. https://doi.org/10.1177/00131644211069906
24.
HuangJ. L.BowlingN. A.LiuM.LiY. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311. https://doi.org/10.1007/s10869-014-9357-6
25.
JooS. H.LeeP.StarkS. (2017). Evaluating anchor-item designs for concurrent calibration with the GGUM. Applied Psychological Measurement, 41(2), 83–96. https://doi.org/10.1177/0146621616673997
26.
JooS. H.LeeP.StarkS. (2018). Development of information functions and indices for the GGUM‐RANK multidimensional forced choice IRT model. Journal of Educational Measurement, 55(3), 357–372. https://doi.org/10.1111/jedm.12183
27.
JooS. H.LeeP.StarkS. (2020). Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring. Behavior Research Methods, 52(2), 761–772. https://doi.org/10.3758/s13428-019-01274-6
28.
JooS. H.LeeP.StarkS. (2022). The explanatory generalized graded unfolding model: Incorporating collateral information to improve the latent trait estimation accuracy. Applied Psychological Measurement, 46(1), 3–18. https://doi.org/10.1177/01466216211051717
29.
JooS. H.LeeP.StarkS. (2023). Modeling multidimensional forced choice measures with the Zinnes and Griggs pairwise preference item response theory model. Multivariate Behavioral Research, 58(2), 241–261. https://doi.org/10.1080/00273171.2021.1960142
30.
KamC. C. S. (2019). Careless responding threatens factorial analytic results and construct validity of personality measure. Frontiers in Psychology, 10(1), 1258. https://doi.org/10.3389/fpsyg.2019.01258
31.
KubingerK. D. (2009). Applications of the linear logistic test model in psychometric research. Educational and Psychological Measurement, 69(2), 232–244. https://doi.org/10.1177/0013164408322021
32.
LeeP.JooS.SonM. (2023). Detecting careless respondents in multidimensional forced choice data: An application of lz person-fit statistic to the TIRT model. Journal of Business and Psychology, 39(3), 1–24. https://doi.org/10.1007/s10869-023-09911-w
33.
LeeP.JooS. H.LeeS. (2019). Examining stability of personality profile solutions between Likert-type and multidimensional forced choice measure. Personality and Individual Differences, 142(1), 13–20. https://doi.org/10.1016/j.paid.2019.01.022
34.
LeeP.JooS. H.StarkS. (2021). Detecting DIF in multidimensional forced choice measures using the Thurstonian item response theory model. Organizational Research Methods, 24(4), 739–771. https://doi.org/10.1177/1094428120959822
35.
LeeP.JooS. H.StarkS.ChernyshenkoO. S. (2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
36.
LeeP.JooS. H.ZhouS.SonM. (2022). Investigating the impact of negatively keyed statements on multidimensional forced-choice personality measures: A comparison of partially ipsative and IRT scoring methods. Personality and Individual Differences, 191(1), 111555. https://doi.org/10.1016/j.paid.2022.111555
37.
LeeP.SonM.ZhouS.JooS.JiaZ.ChengV. (2025). The journey of forced choice measurement over 80 years: Past, present, and future. Organizational Research Methods, 28(4), Article 10944281251350687. https://doi.org/10.1177/10944281251350687
38.
LinY.BrownA.WilliamsP. (2023). Multidimensional forced-choice CAT with dominance items: An empirical comparison with optimal static testing under different desirability matching. Educational and Psychological Measurement, 83(2), 322–350. https://doi.org/10.1177/00131644221077637
39.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83. https://doi.org/10.1016/j.jrp.2013.09.008
MislevyR. J.BeatonA. E.KaplanB.SheehanK. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161. https://doi.org/10.1111/j.1745-3984.1992.tb00371.x
42.
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
43.
NyeC. D.JooS. H.ZhangB.StarkS. (2020). Advancing and evaluating IRT model data fit indices in organizational research. Organizational Research Methods, 23(3), 457–486. https://doi.org/10.1177/1094428119833158
44.
O'NeillT. A.LewisR. J.LawS. J.LarsonN.HancockS.RadanJ.CarswellJ. J. (2017). Forced-choice pre-employment personality assessment: Construct validity and resistance to faking. Personality and Individual Differences, 115(1), 120–127. https://doi.org/10.1016/j.paid.2016.03.075
45.
OswaldF. L.HoughL. M. (2011). Personality and its assessment in organizations: Theoretical and empirical developments. In ZedeckS. (Ed.), APA handbook of industrial and organizational psychology: Vol. 2. Selecting and developing members for the organization (pp. 153–184). American Psychological Association.
46.
PatzR. J.JunkerB. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.2307/1165199
47.
PengS.ManK.VeldkampB. P.CaiY.TuD. (2023). A mixture model for random responding behavior in forced-choice noncognitive assessment: Implication and application in organizational research. Organizational Research Methods, 27(4), 10944281231181642. https://doi.org/10.1177/10944281231181642
48.
PoropatA. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996
49.
RobertsG. O.RosenthalJ. S. (2001). Optimal scaling for various Metropolis-Hastings algorithms. Statistical Science, 16(4), 351–367. https://doi.org/10.1214/ss/1015346320
50.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001
51.
RosenthalJ. S. (2011). Optimal proposal distributions and adaptive MCMC. In BrooksS.GelmanA.JonesG.MengX. L. (Eds.), Handbook of Markov chain Monte Carlo. Chapman and Hall–CRC Press.
52.
SalgadoJ. F. (1997). The Five Factor Model of personality and job performance in the European Community. The Journal of Applied Psychology, 82(1), 30–43. https://doi.org/10.1037/0021-9010.82.1.30
53.
SalgadoJ. F.LadoM. (2018). Faking resistance of a quasi-ipsative forced-choice personality inventory without algebraic dependence. Revista de Psicología del Trabajo y de las Organizaciones, 34(3), 213–216. https://doi.org/10.5093/jwop2018a23
54.
SalgadoJ. F.TaurizG. (2014). The Five-Factor Model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23(1), 3–30. https://doi.org/10.1080/1359432x.2012.716198
55.
SchmittN.OswaldF. L. (2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. The Journal of Applied Psychology, 91(3), 613–621. https://doi.org/10.1037/0021-9010.91.3.613
56.
SpeerA. B.WegmeyerL. J.TenbrinkA. P.DelacruzA. Y.ChristiansenN. D.SalimR. M. (2023). Comparing forced-choice and single-stimulus personality scores on a level playing field: A meta-analysis of psychometric properties and susceptibility to faking. The Journal of Applied Psychology, 108(11), 1812–1833. https://doi.org/10.1037/apl0001099
57.
StarkS.ChernyshenkoO. S.ChanK. Y.LeeW. C.DrasgowF. (2001). Effects of the testing situation on item responding: Cause for concern. The Journal of Applied Psychology, 86(5), 943–953. https://doi.org/10.1037/0021-9010.86.5.943
58.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
59.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487. https://doi.org/10.1177/1094428112444611
60.
StarkS.DrasgowF. (2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26(2), 208–227. https://doi.org/10.1177/01421602026002007
61.
TuN.JooS.StarkS. (2025). Examining the tradeoffs of exposure control and collateral information with multidimensional forced-choice computerized adaptive testing. Behavior Research Methods, 57(7), 207. https://doi.org/10.3758/s13428-025-02712-4
62.
van den NoortgateW.De BoeckP. (2005). Assessing and explaining differential item functioning using logistic mixed models. Journal of Educational and Behavioral Statistics, 30(4), 443–464. https://doi.org/10.3102/10769986030004443
63.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z
64.
von DavierM.SinharayS. (2013). Analytics in international large-scale assessments: Item response theory and population models. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), Handbook of international large-scale assessment (pp. 155–274). CRC Press.
65.
von DavierM.SinharayS.OranjeA.BeatonA. (2006). The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions. In RaoC. R.SinharayS. (Eds.), Handbook of statistics (Vol. 26, pp. 1039–1055). Elsevier. https://doi.org/10.1016/s0169-7161(06)26032-2
66.
WangQ.ZhengY.LiuK.CaiY.PengS.TuD. (2023). Item selection methods in multidimensional computerized adaptive testing for forced-choice items using Thurstonian IRT model. Behavior Research Methods, 56(2), 1–15. https://doi.org/10.3758/s13428-022-02037-6
67.
WangW. C.QiuX. L.ChenC. W.RoS.JinK. Y. (2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183
68.
WilsonM.De BoeckP. (2004). Descriptive and explanatory item response models. In Explanatory item response models (pp. 43–74). Springer.
69.
WilsonM.ZhengX.McGuireL. (2012). Formulating latent growth using an explanatory item response model approach. Journal of Applied Measurement, 13(1), 1–22.
70.
ZhangB.SunT.DrasgowF.ChernyshenkoO. S.NyeC. D.StarkS.WhiteL. A. (2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569–590. https://doi.org/10.1177/1094428119836486
71.
ZhangB.TuN.AngraveL.ZhangS.SunT.TayL.LiJ. (2023). The Generalized Thurstonian Unfolding Model (GTUM): Advancing the modeling of forced-choice data. Organizational Research Methods, 27(4), Article 10944281231210481. https://doi.org/10.1177/10944281231210481
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
