This study compared maximum a posteriori (MAP), expected a posteriori (EAP), and Markov Chain Monte Carlo (MCMC) approaches to computing person scores from the Multi-Unidimensional Pairwise Preference Model. The MCMC approach used the No-U-Turn sampling (NUTS). Results suggested the EAP with fully crossed quadrature and the NUTS outperformed the others when there were fewer dimensions. In addition, the NUTS produced the most accurate estimates in larger dimension conditions. The number of items per dimension had the largest effect on person parameter recovery.
AonH. (2015). Trends in global employee engagement report. Aon Corp.
2.
BezansonJ.EdelmanA.KarpinskiS.ShahV. B. (2017). Julia: A fresh approach to numerical computing.
3.
BrooksS. P.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational & Graphical Statistics, 7(4), 434–455. https://doi.org/10.1080/10618600.1998.10474787
4.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502.
5.
CaoM.DrasgowF. (2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/APL0000414
6.
DrasgowF.StarkS.ChernyshenkoO. S.NyeC. D.HulinC. L.WhiteL. A. (2012). Development of the tailored adaptive personality assessment System (TAPAS) to support army personnel selection and classification decisions. https://apps.dtic.mil/sti/citations/ADA564422
7.
GeH.XuK.GhahramaniZ. (2018). Turing: A language for flexible probabilistic inference. In Proceedings of the twenty-first international conference on artificial intelligence and statistics (pp. 1682–1690). Available at: https://proceedings.mlr.press/v84/ge18b.html
8.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistics Science, 7(4), 457–472.
9.
HoffmanM.GelmanA. (2014). The No-U-Turn Sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623. https://doi.org/10.48550/arXiv.1111.4246
10.
HontangasP. M.de la TorreJ.PonsodaV.LeenenI.MorilloD.AbadF. J. (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851
11.
JooS. H.LeeP.StarkS. (2020). Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring. Behavior Research Methods, 52(2), 761–772.
12.
LeeP.JooS.-H.StarkS.ChernyshenkoO. S. (2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
13.
LuoY.JiaoH. (2018). Using the stan program for bayesian item response theory. Educational and Psychological Measurement, 78(3), 384–408. https://doi.org/10.1177/0013164417693666
14.
McElreathR. (2020). Statistical rethinking (2nd ed.). Chapman and Hall/CRC.
15.
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
16.
PressW. H.TeukolskyS. A.VetterlingW. T.FlanneryB. P. (2007). Numerical recipes: The art of scientific computing: Cambridge University Press. Available at: https://doi.org/10.1142/S0218196799000199
17.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32.
18.
SchulteN.HollingH.BürknerP. C. (2020). Can high-dimensional questionnaires resolve the ipsativity issue of forced-choice response formats?Educational and Psychological Measurement, 81(2), 262–289. https://doi.org/10.1177/0013164420934861
19.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
20.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487. https://doi.org/10.1177/1094428112444611
21.
TuN.JooS.LeeP.StarkS. (2022). Comparison of parameter estimation approaches for multi-unidimensional pairwise preference tests. Behavior Research Methods, 55(6), 2764–2768. https://doi.org/10.3758/s13428-022-01927-z
22.
ZhangB.SunT.DrasgowF.ChernyshenkoO. S.NyeC. D.StarkS.WhiteL. A. (2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569–590. https://doi.org/10.1177/1094428119836486
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
