A computerized adaptive testing (CAT) solution for tests with multidimensional pairwise-comparison (MPC) items, aiming to measure career interest, value, and personality, is rare. This paper proposes new item selection and exposure control methods for CAT with dichotomous and polytomous MPC items and present simulation study results. The results show that the procedures are effective in selecting items and controlling within-person statement exposure with no loss of efficiency. Implications are discussed in two applications of the proposed CAT procedures: a work attitude test with dichotomous MPC items and a career interest assessment with polytomous MPC items.
AitchisonJ. (1982). The statistical analysis of compositional data. Journal of the Royal Statistical Society. Series B (Methodological), 44(2), 139–177. https://doi.org/10.1111/j.2517-6161.1982.tb01195.x
ChenC.-W.WangW.-C.ChiuM.-M.RoS (2020). Item selection and exposure control methods for computerized adaptive testing with multidimensional ranking items. Journal of Educational Measurement, 57(3), 343–369. https://doi.org/10.1111/jedm.12252
6.
ChristiansenN. D.BurnsG. N.MontgomeryG. E (2009). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267–307. https://doi.org/10.1207/s15327043hup1803_4
7.
HetterR. D.SympsonJ. B. (1997). Item-exposure in CAT-ASVAB. In SandsW. A.WatersJ. R.,McBrideJ. R. (Eds), Computerized adaptive testing: From inquiry to operation. American Psychological Association (pp. 141–144). https://doi.org/10.1037/10244-014
8.
HontangasP. M.de la TorreJ.PonsodaV.LeenenI.MorilloD.AbadF. J (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851
9.
HontangasP. M.LeenenI.de la TorreJ.PonsodaV.MorilloD.AbadF. J (2016). Traditional scores versus IRT estimates on forced-choice tests based on a dominance model. Psicothema, 28(1), 76–82. https://doi.org/10.7334/psicothema2015.204
10.
JacksonD. N.WroblewskiV. R.AshtonM. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution?Human Performance, 13(4), 371–388. https://doi.org/10.1207/s15327043hup1304_3
11.
JooS.-H.LeeP.StarkS. (2020). Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring. Behavior Research Methods, 52(2), 761–772. https://doi.org/10.3758/s13428-019-01274-6
12.
MatthewsG.OddyK. (1997). Ipsative and normative scales in adjectival measurement of personality: Problems of bias and discrepancy. International Journal of Selection and Assessment, 5(3), 169–182. https://doi.org/10.1111/1468-2389.00057
13.
MeadeA. W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77(4), 531–551. https://doi.org/10.1348/0963179042596504
14.
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
15.
MulderJ.van der LindenW. J (2009). Multidimensional adaptive testing with optimal design criteria for item selection. Psychometrika, 74(2), 273–296. https://doi.org/10.1007/s11336-008-9097-5
16.
QiuX.-L.WangW.-C. (2016, April). Item response theory models for ipsative tests with polytomous multidimensional forced-choice items. Paper presented at the annual meeting of the National Council on Measurement in Education.
17.
RevueltaJ.PonsodaV. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4), 311–327. https://doi.org/10.1111/j.1745-3984.1998.tb00541.x
StarkS.ChernyshenkoO. S. (2011). Computerized adaptive testing with the zinnes and griggs pairwise preference ideal point model. International Journal of Testing, 11(3), 231–247. https://doi.org/10.1080/15305058.2011.561459
20.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: An application to the problem of faking in personality assessment. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
21.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487. https://doi.org/10.1177/1094428112444611
22.
StockingM. L.LewisC. (1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 23(1), 57–75. https://doi.org/10.3102/10769986023001057
23.
ThissenD (1990). Reliability and measurement precision. In WainerH. (Ed), Computerized adaptive testing: A primer (pp. 161–186). Erlbaum.
24.
van der LindenW. J. (1996). Assembling tests for the measurement of multiple traits. Applied Psychological Measurement, 20(4), 373–388. https://doi.org/10.1177/014662169602000405
25.
WainerH (2000). Computerized adaptive testing: A primer (2nd ed). Lawrence Erlbaum Associates.
26.
WangC.ChangH.-H. (2011). Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika, 76(3), 363–384. https://doi.org/10.1007/s11336-011-9215-7
27.
WangW.-C.QiuX.-L.ChenC.-W.RoS.JinK.-Y (2017). Item response theory models for ipsative tests with multidimensional pairwise-comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
