Restricted accessResearch articleFirst published online 2025
A Generalized Forced-Choice Diagnostic Classification Model Based on Thurstone’s Law of Comparative Judgment in Noncognitive Test: Theory and Application
In the context of single-stimulus (SS) formats, diagnostic classification models (DCMs) have been employed in noncognitive assessments to obtain fine-grained feedback regarding the respondent’s attributes. However, SS format is susceptible to response biases and faking. As an alternative, forced-choice (FC) format has been widely applied in noncognitive assessments, as it can effectively control response biases and fakability that occur in SS formats. Consequently, the incorporation of the FC format into the DCMs in the noncognitive scenario represents an effective means of enhancement. To the best knowledge, only Huang developed a forced-choice diagnostic classification model (FC-DCM) based on simplified modeling. However, there are several important areas where the FC-DCM can be improved. Therefore, this study aims to propose a generalized forced-choice diagnostic classification model (GFC-DCM) for FC blocks within the framework of generalized DCMs, which can not only provide fine-grained diagnostic information about specific attributes but also control response biases and fakability. To validate the performance of the GFC-DCM, a simulation study was conducted in which the sample size, number of statements, FC format, number of attributes, and distribution of attributes were manipulated. The simulation study demonstrated (a) the GFC-DCM achieved high classification accuracy and satisfactory statement parameter estimation accuracy and (b) the number of statements and the FC format exerted a substantial influence on the outcomes. An empirical study demonstrated the GFC-DCM was superior in terms of model convergence and fit, empirical reliability, and diagnostic consistency compared to existing models.
BenschD.MaaßU.GreiffS.HorstmannK. T.ZieglerM. (2019). The nature of faking: A homogeneous and predictable construct?Psychological Assessment, 31(4), 532–544. https://doi.org/10.1037/pas0000619
2.
BrooksS. P.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434. https://doi.org/10.2307/1390675
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
5.
BrownA.Maydeu-OlivaresA. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44(4), 1135–1147. https://doi.org/10.3758/s13428-012-0217-x
6.
BrownA.Maydeu-OlivaresA. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52. https://doi.org/10.1037/a0030641
7.
BürknerP.-C.SchulteN.HollingH. (2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063
8.
ChiuC.-Y.DouglasJ. A.LiX. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665. https://doi.org/10.1007/s11336-009-9125-0
9.
ChiuC.-Y.SunY. J.BianY. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355–375. https://doi.org/10.1007/s11336-017-9595-4
10.
ChristiansenN. D.BurnsG. N.MontgomeryG. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267–307. https://doi.org/10.1207/s15327043hup1803_4
de la TorreJ.DouglasJ. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/10.1007/bf02295640
13.
de la TorreJ.van der ArkL. A.RossiG. (2017). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4), 281–296. https://doi.org/10.1080/07481756.2017.1327286
14.
de ValpineP.PaciorekC.TurekD.MichaudN.Anderson-BergmanC.ObermeyerF.Wehrhahn CortesC.RodrìguezA.Temple LangD.PaganinS. (2024). NIMBLE user manual (R package manual version 1.2.0). https://doi.org/10.5281/zenodo.1211190. https://r-nimble.org
15.
DonovanJ. J.DwightS. A.HurtzG. M. (2003). An assessment of the prevalence, severity, and verifiability of entry-level applicant faking using the randomized response technique. Human Performance, 16(1), 81–106. https://doi.org/10.1207/s15327043hup1601_4
16.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
17.
GordonR. A. (1987). Social desirability bias: A demonstration and technique for its reduction. Teaching of Psychology, 14(1), 40–42. https://doi.org/10.1207/s15328023top1401_11
18.
GuY.XuG. (2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4). https://doi.org/10.1214/19-aos1878
19.
GuY.XuG. (2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449–472. https://doi.org/10.5705/ss.202018.0410
20.
HensonR. A.TemplinJ. L.WillseJ. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5
21.
HuJ.ConnellyB. S. (2021). Faking by actual applicants on personality tests: A meta-analysis of within-subjects studies. International Journal of Selection and Assessment, 29(3–4), 412–426. https://doi.org/10.1111/ijsa.12338
22.
HuangH.-Y. (2022). Diagnostic classification model for forced-choice items and noncognitive tests. Educational and Psychological Measurement, 83(1), 146–180. https://doi.org/10.1177/00131644211069906
23.
JacksonD. N.WroblewskiV. R.AshtonM. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution?Human Performance, 13(4), 371–388. https://doi.org/10.1207/s15327043hup1304_3
24.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064
25.
KreitchmannR. S.SorrelM. A.AbadF. J. (2023). On bank assembly and block selection in multidimensional forced-choice adaptive assessments. Educational and Psychological Measurement, 83(2), 294–321. https://doi.org/10.1177/00131644221087986
26.
LeeH.SmithW. Z. (2020). A Bayesian random block item response theory model for forced-choice formats. Educational and Psychological Measurement, 80(3), 578–603. https://doi.org/10.1177/0013164419871659
27.
LeeP.JooS.-H.StarkS. (2020). Detecting DIF in multidimensional forced choice measures using the Thurstonian item response theory model. Organizational Research Methods, 42(3), 212–237. https://doi.org/10.1177/1094428120959822
28.
LeeP.JooS.-H.StarkS.ChernyshenkoO. S. (2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
29.
LeeY. S.ParkY. S.TaylanD. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144–177. https://doi.org/10.1080/15305058.2010.534571
30.
LoeysT.RosseelY.BatenK. (2011). A joint modeling approach for reaction time and accuracy in psycholinguistic experiments. Psychometrika, 76(3), 487–503. https://doi.org/10.1007/s11336-011-9211-y
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
34.
PengS.CaiY.WangD.LuoF.TuD. (2022). A generalized diagnostic classification modeling framework integrating differential speediness: Advantages and illustrations in psychological and educational testing. Multivariate Behavioral Research, 57(6), 1–20. https://doi.org/10.1080/00273171.2021.1928474
35.
PengS.ManK.VeldkampB. P.CaiY.TuD. (2023). A mixture model for random responding behavior in forced-choice noncognitive assessment: Implication and application in organizational research. Organizational Research Methods, 27(3), 414–442. https://doi.org/10.1177/10944281231181642
36.
R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/
37.
RosseJ. G.StecherM. D.MillerJ. L.LevinR. A. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83(4), 634–644. https://doi.org/10.1037/0021-9010.83.4.634
38.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
39.
SavaF. A.DindeleganC.SălăgeanN. (2020). The utility of the personality clinical form indicators of response distortion: Receiver operating characteristic analysis. Romanian Journal of Applied Psychology, 22(1), 1–8.
SpeerA. B.SpeerA. B.TenbrinkA. P.DelacruzA. Y.ChristiansenN. D.SalimR. M. (2023). Comparing forced-choice and single-stimulus personality scores on a level playing field: A meta-analysis of psychometric properties and susceptibility to faking. Journal of Applied Psychology, 108(11), 1812–1833. https://doi.org/10.1037/apl0001099
42.
SpiegelhalterD. J.BestN. G.CarlinB. P.van der LindeA. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639. https://doi.org/10.1111/1467-9868.00353
43.
StarkS. (2002). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment: The generalized graded unfolding model for multi-unidimensional paired comparison responses. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63(2-B), 1084.
44.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
45.
StarkS.ChernyshenkoO. S.DrasgowF.NyeC. D.WhiteL. A.HeffnerT.FarmerW. L. (2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26(3), 153–164. https://doi.org/10.1037/mil0000044
46.
TanZ.de la TorreJ.MaW.HuhD.LarimerM. E.MunE.-Y. (2022). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses. Prevention Science, 24(3), 480–492. https://doi.org/10.1007/s11121-022-01346-8
47.
TemplinJ.BradshawL. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275. https://doi.org/10.1007/s00357-013-9129-4
48.
TemplinJ. L.HensonR. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989x.11.3.287
WaltonK. E.CherkasovaL.RobertsR. D. (2020). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment, 27(4), 706–718. https://doi.org/10.1177/1073191119843585
51.
WangD.GaoX.CaiY.TuD. (2019). Development of a new instrument for depression with cognitive diagnosis models. Frontiers in Psychology, 10, Article 1306. https://doi.org/10.3389/fpsyg.2019.01306
52.
WangW.-C.QiuX.-L.ChenC.-W.RoS.JinK.-Y. (2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183
53.
WatanabeS. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594.
54.
WetzelE.FrickS.BrownA. (2021). Does multidimensional forced-choice prevent faking? Comparing the susceptibility of the multidimensional forced-choice format and the rating scale format to faking. Psychological Assessment, 33(2), 156–170. https://doi.org/10.1037/pas0000971
55.
WilliamsM. M.RogersR.SharfA. J.RossC. A. (2018). Faking good: An investigation of social desirability and defensiveness in an inpatient sample with personality disorder traits. Journal of Personality Assessment, 101(3), 253–263. https://doi.org/10.1080/00223891.2018.1455691
56.
XuG.ShangZ. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295. https://doi.org/10.1080/01621459.2017.1340889
57.
ZhanP.ManK.WindS. A.MaloneJ. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736–776. https://doi.org/10.3102/10769986221111085
58.
ZhangB.SunT.DrasgowF.ChernyshenkoO. S.NyeC. D.StarkS.WhiteL. A. (2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569–590. https://doi.org/10.1177/1094428119836486
59.
ZhangB.TuN.AngraveL.ZhangS.SunT.TayL.LiJ. (2023). The generalized Thurstonian unfolding model (GTUM): Advancing the modeling of forced-choice data. Organizational Research Methods, 27(4), 713–747. https://doi.org/10.1177/10944281231210481
60.
ZhangJ.CuiS.XuY.CuiT.BarnhartW.R.JiF.NagataJ.M.HeJ. (2025). Introducing diagnostic classification modeling as an unsupervised method for screening probable eating disorders. Assessment, 32(3), 405–416. https://doi.org/10.1177/10731911241247483
61.
ZhuY.XuJ.WangD.LiX.CaiY.TuD. (2025). A ranking forced choice diagnostic classification model for psychological assessment using forced choice questionnaires. British Journal of Mathematical and Statistical Psychology, 78(2), 617–646. https://doi.org/10.1111/bmsp.12376
62.
ZickarM. J.GibbyR. E.RobieC. (2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed-model item response theory. Organizational Research Methods, 7(2), 168–190. https://doi.org/10.1177/1094428104263674
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
