A Generalized Forced-Choice Diagnostic Classification Model Based on Thurstone’s Law of Comparative Judgment in Noncognitive Test: Theory and Application

Abstract

In the context of single-stimulus (SS) formats, diagnostic classification models (DCMs) have been employed in noncognitive assessments to obtain fine-grained feedback regarding the respondent’s attributes. However, SS format is susceptible to response biases and faking. As an alternative, forced-choice (FC) format has been widely applied in noncognitive assessments, as it can effectively control response biases and fakability that occur in SS formats. Consequently, the incorporation of the FC format into the DCMs in the noncognitive scenario represents an effective means of enhancement. To the best knowledge, only Huang developed a forced-choice diagnostic classification model (FC-DCM) based on simplified modeling. However, there are several important areas where the FC-DCM can be improved. Therefore, this study aims to propose a generalized forced-choice diagnostic classification model (GFC-DCM) for FC blocks within the framework of generalized DCMs, which can not only provide fine-grained diagnostic information about specific attributes but also control response biases and fakability. To validate the performance of the GFC-DCM, a simulation study was conducted in which the sample size, number of statements, FC format, number of attributes, and distribution of attributes were manipulated. The simulation study demonstrated (a) the GFC-DCM achieved high classification accuracy and satisfactory statement parameter estimation accuracy and (b) the number of statements and the FC format exerted a substantial influence on the outcomes. An empirical study demonstrated the GFC-DCM was superior in terms of model convergence and fit, empirical reliability, and diagnostic consistency compared to existing models.

Keywords

diagnostic classification model forced choice response bias generalized framework

Get full access to this article

View all access options for this article.

References

Bensch

Maaß

Greiff

Horstmann

K. T.

Ziegler

(2019). The nature of faking: A homogeneous and predictable construct? Psychological Assessment, 31(4), 532–544. https://doi.org/10.1037/pas0000619

Brooks

S. P.

Gelman

(1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434. https://doi.org/10.2307/1390675

Brown

(2016). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135–160. https://doi.org/10.1007/s11336-014-9434-9

Brown

Maydeu-Olivares

(2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112

Brown

Maydeu-Olivares

(2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44(4), 1135–1147. https://doi.org/10.3758/s13428-012-0217-x

Brown

Maydeu-Olivares

(2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52. https://doi.org/10.1037/a0030641

Bürkner

P.-C.

Schulte

Holling

(2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063

Chiu

C.-Y.

Douglas

J. A.

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665. https://doi.org/10.1007/s11336-009-9125-0

Chiu

C.-Y.

Sun

Y. J.

Bian

(2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355–375. https://doi.org/10.1007/s11336-017-9595-4

10.

Christiansen

N. D.

Burns

G. N.

Montgomery

G. E.

(2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267–307. https://doi.org/10.1207/s15327043hup1803_4

11.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

12.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/10.1007/bf02295640

13.

de la Torre

van der Ark

L. A.

Rossi

(2017). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4), 281–296. https://doi.org/10.1080/07481756.2017.1327286

14.

de Valpine

Paciorek

Turek

Michaud

Anderson-Bergman

Obermeyer

Wehrhahn Cortes

Rodrìguez

Temple Lang

Paganin

(2024). NIMBLE user manual (R package manual version 1.2.0). https://doi.org/10.5281/zenodo.1211190. https://r-nimble.org

15.

Donovan

J. J.

Dwight

S. A.

Hurtz

G. M.

(2003). An assessment of the prevalence, severity, and verifiability of entry-level applicant faking using the randomized response technique. Human Performance, 16(1), 81–106. https://doi.org/10.1207/s15327043hup1601_4

16.

Gelman

Rubin

D. B.

(1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136

17.

Gordon

R. A.

(1987). Social desirability bias: A demonstration and technique for its reduction. Teaching of Psychology, 14(1), 40–42. https://doi.org/10.1207/s15328023top1401_11

18.

(2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4). https://doi.org/10.1214/19-aos1878

19.

(2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449–472. https://doi.org/10.5705/ss.202018.0410

20.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5

21.

Connelly

B. S.

(2021). Faking by actual applicants on personality tests: A meta-analysis of within-subjects studies. International Journal of Selection and Assessment, 29(3–4), 412–426. https://doi.org/10.1111/ijsa.12338

22.

Huang

H.-Y.

(2022). Diagnostic classification model for forced-choice items and noncognitive tests. Educational and Psychological Measurement, 83(1), 146–180. https://doi.org/10.1177/00131644211069906

23.

Jackson

D. N.

Wroblewski

V. R.

Ashton

M. C.

(2000). The impact of faking on employment tests: Does forced choice offer a solution? Human Performance, 13(4), 371–388. https://doi.org/10.1207/s15327043hup1304_3

24.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

25.

Kreitchmann

R. S.

Sorrel

M. A.

Abad

F. J.

(2023). On bank assembly and block selection in multidimensional forced-choice adaptive assessments. Educational and Psychological Measurement, 83(2), 294–321. https://doi.org/10.1177/00131644221087986

26.

Lee

Smith

W. Z.

(2020). A Bayesian random block item response theory model for forced-choice formats. Educational and Psychological Measurement, 80(3), 578–603. https://doi.org/10.1177/0013164419871659

27.

Lee

Joo

S.-H.

Stark

(2020). Detecting DIF in multidimensional forced choice measures using the Thurstonian item response theory model. Organizational Research Methods, 42(3), 212–237. https://doi.org/10.1177/1094428120959822

28.

Lee

Joo

S.-H.

Stark

Chernyshenko

O. S.

(2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294

29.

Lee

Y. S.

Park

Y. S.

Taylan

(2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144–177. https://doi.org/10.1080/15305058.2010.534571

30.

Loeys

Rosseel

Baten

(2011). A joint modeling approach for reaction time and accuracy in psycholinguistic experiments. Psychometrika, 76(3), 487–503. https://doi.org/10.1007/s11336-011-9211-y

31.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212. https://doi.org/10.1007/bf02294535

32.

Millon

Davis

Millon

Grossman

(2009). Millon Clinical Multiaxial Inventory-III (MCMI-III) (4th ed.). Pearson Assessments.

33.

Morillo

Leenen

Abad

F. J.

Hontangas

de la Torre

Ponsoda

(2016). A dominance variant under the multi-unidimensional pairwise-preference framework. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226

34.

Peng

Cai

Wang

Luo

(2022). A generalized diagnostic classification modeling framework integrating differential speediness: Advantages and illustrations in psychological and educational testing. Multivariate Behavioral Research, 57(6), 1–20. https://doi.org/10.1080/00273171.2021.1928474

35.

Peng

Man

Veldkamp

B. P.

Cai

(2023). A mixture model for random responding behavior in forced-choice noncognitive assessment: Implication and application in organizational research. Organizational Research Methods, 27(3), 414–442. https://doi.org/10.1177/10944281231181642

36.

R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/

37.

Rosse

J. G.

Stecher

M. D.

Miller

J. L.

Levin

R. A.

(1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83(4), 634–644. https://doi.org/10.1037/0021-9010.83.4.634

38.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

39.

Sava

F. A.

Dindelegan

Sălăgean

(2020). The utility of the personality clinical form indicators of response distortion: Receiver operating characteristic analysis. Romanian Journal of Applied Psychology, 22(1), 1–8.

40.

Schwarz

(1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

41.

Speer

A. B.

Speer

A. B.

Tenbrink

A. P.

Delacruz

A. Y.

Christiansen

N. D.

Salim

R. M.

(2023). Comparing forced-choice and single-stimulus personality scores on a level playing field: A meta-analysis of psychometric properties and susceptibility to faking. Journal of Applied Psychology, 108(11), 1812–1833. https://doi.org/10.1037/apl0001099

42.

Spiegelhalter

D. J.

Best

N. G.

Carlin

B. P.

van der Linde

(2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639. https://doi.org/10.1111/1467-9868.00353

43.

Stark

(2002). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment: The generalized graded unfolding model for multi-unidimensional paired comparison responses. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63(2-B), 1084.

44.

Stark

Chernyshenko

O. S.

Drasgow

(2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988

45.

Stark

Chernyshenko

O. S.

Drasgow

Nye

C. D.

White

L. A.

Heffner

Farmer

W. L.

(2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26(3), 153–164. https://doi.org/10.1037/mil0000044

46.

Tan

de la Torre

Huh

Larimer

M. E.

Mun

E.-Y.

(2022). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses. Prevention Science, 24(3), 480–492. https://doi.org/10.1007/s11121-022-01346-8

47.

Templin

Bradshaw

(2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275. https://doi.org/10.1007/s00357-013-9129-4

48.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989x.11.3.287

49.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 101(2), 266–270. https://doi.org/10.1037/0033-295X.101.2.266

50.

Walton

K. E.

Cherkasova

Roberts

R. D.

(2020). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment, 27(4), 706–718. https://doi.org/10.1177/1073191119843585

51.

Wang

Gao

Cai

(2019). Development of a new instrument for depression with cognitive diagnosis models. Frontiers in Psychology, 10, Article 1306. https://doi.org/10.3389/fpsyg.2019.01306

52.

Wang

W.-C.

Qiu

X.-L.

Chen

C.-W.

Jin

K.-Y.

(2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183

53.

Watanabe

(2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594.

54.

Wetzel

Frick

Brown

(2021). Does multidimensional forced-choice prevent faking? Comparing the susceptibility of the multidimensional forced-choice format and the rating scale format to faking. Psychological Assessment, 33(2), 156–170. https://doi.org/10.1037/pas0000971

55.

Williams

M. M.

Rogers

Sharf

A. J.

Ross

C. A.

(2018). Faking good: An investigation of social desirability and defensiveness in an inpatient sample with personality disorder traits. Journal of Personality Assessment, 101(3), 253–263. https://doi.org/10.1080/00223891.2018.1455691

56.

Shang

(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295. https://doi.org/10.1080/01621459.2017.1340889

57.

Zhan

Man

Wind

S. A.

Malone

(2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736–776. https://doi.org/10.3102/10769986221111085

58.

Zhang

Sun

Drasgow

Chernyshenko

O. S.

Nye

C. D.

Stark

White

L. A.

(2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569–590. https://doi.org/10.1177/1094428119836486

59.

Zhang

Angrave

Zhang

Sun

Tay

(2023). The generalized Thurstonian unfolding model (GTUM): Advancing the modeling of forced-choice data. Organizational Research Methods, 27(4), 713–747. https://doi.org/10.1177/10944281231210481

60.

Zhang

Cui

Barnhart

W.R.

Nagata

J.M.

(2025). Introducing diagnostic classification modeling as an unsupervised method for screening probable eating disorders. Assessment, 32(3), 405–416. https://doi.org/10.1177/10731911241247483

61.

Zhu

Wang

Cai

(2025). A ranking forced choice diagnostic classification model for psychological assessment using forced choice questionnaires. British Journal of Mathematical and Statistical Psychology, 78(2), 617–646. https://doi.org/10.1111/bmsp.12376

62.

Zickar

M. J.

Gibby

R. E.

Robie

(2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed-model item response theory. Organizational Research Methods, 7(2), 168–190. https://doi.org/10.1177/1094428104263674

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB