Proposed is a new method of scoring multidimensional forced-choice (MFC) questionnaires referred to as the dominant trait profile (DTP) method. The DTP method identifies a dominant response vector (DRV) for each trait—a vector of binary scores for preferences in item pairs within MFC blocks from the perspective of a respondent for whom the trait under consideration dominates over the other traits being measured. The respondents’ observed response vectors are matched to the DRV for each trait to produce (1/0) matching scores that are then analyzed via latent trait modeling, with scaling options (a) bounded D-scale (from 0 to 1), or (b) item response theory logit scale. The DTP method allows for the comparison of individuals on a trait of interest, as well as their standing in relation to a dominant trait “standard” (criterion). The study results indicate that DTP-based trait estimates are highly correlated with those produced by the popular Thurstonian item response theory model and the Zinnes and Griggs pairwise preference item response theory model, while avoiding the complexity of their designs and some computations issues.
AndrichD. (1995). Hyperbolic cosine latent trait models for unfolding direct responses and pairwise preferences. Applied Psychological Measurement, 19(3), 269–290. https://doi.org/10.1177/014662169501900306.
2.
AtanasovV.DimitrovD. M. (2018). DELTA: A computer program for test scoring and psychometric analysis under the D-scoring method of measurement. Education & Training Evaluation Commission (ETEC).
3.
BartramD. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272.
4.
BroadfootA. (2008). Comparing the dominance approach to the ideal point approach in the measurement and predictability of personality [Psychology PhD dissertations 120]. https://scholarworks.bgsu.edu/psychology_diss/120
BrownA.Maydeu-OlivaresA. (2010). Issues that should not be overlooked in the dominance versus ideal point controversy. Industrial and Organizational Psychology, 3(4), 489–493. https://doi.org/10.1111/j.1754-9434.2010.01277.x
7.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112
8.
BrownA.Maydeu-OlivaresA. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44(4), 1135–1147. https://doi.org/10.3758/s13428-012-0217-x
9.
BürknerP.-C.SchulteN.HollingH. (2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063
10.
CaiL.ThissenD.du ToitS. H. C. (2011). IRTPRO for Windows [Computer software]. Scientific Software International.
11.
CaoM.SongQ. C.TayL. (2018). Detecting curvilinear relationships: A comparison of scoring approaches based on different item response models. International Journal of Testing, 18, 178–205. https://doi.org/10.1080/15305058.2017.1345913
12.
CarterN. T.DalalD. K.GuanL.LopilatoA. C.WithrowS. A. (2017). Item response theory scoring and the detection of curvilinear relationships. Psychological Methods, 22, 191–203. https://doi.org/10.1037/met0000101
13.
CheungM. W. L.ChanW. (2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling, 9, 55–77. https://doi.org/10.1207/S15328007SEM0901_4
14.
de la TorreJ.PonsodaV.LeenenI.HontangasP. (2012). Some extensions of the multiunidimensional pairwise preference model [Paper presentation]. Paper presented at the 26th Annual Meeting of the Society for Industrial and Organizational Psychology, Chicago, IL.
15.
DimitrovD. M. (2023). D-scoring method of measurement: Classical and latent frameworks. Routledge.
16.
DimitrovD. M.AtanasovD. V. (2021). Latent D-scoring modeling: Estimation of item and person parameters. Educational and Psychological Measurement, 81(2), 388–404.
17.
DoornikJ. A. (2009). An object-oriented matrix programming language Ox 6 [Computer software]. Timberlake Consultants Press.
18.
EmbretsonS. E.ReiseS. (2000). Item response theory for psychologists. Erlbaum.
19.
ForeroG. G.Maydeu-OlivaresA. (2009). Estimation of IRT graded response models: Limited versus full information methods. Psychological Methods, 14(3), 275–299. https://doi.org/10.1037/a0015825
20.
HoganR.JohnsonJ.BriggsS. (1997). Handbook of personality psychology (1st ed.). Academic Press.
21.
HontangasP. M.de la TorreJ.PonsodaV.LeenenI.MorilloD.AbadF. J. (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851
22.
HoughL. M.EatonN. K.DunnetteM. D.KampJ. D.McCloyR. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75(5), 581–595.
23.
JooS.-H.LeeP.StarkS. (2021). Modeling multidimensional forced-choice measures with the Zinnes and Griggs pairwise preference item response theory model. Multivariate Behavioral Research, 58(2), 241–261. https://doi.org/10.1080/00273171.2021.1960142
24.
LeeP.JooS.-H.StarkS.ChernyshenkoO. S. (2019). GGUM-Rank statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294
25.
LiuC.-W.WangW.-C. (2019). A general unfolding IRT model for multiple response styles. Applied Psychological Measurement, 43(3), 195–210. https://doi.org/10.1177/0146621618762743
26.
MuthénL. K.MuthénB. O. (1998–2020). Mplus: Statistical analysis with latent variables: User’s guide (Version 8).
27.
NgV.LeeP.HoM. H. R.KuykendallL.StarkS.TayL. (2021). The development and validation of a multidimensional forced-choice format character measure: Testing the Thurstonian IRT approach. Journal of Personality Assessment, 103, 224–237.
PatzR. J.JunkerB. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.3102/10769986024002146
30.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3–32.
31.
RobitzschA. (2021). On the equivalence of the latent D-scoring model and the two-parameter logistic item response theory model. Mathematics. https://doi.org/10.3390/math9131465
32.
SchmittN.OswaldF. L. (2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. Journal of Applied Psychology, 91(3), 613–621. https://doi.org/10.1037/0021-9010.91.3.613
33.
StarkS.ChernyshenkoO. S.ChanK. Y.LeeW. C.DrasgowF. (2001). Effects of the testing situation on item responding: Cause for concern. The Journal of Applied Psychology, 86(5), 943–953.
34.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model, Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
35.
StarkS.DrasgowF. (2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26(2), 208–227. https://doi.org/10.1177/01421602026002007
36.
TayL.AliU. S.DrasgowF.WilliamsB. (2011). Fitting IRT models to dichotomous and polytomous data: Assessing the relative model–data fit of ideal point and dominance models. Applied Psychological Measurement, 35(4), 280–295. https://doi.org/10.1177/0146621610390674
37.
TayL.DrasgowF. (2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15, 363–384. https://doi.org/10.1177/1094428112439709
38.
TayL.NgV. (2018). Ideal point modeling of non-cognitive constructs: Review and recommendations for research. Frontiers in Psychology, 9, 2423. https://doi.org/10.3389/fpsyg.2018.02423
39.
ThurstoneL. L. (1927a). A law of comparative judgment. Psychological Review, 34(4), 273–286.
40.
ThurstoneL. L. (1927b). The method of paired comparisons for social values. Journal of Abnormal Social Psychology, 21, 384–400.
41.
ThurstoneL. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529–554.
42.
TierneyL. (1994). Markov chains for exploring posterior distributions. The Annals of Statistics, 22(4), 1701–1728. https://doi.org/10.1214/aos/1176325750
43.
WetzelE.FrickS.GreiffS. (2020). The multidimensional forced-choice format as an alternative for rating scales. European Journal of Psychological Assessment, 36(4), 511–515. https://doi.org/10.1027/1015-5759/a000609
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
