Forced-choice (FC) measures are gaining popularity as an alternative assessment format to single-statement (SS) measures. However, a fundamental question remains to be answered: Do FC and SS instruments measure the same underlying constructs? In addition, FC measures are theorized to be more cognitively challenging, so how would this feature influence respondents’ reactions to FC measures compared to SS? We used both between- and within-subjects designs to examine the equivalence of the FC format and the SS format. As the results illustrate, FC measures scored by the multi-unidimensional pairwise preference (MUPP) model and SS measures scored with the generalized graded unfolding model (GGUM) showed strong equivalence. Specifically, both formats demonstrated similar marginal reliabilities and test-retest reliabilities, high convergent validities, good discriminant validities, and similar criterion-related validities with theoretically relevant criteria. In addition, the formats had little differential impact on respondents’ general emotional and cognitive reactions except that the FC format was perceived to be slightly more difficult and more time-saving.
AckermanR. A.DonnellanM. B.RobertsB. W.FraleyR. C. (2016). The effect of response format on the psychometric properties of the Narcissistic Personality Inventory: Consequences for item meaning and factor structure. Assessment, 23(2), 203–220.
2.
Anguiano-CarrascoC.MacCannC.GeigerM.SeybertJ. M.RobertsR. D. (2015). Development of a forced-choice measure of typical-performance emotional intelligence. Journal of Psychoeducational Assessment, 33(1), 83–97.
3.
BartramD. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15(3), 263–272.
4.
BartramD.BrownA. (2004). Online testing: Mode of administration and the stability of OPQ 32i scores. International Journal of Selection and Assessment, 12(3), 278–284.
5.
BosticT. J.RubioD. M.HoodM. (2000). A validation of the subjective vitality scale using structural equation modeling. Social Indicators Research, 52(3), 313–324.
6.
BowenC. C.MartinB. A.HuntS. T. (2002). A comparison of ipsative and normative approaches for ability to control faking in personality questionnaires. International Journal of Organizational Analysis, 10(3), 240–259.
7.
BrownA. (2016a). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135–160.
BrownA.InceogluI.LinY. (2017). Preventing rater biases in 360-degree feedback by forcing choice. Organizational Research Methods, 20(1), 121–148.
10.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502.
11.
BrownA.Maydeu-OlivaresA. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52.
12.
BrownA.Maydeu-OlivaresA. (2017). Ordinal factor analysis of graded-preference questionnaire data. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 516–529.
13.
BürknerP. C.SchulteN.HollingH. (in press). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement. Advanced online publication. doi:0013164419832063
14.
CaoM.DrasgowF. (in press). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology. doi:10.1037/apl0000414
15.
CaoM.DrasgowF.ChoS. (2015). Developing ideal intermediate personality items for the ideal point model. Organizational Research Methods, 18(2), 252–275.
16.
CaoM.SongQ. C.TayL. (2018). Detecting curvilinear relationships: A comparison of scoring approaches based on different item response models. International Journal of Testing, 18(2), 178–205.
17.
CarterN. T.DalalD. K. (2010). An ideal point account of the JDI Work satisfaction scale. Personality and Individual Differences, 49(7), 743–748.
18.
CarterN. T.DalalD. K.GuanL.LoPilatoA. C.WithrowS. A. (2017). Item response theory scoring and the detection of curvilinear relationships. Psychological Methods, 22(1), 191–203.
19.
ChernyshenkoO. S.StarkS.DrasgowF.RobertsB. W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19(1), 88–106.
20.
ChernyshenkoO. S.StarkS.PrewettM. S.GrayA. A.StilsonF. R.TuttleM. D. (2009). Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats. Human Performance, 22(2), 105–127.
21.
ChoS.DrasgowF.CaoM. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241–1252.
22.
DienerE. D.EmmonsR. A.LarsenR. J.GriffinS. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75.
23.
DonovanJ. J.DwightS. A.SchneiderD. (2014). The impact of applicant faking on selection measures, hiring decisions, and employee performance. Journal of Business and Psychology, 29(3), 479–493.
24.
DrasgowF.ChernyshenkoO. S.StarkS. (2010). 75 years after Likert: Thurstone was right!Industrial and Organizational Psychology, 3(4), 465–476.
25.
DrasgowF.StarkS.ChernyshenkoO. S.NyeC. D.HulinC. L.WhiteL. A. (2012). Development of the Tailored Adaptive Personality Assessment System (TAPAS) to support army personnel selection and classification decisions. Urbana, IL: Drasgow Consulting Group.
26.
DueberD. M.LoveA. M.TolandM. D.TurnerT. A. (2019). Comparison of single-response format and forced-choice format instruments using Thurstonian item response theory. Educational and Psychological Measurement, 79, 108–128.
27.
DuijsensI. J.DiekstraR. W. (1995). The 23bb5: a new bipolar big five questionnaire. Personality and Individual Differences, 19(5), 753–755.
28.
DunlapW. P.CortinaJ. M.VaslowJ. B.BurkeM. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1(2), 170–177.
29.
FraserK.MaI.TeterisE.BaxterH.WrightB.McLaughlinK. (2012). Emotion, cognitive load and learning outcomes during simulation training. Medical Education, 46(11), 1055–1062.
30.
GoldbergL. R. (1990). An alternative “description of personality”: The big-five factor structure. Journal of Personality and Social Psychology, 59(6), 1216–1229.
31.
GreenB. F.BockR. D.HumphreysL. G.LinnR. L.ReckaseM. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21(4), 347–360.
32.
GuenoleN.BrownA. A.CooperA. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526.
33.
HausknechtJ. P.DayD. V.ThomasS. C. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57(3), 639–683.
34.
HicksL. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167–184.
35.
JoanesD. N.GillC. A. (1998). Comparing measures of sample skewness and kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician), 47(1), 183–189.
36.
JohnO. P.SrivastavaS. (1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In PervinL. A.JohnO. P. (Eds.), Handbook of personality: Theory and research (pp. 102–138). New York, NY: Guilford.
37.
JudgeT. A.ErezA.BonoJ. E.ThoresenC. J. (2003). The Core Self-Evaluations Scale: Development of a measure. Personnel Psychology, 56(2), 303–331.
38.
JudgeT. A.HellerD.MountM. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87(3), 530–541.
39.
JudgeT. A.Van VianenA. E.De PaterI. E. (2004). Emotional stability, core self-evaluations, and job outcomes: A review of the evidence and an agenda for future research. Human Performance, 17(3), 325–346.
40.
KrumpalI. (2013). Determinants of social desirability bias in sensitive surveys: A literature review. Quality & Quantity, 47(4), 2025–2047.
41.
LandyF. J.FarrJ. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.
42.
LeeP.JooS. H.LeeS. (2019). Examining stability of personality profile solutions between Likert-type and multidimensional forced choice measure. Personality and Individual Differences, 142, 13–20.
43.
LeeP.JooS. H.StarkS.ChernyshenkoO. S. (2018). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement. Advance online publication.
44.
LeeP.LeeS.StarkS. (2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individual Differences, 123, 229–235.
45.
LikertR. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55.
46.
LöckenhoffC. E.SutinA. R.FerrucciL.CostaP. T.Jr (2008). Personality traits and subjective health in the later years: The association between NEO-PI-R and SF-36 in advanced age is influenced by health status. Journal of Research in Personality, 42(5), 1334–1346.
47.
McCloyR. A.HeggestadE. D.ReeveC. L. (2005). A silk purse from the sow’s ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8(2), 222–248.
48.
MeglinoB. M.RavlinE. C. (1998). Individual values in organizations: Concepts, controversies, and research. Journal of Management, 24(3), 351–389.
49.
MilliganM. (2003). Optimism and the five-factor model of personality, coping, and health behavior. Dissertation Abstracts International, 64(11-B), 5830. (UMI 3112405)
50.
NyeC. D.DrasgowF.ChernyshenkoO. S.StarkS.KubisiakU. C.WhiteL. A.JoseI. (2012), Assessing the Tailored Adaptive Personality Assessment System (TAPAS) as an MOS qualification instrument(Tech. Rep. No. 1312). Ft.Belvoir, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
51.
PavlovG.Maydeu-OlivaresA.FairchildA. J. (2018). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods. Advance online publication.
52.
PinquartM. (2001). Correlates of subjective health in older adults: A meta-analysis. Psychology and Aging, 16(3), 414–426.
53.
PodsakoffP. M.MacKenzieS. B.LeeJ. Y.PodsakoffN. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.
54.
RevelleW. (2016). How to: Use the psych package for factor analysis and data reduction. Evanston, IL: Northwestern University, Department of Psychology.
55.
RobertsJ. S.LaughlinJ. E. (1996). A unidimensional item response model for unfolding responses from a graded disagree-agree response scale. Applied Psychological Measurement, 20(3), 231–255.
56.
RobertsJ. S.LaughlinJ. E.WedellD. H. (1999). Validity issues in the Likert and Thurstone approaches to attitude measurement. Educational and Psychological Measurement, 59(2), 211–233.
57.
RundquistE. A. (1946, September). The forced-choice technique and rating scales. Paper presented at the American Psychological Association, Philadelphia.
58.
SassR.FrickS.ReipsU. D.WetzelE. (2018). Taking the test taker’s perspective: Response process and test motivation in multidimensional forced-choice versus rating scale instruments. Assessment. Advance online publication.
59.
SaucierG. (2002). Orthogonal markers for orthogonal factors: The case of the Big Five. Journal of Research in Personality, 36(1), 1–31.
60.
SchmitM. J.RyanA. M. (1993). The big five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78(6), 966–974.
61.
SeybertJ. (2013). A new item response theory model for estimating person ability and item parameters for multidimensional rank order responses(Unpublished doctoral dissertation). University of South Florida, Tampa.
62.
SharpeJ. P.MartinN. R.RothK. A. (2011). Optimism and the big five factors of personality: Beyond neuroticism and extraversion. Personality and Individual Differences, 51(8), 946–951.
63.
SmitherJ. W.ReillyR. R.MillsapR. E.PearlmanK.StoffeyR. W. (1993). Applicant reactions to selection procedures. Personnel Psychology, 46(1), 49–76.
64.
SotoC. J.JohnO. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117–143.
65.
SpearmanC. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.
66.
StantonJ. M.SinarE. F.BalzerW. K.JulianA. L.ThoresenP.AzizS.…SmithP. C. (2002). Development of a compact measure of job satisfaction: The abridged Job Descriptive Index. Educational and Psychological Measurement, 62(1), 173–191.
67.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203.
68.
StarkS.ChernyshenkoO. S.DrasgowF.NyeC. D.WhiteL. A.HeffnerT.FarmerW. L. (2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26(3), 153–164.
69.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487.
70.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?Journal of Applied Psychology, 91(1), 25–39.
71.
SteelP.SchmidtJ.ShultzJ. (2008). Refining the relationship between personality and subjective well-being. Psychological Bulletin, 134(1), 138–161.
72.
StephanY.DemulierV.TerraccianoA. (2012). Personality, self-rated health, and subjective age in a life-span sample: The moderating role of chronological age. Psychology and Aging, 27(4), 875–880.
73.
SunT. (2017). When matches are ideal: Fitting measurement models to adult attachment data(Unpublished master’s thesis). University of Illinois at Urbana-Champaign.
74.
TayL.DrasgowF.RoundsJ.WilliamsB. A. (2009). Fitting measurement models to vocational interest data: Are dominance models ideal?Journal of Applied Psychology, 94(5), 1287–1304.
75.
TenopyrM. L. (1988). Artifactual reliability of forced-choice scales. Journal of Applied Psychology, 73(4), 749–751.
76.
ThompsonE. R. (2007). Development and validation of an internationally reliable short-form of the Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology, 38(2), 227–242.
77.
ThurstoneL. L. (1927). A law of comparative judgment. Psychological Review, 79, 281–299.
78.
TourangeauR.RasinskiK. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314.
79.
UsamiS.SakamotoA.NaitoJ.AbeY. (2016). Developing pairwise preference-based personality test and experimental investigation of its resistance to faking effect by item response model. International Journal of Testing, 16(4), 288–309.
80.
ViswesvaranC.OnesD. S. (1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59(2), 197–210.
81.
WeijtersB.GeuensM.SchillewaertN. (2010). The stability of individual response styles. Psychological Methods, 15(1), 96–110.
82.
WetzelE.BöhnkeJ. R.BrownA. (2016). Response biases. In LeongF. T. L.BartramD.CheungF.GeisingerK. F.IliescuD. (Eds.), The ITC international handbook of testing and assessment (pp. 349–363). New York, NY: Oxford University Press.
83.
WetzelE.LüdtkeO.ZettlerI.BöhnkeJ. R. (2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3), 279–291.
84.
WetzelE.RobertsB. W.FraleyR. C.BrownA. (2016). Equivalence of Narcissistic Personality Inventory constructs and correlates across scoring approaches and response formats. Journal of Research in Personality, 61, 87–98.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
