The multidimensional forced-choice (MFC) format has been proposed as an alternative to the rating scale (RS) response format. However, it is unclear how changing the response format may affect the response process and test motivation of participants. In Study 1, we investigated the MFC response process using the think-aloud technique. In Study 2, we compared test motivation between the RS format and different versions of the MFC format (presenting 2, 3, 4, and 5 items simultaneously). The response process to MFC item blocks was similar to the RS response process but involved an additional step of weighing the items within a block against each other. The RS and MFC response format groups did not differ in their test motivation. Thus, from the test taker’s perspective, the MFC format is somewhat more demanding to respond to, but this does not appear to decrease test motivation.
ArveyR. D.StricklandW.DraudenG.MartinC. (1990). Motivational components of test taking. Personnel Psychology, 43, 695-716.
2.
BarrickM. R.MountM. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
3.
BartramD. (2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15, 263-272. doi:10.1111/j.1468-2389.2007.00386.x
4.
BaumertJ.DemmrichA. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16, 441-462.
5.
BaumgartnerH.SteenkampJ.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 28, 143-156.
6.
BöckenholtU. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17, 665-678. doi:10.1037/a0028111
7.
BrownA.InceogluI.LinY. (2017). Preventing rater biases in 360-degree feedback by forcing choice. Organizational Research Methods, 20, 121-148. doi:10.1177/1094428116668036
8.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460-502. doi:10.1177/0013164410375112
9.
BrownA.Maydeu-OlivaresA. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36-52. doi:10.1037/a0030641
10.
ChristiansenN. D.BurnsG. N.MontgomeryG. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18, 267-307. doi:10.1207/s15327043hup1803_4
11.
DuckworthA. L.QuinnP. D.LynamD. R.LoeberR.Stouthamer-LoeberM. (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences of the United States of America, 108, 7716-7720. doi:10.1073/pnas.1018601108
12.
EklöfH. (2007). Test-taking motivation and mathematics performance in TIMSS 2003. International Journal of Testing, 7, 311-326.
13.
FinnR. H. (1970). A note on estimating reliability of categorical data. Educational and Psychological Measurement, 30, 71-76. doi:10.1177/001316447003000106
14.
FunkeF.ReipsU.-D. (2012). Why semantic differentials in Web-based research should be made from visual analogue scales and not from 5-point scales. Field Methods, 24, 310-327. doi:10.1177/1525822X12444061
15.
HeggestadE. D.MorrisonM.ReeveC. L.McCloyR. A. (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91, 9-24. doi:10.1037/0021-9010.91.1.9
16.
JacksonD. N.WroblewskiV. R.AshtonM. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution?Human Performance, 13, 371-388. doi:10.1207/S15327043HUP1304_3
17.
JonesD. N.PaulhusD. L. (2014). Introducing the short Dark Triad (SD3): A brief measure of dark personality traits. Assessment, 21, 28-41. doi:10.1177/1073191113514105
18.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236. doi:10.1002/acp.2350050305
19.
KuhlmannT.DantlgraberM.ReipsU.-D. (2017). Investigating measurement equivalence of visual analogue scales and Likert-type scales in Internet-based personality questionnaires. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-016-0850-x
20.
MarshH. W. (1984). Experimental manipulations of university student motivation and their effects on examination performance. British Journal of Educational Psychology, 54, 206-213.
21.
Maydeu-OlivaresA.BrownA. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45, 935-974. doi:10.1080/00273171.2010.531231
22.
McDonaldR. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
23.
O’NeillT. A.GoffinR. D.GellatlyI. R. (2010). Test-taking motivation and personality test validity. Journal of Personnel Psychology, 9, 117-125. doi:10.1027/1866-5888/a000012
24.
PaulhusD. L. (2002). Socially desirable responding: The evolution of a construct. In BraunH. I.JacksonD. N.WileyD. E. (Eds.), The role of constructs in psychological and educational measurement (pp. 49-69). Mahwah, NJ: Lawrence Erlbaum.
25.
PeerE.BrandimarteL.SamatS.AcquistiA. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153-163. doi:10.1016/j.jesp.2017.01.006
26.
PodsakoffP. M.MacKenzieS. B.LeeJ. Y.PodsakoffN. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903. doi:10.1037/0021-9101.88.5.879
27.
R Core Team. (2013). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org/
28.
SalgadoJ. F.RemeseiroC.IglesiasM. (1996). Personality and test taking motivation. Psicothema, 8, 553-562.
29.
SalgadoJ. F.TáurizG. (2014). The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23, 3-30. doi:10.1080/1359432X.2012.716198
30.
SavilleP.HoldsworthR.NyfieldG.CrampL.MabeyW. (1993). Occupational Personality Questionnaires: Concept model manual and user’s guide. Esher, England: Saville & Holdsworth Ltd.
31.
SchmitM. J.RyanA. M. (1997). Applicant withdrawal: The role of test-taking attitudes and racial differences. Personnel Psychology, 50, 855-876. doi:10.1111/j.1744-6570.1997.tb01485.x
32.
SotoC. J.JohnO. P.GoslingS. D.PotterJ. (2008). The developmental psychometrics of big five self-reports: acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology, 94, 718-737. doi:10.1037/0022-3514.94.4.718
33.
StiegerS.ReipsU.-D. (2010). What are participants doing while filling in an online questionnaire: A paradata collection tool and an empirical study. Computers in Human Behavior, 26, 1488-1495. doi:10.1016/j.chb.2010.05.013
34.
TourangeauR.RasinskiK. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314. doi:10.1037/0033-2909.103.3.299
35.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
36.
Van VaerenberghY.ThomasT. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195-217. doi:10.1093/Ijpor/Eds021
37.
WetzelE.CarstensenC. H.BöhnkeJ. R. (2013). Consistency of extreme response style and non-extreme response style across traits. Journal of Research in Personality, 47, 178-189. doi:10.1016/j.jrp.2012.10.010
38.
WetzelE.FrickS. (2017). The Big Five Triplets—Development of a multidimensional forced-choice questionnaire. Manuscript in preparation.
39.
WetzelE.LüdtkeO.ZettlerI.BöhnkeJ. R. (2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23, 279-291. doi:10.1177/1073191115583714
40.
WolfL. F.SmithJ. F. (1995). The consequence of consequence: Motivation, anxiety, and test performance. Applied Measurement in Education, 8, 227-242. doi:10.1207/s15324818ame0803_3
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
