An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures

Abstract

Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents’ decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents’ behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.

Keywords

forced-choice response time Thurstonian item response theory model information entropy log-linear model Markov chain Monte Carlo algorithm

Get full access to this article

View all access options for this article.

References

Bartram

(2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment, 15, 263–272. https://doi.org/10.1111/j.1468-2389.2007.00386.x

Bassili

J. N.

(1996). The how and why of response latency measurement in telephone surveys. In Schwarz

Sudman

(Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 319–346). Jossey-Bass/Wiley.

Bassili

J. N.

Scott

B. S.

(1996). Response latency as a signal to question problems in survey research. Public Opinion Quarterly, 60(3), 390–399. https://doi.org/10.1086/297760

Birnbaum

(1968). Some latent trait models. In Ford

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–424). Addison-Wesley.

Boehm

Marsman

van der Maas

H. L.

Maris

(2021). An attention-based diffusion model for psychometric analyses. Psychometrika, 86(4), 938–972. https://doi.org/10.1007/s11336-021-09783-0

Brown

Maydeu-Olivares

(2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460–502. https://doi.org/10.1177/0013164410375112

Brown

Maydeu-Olivares

(2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44, 1135–1147. https://doi.org/10.3758/s13428-012-0217-x

Brown

Maydeu-Olivares

(2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. https://doi.org/10.1037/a0030641

Bunji

Okada

(2020). Joint modeling of the two-alternative multidimensional forced-choice personality measurement and its response time by a Thurstonian D-diffusion item response model. Behavior Research Methods, 52, 1091–1107. https://doi.org/10.3758/s13428-019-01302-5

10.

Bürkner

P. C.

(2022). On the information obtainable from comparative judgments. Psychometrika, 87, 1439–1472. https://doi.org/10.1007/s11336-022-09843-z

11.

Bürkner

P. C.

Schulte

Holling

(2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063

12.

Callegaro

Murakami

M. H.

Tepman

Henderson

(2015). Yes–no answers versus check-all in self-administered modes: A systematic review and analyses. International Journal of Market Research, 57(2), 203–224. https://doi.org/10.2501/IJMR-2015-014a

13.

Casey

M. M.

Tryon

W. W.

(2001). Validating a double-press method for computer administration of personality inventory items. Psychological Assessment, 13, 521–530. https://doi.org/10.1037/1040-3590.13.4.521

14.

Crowne

D. P.

Marlowe

(1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354. https://doi.org/10.1037/h0047358

15.

de Valpine

Paciorek

Turek

Michaud

Anderson-Bergman

Obermeyer

Wehrhahn Cortes

Rodrìguez

Temple Lang

Paganin

(2022). NIMBLE user manual (R package manual version 0.13.1). https://doi.org/10.5281/zenodo.1211190. https://r-nimble.org

16.

de Valpine

Turek

Paciorek

Anderson-Bergman

Temple Lang

Bodik

(2017). Programming with models: Writing statistical algorithms for general model structures with NIMBLE. Journal of Computational and Graphical Statistics, 26, 403–413. https://doi.org/10.1080/10618600.2016.1172487

17.

Dueber

D. M.

Love

A. M. A.

Toland

M. D.

Turner

T. A.

(2018). Comparison of single-response format and forced-choice format instruments using Thurstonian item response theory. Educational and Psychological Measurement, 79(1), 108–128. https://doi.org/10.1177/0013164417752782

18.

Dunn

Lushene

O’Neil

(1972). Complete automation of the MMPI and a study of its response latencies. Journal of Consulting and Clinical Psychology, 39, 381–387. https://doi.org/10.1037/h0033855

19.

Edwards

A. L.

(1959). Edwards personal preference schedule (2nd ed.). Psyschological Corp.

20.

Entink

R. H. K.

van der Linden

W. J.

Fox

J. P.

(2009). A Box–Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62(3), 621–640. https://doi.org/10.1348/000711008X374126

21.

Fan

Z. W.

Wang

Chang

H. H.

Douglas

(2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37(5), 655–670. https://doi.org/10.3102/1076998611422912

22.

Fazio

(1995). Attitudes as object-evaluation associations: Determinants, consequences, and correlates of attitude accessibility. In Petty

Krosnick

(Eds.), Attitude strength: Antecedents and consequences (pp. 247–282). Lawrence Erlbaum.

23.

Ferrando

P. J.

(2006). Person-item distance and response time: An empirical study in personality measurement. Psicológica, 27(1), 137–148.

24.

Ferrando

P. J.

Lorenzo-Seva

(2007). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31(6), 525–543. https://doi.org/10.1177/0146621606295197

25.

Gordon

R. A.

(1987). Social desirability bias: A demonstration and technique for its reduction. Teaching of Psychology, 14(1), 40–42. https://doi.org/10.1207/s15328023top1401_11

26.

Griffith

R. L.

Chmielowski

Yoshita

(2007). Do applicants fake? An examination of the frequency of applicant faking behavior. Personnel Review, 36(3), 341–355. https://doi.org/10.1108/00483480710731310

27.

Henninger

Plieninger

(2020). Different styles, different times: How response times can inform our knowledge about the response process in rating scale measurement. Assessment, 28(5), 1301–1319. https://doi.org/10.1177/1073191119900003

28.

Hicks

L. E.

(1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167–184. https://doi.org/10.1037/h0029780

29.

Holden

R. R.

(1995). Response latency detection of fakers on personnel tests. Canadian Journal of Behavioural Science, 27, 343–355. https://doi.org/10.1037/0008-400X.27.3.343

30.

Holden

R. R.

Hibbs

(1995). Incremental validity of response latencies for detecting fakers on a personality test. Journal of Research in Personality, 29, 362–372. https://doi.org/10.1006/jrpe.1995.1021

31.

Holden

R. R.

Kroner

D. G.

(1992). Relative efficacy of differential response latencies for detecting faking on a self-report measure of psychopathology. Psychological Assessment, 4(2), 170–173. https://doi.org/10.1037/1040-3590.4.2.170

32.

Holden

R. R.

Lambert

C. E.

(2015). Response latencies are alive and well for identifying fakers on a self-report personality inventory: A reconsideration of van Hooft and Born (2012). Behavior Research, 47, 1436–1442. https://doi.org/10.3758/s13428-014-0524-5

33.

Hontangas

P. M.

Torre

D. L. J.

Ponsoda

Leenen

Morillo

Abad

F. J.

(2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851

34.

Houston

J. S.

Borman

W. C.

Farmer

W. F.

Bearden

R. M.

(2006). Development of the navy computer adaptive personality scales (NCAPS) (Technical report no. 06-2). Navy Personnel Research, Studies, and Technology Division, Bureau of Naval Personnel.

35.

Jansen

(1997). Rasch’s model for reading speed with manifest explanatory variables. Psychometrika, 62, 393–409. https://doi.org/10.1007/BF02294558

36.

Kreitchmann

R. S.

Sorrel

M. A.

Abad

F. J.

(2023). On bank assembly and block selection in multidimensional forced-choice adaptive assessments. Educational and Psychological Measurement, 83(2), 294–321. https://doi.org/10.1177/00131644221087986

37.

Kuiper

(1981). Convergent evidence for the self as a prototype: The “Inverted-U RT Effect” for self and other judgements. Personality and Social Psychology Bulletin, 7, 438–443. https://doi.org/10.1177/014616728173012

38.

Kuncel

(1973). Response processes and relative location of subject and item. Educational and Psychological Measurement, 33, 545–563. https://doi.org/10.1177/001316447303300302

39.

Lin

(2022). Reliability estimates for IRT-based forced-choice assessment scores. Organizational Research Methods, 25(3), 575–590. https://doi.org/10.1177/1094428121999086

40.

Lee

Stark

(2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individual Differences, 123, 229–235. https://doi.org/10.1016/j.paid.2017.11.031

41.

MacCann

Ziegler

Roberts

R. D.

(2011). Faking in personality assessment. In Ziegler

MacCann

Roberts

R. D.

(Eds.), New perspectives on faking in personality assessment (pp. 309–329). Oxford University Press.

42.

McCrae

R. R.

Costa

P. T.

(1992). Discriminant validity of NEO-PIR facet scales. Educational and Psychological Measurement, 52, 229–237. https://doi.org/10.1177/001316449205200128

43.

Morillo

Leenen

Abad

F. J.

Hontangas

P. M.

Torre

de la Ponsoda

(2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40, 500–516. https://doi.org/10.1177/0146621616662226

44.

Neubauer

Malle

(1997). Questionnaire response latencies: Implications for personality assessment and self-schema theory. European Journal of Psychological Assessment, 13, 109–117.

45.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Elsevier. https://doi.org/10.1016/B978-0-12-590241-0.50006-X

46.

Ranger

(2013). Modeling responses and response times in personality tests with rating scales. Psychological Test and Assessment Modeling, 55, 361–382.

47.

Ranger

Kuhn

J. T.

(2012). A flexible latent trait model for response times in tests. Psychometrika, 77(1), 31–47. https://doi.org/10.1007/s11336-011-9231-7

48.

Ranger

Ortner

(2011). Assessing personality traits through response latencies using IRT. Educational and Psychological Measurement, 71, 389–406. https://doi.org/10.1177/0013164410382895

49.

Ratcliff

Smith

P. L.

Brown

S. D.

McKoon

(2016). Diffusion decision model: Current issues and history. Trends in Cognitive Sciences, 20(4), 260–281. https://doi.org/10.1016/j.tics.2016.01.007

50.

R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

51.

Salgado

J. F.

Tauriz

(2014). The Five-Factor Model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23(1), 3–30. https://doi.org/10.1080/1359432X.2012.716198

52.

Schmitt

Oswald

F. L.

(2006). The impact of corrections for faking on the validity of noncognitive measures in selection settings. Journal of Applied Psychology, 91(3), 613–621. https://doi.org/10.1037/0021-9010.91.3.613

53.

Shannon

C. E.

(1950). The mathematical theory of communication. Bell Labs Technical Journal, 3(9), 31–32. https://doi.org/10.1063/1.3067010

54.

Silver

N. C.

Dunlap

W. P.

(1987). Averaging correlation coefficients: Should Fisher’s z transformation be used? Journal of Applied Psychology, 72(1), 146–148. https://doi.org/10.1037/0021-9010.72.1.146

55.

Stark

Chernyshenko

O. S.

Drasgow

(2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29, 184–203. https://doi.org/10.1177/0146621604273988

56.

Stark

Chernyshenko

O. S.

Drasgow

Nye

C. D.

White

L. A.

Heffner

Farmer

W. L.

(2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26, 153–164. https://doi.org/10.1037/mil0000044

57.

Temple

D. E.

Geisinger

K. F.

(1990). Response latency to computer-administered inventory items as an indicator of emotional arousal. Journal of Personality Assessment, 54, 289–297.

58.

Thissen

(1983). Timed testing: An approach using item response theory. In Weiss

D. J.

(Ed.), New horizons in testing (pp. 179–203). Academic Press.

59.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 101(2), 266–270. https://doi.org/10.1037/0033-295X.101.2.266

60.

Tyron

Mulloy

(1993). Further validation of computer-assessed response time to emotionally evocative stimuli. Journal of Personality Assessment, 61, 231–236.

61.

van der Linden

W. J.

(2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics, 33(1), 5–20. https://doi.org/10.3102/1076998607302626

62.

van der Linden

W. J.

Entink

R. H. K.

Fox

J. P.

(2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34(5), 327–347. https://doi.org/10.1177/0146621609349800

63.

Van der Linden

W. J.

Guo

(2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384. https://doi.org/10.1007/s11336-007-9046-8

64.

van der Linden

W. J.

van Krimpen-Stoop

E. M.

(2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265. https://doi.org/10.1007/BF02294800

65.

Van Herk

Poortinga

Verhallen

. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346–360. https://doi.org/10.1177/0022022104264126

66.

Walton

K. E.

Cherkasova

Roberts

R. D.

(2020). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment, 27(4), 706–718. https://doi.org/10.1177/1073191119843585

67.

Wang

Chang

H. H.

Douglas

J. A.

(2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66(1), 144–168. https://doi.org/10.1111/j.2044-8317.2012.02045.x

68.

Wetzel

Lüdtke

Zettler

Böhnke

J. R.

(2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3), 279–291. https://doi.org/10.1177/1073191115583714

69.

Wise

S. L.

Kong

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2

70.

Wong

C. S.

Law

K. S.

Wong

P. M.

(2004). Development and validation of a forced choice emotional intelligence measure for Chinese respondents in Hong Kong. Asia Pacific Journal of Management, 21(4), 535–559. https://doi.org/10.1023/B:APJM.0000048717.31261.d0

71.

Zhang

Sun

Drasgow

Chernyshenko

O. S.

Nye

C. D.

Stark

White

L. A.

(2019). Though forced, still valid: Psychometric equivalence of forced-choice and single-item measures. Organizational Research Methods, 23(3), 569–590. https://doi.org/10.1177/1094428119836486