Extreme Response Style: A Simulation Study Comparison of Three Multidimensional Item Response Models

Abstract

Several multidimensional item response models have been proposed for survey responses affected by response styles. Through simulation, this study compares three models designed to account for extreme response tendencies: the IRTree Model, the multidimensional nominal response model, and the modified generalized partial credit model. The modified generalized partial credit model results in the lowest item mean squared error (MSE) across simulation conditions of sample size (500, 1,000), survey length (10, 20), and number of response options (4, 6). The multidimensional nominal response model is equally suitable for surveys measuring one substantive trait using responses to 10 four-option, forced-choice Likert-type items. Based on data validation, comparison of item MSE, and posterior predictive model checking, the IRTree Model is hypothesized to account for additional sources of construct-irrelevant variance.

Keywords

MIRT extreme response style MCMC questionnaires

Get full access to this article

View all access options for this article.

References

Baumgartner

Steenkamp

J.-B. E. M

. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38, 143-156.

Becker

Bjorner

J. B.

Fliege

Klapp

B. F.

Rose

Walter

O. B.

(2005). Development of a Computer-Adaptive Test for Depression (D-CAT). Quality of Life Research, 14, 2277-2291.

Berg

I. A.

Collier

J. S.

(1953). Personality and group differences in extreme response sets. Educational and Psychological Measurement, 13, 164-169.

Bishop

G. F.

(1987). Experiments with the middle response alternative in survey questions. Public Opinion Quarterly, 51, 220-232.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Böckenholt

(2017). Measuring response styles in Likert items. Psychological Methods, 22, 69-83.

Böckenholt

Meiser

(2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70, 159-181.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335-352.

Bolt

D. M.

Newton

J. R.

(2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71, 814-833.

10.

Cabooter

Millet

Weijters

Pandelaere

(2016). The “I” in extreme responding. Journal of Consumer Psychology, 26, 510-523.

11.

Cabooter

Weijters

De Beuckelaer

Davidov

(2016). Is extreme response style domain specific? Findings from two studies in four countries. Quality & Quantity, 51, 2605-2622.

12.

Cronbach

L. J.

(1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475-494.

13.

Cronbach

L. J.

(1950). Further evidence on response sets and test design. Educational and Psychological Measurement, 10, 3-31.

14.

Cronbach

L. J.

Snow

R. E.

Wiley

D. E.

(1991). Improving inquiry in social science: A volume in honor of Lee J. Cronbach. Hillsdale, NJ: Lawrence Erlbaum.

15.

de Jong

M. G.

Steenkamp

J.-B. E. M.

Fox

J.-P.

Baumgartner

. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45, 104-115. doi:10.1509/jmkr.45.1.104

16.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.

17.

Falk

C. F.

Cai

(2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21, 328-347. doi:10.1037/met0000059

18.

Greenleaf

E. A.

(1992a). Improving rating scale measures by detecting and correcting bias components in some response styles. Journal of Marketing Research, 29, 176-188.

19.

Greenleaf

E. A.

(1992b). Measuring extreme response style. Public Opinion Quarterly, 56, 328-351.

20.

Hambleton

R. K.

Swaminathan

(1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff.

21.

Hamilton

D. L.

(1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69, 192-203.

22.

Jackson

D. N.

Messick

(1958). Content and style in personality assessment. Psychological Bulletin, 55, 243-252.

23.

Jeon

De Boeck

(2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48, 1070-1085.

24.

Jeon

De Boeck

van der Linden

(2017). Modeling answer change behavior: An application of a generalized item response tree model. Journal of Educational and Behavioral Statistics, 42, 467-490. doi:10.3102/1076998616688015

25.

Jin

K.-Y.

Wang

W.-C.

(2014). Generalized IRT models for extreme response style. Educational and Psychological Measurement, 74, 116-138.

26.

Johns

(2005). One size doesn’t fit all: Selecting response scales for BES attitude items. Journal of Elections, Public Opinion and Parties, 15, 237-264.

27.

Johnson

T. R.

Bolt

D. B.

(2010). On the use of factor-analytic multinominal logit item response models to account for individual differences in response style. Journal of Educational and Behavioral Statistics, 35, 92-114.

28.

Kalton

Roberts

Holt

(1980). The effects of offering a middle response option with opinion questions. Journal of the Royal Statistical Society, Series D (The Statistician), 29, 65-78.

29.

Kieruj

N. D.

Moors

(2010). Variations in response style behavior by response scale format in attitude research. International Journal of Public Opinion Research, 22, 320-342. doi:10.1093/ijpor/edq001

30.

Krosnick

J. A.

Holbrook

A. L.

Berent

M. K.

Carson

R. T.

Hanemann

W. M.

Kopp

R. J.

. . . Conaway

(2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice?Public Opinion Quarterly, 66, 371-403.

31.

Levy

Mislevy

R. J.

Sinharay

(2009). Posterior predictive model checking for multidimensionality in item response theory. Applied Psychological Measurement, 33, 519-537. doi:10.1177/0146621608329504

32.

Lewis

N. A.

Taylor

J. A.

(1955). Anxiety and extreme response preferences. Educational and Psychological Measurement, 15, 111-116.

33.

K.-Y.

(2001). Interpersonal harmony and the values of forbearance: Understanding generation gap through interpersonal conflicts (NSC Research Report No. NSC90-2413-H-031-006-SSS). Taipei, Taiwan: National Science Council.

34.

Marin

Gamba

R. J.

Marin

B. V.

(1992). Extreme response style and acquiescence among Hispanics: The role of acculturation and education. Journal of Cross-Cultural Psychology, 23, 498-509.

35.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

36.

Moors

(2003). Diagnosing response style behavior by means of a latent-class factor approach. Socio-demographic correlates of gender role attitudes and perceptions of ethnic discrimination reexamined. Quality & Quantity, 37, 277-302. doi:10.1023/A:1024472110002

37.

Moors

(2010). Ranking the ratings: A latent-class regression model to control for overall agreement in opinion research. International Journal of Public Opinion Research, 22, 93-119. doi:10.1093/ijpor/edp036

38.

Moors

(2012). The effect of response style bias on the measurement of transformational, transactional, and laissez-faire leadership. European Journal of Work and Organizational Psychology, 21, 271-298. doi:10.1080/1359432X.2010.550680

39.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176. doi:10.1177/014662169201600206

40.

Myers

A. J.

Ames

A. J.

Leventhal

B. C.

Holzman

M. A.

(2018, April). An item response tree model for validating rubric scoring processes. Paper presented at the Annual meeting of the National Council of Measurement in Education, New York, NY.

41.

Naemi

B. D.

Beal

D. J.

Payne

S. C.

(2009). Personality predictors of extreme response style. Journal of Personality, 77, 261-286. doi:10.1111/j.1467-6494.2008.00545.x

42.

National Center for Education Statistics. (2008). NAEP technical documentation: The generalized partial credit model. Retrieved from https://nces.ed.gov/nationsreportcard/tdw/analysis/scaling_models_gen.aspx

43.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of social psychological attitudes, Vol. 1: Measures of personality and social psychological attitudes (pp. 17-59). San Diego, CA: Academic Press.

44.

Reeve

B. B.

(2002). An introduction to modern measurement theory. National Cancer Institute. Retrieved from https://pdfs.semanticscholar.org/d6b1/0ae949ff4d89b2bfe27a36e40fd83e7aeb6e.pdf

45.

Revuelta

(2014). Multidimensional item response model for nominal variables. Applied Psychological Measurement, 38, 549-562. doi:10.1177/0146621614536272

46.

Rossi

P. E.

Gilula

Allenby

G. M.

(2001). Overcoming scale usage heterogeneity: A Bayesian hierarchical approach. Journal of the American Statistical Association, 96, 20-31.

47.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 35, 139.

48.

Schuman

Presser

(1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York, NY: Academic Press.

49.

Stone

C. A.

Zhu

(2015). Bayesian analysis of item response theory models using SAS. Cary, NC: SS Institute.

50.

Thissen

(1993). Repealing rules that no longer apply to psychological measurement. In Frederiksen

Mislevy

R. J.

Bejar

I. I.

(Eds.), Test theory for a new generation of tests (pp. 79-97). Hillsdale, NJ: Lawrence Erlbaum.

51.

Thissen

Cai

Bock

R. D.

(2010). The nominal categories item response model. In Ostini

M. L. N. R.

(Ed.), Handbook of polytomous item response theory models (pp. 43-75). New York, NY: Routledge.

52.

Thissen-Roe

Thissen

(2013). A two-decision model for responses to Likert-type Items. Journal of Educational and Behavioral Statistics, 38, 522-547.

53.

van Herk

Poortinga

Y. H.

Verhallen

T. M. M

. (2004). Response styles in rating scales: Evidence of method bias in data from 6 EU countries. Journal of Cross-Cultural Psychology, 35, 346-360.

54.

van Rosmalen

van Herk

Groenen

P. J. F

. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47, 157-172. doi:10.1509/jmkr.47.1.157

55.

Van Vaerenbergh

Thomas

T. D

. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195-217. doi:10.1093/ijpor/eds021

56.

Weijters

Schillewaert

Geuens

(2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36, 409-422. doi:10.1007/s11747-007-0077-6

57.

Yen

W. M.

Fitzpatrick

A. R.

(2006). Item response theory. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 111-153). Westport, CT: American Council on Education.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB