Verbal rating scale polarity and verbalizations of the middle category that do not match the polarity in agreement rating scales were investigated. Two randomized web survey experiments were conducted using a probability panel of German Internet users. The classical bipolar “disagree/agree” verbalization was compared with the unipolar “do not agree/agree” alternative. In both experiments, attitudes on gender roles were measured using a different number of rating scale categories (seven vs. five) in each. After controlling for variables related to satisficing, biases were lowest with the unipolar “do not agree/agree” rating scale in which the verbalization of the middle category was also unipolar. Reliability decreased when the verbalization of the middle category did not suit the rating scale polarity. Researchers should pay attention to the polarity of rating scales and use verbalization of the middle category consistent with the rating scale polarity.
AlwinD. F.KrosnickJ. A. (1991). The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods & Research, 20, 139–181.
BassiliJ. N.KrosnickJ. A. (2000). Do strength-related attitude properties determine susceptibility to response effects? New evidence from response latency, attitude extremity, and aggregate indices. Political Psychology, 21, 107–132.
4.
BishopG. F. (1990). Issue involvement and response effects in public opinion surveys. Public Opinion Quarterly, 54, 209. doi:10.1086/269198
5.
BlessH.BohnerG.FellhauerR. F.SchwarzN. (1991). Need for cognition: Eine skala zur erfassung von freude und engagement bei denkaufgaben. Mannheim(ZUMA-Arbeitsbericht 1991/06). Retrieved fromhttp://nbn-resolving.de/urn:nbn:de:0168-ssoar-68892
6.
BraunM. (2006). Funktionale Äquivalenz in interkulturell vergleichenden Umfragen: Mythos und Realität. Mannheim, Germany: ZUMA.
7.
CacioppoJ. T.PettyR. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131. doi:10.1037/0022-3514.42.1.116
8.
CostaP. T.JrMcCraeR. R. (1992). Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) professional manual. Odessa, FL: Psychological Assessment Resources.
German Longitudinal Election Study: Langfrist-Panel. (2013–2017). Fragebogendokumentation: Haupterhebung 2013 (Nachwahl) ZA5770, Version 1.0.0. Retrieved February 2, 2019, fromhttps://www.gesis.org/wahlen/gles/dokumente/
12.
HenkelJ.NeußN. (2015). Die verankerung der transitionsthematik in studium und ausbildung. Frühe Bildung, 4, 25–32.
13.
ISSP2014 - Citizenship II Questionnaire. ZA6670_q_de.pdf. doi:0.4232/1.12590
14.
KaltonG.RobertsJ.HoltT. (1980). The effects of offering a middle response option with opinion questions. Journal of the Royal Statistical Society. Series D (The Statistician), 29, 65–78.
15.
KaplanK. J. (1972). On the ambivalence-indifference problem in attitude theory and measurement: A suggested modification of the semantic differential technique. Psychological Bulletin, 77, 361–372.
16.
KierujN. D.MoorsG. (2010). Variations in response style behavior by response scale format in attitude research. International Journal of Public Opinion Research, 22, 320–342. doi:10.1093/ijpor/edq001
17.
KlopferF. J.MaddenT. M. (1980). The middlemost choice on attitude items: Ambivalence, neutrality, or uncertainty?Personality and Social Psychology Bulletin, 6, 97–101. doi:10.1177/014616728061014
18.
KrebsD. (2012). The impact of response format on attitude measurement. In SalzbornS.DavidovE.ReineckeJ. (Eds.), Methods, theories, and empirical applications in the social sciences (pp. 105–113). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
19.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236. doi:10.1002/acp.2350050305
20.
KrosnickJ. A.FabrigarL. R. (1997). Designing rating scales for effective measurement in surveys. In LybergL. E.BiemerP. P.CollinsM.De LeeuwE. D.DippoC.SchwarzN., et al. (Eds.), Survey measurement and process quality (pp. 141–164). New York: Wiley.
21.
KrosnickJ. A.PresserS. (2009). Question and questionnaire design. In MarsdenP. V.WrightJ. D. (Eds.), Handbook of survey research (pp. 263–313). Bingley, England: Emerald Group.
22.
KrosnickJ. A.SchumanH. (1988). Attitude intensity, importance, and certainty and susceptibility to response effects. Journal of Personality and Social Psychology, 54, 940–952. doi:10.1037/0022-3514.54.6.940
MenoldN. (2017). Rating-scale labeling in online surveys: An experimental comparison of verbal and numeric rating scales with respect to measurement quality and respondents’ cognitive processes. Sociological Methods & Research. doi:10.1177/0049124117729694
26.
MenoldN.BognerK. (2014). Gestaltung von ratingskalen in Fragebögen. GESIS Survey Guidelines. doi:10.15465/sdm-sg_015
27.
MenoldN.KaczmirekL.LenznerT.NeusarA. (2014). How do respondents attend to verbal labels in rating scales?Field Methods, 26, 21–39. doi:10.1177/1525822X13508270
28.
MenoldN.RaykovT. (2015). Can reliability of multiple component measuring instruments depend on response option presentation mode?Educational and Psychological Measurement, 76, 454–469. doi:10.1177/0013164415593602
29.
MõttusR.AllikJ.RealoA.RossierJ.ZeccaG.Ah-KionJ.…JohnsonW. (2012). The effect of response style on self-reported conscientiousness across 20 countries. Personality and Social Psychology Bulletin, 38, 1423–1436. doi:10.1177/0146167212451275
30.
MuthénL. K.MuthénB. O. (Eds.). (2014). Mplus user’s guide. Los Angeles, CA: Author.
31.
O’MuircheartaighC. A.GaskellG. D.WrightD. B. (1995). Weighing anchors: Verbal and numeric labels for response scales. Journal of Official Statistics, 11, 295–307.
32.
PaulhusD. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverP. R.WrightsmanL. S. (Eds.), Measures of personality and social psychological attitudes (1st ed., pp. 17–59). San Diego, CA: Academic Press.
33.
RammstedtB.JohnO. P. (2005). Kurzversion des big five inventory (BFI-K): Entwicklung und validierung eines ökonomischen Inventars zur Erfassung der fünf Faktoren der Persönlichkeit. Diagnostica, Jg, 51, 195–206. doi:10.1026/0012-1924.51.4.195
34.
RattingerH.KrämerJ. (1995). Wahlnorm und Wahlbeteiligung in der Bundesrepublik Deutschland: Eine Kausalanalyse. Politische Vierteljahresschrift, 36, 267–285.
35.
RaykovT.MarcoulidesG. A. (2011). Introduction to psychometric theory. New York: Taylor & Francis.
36.
RohrmannB. (1978). Empirische studien zur Entwicklung von Antwortskalen für die sozialwissenschaftliche Forschung. Zeitschrift für Sozialpsychologie, 9, 222–245.
37.
RuëlH. J.BondaroukT. V.van der VeldeM. (2007). The contribution of e-HRM to HRM effectiveness. Employee Relations, 29, 280–291.
38.
SarisW. E.RevillaM. A.KrosnickJ. A.ShaefferE. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4, 61–79.
39.
SchaefferN. C.PresserS. (2003). The science of asking questions. Annual Review of Sociology, 29, 65–88. doi:10.1146/annurev.soc.29.110702.110112
40.
SchumanH.PresserS. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Thousand Oaks, CA: Sage.
41.
SchwarzN.HipplerH.-J. (1994). The numeric values of rating scales: A comparison of their impact in mail surveys and telephone interviews. Mannheim, Germany: ZUMA.
42.
SchwarzN.KnäuperB.HipplerH-J.Noelle-NeumannE.ClarkL. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570–582. doi:10.1086/269282
43.
SchweizerK. (2011). On the changing role of Cronbach’s α in the evaluation of the quality of a measure. European Journal of Psychological Assessment, 27, 143–144.
44.
StegmannS.DickR. V.UlrichJ.CharalambousJ.MenzelB.EgoldN.Tai-ChiT. (2010). Der work design questionnaire: Vorstellung und erste validierung einer duetschen version. Zeirschrift für Arbeits-u. Organisationpsychologie, 54, 1–28. doi:10.1026/0932-4089/a000002
45.
StruminskayaB.KaczmirekL.SchaurerI.BandillaW. (2014). Assessing representativeness of a probability-based online panel in Germany. In CallegaroM.BakerR.BethlehemG. A. J.KrosnickJ. A.LavrakasP. J. (Eds.), Online panel research: A data quality perspective (pp. 61–84). New York: Wiley.
46.
SturgisP.RobertsC.SmithP. (2014). Middle alternatives revisited: How the neither/nor response acts as a way of saying “I don’t know”?Sociological Methods & Research, 43, 15–38.
47.
TourangeauR.RipsL. J.RasinskiK. A. (2009). The psychology of survey response(10 print). Cambridge, England: Cambridge University Press.
48.
WengL.-J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956–972. doi:10.1177/0013164404268674
49.
YooS.-J. (2010). Two types of neutrality: Ambivalence versus indifference and political participation. The Journal of Politics, 72, 163–177. doi:10.1017/s0022381609990545
50.
ZallerJ.FeldmanS. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36, 579–616.
