Using Only Numeric Labels Instead of Verbal Labels: Stripping Rating Scales to Their Bare Minimum in Web Surveys

Abstract

With the increasing use of smartphones in web surveys, considerable efforts have been devoted to reduce the amount of screen space taken up by questions. An emerging stream of research in this area is aimed at optimizing the design elements of rating scales. One suggestion that has been made is to completely abandon verbal labels and use only numeric labels instead. This approach deliberately shifts the task of scale interpretation to the respondents and reduces the information given to them with an intention to reduce their response burden while still preserving the scale meaning. Following prior research, and by drawing on the established model of the cognitive response process, we critically tested these assumptions. Based on a web survey experiment, we found that omitting verbal labels and using only numeric labels instead pushed respondents to focus their responses on the endpoints of a rating scale. Moreover, drawing on response time paradata, we showed that their response burden was not reduced when presented with only numeric labels; quite the opposite was the case, especially when respondents answered the scale with only numeric labels for the first time, which seemed to entail additional cognitive effort. Based on our findings, we advise against using only numeric labels for rating scales in web surveys.

Keywords

response styles satisficing data quality attitude scales paradata web surveys

Get full access to this article

View all access options for this article.

References

Alwin

D. F.

Krosnick

J. A.

(1991). The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods & Research, 20(1), 139–181.

American Association for Public Opinion Research. (2016). Standard definitions: Final dispositions of case codes and outcome rates for surveys. The American Association for Public Opinion Research.

Antoun

Couper

M. P.

Conrad

F. G.

(2017). Effects of mobile versus pc web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81(S1), 280–306.

Baumgartner

Steenkamp

J. E. M.

(2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156.

Beatty

P. C.

Herrmann

(2002). To answer or not to answer: Decision processes related to survey item nonresponse. In Groves

R. M.

Dillmann

D. A.

Eltinge

J. L.

Little

R. J. A.

(Eds.), Survey nonresposne (pp. 71–86). Wiley.

Cabooter

E. F.

(2010). The impact of situational and dispositional variables on response styles with respect to attitude measures [PhD thesis, Universiteit Gent]. http://lib.ugent.be/catalog/rug01:001809333

Callegaro

DiSogra

(2008). Computing response metrics for online panels. Public Opinion Quarterly, 72(5), 1008–1032.

Cernat

Liu

(2019). Radio buttons in web surveys: Searching for alternatives. International Journal of Market Research, 61(3), 266–286.

Couper

M. P.

Peterson

G. J.

(2017). Why do web surveys take longer on smartphones? Social Science Computer Review, 35(3), 357–377.

10.

Dixon

Tucker

(2010). Survey nonresponse. In Marsden

P. V.

Wright

J. D.

(Eds.), Handbook of survey research (pp. 593–630). Emerald.

11.

Fuchs

Busse

(2009). The coverage bias of mobile web surveys across European countries. International Journal of Internet Science, 4(1), 21–33.

12.

Greszki

Meyer

Schoen

(2015). Exploring the effects of removing ‘too fast’ responses and respondents from web surveys. Public Opinion Quarterly, 79(2), 471–503.

13.

Gummer

Quoß

Roßmann

(2019). Does increasing mobile device coverage reduce heterogeneity in completing web surveys on smartphones? Social Science Computer Review, 37(3), 371–384.

14.

Gummer

Vogel

Kunz

Roßmann

(2020). Let’s put a smile on that scale: Findings from three web survey experiments. International Journal of Market Research, 62(1), 18–26.

15.

Hanson

(2015). Comparing agreement and item-specific response scales: Results from an experiment. Social Research Practice, 1, 17–25.

16.

Bartram

Inceoglu

van de Vijver

F. J. R.

(2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45(7), 1029–1045.

17.

Hofmans

Theuns

Baekelandt

Mairesse

Schillewaert

Cools

(2007). Bias and changes in perceived intensity of verbal qualifiers effected by scale orientation. Survey Research Methods, 1(2), 97–108.

18.

Höhne

J. K.

Lenzner

(2018). New insights on the cognitive processing of agree/disagree and item-specific questions. Journal of Survey Statistics and Methodology, 6(3), 401–417.

19.

Höhne

J. K.

Revilla

M. A.

Lenzner

(2018). Comparing the performance of agree/disagree and item-specific questions across PCs and smartphones. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 14(3), 109–118. https://doi.org/10.1027/1614-2241/a000151

20.

Keusch

Yan

(2017). Web versus mobile web: An experimental study of device effects and self-selection effects. Social Science Computer Review, 35(6), 751–769. https://doi.org/10.1177/0894439316675566

21.

Knol

M. J.

Pestman

W. R.

Grobbee

D. E.

(2011). The (mis)use of overlap of confidence intervals to assess effect modification. European Journal of Epidemiology, 26, 253–254.

22.

Knowles

E. S.

Condon

C. A.

(1999). Why people say ‘yes’: A dual-process theory of acquiescence. Journal of Personality and Social Psychology, 77(2), 379–386. https://doi.org/10.1037/0022-3514.77.2.379

23.

Krebs

Bachner

Y. G.

(2018). Effects of rating scale direction under the condition of different reading direction. Methods, Data, Analyses, 12(1), 105–126.

24.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

25.

Krosnick

J. A.

(1999). Survey research. Annual Review of Psychology, 50, 537–567.

26.

Krosnick

J. A.

Alwin

D. F.

(1988). A test of the form-resistant correlation hypothesis: Ratings, rankings, and the measurement of values. Public Opinion Quarterly, 52(4), 526–538.

27.

Krosnick

J. A.

Fabrigar

L. R.

(1997). Designing rating scales for effective measurement in surveys. In Lyberg

Biemer

Collins

De Leeuw

Dippo

Schwarz

Trewin

(Eds.), Survey measurement and process quality (pp. 141–164). Wiley.

28.

Krosnick

J. A.

Holbrook

A. L.

Berent

M. K.

Carson

R. T.

Hanemann

W. M.

Kopp

R. J.

Mitchell

Presser

Ruud

P. A.

Smith

V. K.

Moody

W. R.

Green

M. C.

Conaway

(2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66(3), 371–403.

29.

Krosnick

J. A.

Judd

C. M.

Wittenbrink

(2005). The measurement of attitudes. In Albarracin

Johnson

B. T.

Zanna

M. P.

(Eds.), The handbook of attitudes (pp. 21–76). Lawrence Erlbaum Associates.

30.

Lelkes

Weiss

(2015, July–September). Much ado about acquiescence: The relative validity and reliability of construct-specific and agree–disagree questions. Research and Politics, 1–8. https://doi.org/10.1177/2053168015604173

31.

Liu

Lee

Conrad

F. G.

(2015). Comparing extreme response styles between agree-disagree and item-specific scales. Public Opinion Quarterly, 79(4), 952–975. https://doi.org/10.1093/poq/nfv034

32.

Lugtig

Toepoel

(2016). The use of PCs, smartphones, and tablets in a probability-based panel survey. Social Science Computer Review, 34(1), 78–94. https://doi.org/10.1177/0894439315574248

33.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725–743. https://doi.org/10.1177/0894439313485201

34.

McCarty

J. A.

Shrum

L. J.

(2000). The measurement of personal values in survey research: A test of alternative rating procedures. Public Opinion Quarterly, 64(3), 271–298.

35.

Menold

(2020). Rating-scale labeling in online surveys: An experimental comparison of verbal and numeric rating scales with respect to measurement quality and respondents’ cognitive processes. Sociological Methods & Research, 49(1), 79–107.

36.

Menold

Kaczmirek

Lenzner

Neusar

(2014). How do respondents attend to verbal labels in rating scales? Field Methods, 26(1), 21–39. https://doi.org/10.1177/1525822X13508270

37.

Mohorko

de Leeuw

Hox

(2013). Internet coverage and coverage bias in Europe: Developments across countries and over time. Journal of Official Statistics, 29(4), 609–622. https://doi.org/10.2478/jos-2013-0042

38.

Moors

Kieruj

N. D.

Vermund

J. N.

(2014). The effect of labelling and numbering of response scales on the likelihood response bias. Sociological Methodology, 44(1), 369–399.

39.

O’Muircheartaigh

Gaskell

Wright

D. B.

(1995). Weighing anchors: Verbal and numeric labels of rating scales. Journal of Official Statistics, 11(3), 295–307.

40.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press.

41.

Poushter

(2016). Smartphone ownership and internet usage continues to climb in emerging economies. Pew Research Center.

42.

Rammstedt

Danner

Bosnjak

(2017). Acquiescence response styles: A multilevel model explaining individual-level and country-level differences. Personality and Individual Differences, 107(1), 190–194.

43.

Roßmann

Gummer

Silber

(2018). Mitigating satisficing in cognitively demanding grid questions: Evidence from two web-based experiments. Journal of Survey Statistics and Methodology, 6(3), 376–400.

44.

Saris

Revilla

Krosnick

J. A.

Shaeffer

E. M.

(2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61–79.

45.

Schenker

Gentleman

J. F.

(2001). On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician, 55(3), 182–186.

46.

Schlosser

Höhne

J. K.

(2018). ECSP—Embedded Client Side Paradata. https://zenodo.org/record/1218941

47.

Schuler

Thornton

G. C.

III Frintrup

Mueller-Hanson

(2004). AMI: Achievement Motivation Inventory. Technical and user’s manual. Hogrefe & Huber.

48.

Schwarz

Hippler

H.-J.

(1995). The numeric values of rating scales: A comparison of their impact in mail surveys and telephone interviews. International Journal of Public Opinion Research, 7(1), 72–74.

49.

Schwarz

Knäuper

Hippler

H.-J.

Noelle-Neumann

Clark

(1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55(4), 570–582.

50.

Stange

Barry

Smyth

Olson

K. M.

(2018). Effects of smiley face scales on visual processing of satisfaction questions in web surveys. Social Science Computer Review, 36(6), 756–766. https://doi.org/10.1177/0894439316674166

51.

Thomas

R. K.

Barlas

F. M.

(2018). We’ve got your number: Can numeric labels replace semantic labels in scales? [Paper presentation]. 19th General Online Research Conference, Cologne, Germany.

52.

Thomas

R. K.

Barlas

F. M.

(2019). New mobile-friendly labels: Using numbers as labels [Paper presentation]. 8th Conference of the European Survey Research Conference, Zagreb, Croatia.

53.

Toepoel

Dillman

D. A.

(2010). How visual design affects the interpretability of survey questions. In Das

Ester

Kaczmirek

(Eds.), Social and behavioral research and the internet (pp. 165–190). Routledge.

54.

Toepoel

Dillman

D. A.

(2011). Words, numbers, and visual heuristics in web surveys: Is there a hierarchy of importance? Social Science Computer Review, 29(2), 193–207.

55.

Toepoel

Funke

(2018). Sliders, visual analogue scales, or buttons: Influence of formats and scales in mobile and desktop surveys. Mathematical Population Studies, 25(2), 112–122.

56.

Toepoel

Vermeeren

Metin

(2019). Smileys, stars, hearts, buttons, tiles or grids: Influence of response format on substantive response, questionnaire experience and response time. Bulletin of Sociological Methodology, 142, 57–74.

57.

Tourangeau

Couper

M. P.

Conrad

F. G.

(2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393. https://doi.org/10.1093/po1/nfh035

58.

Tourangeau

Couper

M. P.

Conrad

F. G.

(2007). Color, labels, and interpretive heuristics for response scales. Public Opinion Quarterly, 71(1), 91–112. https://doi.org/10.1093/poq/nfl046

59.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

60.

Weijters

Baumgartner

Schillewaert

(2013). Reversed item bias: An integrative model. Psychological Methods, 18(3), 320–334. https://doi.org/10.1037/a0032121

61.

Weijters

Cabooter

Schillewaert

(2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236–247.

62.

Yan

Keusch

(2015). The effects of the direction of rating scales on survey responses in a telephone survey. Public Opinion Quarterly, 79(1), 145–165.

63.

Zhang

Conrad

F. G.

(2014). Speeding in web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods, 8(2), 127–135.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB