Previous research shows that the direction of rating scales can influence participants’ response behavior. Studies also suggest that the device used to complete online surveys might affect the susceptibility to these effects due to the different question layouts (e.g., horizontal grids vs. vertical individual questions). This article contributes to previous research by examining scale direction effects in an online multi-device survey conducted with panelists in Spain. In this experiment, respondents were randomly assigned to two groups where the scale direction was manipulated (incremental vs. decremental). Respondents completed the questionnaire using the device of their choosing (57.8% used PCs; 36.5% used smartphones; and 5.7% used tablets). The results show that scale direction influenced response distributions but did not significantly affect data quality. In addition, our findings indicate that scale direction effects were comparable across devices. Findings are discussed and implications are highlighted.
Get full access to this article
View all access options for this article.
References
1.
BelsonW. A.1966. The effects of reversing the presentation order of verbal rating scales. Journal of Advertising Research6:30–37.
2.
BishopG. F.SmithA.. 2001. Response-order effects and the early Gallup split-ballots. Public Opinion Quarterly65:479–505.
3.
ChangL.KrosnickJ. A.. 2009. National surveys via RDD telephone interviewing versus the Internet. Comparing simple representativeness and response quality. Public Opinion Quarterly73:641–78.
4.
ChristianL. M.ParsonsN. L.DillmanD. A.. 2009. Designing scalar questions for web surveys. Sociological Methods & Research37:393–425.
5.
CornesseC.BlomA. G.. 2020. Response quality in nonprobability and probability-based online panels. Sociological Methods & Research1–30.
6.
DeCastellarnauA.2018. A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity52:1523–59.
7.
DiedenhofenB.2016. cocron: Statistical comparisons of two or more alpha coefficients (Version 1.0-1).
8.
EkehammarB.AkramiN.ArayaT.. 2000. Development and validation of Swedish classical and modern sexism scales. Scandinavian Journal of Psychology41:307–14.
GalesicM.TourangeauR.CouperM. P.ConradF.. 2008. Eye-tracking data: New insights on response order effects and other signs of cognitive shortcuts in survey responding. Public Opinion Quarterly72:892–913.
11.
GarbarskiD.SchaefferN. C.DykemaJ.. 2018. The effects of features of survey measurement on self-rated health: Response option order and scale orientation. Applied Research in Quality of Life14:545–60.
12.
HöhneJ. K.KrebsD.. 2017. Scale direction effects in agree/disagree and item-specific questions: A comparison of question formats. International Journal of Social Research Methodology21:91–103.
13.
HöhneJ. K.LenznerT.. 2015. Investigating response order effects in web surveys using eye-tracking. Psihologij48:361–77.
14.
IsraelG. D.2006. Visual cues and response format effects in mail surveys. Revised version of the paper presented at the Annual Meeting of the Southern Rural Sociological Association, Orlando, FL, February4–8.
15.
KeuschF.YanT.. 2018. The influence of rating scale direction on reliability. Paper presented at the annual conference of the American Association for Public Opinion Research, Denver, CO, May16–19.
16.
KrebsD.Hoffmeyer-ZlotnikJ. H. P.. 2010. Positive first or negative first? Effects of the order of answering categories on response behavior. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences6:118–27.
17.
KrebsD.HöhneJ. K.. 2019. Exploring scale direction effects and response behavior across PC and smartphone behavior across PC and smartphone surveys. Journal of Survey Statistics1–19.
18.
KrosnickJ. A.1991. Response strategies for coping with the demands of attitude measures in surveys. Applied Cognitive Psychology5:213–36.
19.
LiuM.CernatA.. 2018. Item-by-item versus matrix questions: A web survey experiment. Social Science Computer Review36:690–706.
20.
LiuM.KeuschF.. 2017. Effects of scale direction on response style of ordinal rating scales. Journal of Official Statistics33:137–54.
21.
MalhotraN.2008. Completion time and response order effects in web surveys. Public Opinion Quarterly72:914–34.
22.
MavletovaA.CouperM. P.LebedevD.. 2018. Grid and item-by-item formats in PC and mobile web surveys. Social Science Computer Review36:647–68.
23.
MenoldN.BognerK.. 2014. Gestaltung von Ratingskalen in Fragebögen. SDM Survey Guidelines. Mannheim, Germany: GESIS.
24.
MenoldN.BognerK.. 2016. Design of rating scales in questionnaires. GESIS Survey Guidelines. Mannheim, Germany: GESIS—Leibniz Institute for the Social Sciences.
25.
RammstedtB.KrebsD.. 2007. Does response scale format affect the answering of personality scales? Assessing the big five dimensions of personality with different response scales in a dependent sample. European Journal of Psychological Assessment23:32–38.
26.
RevillaM.ToninelliD.OchoaC.. 2017. An experiment comparing grids and item-by-item formats in web surveys completed through PCs and smartphones. Telematics and Informatics34:30–42.
27.
SalzbergerT.KollerM.. 2013. Towards a new paradigm of measurement in marketing. Journal of Business Research66:1307–17.
28.
SalzbergerT.KollerM.. 2019. The direction of the response scale matters—accounting for the unit of measurement. European Journal of Marketing53:871–91.
29.
SchwarzN.HipplerH.-J.. 2004. Response alternatives: The impact of their choice and presentation order. In Measurement errors in surveys, eds. BiemerP. P.GrovesR. M.LybergL. E.MathiowetzN. A.SudmanS., 41–56. Hoboken, NJ: Wiley-Interscience.
30.
SchwarzN.HipplerH.-J.Noelle-NewmannE.. 1992. A cognitive model of response-order effects in survey measurement. In Context effects in social and psychological research, eds. SchwarzN.SudmanS., 187–201. New York: Springer.
31.
SmythJ. D.IsraelG. D.NewberryM. G.HullR. G.. 2019. Effects of stem and response order on response patterns in satisfaction ratings. Field Methods31:260–76.
32.
StapletonC.2013. The smart(phone) way to collect survey data. Survey Practice6:1–8.
33.
SternM.DillmanD. A.SmythJ. D.. 2007. Visual design, order effects, and respondent characteristics in a self-administered survey. Journal of Official Statistics33:137–54.
34.
ToepoelV.DasM.van SoestA.. 2009. Design of web questionnaires: The effect of layout in rating scales. Journal of Official Statistics25:509–28.
35.
UnangstJ.AmayaA. E.SandersH. L.HowardJ.FerrellA.KaronS.DeberJ. A.. 2020. A process for decomposing total survey error in probability and nonprobability surveys: A case study comparing health statistics in US internet panels. Journal of Survey Statistics and Methodology8:62–88.
36.
WengL.-J.ChengC.-P.. 2000. Effects of response order on Likert-type scales. Educational and Psychological Measurement60:908–24.
37.
YanT.HuM.. 2018. Examining translation and respondents’ use of response scales in 3MC surveys: Multinational, multiregional, and multicultural contexts (3MC). In Advances in comparative survey methods: Multinational, multiregional, and multicultural contexts (3MC), eds. JohnsonT. P.PennellB.StoopI. A. L.DorerB., 501–18. Hoboken, NJ: John Wiley & Sons Inc.
38.
YanT.KeuschF.. 2015. The effects of the direction of rating scales on survey responses in a telephone survey. Public Opinion Quarterly79:145–65.
39.
YanT.KeuschF.. 2018. Direction of agree-disagree rating scales and data quality. Paper presented at the annual conference of the American Association for Public Opinion Research. Denver, CO, May16–19.
40.
YanT.KeuschF.HeL.. 2018. The impact of question and scale characteristics on scale direction effects. Survey Practice11:1–9.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
