Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes
Restricted accessResearch articleFirst published online February, 2020
Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes
Unlike other data collection modes, the effect of labeling rating scales on reliability and validity, as relevant aspects of measurement quality, has seldom been addressed in online surveys. In this study, verbal and numeric rating scales were compared in split-ballot online survey experiments. In the first experiment, respondents’ cognitive processes were observed by means of eye tracking, that is, determining the respondent’s fixations in different areas of the screen. In the remaining experiments, data for reliability and validity analysis were collected from a German adult sample. The results show that respondents needed more fixations and more time to endorse a category when a rating scale had numeric labels. Cross-sectional reliability was lower and some hypotheses with respect to the criterion validity could not be supported when numeric rating scales were used. In conclusion, theoretical considerations and the empirical results contradict the current broad usage of numeric scales in online surveys.
AckermanRakefetGoldsmithMorris. 2011. “Metacognitive Regulation of Text Learning: On Screen Versus on Paper.” Journal of Experimental Psychology: Applied17:18–32.
AltemeyerBob. 1981. Right-wing Authoritarianism. Winnipeg, Canada: University of Manitoba Press.
4.
AlwinDuane F.KrosnickJon A.. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods & Research20:139–81.
5.
AndrewsFrank M.1984. “Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach.” Public Opinion Quarterly48:409–42.
6.
BeierleinConstanzeAsbrockFrankKauffMathiasSchmidtPeter. 2014. “Die Kurzskala Autoritarismus (KSA-3): “Ein ökonomisches Messinstrument zur Erfassung dreier Subdimensionen autoritärer Einstellungen.” in Zusammenstellung sozialwissenschaftlicher Items und Skalen, edited byDannerD.Glöckner-RistA.Mannheim: GESIS. Retrieved September 7, 2017 (http://zis.gesis.org/skala/Beierlein-Asbrock-Kauff-Schmidt-Kurzskala-Autoritarismus-(KSA-3).
7.
BendigA. W.HughesJ. B.II. 1953. “Effect of Amount of Verbal Anchoring and Number of Rating-scale Categories upon Transmitted Information.” Journal of Experimental Psychology46:87–90.
8.
BernardinH. J.LaShellsMary B.SmithPatricia C.AlvaresKenneth M.. 1976. “Behavioral Expectation Scales: Effects of Developmental Procedures and Formats.” Journal of Applied Psychology61:75–79.
9.
BraunMichael. 2006. Funktionale Äquivalenz in interkulturell vergleichenden Umfragen. Mythos und Realität. Mannheim, Germany: ZUMA.
ChurchillGilbert A.PeterJ. P.. 1984. “Research Design Effects on The Reliability of Rating Scales: A Meta-analysis.” Journal of Marketing Research21:360–75.
12.
CohenJacob. 1988. Statistical Power Analysis for the Behavioral Sciences. New York, NY: Erlbaum.
13.
CronbachLee J.1951. “Coefficient Alpha and The Internal Structure of Tests.” Psychometrika16:297–334.
14.
DuncanLauren E.PetersonBill E.WinterDavid G.. 1997. “Authoritarianism and Gender Roles: Toward A Psychological Analysis of Hegemonic Relationships.” Personality and Social Psychology Bulletin23:41–49.
15.
EpsteinSeymour. 1990. “Cognitive Experiential Self-theory.” Pp. 165–92 in Handbook of personality: Theory and research, edited byPervinLavrence. New York, NY: Guilford Press.
16.
FinnR. H.1972. “Effects of Some Variations in Rating Scale Characteristics on the Means and Reliabilities of Ratings.” Educational and Psychological Measurement32:255–65.
17.
FisseniHermann J.1997. Lehrbuch der Psychologischen Diagnostik. Göttingen, Germany: Hogrefe.
KrosnickJon A.AlwinDuane F.. 1987. “An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement.” Public Opinion Quarterly51:201.
20.
KrosnickJon A.BerentMatthew K.. 1993. “Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format.” American Journal of Political Science37:941–64.
21.
KrosnickJon A.FabrigarLeandre R.. 1997. “Designing Rating Scales for Effective Measurement in Surveys.” Pp. 141–64 in Survey Measurement and Process Quality, edited byLybergL. E.BiemerP.CollinsM.de LeeuwE.DippoC.SchwarzN.TrewinD.. Hoboken, NJ: John Wiley & Sons, Inc.
22.
LenznerTimoKaczmirekLarsGalesicMirta. 2014. “Left Feels Right: A Usability Study on the Position of Answer Boxes in Web Surveys.” Social Science Computer Review32:743–64.
23.
LordFrederic M.NovickMelvin R.. 1968. Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
24.
McDonaldRoderick P.1999. Test Theory. A Unified Treatment. Mahwah, NJ: Erlbaum.
25.
MenoldNataljaKaczmirekLarsLenznerTimoNeusarAleš. 2014. “How Do Respondents Attend to Verbal Labels in Rating Scales?” Field Methods26:21–39.
26.
MenoldNataljaKemperChristoph. 2015. “The Impact of Frequency Rating Scale Formats on the Measurement of Latent Variables in Web Surveys—An Experimental Investigation Using A Measure of Affectivity as an Example.” Psihologija48:431–49.
27.
MenoldNataljaTauschAnja. 2016. “Measurement of Latent Variables With Different Rating Scales: Testing Reliability and Measurement Equivalence by Varying the Verbalization and Number of Categories.” Sociological Methods & Research45: 678–699.
28.
NewsteadStephen E.ArnoldJohn. 1989. “The Effect of Response Format on Ratings of Teaching.” Educational and Psychological Measurement49:33–43.
29.
ParducciAllen. 1983. “Category Ratings and the Relational Character of Judgment.” Pp. 262–82 in Modern Issues in Perception, edited byGeisslerH.-G.BulfartH. F. J. M.LeeuwenbergE. L. H.SarrisV. Berlin, Germany: VEB Deutscher Verlag der Wissenschaften.
30.
PetersDavid L.McCormickErnest J.. 1966. “Comparative Reliability of Numerically Anchored Versus Job-task Anchored Rating Scales.” Journal of Applied Psychology50:92–96.
31.
RammstedtBeatriceBeierleinConstanzeBrählerElmarEidMichaelHartigJohannesKerstingMartinLiebigStefanLukasJosefMayerAnne-KathrinMenoldNataljaSchuppJürgenWeichselgartnerErich. 2015. “Quality Standards for the Development, Application, and Evaluation of Measurement Instruments in Social Science Survey Research. Prepared and written by the Quality Standards Working Group.” RatSWD Working Papers 245. Berlin: German Data Forum (RatSWD). Retrieved September 7, 2017 (https://www.ratswd.de/dl/RatSWD_WP_245.pdf
32.
RattingerHansRoßteutscherSigridSchmitt-BeckRüdigerWeßelsBernhard. 2011. “German Longitudinal Election Study—Langfrist-Online-Tracking, T14, 23.05.-03.06.2011; Nachbefragung: 03.06.-13.06.2011.” ZA5347, Version 1.0.0. Cologne: GESIS. Retrieved September 7, 2017 (https://dbk.gesis.org/dbksearch/sdesc2.asp?db=e&no=534).
RaykovTenko. 2001. “Bias of Coefficient α for Fixed Congeneric Measures with Correlated Errors.” Applied Psychological Measurement25:69–76.
35.
RaykovTenko. 2012. “Scale Construction and Development Using Structural Equation Modeling.” Pp. 472–92 in Handbook of Structural Equation Modeling, edited byHHoyle Rick. New York, NY: The Guilford Press.
36.
RaykovTenkoMarcoulidesGeorge A.. 2011. Introduction to Psychometric Theory. New York, NY: Taylor & Francis.
37.
SarisWillem E.GallhoferIrmtraud N.. 2007. Design, Evaluation, and Analysis of Questionnaires for Survey Research. Hoboken, NJ: John Wiley & Sons, Inc.
38.
SchweizerKarl. 2011. “On the Changing Role of Cronbach’s α in the Evaluation of the Quality of a Measure.” European Journal of Psychological Assessment27:143–144.
39.
ToepoelVeraDillmanDon A.. 2011. “Words, Numbers, and Visual Heuristics in Web Surveys: Is There a Hierarchy of Importance?” Social Science Computer Review29:193–207.
40.
TourangeauRogerCouperMick P.ConradFrederick G.. 2007. “Color, Labels, and Interpretive Heuristics for Response Scales.” Public Opinion Quarterly71:91–112.
41.
TourangeauRogerRipsLance J.RasinskiKenneth. 2000. The Psychology of Survey Response. New York, NY: Cambridge University Press.
42.
VygotskyLev S.1934. “Thinking and Speech.” New York, NY: Plenum Press.
43.
WengLi-Jen. 2004. “Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability.” Educational and Psychological Measurement64:956–72.
44.
WindschitlPaul D.WellsGary L.. 1996. “Measuring Psychological Uncertainty: Verbal Versus Numeric Methods.” Journal of Experimental Psychology: Applied2:343–64.
45.
WoutersKristelMaesschalckJeroenPeetersCarel F. W.RoosenMarijke. 2014. “Methodological Issues in the Design of Online Surveys for Measuring Unethical Work Behavior: Recommendations on the Basis of a Split-Ballot Experiment.” Journal of Business Ethics120:275–89.
