Web surveys are struggling to attract and retain respondents due to high burden and competition for the users’ attention. One possible solution to this issue is the improvement of the visual design of surveys. In this article, we evaluate the impact of visual aids such as smiley faces, stars, hearts, and thumbs as alternatives to traditional radio buttons. We use an experimental design in a nonprobability online survey to investigate how the new designs compare with radio buttons and how the results might interact with device used for completion (PC vs mobile), the use of labels, the type of response scale (bipolar vs unipolar), and the number of response categories (5 vs 7 point). While we do not find big differences in response, quality and experience, there seem to be some indication that the use of smiley faces leads to worse data quality.
AlwinD. F.KrosnickJ. A. (1991). The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods & Research, 20, 139–181. doi:10.1177/0049124191020001005
2.
BrownT. (2006). Confirmatory factor analysis for applied research (1st ed.). New York, NY: Guilford Press.
3.
ChristianL. M.DillmanD. A. (2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68, 57–80. doi:10.1093/poq/nfh004
4.
CouperM. P. (2008). Designing effective web surveys. Cambridge, UK: Cambridge University Press.
5.
CouperM. P.TourangeauR.ConradF. G.CrawfordS. D. (2004). What they see is what we get response options for web surveys. Social Science Computer Review, 22, 111–127.
6.
CouperM. P.TourangeauR.ConradF. G.SingerE. (2006). Evaluating the effectiveness of visual analog scales: A web experiment. Social Science Computer Review, 24, 227–245. doi:10.1177/0894439305281503
7.
CouperM. P.TourangeauR.KenyonK. (2004). Picture this! Exploring visual effects in web surveys. Public Opinion Quarterly, 68, 255–266. doi:10.1093/poq/nfh013
8.
CouperM. P.TraugottM. W.LamiasM. J. (2001). Web survey design and administration. Public Opinion Quarterly, 65, 230–253.
9.
CrawfordS. D.CouperM. P.LamiasM. J. (2001). Web surveys perceptions of burden. Social Science Computer Review, 19, 146–162. doi:10.1177/089443930101900202
10.
DavidovE.MeulemanB.CieciuchJ.SchmidtP.BillietJ. (2014). Measurement equivalence in cross-national research. Annual Review of Sociology, 40, 55–75. doi:10.1146/annurev-soc-071913–043137
11.
FunkeF. (2016). A web experiment showing negative effects of slider scales compared to visual analogue scales and radio button scales. Social Science Computer Review, 34, 244–254. doi:10.1177/0894439315575477
12.
FunkeF.ReipsU.-D.ThomasR. K. (2011). Sliders for the smart: Type of rating scale on the web interacts with educational level. Social Science Computer Review, 29, 221–231.
13.
HealeyB. (2007). Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in web surveys. Social Science Computer Review, 25, 111–128. doi:10.1177/0894439306293888
14.
HeerweghD.LoosveldtG. (2002). An evaluation of the effect of response formats on data quality in web surveys. Social Science Computer Review, 20, 471–484. doi:10.1177/089443902237323
15.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118
16.
KeuschF.ZhangC. (2017). A review of issues in gamified surveys. Social Science Computer Review, 35(2), 147–166.
17.
KrosnickJ.PresserS. (2010). Questions and questionnaire design. In MarsdenP. V.WrightJ. D. (Eds.), Handbook of survey research (pp. 263–313, 2nd ed.). Bingley, UK: Emerald.
18.
LavieT.TractinskyN. (2004). Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies, 60, 269–298. doi:10.1016/j.ijhcs.2003.09.002
19.
LindgaardG.FernandesG.DudekC.BrownJ. (2006). Attention web designers: You have 50 milliseconds to make a good first impression!Behaviour & Information Technology, 25, 115–126. doi:10.1080/01449290500330448
20.
LiuM. (2017a). Labelling and direction of slider questions: Results from web survey experiments. International Journal of Market Research, 59, 601–624.
21.
LiuM. (2017b). Web survey experiments on matrix questions. Computers in Human Behavior, 67, 61–72. Doi:10.1016/j.chb.2016.10.017
22.
LiuM.CernatA. (2016). Item-by-item versus matrix questions: A web survey experiment. Social Science Computer Review, 31, 322–345. doi:10.1177/0894439316674459
23.
LiuM.WronskiL. (2017). Examining completion rates in web surveys via over 25,000 real-world surveys. Social Science Computer Review, 36, 116–124. doi:10.1177/0894439317695581
24.
MavletovaA.CouperM. P. (2014). Mobile web survey design: Scrolling versus paging, SMS versus E-mail Invitations. Journal of Survey Statistics and Methodology, 2, 498–518.
25.
NormanD. A. (2008). Introduction to this special section on beauty, goodness, and usability. Human–Computer Interaction, 19, 311–318. doi:10.1207/s15327051hci1904_1
26.
PeytchevA.CouperM. P.McCabeS. E.CrawfordS. D. (2006). Web survey design paging versus scrolling. Public Opinion Quarterly, 70, 596–607.
27.
StangeM.BarryA.SmythJ.OlsonK. (2016). Effects of smiley face scales on visual processing of satisfaction questions in web surveys. Social Science Computer Review, 0894439316674166.
28.
SteenkampJ. E. M.BaumgartnerH. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–107. doi:10.1086/209528
29.
TourangeauR.ConradF. G.CouperM. P.YeC. (2014). The effects of providing examples in survey questions. Public Opinion Quarterly, 78, 100–125. doi:10.1093/poq/nft083
30.
TourangeauR.CouperM. P.ConradF. (2004). Spacing, position, and order interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68, 368–393. doi:10.1093/poq/nfh035
31.
WestS.TaylorA.WuW. (2012). Model fit and model selection in structural equation modelling. In HoyleR. (Ed.), Handbook of structural equation modeling (pp. 209–231, 1st ed.). New York, NY: Guilford Press.
