Much research has been done comparing grids and item-by-item formats. However, the results are mixed, and more research is needed especially when a significant proportion of respondents answer using smartphones. In this study, we implemented an experiment with seven groups (n = 1,476), varying the device used (PC or smartphone), the presentation of the questions (grids, item-by-item vertical, item-by-item horizontal), and, in the case of smartphones only, the visibility of the “next” button (always visible or only visible at the end of the page, after scrolling down). The survey was conducted by the Netquest online fieldwork company in Spain in 2016. We examined several outcomes for three sets of questions, which are related to respondent behavior (completion time, lost focus, answer changes, and screen orientation) and data quality (item missing data, nonsubstantive responses, instructional manipulation check failure, and nondifferentiation). The most striking difference found is for the placement of the next button in the smartphone item-by-item conditions: When the button is always visible, item missing data are substantially higher.
AndreadisI. (2015). Comparison of response times between desktop and smartphone users. In ToninelliD.PinterR.de PedrazaP. (Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies (pp. 1–10). London, England: Ubiquity Press. Retrieved fromhttp://dx.doi.org/10.5334/bar.e
2.
BakerR. P.CouperM. P.ConradF. G.TourangeauR. (2004). Don’t know and no opinion responses in web surveys. Presented at the RC33 International Conference on Social Science Methodology, Amsterdam, the Netherlands, August.
BellD. S.MangioneC. M.KahnC. E. (2001). Randomized testing of alternative survey formats using anonymous volunteers on the World Wide Web. Journal of the American Medical Informatics Association, 8, 616–620.
CallegaroM. (2010). Do you know which device your respondent has used to take your online survey? Using paradata to collect information on device type. Survey Practice, 3, 6.
CouperM. P. (2016). Grids versus item-by-item designs on smartphones. Presented at the GOR conference, Dresden, Germany.
9.
CouperM. P.AntounC.MavletovaA. (2017). Mobile Web surveys: A total survey error perspective. In BiemerP.EckmanS.EdwardsB.de LeeuwE.KreuterF.LybergL.TuckerC.WestB. (Eds.), Total survey error in practice (pp. 133–154). New York, NY: Wiley.
10.
CouperM. P.PetersonG. (2016). Why do web surveys take longer on smartphones?Social Science Computer Review. doi:10.1177/0894439316629932
11.
CouperM. P.TraugottM.LamiasM. (2001). Web survey design and administration. Public Opinion Quarterly, 65, 230–253.
12.
De BruijneM.WijnantA. (2013). Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review, 31, 482–504.
13.
De BruijneM.WijnantA. (2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78, 951–962.
14.
DillmanD. A. (2007), Mail and internet surveys: The tailored design method (2nd ed.). New York, NY: Wiley.
15.
HansonT. (2017). Adapting questionnaires for smartphones: An experiment on grid format questions. Presented at the GOR conference, Berlin, Germany.
16.
HöhneJ. K.LenznerT. (2015). Investigating response order effects in web surveys using eye tracking. Psihologija, 48, 361–377.
17.
HöhneJ. K.SchlosserS. (2017). Investigating the adequacy of response time outlier definitions in computer-based web surveys using paradata ‘SurveyFocus.’Social Science Computer Review. First Published Online June 1, 2017. DOI: https://doi.org/10.1177/0894439317710450
18.
HöhneJ. K.SchlosserS.KrebsD. (2017). Investigating cognitive effort and response quality of question formats in web surveys. Field Methods, 29, 1–18. DOI: 10.1177/1525822X17710640
19.
KeuschF.YanT.HanS.HeL. (2014). Impact of response scale direction on survey responses in web and mobile web surveys. Presented at the annual conference of the AAPOR, Anaheim, CA.
20.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
MartinssonJ. (2015). Response option order effects; Scale lengths and horizontal or vertical layout. Presented at the annual meeting of the AAPOR, Hollywood, FL, May.
24.
McClainC.CrawfordS. D. (2013). Grid formats, data quality, and mobile device use: Toward a questionnaire design approach. Presented at the annual conference of the AAPOR, Boston, MA.
OppenheimerD. M.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi:10.1016/j.jesp.2009.03.009
27.
PetersonG.GriffinJ.LaFranceJ.LiJ. (2017). Smartphone participation in web surveys: Choosing between the potential for coverage, nonresponse, and measurement error. In BiemerP.EckmanS.EdwardsB.de LeeuwE.KreuterF.LybergL.TuckerC.WestB. (Eds.), Total survey error in practice (pp. 203–233). New York, NY: Wiley.
28.
PetersonG.MechlingJ.LaFranceJ.SwinehartJ.HamG. (2013). Solving the unintentional mobile challenge. Presented at the CASRO Online Research Conference, San Francisco, CA.
29.
PeytchevA. (2004). Web survey design: Effect of layout on measurement error. Proceedings of the RC33 Sixth International Conference on Social Science Methodology, Amsterdam, the Netherlands. Retrieved fromhttp://konference.fdvinfo.net/rc33/2004/Data/PDF/stream_11-02.pdf
30.
PoynterR. (2001). A guide to best practice in online quantitative research. In WestlakeA.SykesW.MannersT.RiggM. (Eds.), The challenge of the internet; Proceedings of the ASC International Conference on survey research methods (pp. 3–19). London, England: Association for Survey Computing.
31.
RevillaM. (in press). Analyzing the survey characteristics, participation, and evaluation across 186 surveys in an Online opt-in panel in Spain. Methods, data, analyses.
32.
RevillaM.ToninelliD.OchoaC. (2017). An experiment comparing grids and item-by-item formats in web surveys completed through PC and smartphones. Telematics and Informatics, 34, 30–42.
33.
RoßmannJ. (2015). RSPEEDINDEX: Stata module to compute a response speed index and perform outlier identification (Version 0.4) [Computer Software]. Chestnut Hill, MA: Boston College. Retrieved fromhttp://econpapers.repec.org/software/bocbocode/s458007.htm
34.
SanchezC. A.BranaghanR. J. (2011). Turning to learn: Screen orientation and reasoning with small devices. Computers in Human Behavior, 27, 793–797. Retrieved fromhttp://dx.doi.org/10.1016/j.chb.2010.11.004
35.
SendelbahA.VehovarV.SlavecA.PetrovčičA. (2016). Investigating respondent multitasking in web surveys using paradata. Computers in Human Behavior, 55, 777–787.
36.
SternM. J. (2008). The use of client-side paradata in analyzing the effects of visual layout on changing responses in web surveys. Field Methods, 20, 377–398. doi:10.1177/1525822X08320421
37.
ThomasR. K.BarlasF. M.GrahamP.SubiasT. (2015). What they can’t see can hurt you: Improving grids for mobile devices. Presented at the CASRO Digital Research Conference, February 11–12, Nashville, TN. Retrieved fromhttp://www.casro.org/?2015DRCFiles
38.
ToepoelV.DasM.van SoestA. (2009). Design of web questionnaires: The effect of layout in rating scales. Journal of Official Statistics, 25, 509–528.
39.
TourangeauR.ConradF. G.CouperM. P. (2013). The science of web surveys. New York, NY: Oxford University Press.
40.
WojtowiczT. (2001). Designing lengthy internet questionnaires: Suggestions and solutions. In WestlakeA.SykesW.MannersT.RiggM. (Eds.), The challenge of the internet; Proceedings of the ASC International Conference on survey research methods (pp. 25–32). London, England: Association for Survey Computing.
41.
YanT.TourangeauR. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
