While grids or matrix questions are a widely used format in PC web surveys, there is no agreement on the format in mobile web surveys. We conducted a two-wave experiment in an opt in panel in Russia, varying the question format (grid format and item-by-item format) and device respondents used for survey completion (smartphone and PC). The 1,678 respondents completed the survey in the assigned conditions in the first wave and 1,079 in the second wave. Overall, we found somewhat higher measurement error in the grid format in both mobile and PC web conditions. We found almost no significant effect of the question format on test–retest correlations between the latent scores in two waves and no differences in breakoff rates between the question formats. The multigroup comparison showed some measurement equivalence between the question formats. However, the difference varied depending on the length of a scale with a longer scale producing some differences in the measurement equivalence between the conditions. The levels of straightlining were higher in the grid than in the item-by-item format. In addition, concurrent validity was lower in the grid format in both PC and mobile web conditions. Finally, subjective indicators of respondent burden showed that the grid format increased reported technical difficulties and decreased subjective evaluation of the survey.
AntounC.CouperM. P.ConradF. G. (2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81, 280–306.
2.
BarlasF. M.ThomasR. K.ButtermoreN. (2017, 3). Mobile-friendly grid questions: The accordion grid as an alternative to the traditional grid. Paper presented at the General Online Research conference, Berlin, Germany.
3.
CallegaroM.Lozar ManfredaK.VehoharV. (2015). Web survey methodology. Los Angeles, CA: Sage.
CernatA.CouperM. P.OfstedalM. B. (2016). Estimation of mode effects in the health and retirement study using measurement models. Journal of Survey Statistics and Methodology, 4, 501–524.
6.
ChenF. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14, 464–504.
7.
CouperM. P.AntounC.MavletovaA. (2017). Mobile web surveys: A total survey error perspective. In BiemerP.EckmanS.EdwardsB.de LeeuwE.KreuterF.LybergL.TuckerC.WestB. (Eds.), Total survey error in practice (pp. 133–154). New York, NY: Wiley.
8.
CouperM. P.PetersonG. J. (2017). Why do web surveys take longer on smartphones?Social Science Computer Review, 35, 357–377.
9.
CouperM. P.TourangeauR.ConradF. G.ZhangC. (2013). The design of grids in web surveys. Social Science Computer Review, 31, 322–345.
10.
CouperM. P.TraugottM.LamiasM. (2001). Web survey design and administration. Public Opinion Quarterly, 65, 230–253.
11.
De BruijneM.MarcelD.van SoestA.WijnantA. (2015). Adapting grid questions for mobile devices (Working paper). Tilburg, the Netherlands: CentERdata (Tilburg University).
12.
De LeeuwE. D. (2017, 717). Mixed mode: Past, present, and the future. Keynote presentation at the European Survey Research Association conference, Lisbon, Portugal.
13.
De LeeuwE. D.HoxJ. J.KlauschT.RobertsA.de JonghA. (2012, 5). Design of web questionnaires: Matrix questions or single question formats. Paper presented at the annual conference of the American Association for Public Opinion Research, Orlando, FL.
14.
DillmanD. A.SmythJ. D.ChristianL. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). New York, NY: Wiley.
15.
FinneyS. J.DiStefanoC. (2006). Non normal and categorical data in structural equation modeling. In HancockG. R.MuellerR. O. (Eds.), Structural equation modeling. A second course (pp. 269–314). Greenwich, CT: Information Age Publishing.
16.
GräfL. (2002). Assessing Internet questionnaires: The online pretest lab. In BattinicB.ReipsU.-D.BosnjakM.WernerA. (Eds.), Online social sciences (pp. 69–79). Seattle, WA: Hogrefe & Huber.
17.
HansonT. (2017, 3). Adapting questionnaires for smartphones: An experiment on grid format questions. Paper presented at the General Online Research conference, Berlin, Germany.
18.
HornJ. L.McArdleJ. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18, 117–144.
19.
HoxJ. J.de LeeuwE. D.ZijlmansE. A. O. (2015). Measurement equivalence in mixed mode surveys. Frontiers in Psychology, 6. doi: 10.3389/fpsyg.2015.00087.
20.
KlauschT.de LeeuwE. D.HoxJ. J.RobertsA.de JonghA. (2012, 3). Matrix vs. single question formats in web surveys: Results from a large scale experiment. Paper presented at the General Online Research conference, Mannheim, Germany.
LatteryK.Park BartoloneG.SaundersT. (2013). Optimizing surveys for smartphones: Maximizing response rates while minimizing bias. Presented at CASRO Online Research Conference, March 6–8, San Francisco, CA.
23.
LiuM. (2017). Web survey experiments on matrix questions. Computers in Human Behavior, 67, 61–72.
24.
LiuM.CernatA. (2018). Item-by-item versus matrix questions: A web survey experiment. Social Science Computer Review, 36, 690–706.
25.
MavletovaA.CouperM. P. (2013). Sensitive topics in PC web and mobile web surveys. Survey Research Methods, 7, 191–205.
26.
MavletovaA.CouperM. P. (2015). A meta-analysis of breakoff rates in mobile web surveys. In ToninelliD.PinterR.de PedrazaP. (Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies (pp. 81–98). London, England: Ubiquity.
27.
McClainC.CrawfordS. D. (2013). Grid formats, data quality, and mobile device use: A questionnaire design approach. Presented at the AAPOR Annual Conference, May 16–19, Boston, MA.
PetersonG.GriffinJ.LaFranceJ.LiJ. (2017). Smartphone participation in web surveys: Choosing between the potential for coverage, nonresponse, and measurement error. In BiemerP.EckmanS.EdwardsB.de LeeuwE.KreuterF.LybergL.TuckerC.WestB. (Eds.), Total survey error in practice (pp. 203–233). New York, NY: Wiley.
30.
PeytchevA. (2005). Web survey design: Effect of layout on measurement error. In van DijkumC.BlasiusJ.DurandC. (Eds.), Recent developments and applications in social research methodology. Proceedings of the RC33 Sixth International Conference on Social Science Methodology, Amsterdam. Retrieved fromhttp://konference.fdvinfo.net/rc33/2004/Data/PDF/stream_11-02.pdf
31.
PhillipsR.LuksS. (2016, 5). Measuring survey behavior of smartphone users. Paper presented at the annual conference of the American Association for Public Opinion Research, Austin, TX.
32.
PoynterR. (2001). A guide to best practice in online quantitative research. In WestlakeA.SykesW.MannersT.RiggM. (Eds.), The challenge of the Internet; Proceedings of the ASC International Conference on Survey Research Methods (pp. 3–19). London, England: Association for Survey Computing.
33.
RevillaM. (2017). Analyzing survey characteristics, participation, and evaluation across 186 surveys in an online opt-in panel in Spain. Methods, Data, and Analysis, 11, 135–162.
34.
RevillaM.CouperM. P. (2017). Comparing grids with vertical and horizontal item-by-item formats for PCs and smartphones. Social Science Computer Review. doi:10.1177/0894439317715626
35.
RevillaM.ToninelliD.OchoaC. (2017). An experiment comparing grids and item-by-item formats in web surveys completed through PCs and smartphones. Telematics and Informatics, 34, 30–42.
36.
SchwartzS. H. (1992). Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. Advances in Experimental Social Psychology, 25, 1–65.
37.
SteenkampJ. E. M.BaumgartnerH. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–107.
38.
SternM.SterrettD.BilgenI. (2016). The effects of grids on web surveys completed with mobile devices. Social Currents, 3, 217–233.
39.
ThomasR. K.BarlasF. M.GrahamP.SubiasT. (2015). What they can’t see can hurt you: Improving grids for mobile devices. Paper presented at the Annual Conference of the American Association for Public Opinion Research, Hollywood, FL.
40.
ToepoelV.DasM.Van SoestA. (2009). Design of web questionnaires: The effects of the number of items per screen. Field Methods, 21, 200–213.
41.
TourangeauR.CouperM. P.ConradF. G. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68, 368–393.
42.
UslanerE. M. (2002). The moral foundations of trust. Cambridge, MA: Cambridge University Press.
43.
VandenbergR.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.
44.
WojtowiczT. (2001). Designing lengthy Internet questionnaires: Suggestions and solutions. In WestlakeA.SykesW.MannersT.RiggM. (Eds.), The Challenge of the Internet; Proceedings of the ASC International Conference on Survey Research Methods (pp. 25–32). London, England: Association for Survey Computing.
45.
YamagishiT.YamagishiM. (1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18, 129–166.
YanT.TourangeauR. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
