Web survey respondents are frequently distracted during survey completion, which potentially affects the quality of data they provide. This article reports on results from a laboratory experiment examining how distractions during web survey completion influence data quality. Participants were randomly assigned to experimental groups using a 2 (device type) × 3 (form of distraction) between-subject factorial design. They were asked to complete a web questionnaire on either a PC or a tablet and were allocated to one of the three distraction conditions: (1) the presence of other people in the room who have a loud conversation, (2) the presence of music, or (3) no distraction. The study examines the effect of distraction on various measures of data quality and attentiveness. While participants felt significantly more distracted in the presence of other people or music, the study found no significant effect of distraction for any of the data quality and attentiveness measures. The findings are encouraging for survey practitioners: Even if web respondents listen to music or are in noisy environments, these forms of distraction generally do not seem to affect the quality of responses they provide.
AizpuruaE.HeidenE. O.ParkK. H.WittrockJ.LoschM. E. (2018). Investigating respondent multitasking and distraction using self-reports and interviewers’ observations in a dual-frame telephone survey. Survey Insights: Methods from the Field, 1–10.
2.
AnsolabehereS.SchaffnerB. F. (2015). Distractions: The incidence and consequences of interruptions for survey respondents. Journal of Survey Statistics and Methodology, 3, 216–239.
3.
AntounC. (2015). Mobile web surveys: A first look at measurement, nonresponse, and coverage errors (Doctoral dissertation). Ann Arbor: University of Michigan.
4.
AntounC.CouperM. P.ConradF. G. (2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81, 280–306.
5.
BerinskyA. J.MargolisM. F.SancesM. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58, 739–753.
6.
BlomA. G.BosnjakM.CornilleauA.CousteauxA.-S.DasM.DouhouS.KriegerU. (2016). A comparison of four probability-based online and mixed-mode panels in Europe. Social Science Computer Review, 34, 8–25.
7.
CacioppoJ. T.PettyR. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.
8.
CacioppoJ. T.PettyR. E.KaoC. F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306–307.
9.
CallegaroM.BakerR. P.BethlehemJ.GöritzA. S.KrosnickJ. A.LavrakasP. J. (2014). Online panel research: A data quality perspective. Chichester, England: Wiley.
10.
CallegaroM.Lozar ManfredaK.VehovarV. (2015). Web survey methodology. London, England: Sage.
11.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
12.
CouperM. P. (2008). Designing effective web surveys. New York, NY: Cambridge University Press.
13.
CouperM. P.PetersonG. J. (2017). Why do web surveys take longer on smartphones?Social Science Computer Review, 35, 357–377.
14.
DunawayJ.SearlesK.SuiM.PaulN. (2018). News attention in a mobile era. Journal of Computer-Mediated Communication, 23, 107–124.
15.
FaulF.ErdfelderE.LangA. G.BuchnerA. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
16.
FrederickS. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.
17.
FurnhamA.BradleyA. (1997). Music while you work: The differential distraction of background music on the cognitive test performance of introverts and extraverts. Applied Cognitive Psychology, 11, 445–455.
18.
GalesicM.TourangeauR.CouperM. P.ConradF. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892–913.
19.
GamesP. A.HowellJ. F. (1976). Pairwise multiple comparison procedures with unequal N’s and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1, 113–125.
20.
GoslingS. D.RentfrowP. J.SwannW. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Psychology, 37, 504–528.
21.
HöhneJ. K.SchlosserS. (2018). Investigating the adequacy of response time outlier definitions in computer-based web surveys using paradata SurveyFocus. Social Science Computer Review, 36, 369–378.
22.
JäckleA.LynnP.BurtonJ. (2015). Going online with a face-to-face household panel: Effects of a mixed mode design on item and unit non-response. Survey Research Methods, 9, 57–70.
23.
JeritJ.BarabasJ.CliffordS. (2013). Comparing contemporaneous laboratory and field experiments on media effects. Public Opinion Quarterly, 77, 256–282.
24.
KahnemanD. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.
25.
KaminskaO.McCutcheonA. L.BillietJ. (2010). Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly, 74, 956–984.
26.
KelloggR. (2012). Fundamentals of cognitive psychology. Los Angeles, CA: Sage.
27.
KennedyC. (2010). Nonresponse and measurement error in mobile phone surveys (Doctoral dissertation). Ann Arbor: University of Michigan.
28.
KennedyC.EverettS. E. (2011). Use of cognitive shortcuts in landline and cell phone surveys. Public Opinion Quarterly, 75, 336–348.
29.
KimK. J.SundarS. S. (2016). Mobile persuasion: Can screen size and presentation mode make a difference to trust?Human Communication Research, 42, 45–70.
30.
KrosnickJ. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
31.
KrosnickJ. A.AlwinD. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.
32.
KrosnickJ. A.NarayanS. S.SmithW. R. (1996). Satisficing in surveys: Initial evidence. In BravermanM. T.SlaterJ. K. (Eds.), Advances in survey research (pp. 29–44). San Francisco, CA: Jossey-Bass.
33.
LavrakasP. J.TompsonT. N.BenfordR. (2010). Investigating data quality in cell phone surveying. Paper presented at the AAPOR Conference, Chicago, IL.
34.
LenznerT.KaczmirekL.LenznerA. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020.
35.
LugtigP.ToepoelV. (2016). The use of PCs, smartphones, and tablets in a probability-based panel survey: Effects on survey measurement error. Social Science Computer Review, 34, 78–94.
36.
LynnP.KaminskaO. (2012). The impact of mobile phones on survey measurement error. Public Opinion Quarterly, 77, 586–605.
37.
MavletovaA. (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31, 725–743.
38.
MavletovaA.CouperM. P. (2013). Sensitive topics in PC web and mobile web surveys: Is there a difference?Survey Research Methods, 7, 191–205.
39.
MillarM. M.DillmanD. A. (2012). Encouraging survey response via smartphones: Effects on respondents’ use of mobile devices and survey response rates. Survey Practice, 5, 1–6.
40.
MolyneuxL. (2017). Mobile news consumption. Digital Journalism, 6, 634–650.
41.
OppenheimerD. M.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
42.
PeytchevA.HillC. A. (2010). Experiments in mobile web survey design. Similarities to other modes and unique considerations. Social Science Computer Review, 28, 319–335.
43.
RatcliffR. (1993). Methods for dealing with response time outliers. Psychological Bulletin, 114, 510–532.
44.
RevillaM.OchoaC. (2015). Open narrative questions in PC and smartphones: Is the device playing a role?Quality & Quantity, 50, 2495–2513.
45.
RevillaM.ToninelliD.OchoaC.LoeweG. (2016). Do online access panels need to allow and adapt surveys to mobile devices?Internet Research, 26, 1209–1227.
46.
SchepersJ. M. (2007). The construction and evaluation of an attention questionnaire. SA Journal of Industrial Psychology, 33, 16–24.
47.
SchoberM. F.ConradF. G.AntounC.EhlenP.FailS.HuppA. L.…ZhangC. (2015). Precision and disclosure in text and voice interviews on smartphones. PLoS One, 10, 1–20.
48.
SendelbahA.VehovarV.SlavecA.PetrovcicA. (2016). Investigating respondent multitasking in web surveys using paradata. Computers in Human Behavior, 55, 777–787.
49.
ToepoelV.LugtigP. (2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of internet users. Social Science Computer Review, 32, 544–560.
50.
TourangeauR. (1984). Cognitive sciences and survey methods. In JabineT. B.StrafM. L.TanurJ. M.TourangeauR. (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington DC: National Academy Press.
51.
TourangeauR.RipsL. J.RasinskiK. A. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
52.
TraftonJ. G.MonkC. A. (2007). Task interruptions. Reviews of Human Factors and Ergonomics, 3, 111–126.
53.
Van VaerenberghY.ThomasT. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195–217.
54.
ZwarunL.HallA. (2014). What’s going on? Age, distraction, and multitasking during online survey taking. Computers in Human Behavior, 41, 236–244.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
