Do Distractions During Web Survey Completion Affect Data Quality? Findings From a Laboratory Experiment

Abstract

Web survey respondents are frequently distracted during survey completion, which potentially affects the quality of data they provide. This article reports on results from a laboratory experiment examining how distractions during web survey completion influence data quality. Participants were randomly assigned to experimental groups using a 2 (device type) × 3 (form of distraction) between-subject factorial design. They were asked to complete a web questionnaire on either a PC or a tablet and were allocated to one of the three distraction conditions: (1) the presence of other people in the room who have a loud conversation, (2) the presence of music, or (3) no distraction. The study examines the effect of distraction on various measures of data quality and attentiveness. While participants felt significantly more distracted in the presence of other people or music, the study found no significant effect of distraction for any of the data quality and attentiveness measures. The findings are encouraging for survey practitioners: Even if web respondents listen to music or are in noisy environments, these forms of distraction generally do not seem to affect the quality of responses they provide.

Keywords

measurement error response quality multitasking attention mobile web survey tablet

Get full access to this article

View all access options for this article.

References

Aizpurua

Heiden

E. O.

Park

K. H.

Wittrock

Losch

M. E.

(2018). Investigating respondent multitasking and distraction using self-reports and interviewers’ observations in a dual-frame telephone survey. Survey Insights: Methods from the Field, 1–10.

Ansolabehere

Schaffner

B. F.

(2015). Distractions: The incidence and consequences of interruptions for survey respondents. Journal of Survey Statistics and Methodology, 3, 216–239.

Antoun

(2015). Mobile web surveys: A first look at measurement, nonresponse, and coverage errors (Doctoral dissertation). Ann Arbor: University of Michigan.

Antoun

Couper

M. P.

Conrad

F. G.

(2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81, 280–306.

Berinsky

A. J.

Margolis

M. F.

Sances

M. W.

(2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58, 739–753.

Blom

A. G.

Bosnjak

Cornilleau

Cousteaux

A.-S.

Das

Douhou

Krieger

(2016). A comparison of four probability-based online and mixed-mode panels in Europe. Social Science Computer Review, 34, 8–25.

Cacioppo

J. T.

Petty

R. E.

(1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.

Cacioppo

J. T.

Petty

R. E.

Kao

C. F.

(1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48, 306–307.

Callegaro

Baker

R. P.

Bethlehem

Göritz

A. S.

Krosnick

J. A.

Lavrakas

P. J.

(2014). Online panel research: A data quality perspective. Chichester, England: Wiley.

10.

Callegaro

Lozar Manfreda

Vehovar

(2015). Web survey methodology. London, England: Sage.

11.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.

12.

Couper

M. P.

(2008). Designing effective web surveys. New York, NY: Cambridge University Press.

13.

Couper

M. P.

Peterson

G. J.

(2017). Why do web surveys take longer on smartphones? Social Science Computer Review, 35, 357–377.

14.

Dunaway

Searles

Sui

Paul

(2018). News attention in a mobile era. Journal of Computer-Mediated Communication, 23, 107–124.

15.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.

16.

Frederick

(2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.

17.

Furnham

Bradley

(1997). Music while you work: The differential distraction of background music on the cognitive test performance of introverts and extraverts. Applied Cognitive Psychology, 11, 445–455.

18.

Galesic

Tourangeau

Couper

M. P.

Conrad

(2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72, 892–913.

19.

Games

P. A.

Howell

J. F.

(1976). Pairwise multiple comparison procedures with unequal N’s and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1, 113–125.

20.

Gosling

S. D.

Rentfrow

P. J.

Swann

W. B.

(2003). A very brief measure of the Big-Five personality domains. Journal of Research in Psychology, 37, 504–528.

21.

Höhne

J. K.

Schlosser

(2018). Investigating the adequacy of response time outlier definitions in computer-based web surveys using paradata SurveyFocus. Social Science Computer Review, 36, 369–378.

22.

Jäckle

Lynn

Burton

(2015). Going online with a face-to-face household panel: Effects of a mixed mode design on item and unit non-response. Survey Research Methods, 9, 57–70.

23.

Jerit

Barabas

Clifford

(2013). Comparing contemporaneous laboratory and field experiments on media effects. Public Opinion Quarterly, 77, 256–282.

24.

Kahneman

(1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.

25.

Kaminska

McCutcheon

A. L.

Billiet

(2010). Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly, 74, 956–984.

26.

Kellogg

(2012). Fundamentals of cognitive psychology. Los Angeles, CA: Sage.

27.

Kennedy

(2010). Nonresponse and measurement error in mobile phone surveys (Doctoral dissertation). Ann Arbor: University of Michigan.

28.

Kennedy

Everett

S. E.

(2011). Use of cognitive shortcuts in landline and cell phone surveys. Public Opinion Quarterly, 75, 336–348.

29.

Kim

K. J.

Sundar

S. S.

(2016). Mobile persuasion: Can screen size and presentation mode make a difference to trust? Human Communication Research, 42, 45–70.

30.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

31.

Krosnick

J. A.

Alwin

D. F.

(1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.

32.

Krosnick

J. A.

Narayan

S. S.

Smith

W. R.

(1996). Satisficing in surveys: Initial evidence. In Braverman

M. T.

Slater

J. K.

(Eds.), Advances in survey research (pp. 29–44). San Francisco, CA: Jossey-Bass.

33.

Lavrakas

P. J.

Tompson

T. N.

Benford

(2010). Investigating data quality in cell phone surveying. Paper presented at the AAPOR Conference, Chicago, IL.

34.

Lenzner

Kaczmirek

Lenzner

(2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020.

35.

Lugtig

Toepoel

(2016). The use of PCs, smartphones, and tablets in a probability-based panel survey: Effects on survey measurement error. Social Science Computer Review, 34, 78–94.

36.

Lynn

Kaminska

(2012). The impact of mobile phones on survey measurement error. Public Opinion Quarterly, 77, 586–605.

37.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31, 725–743.

38.

Mavletova

Couper

M. P.

(2013). Sensitive topics in PC web and mobile web surveys: Is there a difference? Survey Research Methods, 7, 191–205.

39.

Millar

M. M.

Dillman

D. A.

(2012). Encouraging survey response via smartphones: Effects on respondents’ use of mobile devices and survey response rates. Survey Practice, 5, 1–6.

40.

Molyneux

(2017). Mobile news consumption. Digital Journalism, 6, 634–650.

41.

Oppenheimer

D. M.

Meyvis

Davidenko

(2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.

42.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design. Similarities to other modes and unique considerations. Social Science Computer Review, 28, 319–335.

43.

Ratcliff

(1993). Methods for dealing with response time outliers. Psychological Bulletin, 114, 510–532.

44.

Revilla

Ochoa

(2015). Open narrative questions in PC and smartphones: Is the device playing a role? Quality & Quantity, 50, 2495–2513.

45.

Revilla

Toninelli

Ochoa

Loewe

(2016). Do online access panels need to allow and adapt surveys to mobile devices? Internet Research, 26, 1209–1227.

46.

Schepers

J. M.

(2007). The construction and evaluation of an attention questionnaire. SA Journal of Industrial Psychology, 33, 16–24.

47.

Schober

M. F.

Conrad

F. G.

Antoun

Ehlen

Fail

Hupp

A. L.

… Zhang

(2015). Precision and disclosure in text and voice interviews on smartphones. PLoS One, 10, 1–20.

48.

Sendelbah

Vehovar

Slavec

Petrovcic

(2016). Investigating respondent multitasking in web surveys using paradata. Computers in Human Behavior, 55, 777–787.

49.

Toepoel

Lugtig

(2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of internet users. Social Science Computer Review, 32, 544–560.

50.

Tourangeau

(1984). Cognitive sciences and survey methods. In Jabine

T. B.

Straf

M. L.

Tanur

J. M.

Tourangeau

(Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington DC: National Academy Press.

51.

Tourangeau

Rips

L. J.

Rasinski

K. A.

(2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

52.

Trafton

J. G.

Monk

C. A.

(2007). Task interruptions. Reviews of Human Factors and Ergonomics, 3, 111–126.

53.

Van Vaerenbergh

Thomas

T. D.

(2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195–217.

54.

Zwarun

Hall

(2014). What’s going on? Age, distraction, and multitasking during online survey taking. Computers in Human Behavior, 41, 236–244.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.48 MB