The Effects of Response Burden – Collecting Life History Data in a Self-Administered Mixed-Device Survey

Abstract

Collecting life history data is highly demanding and therefore prone to error since respondents must retrieve and provide extensive complex information. Research has shown that response burden is an important factor influencing data quality. We examine whether increases in different measures of response burden in a (mixed-device) online survey lead to adverse effects on the data quality and whether these effects vary by the type of device used (mobile versus non-mobile).

We conducted an experimental study in an online mixed-device survey, for which we developed a questionnaire on the educational and occupational trajectories of secondary-school graduates, undergraduates, and university graduates. To address our research question, we randomly assigned different levels of response burden to the participants and compared different measures on the data quality and response.

We found mixed evidence for unfavourable effects of response burden on the examined outcomes. While some of our results were expected, they were not consistent across all subgroups. Most interestingly, the effects of response burden on outcomes seemed to differ based on the device used. Hence, we conclude that further research is needed to optimise the collection of complex data from different groups of participants.

Keywords

Respondent burden data quality response behavior experiment online survey

References

American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Available at: https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf (accessed December 2020).

Antoun, C., and A. Cernat. 2020. “Factors Affecting Completion Times: A Comparative Analysis of Smartphone and PC Web Surveys.” Social Science Computer Review 38 (4): 477–489. DOI: https://doi.org/10.1177/0894439318823703.

Antoun, C., M.P. Couper, and F.G. Conrad. 2017. “Effects of Mobile Versus PC Web on Survey Response Quality.” Public Opinion Quarterly 81 (S1): 280–306. DOI: https://doi.org/10.1093/poq/nfw088.

Bradburn, N.M. 1979. “Respondent Burden.” In Health Survey Research Methods: Second Biennial Conference, Williamsburg, edited by L. Reeder. Washington, D.C. U.S. Government Printing Office.

Burke, W.J. 2009. “Fitting and Interpreting Cragg’s Tobit Alternative Using Stata.” Stata Journal 9 (4): 584–592. Available at: https://www.stata-journal.com/article.html?article=st0179.

Burton, S., and E. Blair. 1991. “Task Conditions, Response Formulation Processes, and Response Accuracy for Behavioral Frequency Questions in Surveys.” Public Opinion Quarterly 55 (1): 50. DOI: https://doi.org/10.1086/269241.

Busemeyer, J.R., and J.T. Townsend. 1993. “Decision Field Theory: A Dynamic-Cognitive Approach to Decision Making in an Uncertain Environment.” Psychological review 100 (3): 432–459. DOI: https://doi.org/10.1037/0033-295X.100.3.432.

Buskirk, T.D., and C.H. Andrus. 2014. “Making Mobile Browser Surveys Smarter.” Field Methods 26 (4): 322–342. DOI: https://doi.org/10.1177/1525822X14526146.

Clement, S.L., M. Kappelgaard Severin-Nielsen, and D. Shamsiri-Petersen. 2020. “Device Effects on Survey Response Quality. A Comparison of Smartphone, Tablet and PC Responses on a Cross Sectional Probability Sample.” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2020-00020.

10.

Couper, M.P., and F. Kreuter. 2013. “Using Paradata to Explore Item Level Response Times in Surveys.” Journal of the Royal Statistical Society: 176 (1): 271–286. DOI: https://doi.org/10.1111/j.1467-985X.2012.01041.x.

11.

Couper, M.P., and G.J. Peterson. 2017. “Why Do Web Surveys Take Longer on Smartphones?” Social Science Computer Review 35 (3): 357–377. DOI: https://doi.org/10.1177/0894439316629932.

12.

Cragg, J.G. 1971. “Some Statistical Models for Limited Dependent Variables with Application to the Demand for Durable Goods.” Econometrica 39 (5): 829. DOI: https://doi.org/10.2307/1909582.

13.

Crawford, S,D., M.P. Couper, and M.J. Lamias. 2001. “Web Surveys.” Social Science Computer Review 19 (2): 146–162. DOI: https://doi.org/10.1177/089443930101900202.

14.

Daikeler, J., R.L. Bach, H. Silber, and S. Eckman. 2020. “Motivated Misreporting in Smartphone Surveys.” Social Science Computer Review. DOI: https://doi.org/10.1177/0894439319900936.

15.

De Bruijne, M, and M. Oudejans. 2015. “Online Surveys and the Burden of Mobile Responding.” In Survey Measurements: Techniques, Data Quality and Sources of Error, edited by U. Engel: 130–145. Frankfurt, New York: Campus.

16.

De Bruijne, M., and A. Wijnant. 2014. “Mobile Response in Web Panels.” Social Science Computer Review 32 (6): 728–742. DOI: https://doi.org/10.1177/0894439314525918.

17.

Drasch, K., and B. Matthes. 2013. “Improving Retrospective Life Course Data by Combining Modularized Self-Reports and Event History Calendars: Experiences from a Large Scale Survey.” Qual Quant 47 (2): 817–838. DOI: https://doi.org/10.1007/s11135-011-9568-0.

18.

Eckman, S., and F. Kreuter. 2018. “Misreporting to Looping Questions in Surveys: Recall, Motivation and Burden.” Survey Research Methods 12 (1): 59–74. DOI: https://doi.org/10.18148/srm/2018.v12i1.7168.

19.

Galesic, M. 2006. “Dropouts on the Web: Effects of Interest and Burden Experienced During an Online Survey.” Journal of Official Statistics 22 (2): 313–328. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/dropouts-onthe-web-effects-of-interest-and-burden-experienced-during-an-online-survey.pdf.

20.

Galesic, M., and M. Bosnjak. 2009. “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.” Social Science Computer Review 73 (2): 349–360. DOI: https://doi.org/10.1093/poq/nfp031.

21.

Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2011. Survey Methodology. 2nd edition. Hoboken: John Wiley & Sons. Available at: http://gbv.eblib.com/patron/FullRecord.aspx?p=819140.

22.

Guin, T.D.-Le., R. Baker, J. Mechling, and E. Ruyle. 2012. “Myths and Realities of Respondent Engagement in Online Surveys.” International Journal of Market Research 54 (5): 613–633. DOI: https://doi.org/10.2501/IJMR-54-5-613-633.

23.

Gummer, T., F. Quoß, and J. Roßmann. 2019. “Does Increasing Mobile Device Coverage Reduce Heterogeneity in Completing Web Surveys on Smartphones?” Social Science Computer Review 37 (3): 371–384. DOI: https://doi.org/10.1177/0894439318766836.

24.

Gummer, T., and J. Roßmann. 2015. “Explaining Interview Duration in Web Surveys.” Social Science Computer Review 33 (2): 217–234. DOI: https://doi.org/10.1177/0894439314533479.

25.

Hambleton, R.K., H. Swaminathan, and H.J. Rogers. 1991. Fundamentals of Item Response Theroy. California: Sage.

26.

Hedlin, D., H. Lindkvist, H. Bäckström, and J. Erikson. 2008. “An Experiment on Perceived Survey Response Burden Among Businesses.” Journal of Official Statistics 24 (2): 301 – 318. Available at: https://www.scb.se/contentassets/ca21efb41-fee47d293bbee5bf7be7fb3/an-experiment-on-perceived-survey-response-burden-among-businesses.pdf.

27.

Hoogendoorn, A.W., and D. Sikke. 1998. “Response Burden and Panel Attrition.” Journal of Official Statistics 14 (2): 189–205. Available at: https://www.scb.se/contentassets/-ca21efb41fee47d293bbee5bf7be7fb3/response-burden-and-panel-attrition.pdf.

28.

Kaper, E., and W. Saris. 1999. “Effects of Response Burden in a Continuous Survey.” Kwantitative Method 61: 5–17.

29.

Keusch, F., and T. Yan. 2017. “Web Versus Mobile Web.” Social Science Computer Review 35 (6): 751–769. DOI: https://doi.org/10.1177/0894439316675566.

30.

Krebs, D., and J.K. Höhne. 2020. “Exploring Scale Direction Effects and Response Behavior Across Pc and Smartphone Surveys.” Journal of Survey Statistics and Methodology 9: 261. DOI: https://doi.org/10.1093/jssam/smz058.

31.

Krosnick, J.A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5: 213–236. DOI: https://doi.org/10.1002/acp.2350050305.

32.

Lang, S., and J. Carstensen. 2022. “A life history calendar in a CAWI? Evidence from higher education research.” In Survey Methoden in der Hochschulforschung, edited by G. Brandt and S. de Vogel: 121–152. Wiesbaden: Springer VS. DOI: https://doi.org/10.1007/978-3-658-36921-7_6.

33.

Lee, H., S. Kim, M.P. Couper, and Y. Woo. 2019. “Experimental Comparison of PC Web, Smartphone Web, and Telephone Surveys in the New Technology Era.” Social Science Computer Review 37 (2): 234–247. DOI: https://doi.org/10.1177/0894439318756867.

34.

Lenzner, T. 2012. “Effects of Survey Question Comprehensibility on Response Quality.” Field Methods 24 (4): 409–428. DOI: https://doi.org/10.1177/1525822X12448166.

35.

Lugtig, P., and V. Toepoel. 2016. “The Use of PCs, Smartphones, and Tablets in a Probability-Based Panel Survey.” Social Science Computer Review 34 (1): 78–94. DOI: https://doi.org/10.1177/0894439315574248.

36.

Marcus, B., M. Bosnjak, S. Lindner, S. Pilischenko, and A. Schütz. 2007. “Compensating for Low Topic Interest and Long Surveys.” Social Science Computer Review 25 (3): 372–383. DOI: https://doi.org/10.1177/0894439307297606.

37.

Mavletova, A. 2013. “Data Quality in PC and Mobile Web Surveys.” Social Science Computer Review 31 (6): 725–743. DOI: https://doi.org/10.1177/0894439313485201.

38.

Morris, T.P., I.R. White, and P. Royston. 2014. “Tuning Multiple Imputation by Predictive Mean Matching and Local Residual Draws.” BMC medical research methodology 14: 75. DOI: https://doi.org/10.1186/1471-2288-14-75.

39.

Papke, L.E., and J.M. Wooldridge. 1996. “Econometric Methods for Fractional Response Variables with an Application to 401(K) Plan Participation Rates.” Journal of Applied Econometrics 11 (6): 619 – 632. DOI: https://doi.org/10.1002/(SICI)1099-1255(199611)11:6%3C619::AID-JAE418%3E3.0.CO;2-1.

40.

Peytchev, A., and E. Peytcheva. 2017. “Reduction of Measurement Error Due to Survey Length: Evaluation of the Split Questionnaire Design Approach.” Survey Research Methods 11 (4): 361–368. DOI: https://doi.org/10.18148/srm/2017.v11i4.7145.

41.

Reise, S.P., A.T. Ainsworth, and M.G. Haviland. 2005. “Item Response Theory: Fundamentals, Applications, and Promise in Psychological Research.” Current directions in psychological science 14 (2): 95–101. DOI: https://doi.org/10.1111/j.0963-7214.2005.00342.x.

42.

Schlosser, S., and A. Mays. 2018. “Mobile and Dirty.” Social Science Computer Review 36 (2): 212–230. DOI: https://doi.org/10.1177/0894439317698437.

43.

Sharp, L.M., and J. Frankel. 1983. “Respondent Burden: A Test of Some Common Assumptions.” Public Opinion Quarterly 43. DOI: https://doi.org/10.1086/268765.

44.

Sommer, J., B. Diedenhofen, and J. Musch. 2017. “Not to Be Considered Harmful.” Social Science Computer Review 35 (3): 378–387. DOI: https://doi.org/10.1177/0894439316633452.

45.

Toepoel, V., and P. Lugtig. 2014. “What Happens If You Offer a Mobile Option to Your Web Panel? Evidence from a Probability-Based Panel of Internet Users.” Social Science Computer Review 32 (4): 544–560. DOI: https://doi.org/10.1177/0894439313510482.

46.

Tourangeau, R. 2000. “Remembering What Happened: Memory Errors and Survey Reports.” In The Science of Self-Report: Implications for Research and Practice, edited by A.A. Stone. Mahwah, N.J: Lawrence Erlbaum.

47.

Tourangeau, R. 2003. “Cognitive Aspects of Survey Measurement and Mismeasurement.” International Journal of Public Opinion Research 15. DOI: https://doi.org/10.1093/ijpor/15.1.3.

48.

Tourangeau, R. 2018. “The Survey Response Process from a Cognitive Viewpoint.” Quality Assurance in Education 26 (2): 169–181. DOI: https://doi.org/10.1108/QAE-06-2017-0034.

49.

Van Buuren, S. 1999. “Multiple Imputation of Missing Blood Pressure Covariates in Survival Analysis.” Statistics in Medicine 18 (6): 681–694. DOI: https://doi.org/10.1002/(SICI)1097-0258(19990330)18:6,681:AID-SIM71.3.0.CO;2-R.

50.

Van Buuren, S. 2012. Flexible Imputation of Missing Data. Chapman & Hall/CRC interdisciplinary statistics series. Boca Raton, Fla. CRC Press.

51.

Vink, G., L.E. Frank, J. Pannekoek, and S. van Buuren. 2014. “Predictive Mean Matching Imputation of Semicontinuous Variables.” Statistica Neerlandica 68 (1): 61–90. DOI: https://doi.org/10.1111/stan.12023.

52.

Von Hippel, P.T. 2007. “Regression with Missing Ys: An Improved Strategy for Analyzing Multiply Imputed Data.” Sociological Methodology 37 (1): 83–117. DOI: https://doi.org/10.1111/j.1467-9531.2007.00180.x.

53.

Warriner, G.K. 1991. “Accuracy of Self-Reports to the Burdensome Question: Survey Response and Nonresponse Error Trade-Offs.” Quality & Quantity 25: 253–269. DOI: https://doi.org/10.1007/BF00167531.

54.

White, I.R., P. Royston, and A.M. Wood. 2011. “Multiple Imputation Using Chained Equations: Issues and Guidance for Practice.” Statistics in Medicine 30 (4): 377–399. DOI: https://doi.org/10.1002/sim.4067.

55.

Yan, T., S. Fricker, and S. Tsai. 2020. “Response Burden: What Is It and What Predicts It?” In Advances in Questionnaire Design, Development, Evaluation and Testing, edited by P.C. Beatty, D. Collins, L. Kaye, J.-L. Padilla, G.B. Willis, and A. Wilmot: 193–212. Hoboken, NJ, USA: Wiley.

56.

Yan, T., and R. Tourangeau. 2008. “Fast Times and Easy Questions: The Effects of Age, Experience and Question Complexity on Web Survey Response Times.” Applied Cognitive Psychology 22 (1): 51–68. DOI: https://doi.org/10.1002/acp.1331.

57.

Zanon, C., C.S. Hutz, H. Yoo, and R.K. Hambleton. 2016. “An Application of Item Response Theory to Psychological Test Development.” Psicologia: Reflexão e Crítica / Psychology: Research and Review 29 (1): 1. DOI: https://doi.org/10.1186/s41155-016-0040-x.