Effects of Changing Modes on Item Nonresponse in Panel Surveys

Abstract

To investigate the effect of a change from the telephone to the web mode on item nonresponse in panel surveys, we use experimental data from a two-wave panel survey. The treatment group changed from the telephone to the web mode after the first wave, while the control group continued in the telephone mode. We find that when changing to the web, “don’t know” answers increase moderately from a low level, while item refusal increases substantially from a very low level. This is the case for all person groups, although socio-demographic characteristics have some additional effects on giving a don’t know or a refusal when changing mode.

Keywords

telephone web don’t know item refusal

References

AAPOR (The American Association for Public Opinion Research). 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, (9th edition). Illinois: AAPOR. Available at: http://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf (accessed November 2022).

Beatty, P., and D. Herrmann. 2002. “To Answer or not to Answer: Decision Processes Related to Survey Item Non-response.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and J. Little. (pp. 71–85). New York: Wiley.

Bowling, A. 2005. “Mode of Questionnaire Administration can have Serious Effects on Data Quality.” Journal of Public Health 27: 281–291. DOI: https://doi.org/10.1093/-pubmed/fdi031.

Browne, W.J. 2019. “MCMC Estimation in MLwiN, v3.03.” Centre for Multilevel Modelling, University of Bristol. Available at: http://www.bristol.ac.uk/cmm/media/software/mlwin/downloads/manuals/3-03/mcmc-web.pdf (accessed November 2, 2022)

Chang, L., and J.A. Krosnick. 2010. “Comparing Oral Interviewing with Self-Administered Computerized Questionnaires. An Experiment.” Public Opinion Quarterly 74(1): 154–167. DOI: https://doi.org/10.1093/poq/nfp090.

Christian, L.M., D.A. Dillman, and J.D. Smyth. 2007. “The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys.” In Advances in Telephone Survey Methodology, edited by J.M. Lepkowski, C. Tucker, M. Brick, E.D. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link, and R.L. Sangster: 250–275. Hoboken, Wiley.

De Leeuw, E.D. 2005. “To Mix or not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21(2): 233–255. Available at: https://www.scb.se/contentassets/-ca21efb41fee47d293bbee5bf7be7fb3/to-mix-or-not-to-mix-data-collection-modes-in-surveys.pdf.

De Leeuw, E.D., and J. Hox. 2018. “Internet Surveys as Part of a Mixed-Mode Design.” In Social and behavioral research and the Internet: Advances in applied methods and research strategies, edited by M. Das, P. Ester, and L. Kaczmirek. 45–76. New York: Routledge.

De Leeuw, E.D., J. Hox, and A. Boeve. 2016. “Handling do-not-know Answers: Exploring new Approaches in Online and Mixed-mode Surveys.” Social Science Computer Review 34(1): 116–132. DOI: 10.1177/0894439315573744.

10.

Fielding, A., and H. Goldstein. 2006. “Cross-classified and Multiple Membership Structures in Multilevel Models: An Introduction and Review,” University of Birmingham. Available at: http://www.bristol.ac.uk/cmm/team/hg/full-publications/2006/cross-classified-models-review-for-dfes.pdf (accessed November 2022)

11.

Fricker, S., M. Galesic, R. Tourangeau, and T. Yan. 2005. “An Experimental Comparison of Web and Telephone Surveys.” Public Opinion Quarterly 69(3): 370–392. DOI: https://doi.org/10.1093/poq/nfi027.

12.

Gooch, A., and L. Vavreck. 2019. “How Face-to-Face Interviews and Cognitive Skill Affect Item Non-response: A Randomized Experiment Assigning Mode of Interview.” Political Science Research and Methods 7(1): 143–162. DOI: https://doi.org/10.1017/psrm.2016.20.

13.

Greene, J., H. Speizer, and W. Wiitala. 2008. “Telephone and Web: Mixed-Mode Challenge.” Health services research 43(1): 230–248. DOI: 10.1111/j.1475-6773.2007.00747.x.

14.

Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2011. Survey Methodology. Hoboken: Wiley.

15.

Guzy, N., and H. Leitgöb. 2015. “Assessing Mode Effects in Online and Telephone Victimization Surveys.” International Review of Victimology 21(1): 101–131. DOI: https://doi.org/10.1177/0269758014547995.

16.

Juster, F.T., and J.P. Smith. 1997. “Improving the Quality of Economic Data: Lessons from the HRS and AHEAD.” Journal of the American Statistical Association 92(440): 1268–1278. DOI: 10.1080/01621459.1997.10473648.

17.

Holbrook, A., Y.I. Cho, and T. Johnson. 2006. “The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties.” Public Opinion Quarterly 70(4): 565–595. DOI: https://doi.org/10.1093/poq/nfl027.

18.

Keusch, F. 2020. “Gamification in Web Surveys.” In SAGE Research Methods Foundations, edited by P. Atkinson, S. Delamont, A. Cernat, J.W. Sakshaug, and R.A. Williams. 1–8. London: Sage.

19.

Kleiner, B., O. Lipps, and E. Ferrez. 2015. “Language Ability and Motivation Among Foreigners in Survey Responding.” Journal of Survey Statistics and Methodology 3(3): 339–360. DOI: https://doi.org/10.1093/jssam/smv015.

20.

Krosnick, J.A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5: 231–236. DOI: https://doi.org/10.1002/acp.2350050305.

21.

Krumpal, I. 2013. “Determinants of Social Desirability Bias in Sensitive Surveys: a Literature Review.” Quality & Quantity 47(4): 2025–2047. DOI: https://doi.org/10.1007/s11135-011-9640-9.

22.

Leckie, G., and C. Charlton. 2013. “runmlwin – A Program to Run the MLwiN Multilevel Modelling Software from Within Stata.” Journal of Statistical Software 52: 1–40. DOI: https://doi.org/10.18637/jss.v052.i11.

23.

Lipps, O., and G.-A. Monsch. 2022. “Effects of Question Characteristics on Item Non-Response in Telephone and Web Survey Modes.” Field Methods. DOI: https://doi.org/10.1177/1525822X221115838.

24.

Roberts, C., and C. Vandenplas. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2): 303–334. DOI: http://dx.doi.org/10.1515/JOS-2017-0016.

25.

Shoemaker, P.J., M. Eichholz, and E.A. Skewes. 2002. “Item Nonresponse: Distinguishing Between Don’t know and Refuse.” International Journal of Public Opinion Research 14(2): 193–201. DOI: https://doi.org/10.1093/ijpor/14.2.193.

26.

Silber, H., J. Roßmann, T. Gummer, S. Zins, and K.W. Weyandt. 2021. “The Effects of Question, Respondent and Interviewer Characteristics on Two Types of Item Nonresponse.” Journal of the Royal Statistical Society: Series A (Statistics in Society). DOI: https://doi.org/10.1111/rssa.12703.

27.

Sun, H., R. Tourangeau, and S. Presser. 2019. “Panel Effects: Do the Reports of Panel Respondents Get Better or Worse over Time?” Journal of Survey Statistics and Methodology 7(4): 572–588. DOI: https://doi.org/10.1093/jssam/smy021.

28.

Sturgis, P., C. Roberts, and P. Smith. 2014. “Middle Alternatives Revisited: How the Neither/nor Response Acts as a Way of Saying “I don’t know.” Sociological Methods & Research 43(1): 15–38. DOI: https://doi.org/10.1177/0049124112452527.

29.

Tillmann, R., M. Voorpostel, E. Antal, N. Dasoki, H. Klaas, U. Kuhn, F. Lebert, G.-A. Monsch, and V.-A. Ryser. 2021. “The Swiss Household Panel (SHP).” Jahrbücher für Nationalökonomie und Statistik . DOI: https://doi.org/10.1515/jbnst-2021-0039.

30.

Tourangeau, R., and K.A. Rasinski. 1988. “Cognitive Processes Underlying Context Effects in Attitude Measurement.” Psychological bulletin 103(3): 299–314. DOI: https://doi.org/10.1037/0033-2909.103.3.299.

31.

Vannieuwenhuyze, J.T., and G. Loosveldt. 2013. “Evaluating Relative Mode Effects in Mixed-Mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.” Sociological Methods & Research 42(1): 82–104. DOI: https://doi.org/10.1177/0049124112464868.

32.

Voorpostel, M., U. Kuhn, R. Tillmann, G.-A. Monsch, E. Antal, V.-A. Ryser, F. Lebert, H.S. Klaas, and N. Dasoki. 2020. Introducing Web in a Refreshment Sample of the Swiss Household Panel: Main Findings From a Pilot Study. FORS Working Paper Series, paper 2020-2. Lausanne: FORS. Available at: https://serval.unil.ch/resource/serval:BIB_7C300C68F2D1.P001/REF (accessed November 2022)

33.

Voorpostel, M., O. Lipps, and C. Roberts. 2021. “Mixing Modes in Household Panel Surveys: Recent Developments and new Findings.” In Advances in longitudinal survey methodology, edited by P. Lynn. 204–226. Hoboken: Wiley. DOI: https://doi.org/10.1002/9781119376965.ch9.

34.

Young, R. 2012. Don’t Know Responses in Survey Research. Unpublished dissertation. Department of Sociology: Pennsylvania State University. Available at: https://etda.libraries.psu.edu/files/final_submissions/7219 (accessed November 2022)