Abstract
Sequential mixed-mode designs are increasingly considered as an alternative to interviewer-administered data collection, allowing researchers to take advantage of the benefits of each mode. We assess the effects of the introduction of a sequential web-face-to-face mixed-mode design over three waves of a longitudinal survey in which members were previously interviewed face-to-face. Findings are reported from a large-scale randomised experiment carried out on the UK Household Longitudinal Study. No differences are found between the mixed-mode design and face-to-face design in terms of cumulative response rates and only minimal differences in terms of sample composition. On the other hand, potential cost savings are evident.
References
1.
Bianchi, A. and S. Biffignandi. 2014. “Responsive Design for Economic Data in Mixed-Mode Panels.” In Contribution to Sampling Statistics , edited by F. Mecatti, P.L. Conti, and M.G. Ranalli, 85–102. Springer International Publishing.
2.
Bianchi, A. and S. Biffignandi. 2017. “Representativeness in Panel Surveys.” To appear in Mathematical Population Studies .
3.
Bianchi, A. and S. Biffignandi. Forthcoming. “Survey Experiments on Interactions: a Case Study of Incentives and Modes.” In Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment , edited by P.J. Lavrakas, E. de Leeuw, A. Holbrook, C. Kennedy, M.W. Traugott, and B.T. West. Hoboken NJ: John Wiley & Sons.
4.
Biemer, P.P. 2010. “Total Survey Error: Design, Implementation and Evaluation.” Public Opinion Quarterly 74(5): 817–848. Doi: http://dx.doi.org/10.1093/poq/nfq058.
5.
Biemer, P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality . New York: John Wiley.
6.
Bowling, A. 2005. “Mode of Questionnaire Administration Can Have Serious Effects on Data Quality.” Journal of Public Health 27: 281–291.
7.
Buck, N. and S. McFall. 2012. “Understanding Society: Design Overview.” Longitudinal and Life Course Studies 3: 5–17.
8.
Calinescu, M. and B. Schouten. 2015. “Adaptive Survey Designs to Minimize Survey Mode Effects – a Case Study on the Dutch Labor Force Survey.” Survey Methodology 41(2) : 403–425.
9.
Chng, S., M. White, C. Abraham, and S. Skippon. 2016. “Commuting and Wellbeing in London: the Roles of Commute Mode and Local Public Transport Connectivity.” Preventive Medicine 88: 182–188. Doi: http://dx.doi.org/10.1016/j.ypmed.2016.04.014.
10.
Couper, M. 2011. “The Future of Modes of Data Collection.” Public Opinion Quarterly 75: 889–908. Doi: http://dx.doi.org/10.1093/poq/nfr046.
11.
Evandrou, M., J. Falkingham, Z. Feng, and A. Vlachantoni. 2016. “Ethnic Inequalities in Limiting Health and Self-Reported Health in Later Life Revisited.” Journal of Epidemiology & Community Health 70: 653–662. Doi: http://dx.doi.org/10.1136/jech-2015-206074.
12.
Fan, W. and Z. Yan. 2010. “Factors Affecting Response Rates of the Web Surveys: a Systematic Review.” Computers in Human Behavior 26: 132–139.
13.
Farrant, G. and C. O’Muircheartaigh. 1991. “Components of Nonresponse Bias in the British Election Surveys.” In Understanding Political Change , edited by A. Heath, J. Curtice, R. Jowell, S. Evans, J. Field, and S. Witherspoon, 235–249. London: Pergamon Press.
14.
Fong, B. and J. Williams. 2011. “British Crime Survey: Feasibility of Boosting Police Force Area (PFA) Sample Sizes Using Supplementary Recontact Surveys.” Report for the Home Office, TNS-BMRB, London.
15.
Gaia, A. 2014. “Does a Mixed-Mode Design Increase Panel Attrition? Evidence from the UKHLS Innovation Panel.” Paper presented at the Internet Survey Methodology Workshop, Bolzano, December 1–3.
16.
Göritz, A. 2006. “Incentives in Web Studies: Methodological Issues and a Review.” International Journal of Internet Science 1: 58–70.
17.
Göritz, A. 2010. “Using Lotteries, Loyalty Points, and Other Incentives to Increase Participant Response and Completion.” In Advanced methods for conducting online behavioural research , edited by S. Gosling and J. Johnson, 219–233. Washington DC: American Psychological Association. Doi: http://dx.doi.org/10.1037/12076-014.
18.
Göritz, A. 2015. “Incentive Effects.” In Improving Survey Methods: Lessons from Recent Research , edited by U. Engel, B. Jann, P. Lynn, A. Scherpenzeel, and P. Sturgis, 339–350. London: Routledge.
19.
Griffin, D., D. Fischer, and M. Morgan. 2001. “Testing an Internet Response Option for the American Community Survey.” Paper presented at the annual conference of the American Association for Public Opinion Research Montreal.” Quebec, Canada, May 17–20.
20.
Groves, R. and L. Lyberg. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly 74(5): 849–879. Doi: http://dx.doi.org/10.1093/poq/nfq065.
21.
Groves, R.M. and F. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. Doi: https://doi.org/10.1093/poq/nfn011.
22.
Jäckle, A. 2016. “Mode Effects on Measurement in Understanding Society .” Paper presented at the International Panel Survey Methods Workshop, Berlin, June 20–21. Available at: http://www.diw.de/en/diw_01.c.534396.en/program_psmw2016.html (accessed March 2017).
23.
Jäckle, A. and P. Lynn. 2008. “Respondent Incentives in a Multi-Mode Panel Survey: Cumulative Effects on Nonresponse and Bias.” Survey Methodology 34: 105–117.
24.
Jäckle, A., P. Lynn, and J. Burton. 2015. “Going Online with a Face-to-Face Household Panel: Effects of a Mixed Mode Design on Item and Unit Non-Response.” Survey Research Methods 9(1): 57–70. Doi: http://dx.doi.org/10.18148/srm/2015.v9i1.5475.
25.
Janssen, B. 2006. “Web Data Collection in a Mixed Mode Approach: An Experiment.” Paper presented at the European Conference on Quality in Official Statistics (Q2006), Cardiff, April 24–26. Available at: webarchive.nationalarchives.gov.uk/20140721132900/http://ons.gov.uk/about/newsroom/events/q2006––european-conference-on-quality-in-survey-statistics-24-26-april-2006/agenda/session-19-wednesday.pdf (accessed March 2017).
26.
Klausch, T., J. Hox, and B. Schouten. 2015a. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society, Series A 178(4): 945–961. Doi: http://dx.doi.org/10.1111/rssa.12102.
27.
Klausch, T., B. Schouten, and J.J. Hox. 2015b. “Evaluating Bias of Sequential Mixedmode Designs Against Benchmark Surveys.” Sociological Methods & Research : 1–34. Doi: http://dx.doi.org/10.1177/0049124115585362.
28.
Kreuter, F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645: 23–35. Doi: https://doi.org/10.1177/0002716212456815.
29.
Krosnick, J.A. and D.F. Alwin. 1987. “An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement.” Public Opinion Quarterly 51: 201–219. Doi: https://doi.org/10.1086/269029.
30.
Lagerstrøm, B. 2008. “Cost Efficiency in a Mixed-Mode Survey – Evidence from the Norwegian Rent Market Survey.” Paper presented at the 19th International Workshop on Household Survey Nonresponse, Ljubljana, September 15. Available at: http://www.nonresponse.org/db/3/558/Biblliography/Cost%20efficiency%20in%20a%20mixed-mode%20survey%20Evidence%20from%20the%20Norwegian%20Rent%20Marked%20Survey/?&p1=308&p2=74&p3=551 (accessed March 2017).
31.
Laurie, H. and P. Lynn. 2009. “The Use of Respondent Incentives on Longitudinal Surveys.” In Methodology of Longitudinal Surveys , edited by P. Lynn, 205–233. Chichester: Wiley.
32.
Leesti, T. 2010. “Canadian Labour Force Survey Internet Data Collection Pilot Test.” Paper presented at the Fifth Workshop on Labour Force Survey Methodology, Paris, April 15–16.
33.
De Leeuw, E.D. 2005. “To Mix or not to Mix Data Collection in Surveys.” Journal of Official Statistics 21: 233–255.
34.
Lugtig, P. 2014. “Panel Attrition: Separating Stayers, Fast Attriters, Gradual Attriters, and Lurkers.” Sociological Methods and Research 43(4): 699–723. Doi: http://dx.doi.org/10.1177/0049124113520305.
35.
Lynn, P. 2009. “Sample Design for Understanding Society.” Understanding Society Working Paper 2009-01, ISER, University of Essex, Colchester. Available at: www.understandingsociety.ac.uk/research/publications/working-paper/understandingsociety/2009-01 (accessed 16 March 2017).
36.
Lynn, P. 2013. “Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias and Costs.” Journal of Survey Statistics and Methodology 1: 183–205. Doi: http://dx.doi.org/10.1093/jssam/smt015.
37.
Lynn, P. 2014. “Targeted Response Inducement Strategies on Longitudinal Surveys.” In Improving Survey Methods: Lessons from Recent Research , edited by U. Engel, B. Jann, P. Lynn, A. Scherpenzeel, and P. Sturgis. Routledge/Psychology Press.
38.
Lynn, P. Forthcoming. “Tackling Panel Attrition.” In The Palgrave Handbook of Survey Research , edited by D.L. Vannette and J.A. Krosnick. Palgrave.
39.
Lynn, P. and A. Jäckle. Forthcoming. “Mounting Multiple Experiments on Longitudinal Social Surveys: Design and Implementation Considerations.” In Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment , edited by P.J. Lavrakas, E.E. de Leeuw, A. Holbrook, C. Kennedy, M.W. Traugott, and B.T. West. Hoboken NJ: Wiley.
40.
Lynn, P. and P. Lugtig. 2017. “Total Survey Error for Longitudinal Surveys.” In Total Survey Error in Practice , edited by Paul Biemer, Edith de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars Lyberg, Clyde Tucker, and Brady West, 279–298. Hoboken, New Jersey: Wiley.
41.
Lynn, P. and D. Lievesley. 1991. “Drawing General Population Samples in Great Britain.” London: SCPR.
42.
Lynn, P. S.C.N. Uhrig, and J. Burton. 2010. “Lessons from a Randomized Experiment with Mixed-Mode Designs for a Household Panel Survey.” Understanding Society, Working Paper Series, 2010-03.
43.
Martin, P. and P. Lynn. 2011. “The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses.” ISER Working Paper Series, 2011-28. Colchester: Institute for Social and Economic Research, University of Essex. Available at: https://www.iser.essex.ac.uk/research/publications/working-papers/iser/2011-28 (accessed February 5, 2013).
44.
Schoeni, R., F. Stafford, K. McGonagle, and P. Andreski. 2013. “Response Rates in National Panel Surveys.” The Annals of the American Academy of Political and Social Science 645: 60–87. Doi: https://doi.org/10.1177/0002716212456363.
45.
Schwarz, N., F. Strack, H.-J. Hippler, and G. Bishop. 1991. “The Impact of Administration Mode on Response Effects in Survey Measurement.” Applied Cognitive Psychology 5: 193–212.
46.
Souren, M. 2012. “Multi-Mode Surveys at Statistics Netherlands: Implications, Experiences and Open Issues.” Paper presented at Opening Conference of the European Statistical System Network (ESSNet) on Data Collection for Social Surveys using Multiple Modes, Wiesbaden, October 11–12, 2012.
47.
Uhrig, S.C.N. 2008. “The Nature and Causes of Attrition in the British Household Panel Study.” Institute for Social and Economic Research Working Paper 2008-05. Available at: https://www.iser.essex.ac.uk/research/publications/working-papers/iser/2008-05 (accessed 16 March 2017).
48.
Uhrig, S.C.N. 2011. “Using Experiments to Guide Decision Making in Understanding Society: Introducing the Innovation Panel.” In Understanding Society: Early Findings from the First Wave of the UK’s Household Longitudinal Study , edited by S.L. McFall and C. Garrington. Colchester: University of Essex. Available at: http://research.understandingsociety.org.uk/findings/early-findings (accessed 16 March 2017).
49.
Voogt, R. and W. Saris. 2005. “Mixed Mode Designs: Finding the Balance between Nonresponse Bias and Mode Effects.” Journal of Official Statistics 21(3): 367–387.
50.
Voorpostel, M. and V.A. Ryser. 2011. “Mixed Mode Data Collection as a Strategy to Decrease Panel Attrition in the Swiss Household Panel.” FORS Working Paper 2_11. Available at: http://ohs-shp.unil.ch/workingpapers/WP2_11.pdf (accessed 28 March 2017).
51.
Wallace, S., J. Nazroo, and L. Bécares. 2016. “Cumulative Effect of Racial Discrimination on the Mental Health of Ethnic Minorities in the United Kingdom.” American Journal of Public Health 106(7): 1294–1300. Doi: http://dx.doi.org/10.2105/AJPH.2016.303121.
52.
Watson, N. and M. Wooden. 2014. “Re-Engaging with Survey Non-Respondents: Evidence from Three Household Panels.” Journal of the Royal Statistical Society Series A (Statistics in Society) 177(2): 499–522. Doi: http://dx.doi.org/10.1111/rssa.12024.