Weighting Strategies for Combining Data from Dual-Frame Telephone Surveys: Emerging Evidence from Australia

Abstract

Until quite recently, telephone surveys have typically relied on landline telephone numbers. However, with the increasing popularity and affordability of mobile phones, there has been a surge in households that do not have landline connections. Additionally, there has been a decline in the response rates and population coverage of landline telephone surveys, creating a challenge to collecting representative social data. Dual-frame telephone surveys that use both landline and mobile phone sampling frames can overcome the incompleteness of landline-only telephone sampling. However, surveying mobile phone users introduces new complexities in sampling, nonresponse measurement and statistical weighting. This article examines these issues and illustrates the consequences of failing to include mobile-phone-only users in telephone surveys using data from Australia. Results show that there are significant differences in estimates of populations’ characteristics when using information solely from the landline or mobile telephone sample. These biases in the population estimates are significantly reduced when data from the mobile and landline samples are combined and appropriate dual-frame survey estimators are used. The optimal choice of a dual-frame estimation strategy depends on the availability of good-quality information that can account for the differential patterns of nonresponse by frame.

Keywords

Dual-frame telephone surveys mobile phone sampling nonresponse weighting

References

Akter, S. and J. Bennett. 2011. “Household Perceptions of Climate Change and Preferences for Mitigation Action: The Case of the Carbon Pollution Reduction Scheme in Australia.” Climatic Change 109: 417–436. Doi: http://dx.doi.org/10.1007/s10584-011-0034-8.

Arcos, A., M.M. Rueda, M. Trujillo, and D. Molina. 2014. “Review of Estimation Methods for Landline and Cell Phone Surveys.” Sociological Methods and Research 44: 458–485. Doi: http://dx.doi.org/10.1177/0049124114546904.

American Association for Public Opinion Research (AAPOR). 2010. “Cell Phone Task Force Report. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the US With Respondents Reached via Cell Phone Numbers.” AAPOR. Available at: http://www.aapor.org/AAPORKentico/Education-Resources/Reports/Cell-Phone-Task-Force-Report.aspx. (accessed 25 November 2015).

American Association for Public Opinion Research (AAPOR). 2011. “Standard Definitions: Final Disposition of Case Codes and Outcome Rates for Surveys 7th Edition.” AAPOR. Available at: http://www.esomar.org/knowledge-and-standards/research-resources/aapor-standard-definitions.php. (accessed 25 November 2015).

Australian Bureau of Statistics (ABS). 2008. “How Australians Use Their Time: Time spent on cultural activities. ABS cat.no.4153.0.” Canberra: Australian Bureau of Statistics. Available at: http://www.abs.gov.au/ausstats/abs@.nsf/mf/4153.0 (accessed 25 November 2015).

Australian Bureau of Statistics (ABS). 2012. “Basic Community Profiles, Census 2011. Canberra: Australian Bureau of Statistics.” Available at: http://www.abs.gov.au/websitedbs/censushome.nsf/home/communityprofiles. (accessed 25 November 2015).

Australian Bureau of Statistics (ABS). 2014. “Australian Health Survey: Users’ Guide, 2011-13.” ABS cat.no. 4363.0.55.001. Canberra: Australian Bureau of Statistics. Available at: http://www.abs.gov.au/ausstats/abs@.nsf/mf/4363.0.55.001 (accessed 25 November 2015).

Australian Communications and Media Authority (ACMA). 2006, 2011, 2014, 2015. Communications Report, 2005–2006, 2010–2011, 2013–2014 & 2014–2015 Series “Converging Communications Channels: Preferences and Behaviour of Australian Communications Users.” ACMA: Canberra. Available at: www.acma.gov.au/Comms-Report. (accessed 25 November 2015).

Bankier, M.D. 1986. “Estimators Based on Several Stratified Samples With Applications to Multiple Finite Surveys.” Journal of the American Statistical Association 81: 1074–1079. Doi: http://dx.doi.org/10.1080/01621459.1986.10478376.

10.

Barnes, G.M., J.H. Hoffman, M.O. Tidwell, and J.W. Welte. 2015. “Gambling and Substance Use: Co-Occurrence Among Adults in a Recent General Population Study in the United States.” International Gambling Studies 15: 55–71. Doi: http://dx.doi.org/10.1080/14459795.2014.990396.

11.

Barr, M.L., J.J. van Ritten, D.G. Steel, and S.V. Thackway. 2012. “Inclusion of Mobile Phone Numbers Into an Ongoing Population Health Survey in New South Wales, Australia: Design, Methods, Call Outcomes, Costs and Sample Representativeness.” BMC Medical Research Methodology 12: 177–185. Doi: http://dx.doi.org/10.1186/1471-2288-12-177.

12.

Barr, M.L., R.A. Ferguson, P.J. Hughes, and D.G. Steel. 2014. “Developing a Weighting Strategy to Include Mobile Telephone Numbers Into an Ongoing Population Health Survey Using an Overlapping Dual Frame Design With Limited Benchmark Information.” BMC Medical Research Methodology 14: 102–112. Doi: http://dx.doi.org/10.1186/1471-2288-14-102.

13.

Béland, Y. 2002. “Canadian Community Health Survey – Methodological Overview.” Health Reports - Statistics Canada 13: 9–14.

14.

Blumberg, S.J. and J.V. Luke. 2007. “Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults.” Public Opinion Quarterly 71: 734–749. Doi: http://dx.doi.org/10.1093/poq/nfm047.

15.

Blumberg, S.J. and J.V. Luke. 2014. “Wireless Substitution: Early Release Estimates from the National Health Interview Survey, January-June 2014.” National Center for Health Statistics. Available at: http://www.cdc.gov/nchs/nhis/releases.htm (accessed 25 November 2015).

16.

Blumberg, S.J. and J.V. Luke. 2015. “Wireless Substitution: Early Release Estimates from the National Health Interview Survey, July-December 2014.” National Center for Health Statistics. Available at: http://www.cdc.gov/nchs/nhis/releases.htm (accessed 25 November 2015).

17.

Brick, J.M. 2011. “The future of survey sampling.” Public Opinion Quarterly 75: 872–888. Doi: http://dx.doi.org/10.1093/poq/nfr045.

18.

Brick, J.M. 2013. “Unit Nonresponse and Weighting Adjustments: a Critical Review.” Journal of Official Statistics 29: 329 – 353. Doi: http://dx.doi.org/10.2478/jos-2013-0026.

19.

Brick, J.M., I.F. Cervantes, S. Lee, and G. Norman. 2011. “Nonsampling Errors in Dual Frame Telephone Surveys.” Survey Methodology 37: 1–12.

20.

Brick, J.M., S. Dipko, S. Presser, and C. Tucker. 2006. “Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers.” Public Opinion Quarterly 70: 780–793. Doi: http://dx.doi.org/10.1093/poq/nfl031.

21.

Busse, B. and M. Fuchs. 2012. “The Components of Landline Telephone Survey Coverage Bias. The Relative Importance of No-Phone and Mobile-Only Populations.” Quality & Quantity 46: 1209–1225. Doi: http://dx.doi.org/10.1007/s11135-011-9431-3.

22.

Callegaro, M. and T. Possio. 2004. “Mobile Telephone Growth and Coverage Error in Telephone Surveys.” Polis: Research and Studies in Italian Society and Politics 18: 477–506.

23.

Deville, J. and C.-E. Särndal. 1992. “Calibration Estimators in Survey Sampling.” Journal of the American Statistical Association 87: 376–382.

24.

Flores Cervantes, I. and G. Kalton. 2008. “Methods for Sampling Rare Populations in Telephone Surveys.” In Advances in Telephone Survey Methodology, edited by J.M. Lepkowski, C. Tucker, J.M. Brick, E.D. de Leeuw, L. Japec, P. Lavrakas, M.W. Link, and R.L. Sangster, 113–132. Hoboken, NJ: Wiley.

25.

Fuller, W.A. and L.F. Burmeister. 1972. “Estimators for Samples Selected From Two Overlapping Frames.” In Proceedings of the Social Statistics Section of American Statistical Association, August 14–17, 245–249. Alexandria, VA: American Statistical Association.

26.

Gong, G. and F.J. Samaniego. 1981. “Pseudo Maximum Likelihood Estimation: Theory and Applications.” The Annals of Statistics 9: 861–869. Doi: http://dx.doi.org/10.1214/aos/1176345526.

27.

Groves, R.M. and E. Peytcheva. 2008. “The Impact of Non-Response Rates on Non-Response Bias.” Public Opinion Quarterly 72: 167–189. Doi: http://dx.doi.org/10.1093/poq/nfn011.

28.

Hartley, H.O. 1962. “Multiple Frame Surveys.” In Proceedings of the Social Statistics Section of American Statistical Association, date of conference, 203–206. Alexandria, VA: American Statistical Association.

29.

Hartley, H.O. 1974. “Multiple Frame Methodology and Selected Applications.” Sankhya C 36: 99–118.

30.

Holt, D. and T.M.F. Smith. 1979. “Post-Stratification.” Journal of the Royal Statistical Society, Series A 142: 33–46.

31.

Hu, S.S., L. Balluz, M.P. Battaglia, and M.R. Frankel. 2011. “Improving Public Health Surveillance Using a Dual Frame Survey of Landline and Cell Phone Numbers.” American Journal of Epidemiology 173: 703–711. Doi: http://dx.doi.org/10.1093/aje/kwq442.

32.

Hu, F.B., T.Y. Li, G.A. Colditz, W.C. Willett, and J.E. Manson. 2003. “Television Watching and Other Sedentary Behaviours in Relation to Risk of Obesity and Type 2 Diabetes Mellitus in Women.” Journal of the American Medical Association (JAMA) 289: 1785–1791. Doi: http://dx.doi.org/10.1001/jama.289.14.1785.

33.

Iachan, R. and M.L. Dennis. 1993. “A Multiple Frame Approach to Sampling the Homeless and Transient Population.” Journal of Official Statistics 9: 747–764.

34.

Kalton, G. and D.W. Anderson. 1986. “Sampling Rare Populations.” Journal of the Royal Statistical Society, Series A 149: 65–82. Doi: http://dx.doi.org/10.2307/2981886.

35.

Keeter, S., C. Miller, A. Kohut, R.M. Groves, and S. Presser. 2000. “Consequences of Reducing Non-Responses in a National Telephone Survey.” Public Opinion Quarterly 64: 125–148. Doi: http://dx.doi.org/10.1086/317759.

36.

Keeter, S., C. Kennedy, A. Clark, T. Tompson, and M. Mokrzycki. 2007. “What’s Missing From National Landline RDD surveys? The Impact of the Growing Cell-Only Population.” Public Opinion Quarterly 71: 772–792. Doi: http://dx.doi.org/10.1093/poq/nfm053.

37.

Keeter, S., S.C. Kennedy, M. Dimock, J. Best, and P. Craighill. 2006. “Gauging the Impact of Growing Non-Response on Estimates from a National RDD Telephone Survey.” Public Opinion Quarterly 70: 759–779. Doi: http://dx.doi.org/10.1093/poq/nfl035.

38.

Kennedy, C. 2007. “Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD surveys.” Public Opinion Quarterly 71: 750–771. Doi: http://dx.doi.org/10.1093/poq/nfm050.

39.

Kuusela, V., M. Callegaro, and V. Vehovar. 2008. “The Influence of Mobile Telephones on Telephone Surveys.” In Advances in Telephone Survey Methodology, edited by J.M. Lepkowski, C. Tucker, J.M. Brick, E.D. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link, and R.L. Sangster, 87–112. Doi: http://dx.doi.org/10.1002/9780470173404.ch4.

40.

Lee, S., J.M. Brick, E.R. Brown, and D. Grant. 2010. “Growing Cell-Phone Population and Noncoverage Bias in Traditional Random Digit Dial Telephone Health Surveys.” Health Services Research 45: 1121–1139. Doi: http://dx.doi.org/10.1111/j.1475-6773.2010.01120.x.

41.

Little, R.J.A. 1993. “Post-Stratification: A Modeler’s Perspective.” Journal of the American Statistical Association 88: 1001–1012.

42.

Livingston, M., P. Dietze, J. Ferris, D. Pennay, L. Hayes, and S. Lenton. 2013. “Surveying Alcohol and Other Drug Use Through Telephone Sampling: A Comparison of Landline and Mobile Phone Samples.” BMC Medical Research Methodology 13: 41. Doi: http://dx.doi.org/10.1186/1471-2288-13-41.

43.

Lohr, S. 2009. “Multiple Frame Surveys.” In Handbook of Statistics, Sample Surveys: Design, Methods and Applications, edited by D. Pfeffermann and C.R. Rao, 71–88. Amsterdam: North Holland.

44.

Lohr, S.L. 2011. “Alternative Survey Sample Designs: Sampling With Multiple Overlapping Frames.” Survey Methodology 37: 197–213.

45.

Lohr, S. and J. Rao. 2000. “Inference From Dual Frame Surveys.” Journal of the American Statistical Association 95: 271–280. Doi: http://dx.doi.org/10.1080/01621459.2000.10473920.

46.

Lohr, S. and J. Rao. 2006. “Estimation in Multiple-Frame Surveys.” Journal of the American Statistical Association 101: 1019–1030. Doi: http://dx.doi.org/10.1198/0162144506000000195.

47.

Lopez, M.H. and A. Gonzalez-Barrera. 2013. “Inside the 2012 Latino Electorate.” Washington, D.C.: Pew Research Center’s Hispanic Trends Project. Available at: http://www.pewhispanic.org/2013/06/03/inside-the-2012-latino-electorate. (accessed 25 November 2015).

48.

Mecatti, F. 2007. “A Single Frame Multiplicity Estimator for Multiple Frame Surveys.” Survey Methodology 33: 151–157.

49.

Mohorko, A., E. de Leeuw, and J. Hox. 2013. “Coverage Bias in European Telephone Surveys: Developments of Landline and Mobile Phone Coverage Across Countries and Over Time.” Survey Methods: Insights from the Field (SMIF) 2013, 1–13. Doi: http://dx.doi.org/10.13094/SMIF-2013-00002.

50.

Pennay, D.W. 2010. “Profiling the ‘Mobile Only’ Population. Results From a Dual-Frame Telephone Survey Using a Landline and Mobile Phone Sample Frame.” In Proceedings of the Australian Consortium for Social and Political Research Incorporated (ACSPRI) Social Science Methodology Conference, 1–3 December, Sydney, Australia. Available at: http://www.srcentre.com.au/docs/publications/dual_frame-survey_acspri-conference-paper_finalv2.pdf?sfvrsn=0 (accessed 25 November 2015).

51.

Pennay, D.W. and N. Vickers. 2012. “Dual-Frame Omnibus Survey.” Technical and Methodological Summary Report. Social Research Centre Pty Ltd, Melbourne, Australia. Available at: http://www.srcentre.com.au/docs/event-workshop-july-2012/dual-frame-omnibus-technical-report-(pennay).pdf?sfvrsn=2 (accessed 25 November 2015).

52.

Pennay, D.W. and N. Vickers. 2013. “Second Dual-Frame Omnibus Survey.” Technical and Methodological Summary Report. Social Research Centre Pty Ltd, Melbourne, Australia. Available at: http://www.srcentre.com.au/docs/publications/full-report-here.pdf?sfvrsn=0 (accessed 27 November 2015).

53.

Pew Research Centre. 2012. “Assessing the Representativeness of Public Opinion Surveys.” Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/ (accessed 25 November 2015).

54.

Skinner, C.J. 1991. “On Efficiency of Raking Ratio Estimation for Multiple Frame Surveys.” Journal of the American Statistical Association 86: 779–784.

55.

Skinner, C.J. and J.N.K. Rao. 1996. “Estimation in Dual Frame Surveys with Complex Designs.” Journal of the American Statistical Association 91: 349–356.

56.

Steeh, C. 2008. “Telephone Surveys.” In International Handbook of Survey Methodology, edited by E.D. de Leuuw, J.J. Hox, and D.A. Dillman, 221–238. New York: Routledge.

57.

Thomée, S., A. Härenstam, and M. Hagberg. 2011. “Mobile Phone Use and Stress, Sleep Disturbances, and Symptoms of Depression Among Young Adults - A Prospective Cohort Study.” BMC Public Health 11: 66. Doi: http://dx.doi.org/10.1186/1471-2458-11-66.

58.

Tucker, C. and J.M. Brick. 2007. “Household Telephone Service and Usage Patterns in the United States in 2004: Implications for Telephone Samples.” Public Opinion Quarterly 71: 3–22. Doi: http://dx.doi.org/10.1093/poq/nfl047.

59.

Vicente, P. and E. Reis, E. 2009. The “Mobile Only Population in Portugal and its Impact in a Dual Frame Telephone Survey.” Survey Research Methods 3: 105–111.