Using Response Propensity Models to Improve the Quality of Response Data in Longitudinal Studies

Abstract

We review two approaches for improving the response in longitudinal (birth cohort) studies based on response propensity models: strategies for sample maintenance in longitudinal studies and improving the representativeness of the respondents over time through interventions. Based on estimated response propensities, we examine the effectiveness of different re-issuing strategies using Representativity Indicators (R-indicators). We also combine information from the Receiver Operating Characteristic (ROC) curve with a cost function to determine an optimal cut point for the propensity not to respond in order to target interventions efficiently at cases least likely to respond. We use the first four waves of the UK Millennium Cohort Study to illustrate these methods. Our results suggest that it is worth re-issuing to the field nonresponding cases from previous waves although re-issuing refusals might not be the best use of resources. Adapting the sample to target subgroups for re-issuing from wave to wave will improve the representativeness of response. However, in situations where discrimination between respondents and nonrespondents is not strong, it is doubtful whether specific interventions to reduce nonresponse will be cost effective.

Keywords

Representativity indicators ROC curves nonresponse Millennium Cohort Study

References

Alberman, E.D. and H. Goldstein. 1970. “The At Risk Register: A Statistical Evaluation.” British Journal of Preventive and Social Medicine 24: 129–135. Doi: http:/dx.doi.org/10.1136/jech.24.3.129.

Behr, A., E. Bellgardt, and U. Rendtel. 2005. “Extent and Determinants of Panel Attrition in the European Community Household Panel.” European Sociological Review 21: 489–512. Doi: http:/dx.doi.org/10.1093/esr/jci037.

Blom, A.G. 2014. “Setting Priorities: Spurious Differences in Response Rates.” International Journal of Public Opinion Research 26: 245–255. Doi: http:/dx.doi.org/10.1093/ijpor/edt023.

Binder, D. 1983. “On the Variances of Asymptotically Normal Estimators from Complex Surveys.” International Statistical Review 51: 279–292. Doi: http:/dx.doi.org/10.2307/1402588.

Box, G.P. and D.R. Cox. 1964. “An Analysis of Transformations.” Journal of Royal Statistical Society, Series B 26: 211–252.

Browne, W.J. 2009. MCMC Estimation in MLwiN. Centre for Multilevel Modelling, University of Bristol. Available at: http://www.bristol.ac.uk/cmm/software/mlwin/features/mcmc.html (accessed July 17, 2017).

Browne, W.J., F. Steele, M. Golalizadeh, and M.J. Green. 2009. “The Use of Simple Reparameterizations to Improve the Efficiency of Markov Chain Monte Carlo Estimation for Multilevel Models with Applications to Discrete Time Survival Models.” Journal of Royal Statistical Society, Series A 172: 579–598. Doi: http://dx.doi.org/10.1111/j.1467-985X.2009.00586.x.

Calderwood, L., I. Plewis, S.C. Ketende, and T. Mostafa. 2016. “Evaluating the Immediate and Longer-term Impact of a Refusal Conversion Strategy in a Large Scale Longitudinal Study.” Survey Research Methods 10: 225–236. Doi: http://dx.doi.org/10.18148/srm/2016.v10i3.6275.

Couper, M. and M. Ofstedal. 2009. “Keeping in Contact with Mobile Sample Members.” In Methodology of Longitudinal Surveys, edited by P. Lynn, 183–203. Chichester: John Wiley. Doi: http://dx.doi.org/10.1002/9780470743874.ch11.

10.

Durrant, G.B. and F. Steele. 2009. “Multilevel Modelling of Refusal and Non-Contact in Household Surveys: Evidence from six UK Government Surveys.” Journal of the Royal Statistical Society, Series A 172: 361–382. Doi: http://dx.doi.org/10.1111/j.1467-985X.2008.00565.x.

11.

Durrant, G.B., J. D’Arrigo, and F. Steele. 2011. “Using Paradata to Predict Best Times of Contact Conditioning on Household and Interviewer Influences.” Journal of the Royal Statistical Society, Series A 174: 1029–1049. Doi: http://dx.doi.org/10.1111/j.1467-985X.2011.00715.x.

12.

Fumagalli, L., H. Laurie, and P. Lynn. 2013. “Experiments with Methods to Reduce Attrition in Longitudinal Surveys.” Journal of the Royal Statistical Society, Series A 176: 499–519. Doi: http://dx.doi.org/10.1111/j.1467-985X.2012.01051.x.

13.

Groves, R.M. 2006. “Nonresponse Rates and Non-response Bias in Household Surveys.” Public Opinion Quarterly 70: 646–675. Doi: http://dx.doi.org/10.1093/poq/nfl033.

14.

Hawkes, D. and I. Plewis. 2006. “Modelling Non-Response in the National Child Development Study.” Journal of the Royal Statistical Society, Series A 169: 479–491. Doi: http://dx.doi.org/10.1111/j.1467-985X.2006.00401.x.

15.

Lynn, P., O. Kaminska, and H. Goldstein. 2014. “Panel Attrition: How Important is Interviewer Continuity?” Journal of Official Statistics 30: 443–457. Doi: http://dx.doi.org/10.2478/jos-2014-0028.

16.

Kreuter, F., K. Olson, J. Wagner, T. Yan, T.M. Ezzati-Rice, C. Casas-Cordero, M. Lemay, A. Peytchev, R.M. Groves, and T.E. Raghunathan. 2010. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys.” Journal of the Royal Statistical Society, Series A 173: 389–408. Doi: http://dx.doi.org/10.1111/j.1467-985X.2009.00621.x.

17.

Krzanowski, W.J. and D.J. Hand. 2009. ROC Curves for Continuous Data. Boca Raton, Fl: Chapman and Hall/CRC. Doi: http://dx.doi.org/10.1201/9781439800225.

18.

Laurie, H. and P. Lynn. 2009. “The Use of Respondent Incentives on Longitudinal Surveys.” In Methodology of Longitudinal Surveys, edited by P. Lynn, 205–233. Chichester: John Wiley. Doi: http://dx.doi.org/10.1002/9780470743874.ch12.

19.

Lynn, P. 2017. “From Standardised to Targeted Survey Procedures for Tackling Nonresponse and Attrition.” Survey Research Methods 11: 93–103. Doi: http://dx.doi.org/10.18148/srm/2017.v11i1.6734.

20.

McGonagle, K., M. Couper, and R. Schoeni. 2009. “An Experimental Text of a Strategy to Maintain Contact with Families Between Waves of a Panel Study.” Survey Practice 2(5).

21.

Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. New York: OUP. Doi: http://dx.doi.org/10.1198/jasa.2005.s19.

22.

Peytchev, A., S. Riley, J. Rosen, and J. Murphy. 2010. “Reduction of Nonresponse Bias in Surveys Through Case Prioritization.” Survey Research Methods 4: 21–29. Doi: http://dx.doi.org/10.18148/srm/2010.v4i1.3037.

23.

Plewis, I. 2007a. “Non-Response in a Birth Cohort Study: The Case of the Millennium Cohort Study.” International Journal of Social Research Methodology 10: 325–334. Doi: http://dx.doi.org/10.1080/13645570701676955.

24.

Plewis, I. (Ed.). 2007b. The Millennium Cohort Study: Technical Report on Sampling (4th Ed.). London: Institute of Education, University of London. Available at: http://www.cls.ioe.ac.uk/library-media/documents/Technical_Report_on_Sampling_4th_Edition.pdf (accessed July 17, 2017).

25.

Plewis, I., S.C. Ketende, H. Joshi, and G. Hughes. 2008. “The Contribution of Residential Mobility to Sample Loss in a Birth Cohort Study: Evidence from the First two Waves of the Millennium Cohort Study.” Journal of Official Statistics 24: 365–385. Doi: http://dx.doi.org/10.1080/13645570701676955.

26.

Plewis, I., S.C. Ketende, and L. Calderwood. 2012. “Assessing the Accuracy of Response Propensities in Longitudinal Studies.” Survey Methodology 38: 167–171.

27.

Pudney, S. and N. Watson. 2013. If at First you Don’t Succeed? Fieldwork, Panel Attrition, and Health-Employment Inferences in BHPS and HILDA, Institute for Social and Economic Research Working Papers, No. 2013–27, ISER, University of Essex. Available at: https://www.iser.essex.ac.uk/research/publications/working-papers/iser/2013-27 (accessed July 17, 2017).

28.

Rasbash, J., C. Charlton, W.J. Browne, M. Healy, and B. Cameron. 2009. MLwiN Version 2.1. Centre for Multilevel Modelling, University of Bristol. Available at: http://www.bristol.ac.uk/cmm/software/mlwin/download/manuals.html (accessed July 17, 2017).

29.

SAS Institute Inc. 2011. SAS/STAT® 9.3, User’s Guide. Cary, NC: SAS Institute Inc. Available at: https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#titlepage.htm (accessed July 17, 2017).

30.

Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators of the Representativeness of Survey Response.” Survey Methodology 35: 101–113.

31.

Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27: 231–253.

32.

Schouten, B. and N. Shlomo. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85: 143–163. Doi: http://dx.doi.org/10.1111/insr.12159.

33.

Shlomo, N., C. Skinner, and B. Schouten. 2012. “Estimation of an Indicator of the Representativeness of Survey Response.” Journal of Statistical Planning and Inference 142: 201–211. Doi: http://dx.doi.org/10.1016/j/jspi.2011.07.008.

34.

Skinner, C. and J. D’Arrigo. 2011. “Inverse Probability Weighting for Clustered Nonresponse.” Biometrika 98: 953–966. Doi: http://dx.doi.org/10.1093/biomet/asr058.

35.

Watson, N. and M. Wooden. 2009. “Identifying Factors Affecting Longitudinal Survey Response.” In Methodology of Longitudinal Surveys, edited by P. Lynn, 157–182. Chichester: John Wiley. Doi: http://dx.doi.org/10.1002/9780470743874.ch10.

36.

Watson, N. and M. Wooden. 2014. “Re-Engaging with Survey Non-Respondents: Evidence from Three Household Panels.” Journal of the Royal Statistical Society, Series A 177: 499–523. Doi: http://dx.doi.org/10.1111/rssa.12024.