A Note on the Optimum Allocation of Resources to Follow up Unit Nonrespondents in Probability Surveys

Abstract

Common practice to address nonresponse in probability surveys in National Statistical Offices is to follow up every non respondent with a view to lifting response rates. As response rate is an insufficient indicator of data quality, it is argued that one should follow up non respondents with a view to reducing the mean squared error (MSE) of the estimator of the variable of interest. In this article, we propose a method to allocate the nonresponse follow-up resources in such a way as to minimise the MSE under a quasi-randomisation framework. An example to illustrate the method using the 2018/19 Rural Environment and Agricultural Commodities Survey from the Australian Bureau of Statistics is provided.

Keywords

Data quality nonresponse follow-up propensity score weighting adjustment

References

Australian Bureau of Statistics, 2000. Droughts, fires, cyclones, hailstorms and a pandemic. Available at: https://www.abs.gov.au/articles/droughts-fires-cyclones-hailstorms-and-pandemic-march-quarter-2020#businesses.

Beaumont, J.F., 2005. “Calibrated Imputation In Surveys Under A Quasi-Model-Assisted Approach.” Journal of the Royal Statistical Society B67: 445–458. DOI: https://doi.org/10.1111/j.1467-9868.2005.00511.x.

Beaumont, J.F., C. Bocci, and D. Haziza. 2014. “An Adaptive Data Collection Procedure for CallPrioritization.” Journal of Official Statistics 30: 607–621. DOI: http://dx.doi.org/10.2478/jos-2014-0040.

Bethlehem, J.G. 1998. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 3: 251–260.

Breiman, L. 2001. “Random Forests.” Machine Learning 45: 5–32. DOI: https://doi.org/10.1023/A:1010933404324.

Buskirk, T.D., and S. Kolenikov. 2015. “Finding Respondents in the Forest: A Comparison of Logistic Regression and Random Forest Models for Response Propensity Weighting and Stratification.” Survey Methods: from the Field. DOI: http://doi.org/10.13094/SMIF-2015-00003.

Curtin, R., S. Presser, and E. Singer. 2000. “The Effects of Response Rate Changes on the Index of Consumer Sentiment.” Public Opinion Quarterly 64: 413–428. DOI: http://dx.doi.org/10.1086/318638.

Elliott, M.R., R.J.A. Little, and S. Lewitzky. 2000 “Subsampling Callbacks to Improve Survey Efficiency.” Journal of the American Statistical Association 95: 730–738. DOI: https://dx.doi.org/10.2307/2669453.

Gower, J.C. 1971. “A Coefficient of Similarity and Some of Its Properties.” Biometrics 27: 857–874. DOI: https://dx.doi.org/10.2307/2528823.

10.

Groves, R.M., 2006 “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70: 646–675. DOI: https://dx.doi.org/10.1093/poq/nfl033.

11.

Groves, R.M., and M.P. Couper. 1998. Nonresponse in Household Interview Surveys. New York: John Wiley and Sons.

12.

Groves, R.M., and S.G. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society 169: 439–457. DOI: https://doi.org/10.1111/j.1467-985X.2006.00423.x.

13.

Groves, R.M., and E. Peytcheva. 2006. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” In Proceedings of the 17th International Workshop on Household Survey Nonresponse, August 28–30, Omaha, NE, USA. DOI: https://doi.org/10.1093/poq/nfn011.

14.

Groves, R.M., S. Presser, and S. Dipko. 2004. “The Role of Topic Interest in Survey Participation Decisions.” Public Opinion Quarterly 68: 2–31. DOI: http://dx.doi.org/10.1093/poq/nfh002.

15.

Hansen, M.H., and W.N. Hurwitz. 1946. “The Problem of Nonresponse in Sample Surveys.” Journal of the American Statistical Association 41: 517–529.

16.

Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Second Edition, New Youk: Springer.

17.

Hedlin D. 2020. “Is There A ’Safe Area’ Where the Nonresponse Rate Has Only A Modest Effect on Bias Despite Non-Ignorable Nonresponse?” International Statistical Review 88: 642–657. DOI:. https://dx.doi.org/10.1111/insr.12359

18.

Holmberg, A. 2002. “A Multiparameter Perspective on the Choice of Sampling Design in Surveys.” Statistics in Transition 6: 969–994.

19.

Holmberg, A. 2003. “Using Auxiliary Information to Choose between Alternative Sampling Designs in A Survey with Several Key Variables.” In Proceedings of the Section on Challenges in Survey Taking for the Next Decade: Statistics Canada Symposium, October 29, Quebec: Available at: https://www150.statcan.gc.ca/n1/en/pub/11-522-x/2003001/session18/7726-eng.pdf?st=FCpnf4OS. (accessed February 2023).

20.

Holmberg, A., P. Flisberg, and M. Rönnqvist. 2003. On the choice of optimal design in business surveys with several important study variables. Uppsala: Uppsala University. Available at: https://www.diva-portal.org/smash/get/diva2:162729/FULLTEXT01.pdf (accessed February 2023).

21.

Kim, J.K., and J.J. Kim. 2014. “Nonresponse Weighting Adjustment Using Estimated Response Probability.” The Canadian Journal of Statistics 35:501–514. DOI: https://doi.org/10.1002/cjs.5550350403.

22.

Kim, J.K., and K. Morikawa. 2022. An Empirical Likelihood Approach to Reduce Selection Bias in Voluntary Samples. New York: Cornell University. Available at: https://arxiv.org/abs/2211.02998.

23.

Kim, J.K., and J. Shao. 2014. Statistical Methods for Handling Incomplete Data. London: Chapman and Hall.

24.

Lee, B.K., J. Lessler, and E.A. Stuart. 2010. “Improving Propensity Score Weighting Using Machine Learning.” Statistics in Medicine 29: 337–346. DOI: https://dx.doi.org/10.1002/sim.3782.

25.

Little, R.J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review 54: 139–157. DOI: https://dx.doi.org/10.2307/1403140.

26.

Little, R.J.A., and D.B. Rubin. 2019. Statistical Analysis of Missing Data. New York: John Wiley and Sons.

27.

Neusy, E., J.F. Beaumont, W. Yung, M. Hidiroglou, and D. Haziza 2022. “Nonresponse Follow-Up for Business Surveys.” Survey Methodology 48: 95–117. Available at: http://www.statcan.gc.ca/pub/12-001-x/2022001/article/00006-eng.htm.

28.

Oh, H.L., and F.J. Scheuren. 1983 “Weighting Adjustment for Unit Nonresponse.” Incomplete Data in Sample Surveys, 2: 143–184.

29.

Pfeffermann, D., and M. Sverchkov. 1999. “Parametric and Semiparametric Estimation of Regression Models Fitted to Survey Data.” The Indian Journal of Statistics 61: 166–186. Available at: https://www.jstor.org/stable/i25053064.

30.

Phipps, P., and D. Toth 2014. Regression Tree Models for Analyzing Survey Response. U.S. Bureau of Labour Statistics. Washington DC: BLS. Available at: https://www.bls.gov/osmr/research-articles/2014/pdf/st140160.pdf (accessed February 2023).

31.

Särndal, C.E., B. Swensson, and J. Wretman 1992. Model Assisted Sampling. New York: Springer.

32.

Särndal, C.E., and S. Lundström 2005. Estimation in Surveys with Nonresponse. New York: John Wiley and Sons.

33.

Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35: 101–113.

34.

Schouten, B., F. Cobben, and J. Bethlehem. 2011. Handbook of Nonresponse in Household Surveys. New York: John Wiley and Sons.

35.

Sikov, A. 2018. “A Brief Review of Approaches to Non-ignorable Nonresponse: Approaches to Non-ignorable Nonresponse.” International Statistical Review 86: 415–441. DOI: https://doi.org/10.1111/insr.12264.