An Appraisal of Common Reweighting Methods for Nonresponse in Household Surveys Based on the Norwegian Labour Force Survey and the Statistics on Income and Living Conditions Survey

Abstract

Despite increasing efforts during data collection, nonresponse remains sizeable in many household surveys. Statistical adjustment is hence unavoidable. By reweighting the design, weights of the respondents are adjusted to compensate for nonresponse. However, there is no consensus on how this should be carried out in general. Theoretical comparisons are inconclusive in the literature, and the associated simulation studies involve hypothetical situations not all equally relevant to reality. In this article we evaluate the three most common reweighting approaches in practice, based on real data in Norway from the two largest household surveys in the European Statistical System. We demonstrate how cross- examination of various reweighting estimators can help inform the effectiveness of the available auxiliary variables and the choice of the weight adjustment method.

Keywords

Unit nonresponse auxiliary variable selection: inverse propensity weighting generalised regression estimation doubly robust estimation

References

Andersson, P.G. and C.-E. Särndal. 2016. “Calibration for nonresponse treatment: in one or two steps?” Statistical Journal of the IAOS 32: 1–7.

Bang, H. and J.M. Robins. 2005. “Doubly robust estimation in missing data and causal inference models.” Biometrics 61: 962–972. DOI: https://doi.org/10.1111/j.1541-0420.2005.00377.x.

Bethlehem, J.G. 1988. “Reduction of nonresponse bias through regression estimation.” Journal of Official Statistics 4: 251–260. Available at: https://search.proquest.com/openview/90feaa08b1f293c8544d8576e6d94436/1?pq-origsite=gscholar&cbl=105444 (accessed February 2020).

Bethlehem, J., F. Cobben, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. Hoboken: Wiley.

Brick, M.J. 2013. “Unit nonresponse and weighting adjustments: A critical review.” Journal of Official Statistics 29: 329–353. DOI: https://doi.org/10.2478/jos-2013-0026.

Canty, A. and A. Davison. 1999. “Resampling-based Variance Estimation for Labour Force Surveys.” The Statistician 48: 379–391.

Carpenter, J.R., M.G. Kenward, and S. Vansteelandt. 2006. “A comparison of multiple imputation and doubly robust estimation for analyses with missing data.” J. R. Statist. Soc. A 169: 571–584. DOI: https://doi.org/10.1111/j.1467-985X.2006.00407.x.

Cassel, C.M., C.-E. Särndal, and J.H. Wretman. 1983. “Some uses of statistical models in connection with the nonresponse problem.” In Incomplete Data in Sample Surveys, edited by W.G. Madow, I. Olkin, and D.B. Rubin, vol. 3, 143–160. New York: Academic Press.

Cochran, W. 1953. Sampling Techniques. New York: Wiley.

10.

De Leeuw, E. and W. de Heer. 2002. “Trends in household survey nonresponse – a longitudinal and international comparison.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R.J.A. Little, 41–54. New York: Wiley. Available at: https://www.wiley.com/en-us/Survey+Nonresponse-p-9780471396277.

11.

Eltinge, J.L. and I.S. Yansaneh. 1997. “Diagnostics for formation of nonresponse adjustment cells, with an application to income nonresponse in the U.S. consumer expenditure survey.” Survey Methodology 23: 33–40. Available at: https://www150.statcan.gc.ca/n1/en/catalogue/12-001-X19970013103 (accessed February 2020).

12.

Haziza, D. and É. Lesage. 2016. “A discussion of weighting procedures for unit nonresponse.” Journal of Official Statistics 32: 129–145. DOI: https://doi.org/10.1515/jos-2016-0006.

13.

Kalton, G. and I. Flores-Cervantes. 2003. “Weighting methods.” Journal of Official Statistics 19: 81–97. Available at: https://www.scb.se/contentassets/ca21efb41-fee47d293bbee5bf7be7fb3/weighting-methods.pdf (accessed February 2020).

14.

Kalton, G. and D. Kasprzyk. 1986. “The treatment of missing survey data.” Survey Methodology 12: 1–16. Available at: https://www150.statcan.gc.ca/n1/en/catalogue/12-001-X198600114404 (accessed February 2020).

15.

Kang, J.D.Y. and J.L. Schafer. 2007. “Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data.” Statistical Science 22: 523–539. DOI: https://doi.org/10.1214/07-STS227.

16.

Kim, J. and D. Haziza. 2014. “Doubly robust inference with missing data in survey sampling.” Statistica Sinica 24: 375–394. DOI: http://doi.org/10.5705/ss.2012.005.

17.

Kott, P.S. and D. Liao. 2015. “One step or two? Calibration weighting from a complete list frame with nonresponse.” Survey Methodology 41: 165–181. Available at: https://www150.statcan.gc.ca/n1/pub/12-001-x/2015001/article/14172-eng.htm (accessed February 2020).

18.

Little, R.J.A. 1986. “Survey nonresponse adjustments for estimates of means.” International Statistical Review 54: 139–157. DOI: https://doi.org/10.2307/1403140.

19.

Little, R.J. and D.B. Rubin. 1987. Statistical Analysis with Missing Data. New York: Wiley.

20.

Little, R.J. and S. Vartivarian. 2005. “Does weighting for nonresponse increase the variance of survey means?” Survey Methodology 31: 161–168. Available at: https://www150.statcan.gc.ca/n1/en/catalogue/12-001-X20050029046 (accessed February 2020).

21.

Lunceford, J.K. and M. Davidian. 2004. “Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study.” Statist. Med. 23: 2937–2960. DOI: https://doi.org/10.1002/sim.1903.

22.

Lundström, S. and C.-E. Särndal. 1999. “Calibration as a Standard Method for Treatment of Nonresponse.” Journal of Official Statistics 15: 305–327. DOI: https://doi.org/10.1002/sim.1903.

23.

Meyer, B.D., W.K.C. Mok, and J.X. Sullivan. 2015. “Household surveys in crisis.” Journal of Economic Perspectives 29: 199–226. Available at: https://pubs.aeaweb.org/doi/pdf/10.1257/jep.29.4.199 (accessed September 2019).

24.

Oh, H.L. and F.S. Scheuren. 1983. “Weighting adjustments for unit nonresponse.” In Incomplete Data in Sample Surveys, edited by W.G. Madow, I. Olkin, and D.B. Rubin, vol. 2, 143–184. New York: Academic Press.

25.

Robins, B.J.M. and N. Wang. 2000. “Inference for imputation estimators.” Biometrika 87: 113–124. Available at: https://pdfs.semanticscholar.org/3214/538c562c010bdb-b055582e04fc3a4d7b6ce9.pdf (accessed February 2020).

26.

Robins, J.M., A. Rotnitzky, and L.P. Zhao. 1994. “Estimation of regression coefficients when some regressors are not always observed.” J. Am. Statist. Ass. 89: 846–866. DOI: https://doi.org/10.1080/01621459.1994.10476818.

27.

Särndal, C.-E. and S. Lundström. 2005. Estimation in Surveys with Nonresponse. Chichester: Wiley.

28.

Särndal, C.-E. and S. Lundström. 2008. “Assessing auxiliary vectors for control of nonresponse bias in the calibration estimator.” Journal of Official Statistics 24: 167–191. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/assessing-auxiliary-vectors-for-control-of-nonresponse-bias-in-the-calibration-estimator.pdf (accessed February 2020).

29.

Särndal, C.-E. and S. Lundström. 2010. “Design for estimation: Identifying auxiliary vectors to reduce nonresponse bias.” Survey Methodology 36: 131–144. Available at: https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2010002/article/11376-eng.pdf?st=XFmzYjVz (accessed February 2020).

30.

Schouten, B. 2007. “A selection strategy for weighting variables under a not-missing- at-random assumption.” Journal of Official Statistics 23: 51–68. Available at: https://pdfs.semanticscholar.org/634f/ca032e7c166eef11ee65b84192e5aca1e039.pdf (accessed February 2020).

31.

Stoop, I., J. Billiet, A. Koch, and R. Fitzgerald. 2010. Improving Survey Response: Lessons Learned from the European Social Survey. Chichester, UK: Wiley.

32.

Thomsen, I. 1973. “A note on the efficiency of weighting subclass means to reduce the effects of non-response when analyzing survey data.” Statistisk Tidskrift 11: 278–285. Available at: https://www.statistics.no/a/histstat/ano/ano_io73_02.pdf (accessed February 2020).

33.

Thomsen, I. 1978. “A second note on the efficiency of weighting subclass means to reduce the effects of non-response when analyzing survey data.” Statistisk Tidskrift 16: 278–285. Available at: https://www.statistics.no/a/histstat/ano/ano_io73_02.pdf (accessed February 2020).

34.

Thomsen, I. and L.-C. Zhang. 2001. “The effects of using administrative registers in economic short term statistics: The Norwegian Labour Force Survey as a case study.” Journal of Official Statistics 17: 285–294. Available at: https://www.scb.se/content-assets/ca21efb41fee47d293bbee5bf7be7fb3/the-effects-of-using-administrative-regis-ters-in-economic-short-term-statistics-the-norwegian-labour-force-survey-as-a-case-study.pdf (accessed February 2020).

35.

Zhang, L.-C. 1999. “A note on post-stratification when analyzing binary survey data subject to nonresponse.” Journal of Official Statistics 15: 329–334. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/a-note-on-post-stratification-when-analyzing-binary-survey-data-subject-to-nonresponse.pdf (accessed February 2020).

36.

Zhang, L.-C. 2005. “On the bias in gross labour flow estimates due to nonresponse and mis-classification.” Journal of Official Statistics 21: 591–604. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/on-the-bias-in-gross-labour-flow-estimates-due-to-nonresponse-and-misclassification.pdf (accessed February 2020).

37.

Zhang, L.-C., I. Thomsen, and Ø. Kleven. 2013. “On the use of auxiliary and para-data for dealing with non-sampling errors in household surveys.” International Statistical Review 81: 270–288. DOI: https://doi.org/10.1111/insr.12009.