Small Domain Estimation of Census Coverage – A Case Study in Bayesian Analysis of Complex Survey Data

Abstract

Many countries conduct a full census survey to report official population statistics. As no census survey ever achieves 100% response rate, a post-enumeration survey (PES) is usually conducted and analysed to assess census coverage and produce official population estimates by geographic area and demographic attributes. Considering the usually small size of PES, direct estimation at the desired level of disaggregation is not feasible. Design-based estimation with sampling weight adjustment is a commonly used method but is difficult to implement when survey nonresponse patterns cannot be fully documented and population benchmarks are not available. We overcome these limitations with a fully model-based Bayesian approach applied to the New Zealand PES. Although theory for the Bayesian treatment of complex surveys has been described, published applications of individual level Bayesian models for complex survey data remain scarce. We provide such an application through a case study of the 2018 census and PES surveys. We implement a multilevel model that accounts for the complex design of PES. We then illustrate how mixed posterior predictive checking and cross-validation can assist with model building and model selection. Finally, we discuss potential methodological improvements to the model and potential solutions to mitigate dependence between the two surveys.

References

Atkinson, J., C. Salmond, and P. Crampton. 2019. NZ Dep 2013 index of deprivation interim research report. Technical report, Department of Public Health, University of Otago,Wellington. Available at: https://www.otago.ac.nz/wellington/otago730394.pdf. (accessed February 2022).

Brown, J.J., C. Sexton, O. Abbott, and P.A. Smith. 2019. “The framework for estimating coverage in the 2011 census of England and Wales: Combining dual-system estimation with ratio estimation.” Statistical Journal of the International Association of Official Statistics 35: 481–499. DOI: https://doi.org/10.3233/SJI-180426.

Bryant, J., K. Dunstan, P. Graham, N. Matheson-Dunning, E. Shrosbree, and R. Speirs. 2016. Measuring uncertainty in the 2013-base estimated resident population (Statistics New Zealand Working paper: 16–04). Wellington, New Zealand: Statistics New Zealand. Available at: https://www.stats.govt.nz/ (accessed February 2022).

Chandrasekar, C. and W.E. Deming. 1949. “On a method of estimating birth and death rates and the extent of registration.” Journal of the American Statistical Association 44: 101–115. DOI: https://doi.org/10.1080/01621459.1949.10483294.

Chen, C., J. Wakefield, and T. Lumley. 2014. “The use of sampling weights in Bayesian hierarchical models for small area estimation.” Spatial and Spatio-Temporal Epidemiology 11: 33–43. DOI: https://doi.org/10.1016/j.sste.2014.07.002.

Chen, S.X., C.Y. Tang, and V.T. Mule Jr. 2010. “Local post-stratification in dual system accuracy and coverage evaluation for the US census.” Journal of the American Statistical Association 105: 105–119. DOI: https://doi.org/10.1198/jasa.2009.ap08404.

Chipperfield, J., J. Brown, and P. Bell. 2017. “Estimating the count error in the Australian census.” Journal of Official Statistics 33: 43–59. DOI: https://doi.org/10.1515/jos-2017-0003.

Elliott, M.R. and R.J. Little. 2000. “A Bayesian approach to combining information from a census, a coverage measurement survey, and demographic analysis.” Journal of the American Statistical Association 95 (450): 351–362. DOI: https://doi.org/10.1080/01621459.2000.10474205.

Gelman, A., J. Carlin, H. Stern, D. Dunson, D. Vehtari, and A. Rubin. 2014. Bayesian Data Analysis. Boca Raton, FL.: CRC Press.

10.

Gelman, A., and J. Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical models. Cambridge: Cambridge university press.

11.

Gelman, A., A. Jakulin, M.G. Pittau, and Y.-S. Su. 2008. “A weakly informative default prior distribution for logistic and other regression models.” Annals of Applied Statistics 2: 1360–1383. DOI: https://doi.org/10.1214/08-A0AS191.

12.

Gelman, A. and T.C. Little. 1997. “Poststratification into many categories using hierarchical logistic regression.” Survey Methodology 23: 127–135. Available at: https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1997002/article/3616-eng.pdf?st=76F1g34m (accessed July 2022).

13.

Gelman, A., X.-L. Meng, and H. Stern. 1996. “Posterior predictive assessment of model fitness via realized discrepancies.” Statistica Sinica 6: 733–760.

14.

Ghitza, Y., and A. Gelman. 2013. “Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups.” American Journal of Political Science 57: 762–776. DOI: https://doi.org/10.1111/ajps.12004.

15.

Ghosh, M., K. Natarajan, T. Stroud, and B.P. Carlin. 1998. “Generalized linear models for small-area estimation.” Journal of the American Statistical Association 93: 273–282. DOI: https://doi.org/10.1080/01621459.1998.10474108.

16.

Hogan, H.P. 1993. “The 1990 post-enumeration survey: Operations and results.” Journal of the American Statistical Association 88: 1047–1060. DOI: https://doi.org/10.1080/01621459.1993.10476374.

17.

Lax, J.R., and J.H. Phillips. 2009. “How should we estimate public opinion in the States?” American Journal of Political Science 53: 107–121. DOI: https://doi.org/10.1111/j.1540-5907.2008.00360.x.

18.

Little, R.J. 2003. “The Bayesian approach to sample survey inference.” In Analysis of Complex Surveys, edited by R. Chambers and C. Skinner: 49–57. John Wiley and Sons.

19.

Lumley, T., and A. Scott. 2017. “Fitting regression models to survey data.” Statistical Science 32: 265–278. DOI: https://doi.org/10.1214/16-STS605.

20.

Molina, I., B. Nandram, and J. Rao. 2014. “Small area estimation of general parameters with application to poverty indicators: a hierarchical Bayes approach.” Annals of Applied Statistics 8: 852–885. DOI: https://doi.org/10.1214/13-A0AS702.

21.

Mule, T., T. Schellhamer, D. Malec, and J. Maples. 2008. “Using continuous variables as modeling covariates for net coverage estimation.” In JSM Proceedings: Section on Survey Research Methods: 1941–1948. Denver. Available at: http://www.asasrms.org/Proceedings/y2008/Files/301279.pdf (accessed February 2022).

22.

Nandram, B., L. Chen, and B. Manandhar. 2018. “Bayesian analysis of multinomial counts from small areas and sub-areas.” In JSM proceedings: Section on Survey Research Methods: 1140–1162. Vancouver. Available at: http://www.asasrms.org/Proceedings/y2018/files/867100.pdf (accessed February 2022).

23.

Paige, J., G.-A. Fuglstad, A. Riebler, and J. Wakefield. 2020. “Design-and model-based approaches to small-area estimation in a low and middle income country context: comparisons and recommendations.” Journal of Survey Statistics and Methodology. DOI: https://doi.org/10.1093/jssam/smaa011.

24.

Pavlou, M., G. Ambler, S. Seaman, and R.Z. Omar. 2015. “A note on obtaining correct marginal predictions from a random intercepts model for binary outcomes.” BMC Medical Research Methodology 15: 1–6. DOI: http://doi.org/10.1186/s12874-015-0046-6.

25.

Pfeffermann, D. 2013. “New important developments in small area estimation.” Statistical Science 28: 40–68. DOI: https://doi.org/10.1214/12-STS395.

26.

Pfeffermann, D., F.A.D.S. Moura, and P.L.D.N. Silva. 2006. “Multilevel modelling under informative sampling.” Biometrika 93: 943–959. DOI: https://doi.org/10.1093/biomet/93.4.943.

27.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Available at: https://www.R-project.org/ (accessed February 2022).

28.

Rabe-Hesketh, S., and A. Skrondal. 2006. “Multilevel modelling of complex survey data.” Journal of the Royal Statistical Society: 169: 805–827. DOI: https://doi.org/10.1111/j.1467-985X.2006.00426.x.

29.

Rao, J. and I. Molina. 2014. Small-area estimation. Hoboken, NJ: John Wiley & Sons, Inc.

30.

Rao, J., F. Verret, and M.A. Hidiroglou. 2013. “A weighted composite likelihood approach to inference for two-level models from survey data.” Survey Methodology 39(2): 263–282. Available at: https://www150.statcan.gc.ca/n1/en/pub/12-001-x/12-001-x2013002-eng.pdf?st=LsDJmvSV (accessed July 2022).

31.

Rubin, D.B. 1987. Multiple imputation for nonresponse in surveys. Hoboken, NJ: John Wiley & Sons.

32.

Shirley, K.E., and A. Gelman. 2015. “Hierarchical models for estimating state and demographic trends in US death penalty public opinion.” Journal of the Royal Statistical Society 178: 1–28. DOI: https://doi.org/10.1111/rssa.12052.

33.

Si, Y., R. Trangucci, J.S. Gabry, and A. Gelman. 2020. “Bayesian hierarchical weighting adjustment and survey inference.” Survey Methodology 46: 181–214. Available at: https://www150.statcan.gc.ca/n1/pub/12-001-x/2020002/article/00003-eng.htm. (accessed July 2022).

34.

Skrondal, A., and S. Rabe-Hesketh. 2009. “Prediction in multilevel generalized linear models.” Journal of the Royal Statistical Society: 172: 659–687. DOI: https://doi.org/10.1111/j.1467-985X.2009.00587.x.

35.

Stan Development Team. 2020a. R Stan: the R interface to Stan. R package version 2.21.2. Available at: http://mc-stan.org/ (accessed February 2022). Stan Development Team. 2020b.

36.

Stan Modeling Language Users Guide and Reference Manual, version 2.25. Available at: http://mc-stan.org/ (accessed February 2022).

37.

Stats NZ. 2014. Coverage in the 2013 Census based on the New Zealand 2013 Post-enumeration Survey. Wellington: Statistics New Zealand. Available at: https://www.stats.govt.nz/. (accessed February 2022).

38.

Stats NZ. 2019. Overview of statistical methods for adding admin records to the 2018 Census dataset. Wellington, NZ: Statistics New Zealand. Available at: https://www.stats.govt.nz/ (accessed February 2022).

39.

Stats NZ 2020a. Estimated resident population 2018: Data sources and methods. Wellington, NZ: Statistics New Zealand. Available at: https://www.stats.govt.nz/ (accessed February 2022).

40.

Stats NZ. 2020b. Post-enumeration survey 2018: Methods and Results. Wellington, NZ: Statistics New Zealand. Available at: https://www.stats.govt.nz/ (accessed February 2022).

41.

Vehtari, A., A. Gelman, and J. Gabry. 2017. “Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC”. Statistics and Computing 27: 1413–1432. DOI: https://doi.org/10.1007/s11222-016-9696-4.

42.

Yi, G.Y., J. Rao, and H. Li. 2016. “A weighted composite likelihood approach for analysis of survey data under two-level models.” Statistica Sinica 26: 569–587. DOI: https://doi.org/10.5705/ss.2013.383.

43.

You, Y., and B. Chapman. 2006. “Small area estimation using area level models and estimated sampling variances.” Survey Methodology 32: 97–104. Available at: https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006001/article/9263-eng.pdf?st=a4rH5VTf (accessed July 2022).

44.

You, Y. and P. Dick. 2004. “Hierarchical Bayes small area inference to the 2001 census undercoverage estimation.” In JSM Proceedings: Section on Government Statistics: 1836–1840. Available at: http://www.asasrms.org/Proceedings/y2004/files/Jsm2004-000377.pdf (accessed February 2022).