Assessing and Adjusting Bias Due to Mixed-Mode in Aspect of Daily Life Survey

Abstract

The mixed-mode (MM) designs are adopted by NSIs both to contrast declining response and coverage rates and to reduce the cost of the surveys. However, MM introduces several issues that must be addressed both at the design phase, by defining the best collection instruments to contain the measurement error, and at the estimation phase, by assessing and adjusting the mode effect. In the MM surveys, the mode effect refers to the introduction of bias effects on the estimate of the parameters of interest due to the difference in the selection and measurement errors specific to each mode. The switching of a survey from single to mixed-mode is a delicate operation: the accuracy of the estimates must be ensured in order to preserve their consistency and comparability over time. This work focuses on the methods chosen for the evaluation of the mode effect in the Italian National Institute of Statistics (ISTAT) mixed-mode survey “Aspects of Daily Life – 2017”, in the experimental context for which an independent control single-mode (SM) PAPI sample was planned to assess the introduction of the sequential web/PAPI survey. The presented methods aim to analyze the causes that can determine significant differences in the estimates obtained with the SM and MM surveys.

Keywords

Mixed-mode mode effect instrumental variable approach propensity score

References

Austin, P.C., and D.S. Small. 2014. “The use of bootstrapping when using propensity-score matching without replacement: a simulation study.” Statistics in Medicine 33(24) : 4306–4319. Wiley Online Library. DOI: https://doi.org/10.1002/sim.6276.

Ballabio, S., A. Carra, S. Casacci, D. Ferrazza, F. Verrecchia, A. Vitalini, and L.C. Viviano. 2018. “Local decisions and new guidelines of the Official Statistics.” In Proceedings of Q2018 European Conference on Quality in Official Statistics, June 2018, Cracovia, Poland.

Bethlehem, J., F. Cobben, and B. Schouten. 2011. Handbook of nonresponse in household surveys. Wiley Series in Survey Methodology. Hoboken, New Jersey: John Wiley & Sons, Inc. DOI: https://doi.org/10.1002/9780470891056.

Buelens, B., and J.A. van den Brakel. 2014. “Measurement error calibration in mixed-mode Sample Surveys.” Sociological methods and Research 44(3) : 391–426. DOI: https://doi.org/10.1177/0049124114532444.

Buelens, B., J.A. van den Brakel, and B. Schouten. 2018. “Current methodologies to deal with mode effects and mode bias in multi-mode designs.” MIMOD Deliverable 1 – WP2. Available at: https://www.istat.it/en/research-activity/international-research-activity/essnet-and-grants (accessed May 2021).

De Leeuw, E.D. 2005. “To mix or not to mix data collection modes in surveys.” Journal of Official Statistics 21(2) : 233–255. Available at: https://www.scb.se/contentassets/-ca21efb41fee47d293bbee5bf7be7fb3/to-mix-or-not-to-mix-data-collection-modes-in-surveys.pdf (accessed May 2021).

Efron, B. 1979. “Bootstrap methods: Another look at the jackknife.” The Annals of Statistics 7(1) : 1–26. DOI: https://doi.org/10.1214/aos/1176344552.

Efron, B., and R. Tibshirani. 1986. “Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy.” Statistical Science 1(1) : 54–75. DOI: https://doi.org/10.1214/ss/1177013815.

Eurostat. 2014. ESS Guidelines for the Implementation of the ESS Quality and Performance Indicators (QPI). Luxembourg ESTAT / D4/LA D (2014). Available at: https://ec.europa.eu/eurostat/documents/64157/4373903/02-ESS-Quality-and-performance-Indicators-2014.pdf/5c996003-b770-4a7c-9c2f-bf733e6b1f31 (accessed May 2021).

10.

Garofalo, G. 2014. “Il Progetto ARCHIMEDE obiettivi e risultati sperimentali.” Istat Working Paper. Available at: https://www.istat.it/it/files/2014/11/IWP-n.-9-2014.pdf (accessed February 2015).

11.

Hox, J., E.D. de Leeuw, and T. Klausch. 2015. “Mixed Mode Research: Issues in Design and Analysis.” In Total Survey Error in Practice, edited by P.P. Biemer et al.: 511–530. New York: John Wiley & Sons. DOI: http://dx.doi.org/10.1002/9781119041702.ch23.

12.

ISTAT (2019). “Aspetti metodologici dellindagine. Aspetti della vita quotidiana Periodo di riferimento: anno 2017”. Rome: ISTAT. Available at: https://www.istat.it/microdata/download.php?id=/import/fs/pub/wwwarmida/60/2017/01/Nota.pdf (accessed May 2021).

13.

Martin, P., and P. Lynn. 2011. “The effects of mixed mode survey designs on simple and complex analyses.” ISER Working Paper Series 2011–28. Available at: https://ideas.repec.org/p/ese/iserwp/2011-28.html (accessed May 2021).

14.

Rosenbaum, P.R., and D.B. Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.” Biometrika 70: 41–55. DOI: https://doi.org/10.1093/biomet/70.1.41.

15.

Rust, K.F., and J.N.K. Rao. 1996. “Variance estimation for complex surveys using replication techniques.” In Statistical methods in medical research 5(3) : 283–310. DOI: https://doi.org/10.1177%2F096228029600500305.

16.

Schouten, B.F., F. Cobben, and J. Bethlehem. 2009. “Indicators for the representativeness of survey response.” In Survey Methodology 35(1) : 101–113. Available at: Indicators for the representativeness of survey response (statcan.gc.ca).

17.

Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativity of Response.” Journal of Official Statistics 27: 231–253. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/indicators-for-monitoring-and-improving-representativeness-of-response.pdf (accessed May 2021).

18.

Tourangeau, R., and T. Yan. 2007. “Sensitive questions in surveys.” Psychological Bulletin 133(5) : 859–883. DOI: https://doi.org/10.1037/0033-2909.133.5.859.

19.

Vandenplas, C., G. Loosveldt, and J.T.A. Vannieuwenhuyze. 2016. “Assessing the use of mode preference as a covariate for the estimation of measurement effects between modes. A sequential mixed mode experiment.” Methods, data, analyses (mda) 10(2) : 119–142. DOI: https://doi.org/10.12758/mda.2016.011.

20.

Vannieuwenhuyze, J.T A., G. Loosveldt, and G. Molenberghs. 2010. “A Method for Evaluating Mode Effects in Mixed-mode Surveys.” Public Opinion Quarterly 74(5) : 1027–1045. DOI: https://doi.org/10.1093/poq/nfq059.

21.

Wolter, K.M. 2007. Introduction to variance estimation. Statistical for Social and Behavioral Science. New York: Springer Science & Business Media, LLC.