Abstract
R-indicators are increasingly used as nonresponse bias indicators. However, their effectiveness depends on the auxiliary data used in their estimation. Because of this, it is not always clear for practitioners what the magnitude of the R-indicator implies for bias in other survey variables, or how adjustment on auxiliary variables will affect it. In this article, we investigate these potential limitations of R-indicators in a case study using data from the Swiss European Social Survey (ESS5), which included a nonresponse follow-up (NRFU) survey. First, we analyse correlations between estimated response propensities based on auxiliary data from the register-based sampling frame, and responses to survey questions also included in the NRFU. We then examine how these relate to bias detected by the NRFU, before and after adjustment, and to predictions of the risk of bias provided by the R-indicator. While the results lend support for the utility of R-indicators as summary statistics of bias risk, they suggest a need for caution in their interpretation. Even where auxiliary variables are correlated with target variables, more bias in the former (resulting in a larger R-indicator) does not automatically imply more bias in the latter, nor does adjustment on the former necessarily reduce bias in the latter.
References
1.
AAPOR, American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition.
2.
Bethlehem, J.G. 2002. “Weighting Non-response Adjustment Based on Auxiliary Information.” In Survey Non-response , edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.
3.
Beullens, K. and G. Loosveldt. 2012. “Should High response Rates Really be the Primary Objective?” Survey Practice 5(3): 1–5. DOI: https://doi.org/10.29115/SP-2012-0019.
4.
Beullens, K., G. Loosveldt, C. Vandenplas, and I. Stoop. 2018. “Response Rates in the European Social Survey: Increasing, Decreasing, or a Matter of Fieldwork Efforts?” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2018-00003.
5.
Brick, J.M. and M.E. Jones. 2008. “Propensity to respond and nonresponse bias.” METRON – International Journal of Statistics , LXVI(1), 51–73.
6.
Brick, J.M. and D. Williams. 2013. “Explaining rising Non-response Rates in Cross-sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. DOI: https://doi.org/10.1177/0002716212456834.
7.
Brick, J.M. and R. Tourangeau. 2017. “Responsive Survey Designs for Reducing Non-response Bias.” Journal of Official Statistics 33(3): 735–752. DOI: http://dx.doi.org/10.1515/JOS-2017-0034.
8.
Cobben, F. 2009. Non-response in Sample Surveys. Methods for Analysis and Adjustment . PhD thesis, The Hague: Statistics Netherlands. Available at: https://hdl.handle.-net/11245/1.312964 (accessed June 2020).
9.
Cornesse, C. and M. Bosnjak. 2018. “Is there an association between survey characteristics and representativeness? A meta-analysis.” Survey Research Methods 12(1): 1–13. DOI: https://doi.org/10.18148/srm/2018.v12i1.7205.
10.
De Heij, V., B. Schouten, and N. Shlomo. 2015. RISQ manual 2.1. Tools in SAS and R for the computation of R-indicators, partial R-indicators and partial coefficients of variation . RISQ Project. Available at: www.risq-project.eu.
11.
De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Non-response: A Longitudinal and International Comparison.” In Survey Non-response , edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.
12.
Ernst Stähli, M., D. Joye, M. Sapin, A. Pollien, M. Ochsner, and A. van den Hende. 2018. “Non Response Surveys (NRS): ESS 2006, EVS 2008, ESS 2010, MOSAiCH 2011, ESS 2012, ESS 2014.” [Dataset]. Distributed by FORS, Lausanne. https://doi.org/10.23662/FORS-DS-697-1.
13.
ESS Round 5: European Social Survey 2016: ESS-5 2010 Documentation Report. Edition 4.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/docs/round5/survey/ESS5_data_documentation_report_e04_2.pdf (accessed June 2020).
14.
ESS Round 5: European Social Survey Round 5 Data. 2010. Data file edition 3.4 . NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/download.html?file=ESS5CH&c=CH&y=2010.
15.
Groves, R.M. 2006. “Non-response Rates and Non-response Bias in Household surveys.” Public Opinion Quarterly 70, Special Issue: 646–675. DOI: https://doi.org/10.1093/poq/nfl033.
16.
Groves, R.M. and S. Heeringa. 2006. “Responsive design for household surveys: tools for actively controlling survey errors and costs.” Journal of the Royal Statistical Society : Series A (Statistics in Society), 169(3): 439–457. DOI: https://doi.org/10.1111/j.1467-985X.2006.00423.x
17.
Groves, R.M. and E. Peytcheva. 2008. “The Impact of Non-response Rates on Non-response Bias – A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. DOI: https://doi.org/10.1093/poq/nfn011.
18.
Groves, R.M. and M.P. Couper. 1998. Non-Response in Household Interview Survey . New York: John Wiley & Sons.
19.
Groves, R.M., J.M. Brick, M.P. Couper, W. Kalsbeek, B. Harris-Kojetin, F. Kreuter, B.-E. Pennell, T. Raghunathan, B. Schouten, T. Smith, R. Tourangeau, A. Bowers, M. Jans, C. Kennedy, R. Levenstein, K. Olson, E. Peytcheva, S. Ziniel, and J. Wagner. 2008. “Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools.” Survey Practice 1(3): 1–6. DOI: https://doi.org/10.29115/SP-2008-0013.
20.
Gummer, T. and J.E. Blummenstiel. 2018. “Experimental Evidence on Reducing Nonresponse Bias through Case Prioritization: The Allocation of Interviewers.” Field Methods 30(2): 124–139. DOI: https://doi.org/10.1177/1525822X18757967.
21.
Kreuter, F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645(1): 23–35. DOI: https://doi.org/10.1177/0002716212456815.
22.
Kreuter, F. and K. Olson. 2011. “Multiple Auxiliary Variables in Nonresponse Adjustment.” Sociological Methods & Research 40(2): 311–332. DOI: https://doi.org/10.1177/0049124111400042.
23.
Lin, I.-F. and N.C. Schaeffer. 1995. “Using Survey Participants to Estimate the Impact of Nonparticipation.” Public Opinion Quarterly 59(2): 236–258, DOI: https://doi.org/10.1086/269471.
24.
Little, R.J.A. and S. Vartivarian. 2005. “Does Weighting for Non-Response increase the Variance of Survey Means?” Survey Methodology 31(2): 161–68. DOI: https://doi.org/10.2307/1403140.
25.
Little, R.J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review / Revue Internationale de Statistique 54(2), 139–157. JSTOR. DOI: https://doi.org/10.2307/1403140.
26.
Little, R.J.A. and D.B. Rubin. 2014. Statistical Analysis with Missing Data . London, UK: John Wiley & Sons.
27.
Luiten, A. and B. Schouten, 2013. “Tailored Fieldwork Design to increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): 169–189. DOI: https://doi.org/10.1111/j.1467-985X.2012.01080.x.
28.
Matsuo, H., J. Billiet, G. Loosveldt, F. Berglund, and Ø. Kleven. 2010. “Measurement and Adjustment of Non-response Bias based on Non-response Survey: the Case of Belgium and Norway in the European Social Survey Round 3.” Survey Research Methods 4(3): 165–178. DOI: http://dx.doi.org/10.18148/srm/2010.v4i3.3774.
29.
McCaffrey, D.F., G. Ridgeway, and A.R. Morral. 2004. “Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.” Psychological Methods 9: 403–425. DOI: https://doi.org/10.1037/1082-989X.9.4.403.
30.
Moore, J.C., G.B. Durrant, and P.W. Smith. 2018. “Data Set Representativeness during Data Collection in Three UK Social Surveys: Generalizability and the Effects of Auxiliary Covariate Choice.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181(1): 229–248. DOI: https://doi.org/10.1111/rssa.12256.
31.
Nagelkerke, N.J. 1991. “A Note on a General Definition of the Coefficient of Determination.” Biometrika 78(3): 691–692. DOI: https://doi.org/10.1093/biomet/78.3.691.
32.
Nishimura, R., J. Wagner, and M. Elliott. 2016. “Alternative Indicators for the Risk of Non-response Bias: A Simulation Study.” International Statistical Review 84(1): 43–62. DOI: https://doi.org/10.1111/insr.12100.
33.
Olmos, A. and P. Govindsamy. 2015. “A Practical Guide for using Propensity Score Weighting in R.” Practical Assessment, Research & Evaluation , 20(13). Available at: http://pareonline.net/getvn.asp?v=20&n=13. (accessed June 2020).
34.
Olson, K. 2006. “Survey participation, Non-Response Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. DOI: https://doi.org/10.1093/-poq/nfl038.
35.
Olson, K. 2013. “Do Non-response Follow-ups Improve or Reduce Data Quality?: A Review of the Existing Literature.” Journal of the Royal Statistical Society: Series A: Statistics in Society 176(1): 129–145. DOI: http://doi.org/10.1111/j.1467-985X.2012.01042.x.
36.
Peytchev, A., E. Peytcheva, and R.M. Groves. 2010. “Measurement Error, Unit Nonresponse, and Self-Reports of Abortion Experiences.” Public Opinion Quarterly 74: 319–327. DOI: https://doi.org/10.1093/poq/nfq002.
37.
Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Non-response Bias in Survey Estimates.” Journal of Official Statistics 25: 193–201. Available at: https://www.researchgate.net/publication/282119961_Using_Variation_in_Response_Rates_of_Demographic_Subgroups_as_Evidence_of_Nonresponse_Bias_in_Survey_Estimates (accessed June 2020).
38.
Roberts, C. and C. Vandenplas. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2): 303–334. DOI: https://doi.org/10.1515/jos-2017-0016.
39.
Roberts, C., C. Vandenplas, and M. Ernst Stähli. 2014a. “Using Register Data to assess the Impact of Response Enhancement Methods on the Risk of Non-response Bias.” Survey Research Methods 8(2): 67–80. DOI: http://dx.doi.org/10.18148/srm/2014.v8i2.5459.
40.
Roberts, C., N. Allum, and P. Sturgis. 2014b. “Non-response and Measurement error in an online panel: Does additional Effort to Recruit Reluctant Respondents Result in Poorer Quality Data?” In Online Panel Research: A Data Quality Perspective . edited by M. Callegaro, R. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas. Hoboken: Wiley, Survey Methodology Series.
41.
Sakshaug, J.F. and M. Antoni. 2018. “Evaluating the Utility of Indirectly Linked Federal Administrative Records for Nonresponse Bias Adjustment.” Journal of Survey Statistics and Methodology 7(2): 227–249. DOI: https://doi.org/10.1093/jssam/smy009.
42.
Särndal, C.-E. and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36(2): 131–144.
43.
Schouten, B. 2018. “Statistical inference based on randomly generated auxiliary variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80(1): 33–56. DOI: https://doi.org/10.1111/rssb.12242.
44.
Schouten, B. and N. Shlomo. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85(1): 143–163. DOI: https://doi.org/10.1111/insr.12159.
45.
Schouten, B. and F. Cobben. 2007. “R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes.” Statistics Netherlands . Discussion Paper , 07002, CBC, Voorburg. Available at: http://hummedia.manchester.ac.uk/institutes/cmist/risq/schouten-cobben-2007-a.pdf (accessed June 2020).
46.
Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35(1): 101–113. Available at: https://pdfs.semanticscholar.org/aa59/4bf03a7cc219ccc6da01d1e3cb14a125d67a.pdf (accessed June 2020).
47.
Schouten, B., F. Cobben, P. Lundquist, and J. Wagner. 2016. “Does more Balanced Response imply less Non-response Bias?” Journal of the Royal Statistical Society, Series A (Statistics in Society) 179(3): 727–748. DOI: https://doi.org/10.1111/rssa.12152.
48.
Schouten, B., J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80(3): 382–399. DOI: https://doi.org/10.1111/j.1751-5823.2012.00189.x.
49.
Schouten, B., N. Shlomo, and C. Skinner. 2010. “Indicators for Representative Response.” Paper presented at Quality in Official Statistics Conference 2010, Helinski, Finland. Available at: https://q2010.stat.fi/ (accessed June 2020).
50.
Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27(2): 231–253. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/indicators-for-monitoring-and-improving-representativeness-of-response.pdf (accessed June 2020).
51.
Stoop, I.A. 2004. “Surveying Nonrespondents.” Field Methods 16(1): 23–54. DOI: https://doi.org/10.1177/1525822X03259479.
52.
Stoop, I., J. Billiet, A. Koch, and R. Fitzerald. 2010. Improving Survey Response: Lessons Learned from the European Social Survey . London, UK: John Wiley and Sons Ltd.
53.
Tancreto, J.G. and M. Bentley. 2005. “Determining the Effectiveness of Multiple Non-response Follow-up Contact Attempts on Response and Data Quality.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2005. 3626–3632. Minneapolis, MN: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2005f.html (accessed June 2020).
54.
Vandenplas, C., D. Joye, M. Ernst Stähli, and A. Pollien. 2015. “Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland.” Survey Research Methods 9(3): 141–158. DOI: https://doi.org/10.18148/srm/2015.v9i3.6138.
55.
Voogt, R.J.J. and W.E. Saris, 2005. “Mixed Mode Designs: Finding the Balance Between Non-response Bias and Mode Effects.” Journal of Official Statistics , 21(3): 367–387. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/-mixed-mode-designs-finding-the-balance-between-nonresponse-bias-and-mode-effects.pdf (accessed June 2020).
56.
Wagner, J. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76(3): 555–575. DOI: https://doi.org/10.1093/poq/nfs032.
57.
Williams, D. and J.M. Brick. 2017. “Trends in U.S. Face-To-Face Household Survey Nonresponse and Level of Effort.” Journal of Survey Statistics and Methodology 6(2): 186–211. DOI: https://doi.org/10.1093/jssam/smx019.
58.
Yan, T., R. Tourangeau, and Z. Arens. 2004. “When less is more: Are reluctant respondents poor reporters?” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2004. 4632–4651. Toronto: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2004/files/Jsm2004-000169.pdf (accessed July 2020).