Adaptive survey design refers to using targeted procedures to recruit different sampled cases. This technique strives to reduce bias and variance of survey estimates by trying to recruit a larger and more balanced set of respondents. However, it is not well understood how adaptive design can improve data and survey estimates beyond the well-established post-survey adjustment. This paper reports the results of an experiment that evaluated the additional effect of adaptive design to post-survey adjustments. The experiment was conducted in the Detroit Metro Area Communities Study in 2021. We evaluated the adaptive design in five outcomes: 1) response rates, 2) demographic composition of respondents, 3) bias and variance of key survey estimates, 4) changes in significant results of regression models, and 5) costs. The most significant benefit of the adaptive design was its ability to generate more efficient survey estimates with smaller variances and smaller design effects.
American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, (Ninth Edition). Retrieved from: https://www.aapor.org/Standards-Ethics/Standard-Definitions-(1).aspx.
2.
AndridgeR. R.LittleR. J.. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics27(2):153.
3.
AxinnW. G.LinkC. F.GrovesR. M.. 2011. “Responsive Survey Design, Demographic Data Collection, and Models of Demographic Behavior.” Demography48(3):1127-49.
4.
BeaumontJ. F.BocciC.HazizaD.. 2014. “An Adaptive Data Collection Procedure for Call Prioritization.” Journal of Official Statistics30(4):607-21.
5.
BergmannM.ScherpenzeelA.. 2020. “Using Field Monitoring Strategies to Improve Panel Sample Representativeness: Application During Data Collection in the Survey of Health, Ageing and Retirement in Europe (SHARE).” Survey Methods: Insights from the Field. 10.13094/SMIF-2020-00003.
6.
BrickJ. M.2013. “Unit Nonresponse and Weighting Adjustments: A Critical Review.” Journal of Official Statistics29(3):329-53.
7.
BrickJ. M.KennedyC.Flores CervantesI.MercerA.. 2021. “An Adaptive Mode Adjustment for Multimode Household Surveys. Journal of Survey Statistics and Methodology 00:1-24.
8.
BrickJ. M.TourangeauR.. 2017. “Responsive Survey Designs for Reducing Nonresponse Bias.” Journal of Official Statistics33(3):735-52.
9.
BurgerJ.PerryckK.SchoutenB.. 2017. “Robustness of Adaptive Survey Designs to Inaccuracy of Design Parameters.” Journal of Official Statistics33(3):687-708.
10.
ChristensenA. I.LynnP.TolstrupJ. S.. 2019. “Can Targeted Cover Letters Improve Participation in Health Surveys? Results from a Randomized Controlled Trial.” BMC Medical Research Methodology19(1):1-8.
11.
ChunA. Y.HeeringaS. G.SchoutenB.. 2018. “Responsive and Adaptive Design for Survey Optimization.” Journal of Official Statistics34(3):581-97.
12.
CoffeyS.ReistB.MillerP. V.. 2019. “Interventions on-Call: Dynamic Adaptive Design in the2015 National Survey of College Graduates.” Journal of Survey Statistics and Methodology8(4):726-47.
13.
FumagalliL.LaurieH.LynnP.. 2013. “Experiments with Methods to Reduce Attrition in Longitudinal Surveys.” Journal of the Royal Statistical Society: Series A (Statistics in Society)176(2):499-519.
14.
GrovesR. M.HeeringaS. G.. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society)169(3):439-57.
15.
GrovesR. M.McGonagleK. A.. 2001. “A Theory-Guided Interviewer Training Protocol Regarding'survey Participation.” Journal of Official Statistics17(2):249-65.
16.
GrovesR. M.SingerE.CorningA.. 2000. “Leverage-saliency Theory of Survey Participation: Description and an Illustration.” Public Opinion Quarterly64(3):299-308.
17.
HanD.MontaquilaJ. M.BrickJ. M.. 2013. “An Evaluation of Incentive Experiments in a two-Phase Address-Based Sample Mail Survey.” Survey Research Methods7(3):207-18.
18.
JacksonM. T.McpheeC. B.LavrakasP. J.. 2020. “Using Response Propensity Modeling to Allocate Noncontingent Incentives in an Address-Based Sample: Evidence from a National Experiment.” Journal of Survey Statistics and Methodology8(2):385-411.
19.
KaltonG.Flores-CervantesI.. 2003. “Weighting Methods.” Journal of Official Statistics19(2):81-97.
20.
Kish, L. 1995. Survey Sampling. New York: Wiley.
21.
LavrakasP. J.JacksonM.McPheeC.. 2018. “The use of Response Propensity Modeling (RPM) for Allocating Differential Survey Recruitment Strategies: Purpose, Rationale, and Implementation.” Survey Practice11(2). https://doi.org/10.29115/SP-2018-0023.
22.
LinkM. W.BurksA. T.. 2013. “Leveraging Auxiliary Data, Differential Incentives, and Survey Mode to Target Hard-to-Reach Groups in an Address-Based Sample Design.” Public Opinion Quarterly77(3):696-713.
23.
LittleR. J.VartivarianS.. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?” Survey Methodology31(2):161-8.
24.
LittleR. J.WestB. T.BoonstraP. S.HuJ.. 2020. “Measures of the Degree of Departure from Ignorable Sample Selection.” Journal of Survey Statistics and Methodology8(5):932-64.
25.
LiuM.KuriakoseN.CohenJ.ChoS.. 2016. “Impact of web Survey Invitation Design on Survey Participation, Respondents, and Survey Responses.” Social Science Computer Review34(5):631-44.
26.
LuitenA.SchoutenB.. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society)176(1):169-89.
27.
LynnP.2016. “Targeted Appeals for Participation in Letters to Panel Survey Members.” Public Opinion Quarterly80(3):771-82.
28.
LynnP.2017. “From Standardised to Targeted Survey Procedures for Tackling non-Response and Attrition.” Survey Research Methods11(1):93-103.
29.
MercerA.CaporasoA.CantorD.TownsendR.. 2015. “How Much Gets you how Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly79(1):105-29.
30.
MurphyJ.BiemerP.BerryC.. 2018. “Transitioning a Survey to Self-Administration Using Adaptive, Responsive, and Tailored (ART) Design Principles and Data Visualization.” Journal of Official Statistics34(3):625-48.
31.
OlsonK. M.WagnerJ.AndersonR.. 2020. “Survey Costs: Where Are We and What Is the Way Forward?.” Journal of Survey Statistics and Methodology 9(5):921-42.
32.
PeytchevA.2013. “Consequences of Survey Nonresponse.” The ANNALS of the American Academy of Political and Social Science645(1):88-111.
33.
PeytchevA.PrattD.DupreyM.. 2020. “Responsive and Adaptive Survey Design: Use of Bias Propensity During Data Collection to Reduce Nonresponse Bias.” Journal of Survey Statistics and Methodology. 10(1): 131-48.
34.
RosenJ. A.MurphyJ.PeytchevA.HolderT.DeverJ.HergetD.PrattD.. 2014. “Prioritizing low Propensity Sample Members in a Survey: Implications for Nonresponse Bias.” Survey Practice7(1):1-8.
35.
SärndalC.-E.LundquistP.. 2014. “Accuracy in Estimation with Nonresponse: A Function of Degree of Imbalance and Degree of Explanation.” Journal of Survey Statistics and Methodology2(4):361-87.
36.
SärndalC.-E.LundquistP.. 2017. “Inconsistent Regression and Nonresponse Bias: Exploring Their Relationship as a Function of Response Imbalance.” Journal of Official Statistics33(3):709-34.
37.
SärndalC.-E.LundquistP.. 2019. “An Assessment of Accuracy Improvement by Adaptive Survey Design.” Survey Methodology45(2):317-37.
38.
SauerbreiW.SchumacherM.. 1992. “A Bootstrap Resampling Procedure for Model Building: Application to the Cox Regression Model.” Statistics in Medicine11(16):2093-109.
39.
SchoutenB.CobbenF.BethlehemJ.. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology35(1):101-13.
40.
SchoutenB.CobbenF.LundquistP.WagnerJ.. 2016. “Does More Balanced Survey Responseimply Less non-Response Bias?” Journal of the Royal Statistical Society: Series A (Statisticsin Society)179(3):727-48.
SingerE.GrovesR. M.CorningA. D.. 1999. “Differential Incentives: Beliefs About Practices, Perceptions of Equity, and Effects on Survey Participation.” Public Opinion Quarterly63(2):251-60.
43.
SingerE.YeC.. 2013. “The use and Effects of Incentives in Surveys.” The ANNALS of the American Academy of Political and Social Science645(1):112-41.
44.
TourangeauR.Michael BrickJ.LohrS.LiJ.. 2017. “Adaptive and Responsive Survey Designs: A Review and Assessment.” Journal of the Royal Statistical Society: Series A (Statistics in Society)180(1):203-23.
45.
van BerkelK.van der DoefS.SchoutenB.. 2020. “Implementing Adaptive Survey Design With an Application to the Dutch Health Survey.” Journal of Official Statistics36(3):609-29.
46.
WagnerJ.2008. Adaptive survey design to reduce nonresponse bias. PhD thesis, University of Michigan, Ann Arbor, USA.
47.
WagnerJ.WestB. T.KirgisN.LepkowskiJ. M.AxinnW. G.Kruger-NdiayeS. K.. 2012. “Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection.” Journal of Official Statistics28(4):477-99.
48.
West, B. T., W. Chang, and A. Zmich. 2021. “An Experimental Evaluation of Alternative Methods for Case Prioritization in Responsive Survey Design.” Journal of Survey Statistics and Methodology 00:1-22.
49.
Zhang, S. 2022. “Benefits of Adaptive Design Under Suboptimal Scenarios: A Simulation Study.” Journal of Survey Statistics and Methodology 00:1-31.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
