Sage Journals: Discover world-class research

Abstract

In a probability sampling survey, adaptive data collection strategies may be used to obtain a response set that minimizes nonresponse bias within budget constraints. Previous research has stratified the target population into subgroups defined by categories of auxiliary variables observed for the entire population, and tailored strategies to obtain similar response rates across subgroups. However, if the auxiliary variables are weakly correlated with the target survey variables, optimizing data collection for these subgroups may not reduce nonresponse bias and may actually increase the variance of survey estimates. In this paper, we propose a stratification method to identify subgroups by: (1) predicting values of target survey variables from auxiliary variables, and (2) forming subgroups with different response propensities based on the predicted values of target survey variables. By tailoring different data collection strategies to these subgroups, we can obtain a response set with less variation in response propensities across subgroups that are directly relevant to the target survey variables. Given this rationale, we also propose to measure nonresponse bias by the coefficient of variation of response propensities estimated from the predicted target survey variables. A case study using the Dutch Health Survey shows that the proposed stratification method generally produces less variation in response propensities with respect to the predicted target survey variables compared to traditional methods, thereby leading to a response set that better resembles the population.

Keywords

adaptive survey design nonresponse bias stratification Bayesian methods survey optimization

Introduction

The cost of collecting official statistics survey data has been on the rise in recent years, while budgets for these efforts have been decreasing (Luiten, Hox and De Leeuw 2020). This presents a challenge for statistical institutes, as they strive to maintain a balance between cost and data quality in their survey efforts. To address this challenge, many institutes are turning to adaptive survey design (ASD) (Schouten, Peytchev and Wagner 2017); ASD employs non-uniform data collection strategies, such as different survey modes or contact attempts for different population subgroups, to recruit a balanced set of respondents across subgroups on a budget; see Coffey, Reist and Miller (2020); Luiten and Schouten (2013); Van Berkel, Van der Doef and Schouten (2020) for recent applications. A balanced set of respondents more closely resembles a probability sample selected from a population, so that, for example, the distributions of certain characteristics of the respondents (e.g., age) agree with the corresponding distributions in the population (Särndal 2011; Schouten et al. 2016). If a subgroup of sample units (e.g., young adults) is less likely to respond to the survey, the response set is unbalanced and thus not representative of the population. Obtaining a balanced response set within budget through ASD can be framed as an optimization problem, with the objective of finding the optimal allocation of data collection strategies across subgroups to minimize nonresponse bias. Before optimizing for this objective, we first face a critical question:

How to stratify the target population into subgroups effectively and efficiently?

We aim to identify subgroups across which balanced responses minimize nonresponse bias (i.e., effectiveness) without inflating the variance of survey estimates (i.e., efficiency).

ASD is based on historical survey data with varying design features (e.g., mixed-mode surveys). It requires powerful auxiliary information observed for the entire population, such as sampling frames or administrative data (e.g., resident registers), in order to identify subgroups and tailor strategies accordingly before starting data collection for future surveys. Specifically, ASD uses estimated response propensities and costs under different data collection strategies from historical survey data to inform the stratification and optimal allocation of data collection strategies in future surveys. Traditionally, subgroups are the categories or cross-categories of observed auxiliary variables, such as age and sex. The balance of responses across these subgroups can be monitored during data collection. Särndal (2011) and Lundquist and Särndal (2013) derive a function of the coefficient of variation (CV) of response propensities as an indicator of balance. Similarly, Schouten, Cobben and Bethlehem (2009) derive a function of the standard deviation of response propensities as an indicator of representativeness (R-indicators) of a response set. As these indicators are at the survey level, Schouten et al. (2012) and Schouten, Shlomo and Skinner (2011) propose the partial R-indicators that are available at the variable level and at the variable category level to monitor data collection and to identify and prioritize subgroups defined by auxiliary variables. Schouten and Shlomo (2017) further demonstrate the use of partial R-indicators to build nonrespondent profiles, identifying lower-response categories in the auxiliary variables as subgroups. These methods identify subgroups based on auxiliary variables and allocate appropriate data collection strategies to balance responses with respect to auxiliary variables.

Even with a response set that is perfectly balanced with respect to auxiliary variables, nonresponse bias is not fully eliminated (Särndal 2011). Instead, if the selected auxiliary variables are only weakly correlated with the target survey variables, the variance of survey estimates may be inflated (Little and Vartivarian 2005). To illustrate, assuming that older adults are more likely to respond to face-to-face interviews but less likely to respond to web surveys than young adults, responses are unbalanced between age subgroups if web surveys are uniformly employed for the entire sample; follow-up with nonrespondents among older adults through face-to-face interviews may improve the balance. However, if the target survey variable (e.g. frequency of sedentary behavior) is weakly correlated with age, balancing responses between age subgroups may recruit respondents with similar values of the survey variable and thus have a limited effect on improving accuracy and may instead reduce the precision of the survey estimate. To alleviate the problem, alternative methods attempt to consider survey variables for monitoring data collection.

Wagner (2014) proposes to use regression diagnostic measures to identify subgroups of influential sample units that have outliers in the values of target survey variables predicted by auxiliary variables. For example, suppose that smoking behavior is the target survey variable that is predicted to be more heterogeneous among young adults than older adults, limited survey responses from young adults may distort the distribution of smoking behavior and the relationship between smoking behavior and age in the sample; thus, converting nonrespondents among young adults to respondents minimizes nonresponse bias in the estimate of smoking prevalence in the population. However, the likelihood of recruiting target nonrespondents is unknown, making it difficult to allocate field resources and control costs.

To effectively and efficiently use auxiliary information to balance the quality and costs of surveys, we propose a parsimonious stratification method—identifying the subgroups that differ the most in estimated response propensities with respect to the predicted survey variables. Using historical survey data, this method (1) predicts values of target survey variables from available auxiliary variables for respondents and nonrespondents, and (2) identifies subgroups consisting of predicted values of target survey variables that have different estimated response propensities under different data collection strategies. Thus, the identified subgroups capture heterogeneity in estimated response propensities and predicted target survey variables. Balancing responses across these subgroups by allocating appropriate data collection strategies reduces variation in estimated response propensities with respect to the predicted target survey variables. For example, suppose that smoking behavior is the target survey variable and is predicted by the auxiliary variables, a subgroup may consist of sample units that have a probability of smoking higher than 0.4 and are found to be reluctant to participate in face-to-face interviews, preferring instead to receive self-administered surveys; approaching sample units in this subgroup with a mailed invitation and having them complete a web survey may result in a balanced set of respondents with respect to smoking behavior and hence a more accurate and precise estimate of smoking prevalence in the population.

This ASD stratification method conforms to forming strata or cells for nonresponse adjustment following the principle to control both bias and variance. It resembles joint classification of response propensity stratification and predictive mean stratification (Vartivarian and Little 2002) and constructs strata that are homogeneous with respect to target survey variables and response propensities (Bethlehem 1988). Applying such principle to ASD protects against variance inflation in survey estimates by exploiting residual correlations between auxiliary variables and target survey variables at the data collection stage. Such an extension echoes the call to make the data missing at random (MAR) (Rubin 1976) assumption more plausible at the survey design and data collection stage, i.e., responses do not depend on survey variables after conditioning on auxiliary variables (Little 2022).

In addition to introducing a parsimonious stratification method, we propose to measure nonresponse bias by the CV of response propensities estimated from the predicted target survey variables. We use this measure to quantify the remaining nonresponse bias after implementing ASD with the optimal allocation of data collection strategies and to evaluate and compare the effectiveness of using different methods for stratification in minimizing nonresponse bias. We aim to find the ASD with minimal remaining nonresponse bias under the assumption that the underlying stratification is optimal.

The “Methods” section outlines the steps to design ASD using the proposed stratification methods as well as alternative traditional methods for comparison. The “A Case Study on the Dutch Health Survey” section examines the effectiveness of the proposed stratification method in a case study using the Dutch Health Survey. The “Discussion” section concludes with caveats and a discussion of future research directions.

Methods

With the objective of minimizing nonresponse bias, we outline the steps for designing an ASD based on historical survey data, using the proposed stratification methods and traditional methods. The steps we go through are:

Setting the framework for Bayesian ASD in the “Adaptive Strategies and Optimization Problem” section: We identify data collection strategies from historical surveys and introduce the optimization objective functions in which the design parameters are analyzed based on a Bayesian framework to facilitate ASD stratification and optimization.

Stratifying the target population into subgroups in the “Stratify Target Population into Subgroups” section: We choose a number of stratification scenarios corresponding to the proposed and traditional methods. Specific to the proposed stratification method, we create models for predicting target survey variables.

Estimating ASD design parameters in the “Estimate Design Parameters” section: We create models for estimating response propensities and costs and compute the quality and cost indicators.

Optimizing ASD allocation of data collection strategies in the “Optimize Allocation of Strategies” section: We optimize the allocation of data collection strategies and describe how we use the CV of response propensities estimated from predicted survey variables to evaluate performance across stratification scenarios.

Our main contribution lies in steps 2 and 4. We summarize these steps as a practical guide in the “Summary of Steps in the Proposed Method” section.

Adaptive Strategies and Optimization Problem

In this section, we set notation and introduce the optimization objective functions. We do this within a Bayesian framework.

Data Collection Strategies

A survey design involves a variety of features such as sampling design, mode of administration, number of phases, type of questionnaire, and interviewer. A data collection strategy is a sequence of actions corresponding to the choices made for the design features. In this paper, we simplify data collection strategies consisting of two actions, namely two survey modes—web and face-to-face interviews. For example, in a historical survey with a mixed-mode sequential design, the default uniform strategy may be to invite all sample units to participate in the web survey and follow up with nonrespondents by face-to-face visits; an alternative strategy for future rounds may be to stop further efforts with selected nonrespondents at the end of the web survey for budgetary reasons. Instead of following up with all the nonrespondents, ASD determines which (groups of) nonrespondents to follow up.

Let the survey design consist of $T$ phases, labeled $t = 1, 2, \dots, T$ . Let $S_{t}$ denote the collection of all possible actions in phase $t$ and let $s_{t}$ represent the action taken in phase $t$ . The aggregation of data collection strategies from phase one to $T$ is defined as

S : = {(s_{1}, \dots, s_{T}) : s_{t} \in S_{t}, t = 1, 2, \dots, T},

and let

s = (s_{1}, \dots, s_{T})^{'}

denote one possible strategy, i.e., a sequence of actions from phase one to

T

, where

s \in S

Traditional non-ASDs use a single data collection strategy for the entire sample. In contrast, ASD adapts different strategies to different sample units. One key aspect of ASD is stratification, which separates sample units into subgroups with different estimated response propensities under different strategies. For example, a subgroup consisting of older adults may be less likely to respond to web surveys, but may be more likely to respond to face-to-face interviews. Considering the differences in response propensities, we may identify subgroups from historical surveys and adapt different data collection strategies for different subgroups of sample units in future surveys. Details on stratification follow in the “Stratify Target Population into Subgroups” section.

Let the target population consist of $G$ subgroups, labeled $g = 1, 2, \dots, G$ . Every unit in subgroup $g$ is eligible to be approached by any one of the strategies in $S$ . Let $a_{g}$ represent one strategy allocated to subgroup $g$ . The aggregation of allocations of data collection strategies over $G$ subgroups is defined as

A = S^{G} = \underset{G}{\underset{⏟}{S \times \dots \times S}} : = {(a_{1}, \dots, a_{G}) : a_{g} \in S, g = 1, 2, \dots, G},

and let

a = (a_{1}, \dots, a_{G})^{'}

denote one possible allocation for all subgroups, where

a \in A

Optimization Objective and Constraints

After identifying multiple data collection strategies and all possible allocations of strategies to the identified subgroups, we search for the optimal allocation of strategies, denoted by $a_{opt}$ , with the lowest risk of nonresponse while satisfying the response rate (RR) and budget constraints. This is an optimization problem that requires evaluating the indicators of nonresponse bias, RR, and cost for all allocations of strategies in $A$ . These indicators are functions of design parameters (i.e., response propensity and cost) and can be estimated from historical survey data.

While cost is an observed quantity, response propensity is an unobserved, nonzero probability that can be estimated from observed data (Bethlehem, Cobben and Schouten 2011; Little 1986). Nonresponse bias occurs when the response propensity is correlated with target survey variables (Groves 2006). Addressing nonresponse bias implies reducing the correlation between the response propensity and target survey variables. Let $Y$ be the target survey variable, and let ${\hat{\bar{y}}}_{HT}$ be the Horvitz-Thompson estimator for the population mean based on the survey response set. It can be shown that nonresponse bias is approximately equal to

Bias ({\hat{\bar{y}}}_{HT}) \approx \frac{Corr (ρ, Y) S_{ρ} S_{Y}}{\bar{ρ}},

where

ρ

denotes the response propensities,

\bar{ρ}

is the average response propensity,

S_{ρ}

is the standard deviation of the response propensities, and

S_{Y}

is the standard deviation of the target survey variable (see Bethlehem 2009 for details). Schouten, Cobben and Bethlehem (2009) bound the absolute bias, and Van Berkel, Van der Doef and Schouten (2020) link the upper bound to the CV of response propensities as follows

| Bias ({\hat{\bar{y}}}_{HT}) | \leq \frac{S_{ρ} S_{Y}}{\bar{ρ}} = CV (ρ) S_{Y} .

Given the allocation of strategies

a

, the response propensity of unit

i

in the sample can be estimated as

{\hat{ρ}}_{i} (a) = Pr (u_{i} (a) = 1 ∣ {\hat{y}}_{i}),

where

u_{i} (a)

represents the response indicator, and

{\hat{y}}_{i}

represents the predicted value of the target survey variable. Traditionally, we assume that the available auxiliary variables are correlated with target survey variables and estimate response propensities from auxiliary variables. Little (2022) emphasizes the importance of conditioning response propensity on survey variables, allowing for data missing not at random (MNAR) when missingness depends on survey variables. Since survey variables are only observed for respondents, we predict target survey variables from auxiliary variables for respondents and nonrespondents, and then estimate response propensities from the predicted values of target survey variables. Given that surveys are multipurpose (Kish 1988) and that nonresponse bias is associated with specific survey estimates, stakeholders need to decide on the target survey variables that primarily require bias reduction.

We allocate the same strategy to all units in the same subgroup and assume a simplified situation where, given the allocation of strategies $a$ , all units in subgroup $g$ have the same estimated response propensity ${\hat{ρ}}_{g} (a)$ and estimated cost ${\hat{C}}_{g} (a)$ . Based on the estimated response propensities and costs at the subgroup level, we compute the quality and cost indicators—CV of estimated response propensities at the subgroup level, overall RR, and the required budget per respondent (B). To avoid repeatedly constructing the response indicator $u_{i} (a)$ and averaging the estimated response propensity ${\hat{ρ}}_{i} (a)$ by subgroup $g$ given the allocation, we derive these subgroup-level estimated response propensities based on the estimated response propensities under a given strategy ${\hat{ρ}}_{i} (s)$ , which are estimated only once from historical survey data; the same applies to the subgroup-level estimated cost. Details on the models for estimating response propensities and costs, and on the computation of quality and cost indicators follow in the “Estimate Design Parameters” section.

The optimization searches for the optimal allocation from $A$ that minimizes the CV subject to the RR and B constraints

\begin{aligned} min_{a \in A} & E (CV (\hat{ρ} (a))) \\ subject to & E (RR (\hat{ρ} (a))) \geq {RR}_{lower}, \\ Pr (B (\hat{C} (a), \hat{ρ} (a)) \geq B_{upper}) \leq 0.10, \end{aligned}

and let

\hat{ρ} (a) = ({\hat{ρ}}_{1} (a), \dots, {\hat{ρ}}_{G} (a))^{'}

denote the vector of estimated response propensities at the subgroup level and

\hat{C} (a) = ({\hat{C}}_{1} (a), \dots, {\hat{C}}_{G} (a))^{'}

denote the vector of estimated costs at the subgroup level; the required budget per respondent (B) is a function of both estimated response propensities and costs, as we expect the budget to be higher for recruiting respondents with low response propensities; a probabilistic constraint is imposed on B such that no more than 10% of the draws from the posterior distribution of B can exceed the maximum budget (see the next section on the Bayesian framework for details); the boundary values for RR and B are determined by survey stakeholders. For the proposed stratification method, the vector of estimated response propensities is projected onto a subspace defined by predicted survey variables. If all survey variables are continuous so that linear regression is used for prediction, the vector of response propensities is projected onto a subspace defined by auxiliary variables.

We acknowledge that recent research on optimizing data collection incorporates cost into the objective function to maximize the cost-quality trade-off (Coffey and Elliott 2024; Wagner et al. 2023). In our case, the survey stakeholders are more interested in understanding changes in quality while fixing budget, i.e., there is no direct need to reduce budget. Although the RR itself is a poor quality indicator, sufficient RRs can still reduce the risk of nonresponse bias (Hedlin 2020; Schouten, Cobben and Bethlehem 2009). Since the proposed method minimizes nonresponse bias with respect to a limited set of predicted target survey variables, it is necessary to ensure a certain RR to adequately capture variation in other survey variables. Details on optimization follow in the “Optimize Allocation of Strategies” section.

Bayesian Framework for ASD

We adopt the Bayesian framework for analyzing design parameters in survey data collection (Schouten et al. 2018). We fit models for predicting target survey variables and estimating response propensities and costs. Appendix A in the online Supplemental Material specifies the prior and posterior distributions of model parameters. As the posterior distributions cannot be computed analytically, we apply a Gibbs sampler using Markov Chain Monte Carlo (MCMC) methods to sample the model parameters. Appendix B in the online Supplemental Material specifies the full conditionals of the model parameters to facilitate Gibbs sampling.

With the sampled model parameters, we obtain the posterior predictive distributions of target survey variables and treat the expectations as the predicted values for respondents and nonrespondents. We use the predicted values of target survey variables for the proposed stratification method (see the “Stratification Methods” section) and for the evaluation of remaining nonresponse bias after implementing ASD (see the “Determine Optimal Stratification” section). Similarly, we obtain the joint posterior distributions of response propensities and costs and derive the posterior distributions of quality and cost indicators under the given allocation. The optimization searches for the optimal allocation that minimizes the expectation of the CV of estimated response propensities at the subgroup level; meanwhile, the expectation of overall RR is greater than or equal to the specified minimum; and for draws from the posterior distribution of the required budget per respondent, the proportion of draws exceeding the specified maximum is lower than 0.10.

Stratify Target Population into Subgroups

In this section, we introduce the proposed stratification method based on predicted target survey variables. We also introduce competing stratification methods to which we compare the proposed method. We first describe these methods and then elaborate on how we predict target survey variables.

Stratification Methods

We consider five different stratification methods. Table 1 displays the different stratification methods. They vary in the predictors that are used and the indicator that is focused on. Two of the proposed methods use predicted target survey variables, one explaining the response indicator (Response $\hat{Y}$ ) and the other explaining the number of in-person visits (Visit $\hat{Y}$ ). The three other methods are based on the “traditional” stratification using available auxiliary variables directly as predictors. The three methods explain the response indicator (ResponseX), the number of in-person visits (VisitX), and survey costs (CostX). In all methods, we identify subgroups from historical survey data. When implementing ASD for future surveys, provided the same auxiliary variables are available, we can form the same subgroups. In the "Predict Target Survey Variables" section, we explain how we predict target survey variables. We show the results of all five methods in the case study in the “A Case Study on the Dutch Health Survey” section for comparison purposes. However, we only discuss the two proposed survey variable methods in detail throughout the paper, because these are novel.

Table 1.

Stratification Methods Corresponding to Different Outcomes and Predictors in the Classification and Regression Trees (CART).

Outcome	$X$ as predictors	$\hat{Y}$ as predictors
Survey response	ResponseX	Response $\hat{Y}$
#In-person visits	VisitX	Visit $\hat{Y}$
Survey costs	CostX

We use the classification and regression trees (CART) (Breiman et al. 1984) with predicted target survey variables to predict binary response indicators (see Lohr, Hsu and Montaquila 2015). CART can account for potential collinearity and interactions between predicted target survey variables. CART partitions the population into subgroups consisting of different values or combinations of values of predicted target survey variables, which have varying estimated response propensities under a given data collection strategy. For example, an identified subgroup of sample units with a probability of smoking higher than 0.4 may be less likely to respond to face-to-face interviews but may be more likely to respond to web surveys. As the response indicator corresponds to each strategy, we may build multiple CARTs and select one as the stratification that best differentiates between subgroups with varying estimated response propensities under different strategies.

In addition to the response indicator, we also use predicted target survey variables to explain the number of in-person visits to sample units when the face-to-face interview is an action in particular data collection phases. For example, the face-to-face interview can be used as a follow-up action in the subsequent phases for sample units who have not responded in the previous phase of the self-administered survey. The number of visits can serve as a continuous indicator of survey response; increasing the number of visits may be effective in recruiting respondents; if not, it indicates that the sample units are either unreachable or refuse to participate. On the other hand, the number of visits indicates the survey costs, as the more in-person visits, the higher the costs. Compared with the binary response indicator, the number of in-person visits captures variation in response behavior as well as the associated cost. While the focus of this paper is not on minimizing survey costs, for cost-sensitive ASD, we also identify subgroups that are homogeneous in terms of actual survey costs.

Predict Target Survey Variables

Using historical survey data, we model the observed values of target survey variables by auxiliary variables available at the start of data collection and predict the values of target survey variables for respondents and nonrespondents. Let $x_{i} = (x_{1, i}, \dots, x_{m, i})^{'}$ be the vector of $m$ auxiliary variables for sample unit $i$ and let $y_{i} = (y_{1, i}, \dots, y_{j, i})^{'}$ be the vector of $j$ survey variables. If survey variable $k$ is continuous, then predict it by

{\hat{y}}_{k, i} = x_{i}^{'} β_{k},

(1)

where

β_{k}

is an

m

-vector of linear regression parameters. If survey variable

k

is dichotomous, then predict the probability of success by

\hat{Pr} (y_{k, i} = 1 ∣ x_{i}) = Φ (x_{i}^{'} β_{k}),

(2)

where

Φ

is the probit function. For simplicity, survey variables that are neither continuous nor dichotomous are dichotomized. When implementing ASD for future surveys, provided the same auxiliary variables are available, we can use the parameters from regressions fit to historical data to predict the unknown values of target survey variables for all sample units at the start of data collection.

Estimate Design Parameters

In this section, we create models for estimating response propensities and costs per sample unit under each data collection strategy and compute the quality and cost indicators under a given allocation of strategies.

Response Propensities and Costs

We assume that all units within the same subgroup have the same estimated response propensities and costs under the given strategy. Given data collection strategy $s$ , i.e., the sequence of actions from phase one through phase $T$ , we estimate response propensities ${\hat{ρ}}_{i} (s)$ and costs ${\hat{C}}_{i} (s)$ per sample unit; given allocation of strategies $a$ for all subgroups, we average the estimated response propensities and costs for sample units within the same subgroup to derive the subgroup-level estimated response propensities ${\hat{ρ}}_{g} (a)$ and costs ${\hat{C}}_{g} (a)$ . Using historical survey data, we apply the probit model to estimate individual response propensities by modeling binary response indicators at the end of phase $T$ under the given strategy.

Let $g_{i} = (g_{1, i}, \dots, g_{G, i})^{^{'}}$ be the vector of binary indicators of subgroup membership for sample unit $i$ , the response ability under strategy $s$ is estimated as a continuous latent variable

{\hat{Z}}_{i} (s) = α^{'} (s) g_{i},

(3)

where

α (s)

is the vector of regression coefficients; the response propensity is the probability that the estimated latent variable is greater than zero, i.e.,

{\hat{ρ}}_{i} (s) = Pr (u_{i} (s) = 1 ∣ g_{i}) = Pr ({\hat{Z}}_{i} (s) > 0)

Similarly, we apply the linear model to estimate costs per sample unit by modeling the actual survey costs at the end of phase $T$ under the given strategy. The cost under strategy $s$ is estimated as

{\hat{C}}_{i} (s) = γ^{'} (s) g_{i},

(4)

where

γ (s)

is the vector of regression coefficients. For self-administered survey modes, such as web surveys or mail surveys, the cost per sample unit is more or less fixed, which means that it is not necessary to model the costs of these modes.

Based on the estimated response propensities and costs per sample unit, the subgroup-level estimated response propensities and estimated costs for subgroup $g$ under the allocation of strategies $a$ are derived as

{\hat{ρ}}_{g} (a) = \frac{\sum_{i = 1}^{n} δ_{g, i} d_{i} {\hat{ρ}}_{i} (s)}{\sum_{i = 1}^{n} δ_{g, i} d_{i}},

(5)

{\hat{C}}_{g} (a) = \frac{\sum_{i = 1}^{n} δ_{g, i} {\hat{C}}_{i} (s)}{\sum_{i = 1}^{n} δ_{g, i}},

(6)

where

δ_{g, i}

indicates if sample unit

i

is in subgroup

g

, and

d_{i}

denotes the design weight. We include design weights to adjust for sampling designs (e.g., stratified sampling), evaluating whether the resulting response set is balanced with respect to the target population. It should be noted that the subgroup-level estimated costs are the average costs per sample unit in a subgroup. Then, we construct the vector of subgroup-level estimated response propensities

\hat{ρ} (a)

and costs

\hat{C} (a)

under the given allocation of data collection strategies. These vectors are components of the quality and cost indicators used for optimization.

Quality and Cost Indicators

Three functions of subgroup-level estimated response propensities and estimated costs constitute the quality and cost indicators specified in the optimization problem—overall RR, required budget per respondent (B), and CV of subgroup-level estimated response propensities. The overall RR is computed as

RR (\hat{ρ} (a)) = \frac{1}{n} \sum_{g = 1}^{G} n_{g} {\hat{ρ}}_{g} (a),

(7)

and the required budget per respondent (i.e., per completed interview) is computed as

B (\hat{C} (a), \hat{ρ} (a)) = \frac{1}{n} \sum_{g = 1}^{G} \frac{n_{g} {\hat{C}}_{g} (a)}{{\hat{ρ}}_{g} (a)},

(8)

and the CV of subgroup-level estimated response propensities is computed as

CV (\hat{ρ} (a)) = \frac{\sqrt{\frac{1}{n} \sum_{g = 1}^{G} n_{g} ({\hat{ρ}}_{g} (a) - RR (\hat{ρ} (a)))^{2}}}{RR (\hat{ρ} (a))},

(9)

where

n = \sum_{g = 1}^{G} n_{g}

; we specify the required budget per respondent to reflect the fact that recruiting respondents from subgroups with low response propensities requires more budget.

Optimize Allocation of Strategies

In this section, we search for the optimal allocation of data collection strategies to the selected subgroups and introduce a criterion to evaluate the performance of minimizing nonresponse bias.

Optimization through Synthetic Samples

From historical surveys, we may identify varying design features (e.g., survey modes) and construct the aggregation of data collection strategies; we may also identify subgroups with varying estimated response propensities for a given strategy and construct the aggregation of allocations of strategies over all subgroups. For each allocation of data collection strategies, we synthesize the sample based on the observed historical survey data to emulate the implementation of ASD given such allocation; we cannot add more data, but we can remove data to show the effects of interventions during data collection, which is referred to as “experiments in retrospect” (Lundquist and Särndal 2013; Särndal and Lundquist 2014). For example, in a non-adaptive sample, all units that do not respond at the end of the web survey phase receive follow-up face-to-face visits; for subgroups that are only allocated the web survey strategy, we exclude data collected after the web survey phase and record their responses at the end of the web survey phase as final survey responses.

For each stratification method specified in Table 1, we obtain synthetic samples realizing all possible allocations of data collection strategies and compute the quality and cost indicators for each allocation. After evaluating the conditions specified in the optimization problem, we find the optimal allocation that minimizes the CV, i.e., the one that minimizes nonresponse bias.

Determine Optimal Stratification

When comparing the optimal allocations between different stratification methods, it should be noted that the quality and cost indicators are specific to stratification. The underlying models for response propensities and costs are different because predictors, i.e., binary indicators of subgroup membership, vary by stratification methods. We thus propose an overarching criterion to evaluate remaining nonresponse bias given the optimal allocation of strategies—CV of individual response propensities estimated from predicted survey variables. Such criterion quantifies nonresponse bias with respect to survey variables predicted from the observed auxiliary variables, independent of stratification.

The prediction of survey variables has been specified in the “Predict Target Survey Variables” section. Similar to the specification of the response propensity model in the “Response Propensities and Costs” section, we apply the probit model to estimate individual response propensities by modeling the final survey response indicators in the synthetic sample given the optimal allocation but using target survey variables predicted from observed auxiliary variables, $E (Y ∣ X)$ , rather than binary indicators of subgroup membership

{\hat{Z}}_{i} (a_{opt}) = α^{'} (a_{opt}) E (Y ∣ X),

(10)

where

α (a_{opt})

is the vector of regression coefficients; the response propensity is the probability that the estimated latent variable is greater than zero, i.e.,

{\hat{ρ}}_{i} (a_{opt}) = Pr (u_{i} (a_{opt}) = 1 ∣ E (Y ∣ X)) = Pr ({\hat{Z}}_{i} (a_{opt}) > 0)

. The posterior predictive distributions of target survey variables are derived from the posterior distributions of model parameters in (1) and (2). See Appendix A in the online Supplemental Material for details.

The corresponding CV of estimated individual response propensities is computed as

CV (\hat{ρ} (a_{opt})) = \frac{\sqrt{\frac{1}{\sum_{i = 1}^{n} d_{i}} \sum_{i = 1}^{n} d_{i} ({\hat{ρ}}_{i} (a_{opt}) - \frac{1}{\sum_{i = 1}^{n} d_{i}} \sum_{i = 1}^{n} d_{i} {\hat{ρ}}_{i} (a_{opt}))^{2}}}{\frac{1}{\sum_{i = 1}^{n} d_{i}} \sum_{i = 1}^{n} d_{i} {\hat{ρ}}_{i} (a_{opt})},

(11)

where

d_{i}

denotes the design weight. We compute and compare the CV for the optimal allocation under each stratification method. The optimal allocation with the minimum CV indicates that the stratification method behind it is optimal for minimizing nonresponse bias in ASD, when the available auxiliary variables are predictive of the target survey variables.

Summary of Steps in the Proposed Method

To assist application of the proposed stratification method in practice, the steps are summarized as follows:

Identify candidate data collection strategies, e.g. based on the choice of modes, choice of number of calls/visits, or choice of incentive;

Collect historical survey data that includes the candidate data collection strategies and relevant auxiliary variables from the sampling frame or administrative data;

Determine the target survey variables. Predict each target survey variable for each sample unit using the available auxiliary variables (i.e., ${\hat{y}}_{j, i} ∣ x_{i}$ );

Based on the predicted target survey variables ( ${\hat{y}}_{j, i} ∣ x_{i}$ ), use CART to stratify sample units into $G$ subgroups that can explain variation in response behavior as a result of candidate data collection strategies (i.e., with mode response indicator, number of in-person visits, or incentive amount as the dependent variable);

Estimate the response propensity and cost under the candidate data collection strategies for the $i$ th sample unit using the subgroup membership indicator (i.e., ${\hat{ρ}}_{i}, {\hat{C}}_{i} ∣ g_{i}$ );

Allocate the candidate data collection strategies to $G$ different subgroups, with a total of $S^{G}$ possible allocations, where $S$ is the number of strategies;

Given a specific allocation, synthesize a sample and calculate the mean response propensity and cost by subgroup $g$ (i.e., subgroup-level estimated response propensity and cost);

For each synthetic sample corresponding to each alternative allocation, calculate and compare the CV of the subgroup-level estimated response propensities, the overall RR, and the cost per completion (B);

Select the allocation of strategies that provides the best performance in minimizing nonresponse bias per the conditions specified in the optimization problem.

A Case Study on the Dutch Health Survey

In this section, we present a case study using historical data from the Dutch Health Survey to illustrate the steps involved in designing an ASD and to evaluate the effectiveness of the proposed stratification method in minimizing nonresponse bias.

Survey Design and Adaptive Strategies

Survey Design

The Dutch Health Survey is designed to provide a comprehensive understanding of the health, medical contacts, lifestyle, and preventive behaviors of the Dutch population (Statistics Netherlands 2020). The target population is all individuals living in private households. For children under 12 years of age, the survey questions are answered by parents or guardians. Measurement stability and low nonresponse bias of target survey variables are top priorities for the survey stakeholders, as survey results are used to compare health statistics across time.

In this case study, we use data collected from the last three quarters of 2017 and the first quarter of 2018 and exclude respondents under the age of 12 from the sample. We aim to minimize nonresponse bias with respect to three target survey variables:

How is your health in general?

This variable is measured on a 5-point scale: Very good, Good, Goes well, Bad, Very bad. The first two categories are recoded as healthy, and the last three are unhealthy. The item goes well in Dutch has no positive meaning.

Do you ever smoke?

This is a dichotomous variable: Yes/No.

Obesity.

This dichotomous variable is derived from body mass index (BMI) calculated from height and weight measurements (weight in kilos divided by squared height in meters):

How tall are you? It is the height in centimeters, without shoes;

What is your weight in kilos? (We mean the weight before pregnancy.) It’s the weight in whole kilos, without clothes.

For adults, a BMI over 30 indicates obesity. For teenagers, the criteria depend on age.

For ease of illustration and application of stratification methods, we restrict the case study to binary indicators and leave the extension to nominal categorical variables for the future.

Adaptive Strategies

The Dutch Health Survey employs a sequential mixed-mode design, starting with a web survey followed by face-to-face interviews. Specifically, nonrespondents to the web survey receive either short face-to-face interviews (F2F-short) or extended face-to-face interviews (F2F-extended). The F2F-short mode involves three in-person visits and the F2F-extended mode involves additional visits to nonrespondents after the first three visits. We thus identify three data collection strategies—Web, Web + F2F-short, and Web + F2F-short + F2F-extended (shortened as Web + F2F-extended)—corresponding to three phases of data collection that sample units may go through in sequence—Phase 1 (Web), Phase 2 (F2F-short), and Phase 3 (F2F-extended).

For sample units that receive the Web strategy, data collection stops at the end of Phase 1, meaning that they are not followed up even if they do not respond. For sample units that receive the Web + F2F-short strategy, the data collection stops at the end of Phase 2. For sample units that receive the Web + F2F-extended strategy, the data collection stops at the end of Phase 3. Based on the observed survey data, we identify subgroups with varying estimated response propensities to these strategies and search for the optimal allocation of strategies to subgroups.

Stratification and Optimization

We perform all analyses in R (R Core Team 2023). For stratification, we use the rpart package (Therneau and Atkinson 2019) to build the CART. For prediction of target survey variables, we use the MCMCpack package (Martin, Quinn and Park 2011) to fit models and derive posterior predictive distributions. For optimization, we program a Gibbs sampler to fit models for response propensities and costs, and derive the joint posterior distributions of response propensities and costs, as well as posterior distributions of quality and cost indicators. We specify non-informative priors for all model parameters. The convergence properties of the Gibbs sampler can be found in Appendix C in the online Supplemental Material.

Stratification

For each stratification method in Table 1, we identify subgroups using the CART and estimate subgroup-level response propensities and costs. Specific to the proposed stratification methods, we predict target survey variables (health, smoking, and obesity) with categorical auxiliary variables extracted from registers, including age, sex, income, marital status, level of education, migration background, receipt of rent benefits, type of household, and level of urbanization in the area of residence. The results of prediction models for survey variables can be found in Appendix D in the online Supplemental Material.

We then use the predicted values of target survey variables to predict the survey response indicator and the number of in-person visits using the CART. For the Response $\hat{Y}$ method, we predict the survey response indicator at the end of Phase 1, i.e., the response indicator for the Web strategy, because the CART barely differentiates between subgroups with varying estimated response propensities at the end of Phase 2 or Phase 3, suggesting that sample units going through Phase 2 or Phase 3 have similar estimated response propensities. Tables 2 and 3 show the identified subgroups based on the Response $\hat{Y}$ and Visit $\hat{Y}$ methods, respectively. Because this case study aims to illustrate the proposed methods, we do not present identified subgroups based on other methods.

Table 2.

Identified Subgroups Using the Response $\hat{Y}$ Stratification Method.

Subgroup^a	Response rate^b	$Pr$ (Smoking)	$Pr$ (Health)
1 (311)	16.72%	$\geq$ 0.37	$\geq$ 0.64 & < 0.75
2 (546)	23.63%	$\geq$ 0.37	$\geq$ 0.75 & < 0.87
3 (453)	28.26%	$\geq$ 0.37	$\geq$ 0.87
4 (538)	24.54%	$\geq$ 0.37	< 0.64
5 (4,380)	33.47%	$\geq$ 0.20 & < 0.37
6 (6,969)	45.92%	< 0.20

Subgroup size in parentheses (total $n$ =13,197).

Response rate at the end of Phase 1.

Table 3.

Identified Subgroups Using the Visit $\hat{Y}$ Stratification Method.

Subgroup^a	#Visits^b	$Pr$ (Smoking)	$Pr$ (Health)	$Pr$ (Obesity)
1 (3,278)	0.90	< 0.17	< 0.82
2 (1,514)	1.26	$\geq$ 0.17 & < 0.25	< 0.82
3 (1,932)	1.28	< 0.13	$\geq$ 0.82
4 (1,957)	1.59	$\geq$ 0.13 & < 0.25	$\geq$ 0.82
5 (1,783)	1.64	$\geq$ 0.25 & < 0.43		$\geq$ 0.14
6 (680)	1.69	$\geq$ 0.25 & < 0.43	$\geq$ 0.91	< 0.14
7 (1,082)	2.08	$\geq$ 0.25 & < 0.43	< 0.91	< 0.14
8 (971)	2.11	$\geq$ 0.43

Subgroup size in parentheses (total $n$ =13,197).

Average number of visits.

CART splits sample units based on the predicted probability of smoking, health, and obesity. For example, the first subgroup in Table 2 consists of the sample units in which the predicted probability of smoking is greater than or equal to 0.37 and the predicted probability of health is between 0.64 and 0.75. CART utilizes the predicted probability of obesity to predict the number of in-person visits but not the survey response indicator, suggesting that the probability of obesity may not distinguish between subgroups using the Response $\hat{Y}$ method.

Figure 1 shows the subgroup-level estimated response propensities for all subgroups under different data collection strategies, along with the 95% credible intervals. Different subgroups may have varying estimated response propensities to different strategies. Small subgroups exhibit great heterogeneity in estimated response propensities, as indicated by wide credible intervals. The estimated response propensities to the Web mode strategy are distinguishable from estimated response propensities to the other two strategies. Since few sample units reach Phase 3 (Web + F2F-extended), CART rarely distinguish them from those reaching Phase 2 (Web + F2F-short). It is evident that if one data collection strategy is applied uniformly to the entire sample, the variation in estimated response propensities may be substantial. We aim to optimally allocate strategies across subgroups, i.e., to balance estimated response propensities among subgroups while controlling survey costs.

Figure 1.

Estimated response propensities per subgroup and per strategy (Web, Web + F2F-short, and Web + F2F-extended). (a) Response $\hat{Y}$ ; (b) Visit $\hat{Y}$ . F2F-short: short face-to-face interviews; F2F-extended: extended face-to-face interviews.

Optimization

Given the three data collection strategies and identified subgroups, we construct all possible allocations of strategies over all subgroups. For example, using the Response $\hat{Y}$ stratification method, we identify six subgroups and $3^{6} = 729$ possible allocations; using the Visit $\hat{Y}$ method, we identify eight subgroups and $3^{8} = 6, 561$ possible allocations. For each allocation, we synthesize the sample to emulate the implementation of ASD and compute the quality and cost indicators, i.e., the CV of subgroup-level estimated response propensities, the overall RR, and the required budget per respondent (B).

We set a lower limit on the overall RR at 50% and an upper limit on the required budget per respondent at 90% of realized costs. Among all possible allocations of data collection strategies, the optimization filters out allocations that do not meet these constraints and locates the optimal allocation with the minimum CV. For each stratification method, we run the optimization and select the top five optimal allocations and compute the corresponding CV of estimated individual response propensities by (11).

Figure 2 shows the CV values of the top five optimal allocations for each stratification method, along with the 95% credible intervals. The proposed stratification methods Response $\hat{Y}$ and Visit $\hat{Y}$ yield lower CV values, i.e., less variation in estimated response propensities with respect to predicted target survey variables, than the other methods, suggesting that the proposed methods perform well in minimizing nonresponse bias in survey estimates. In contrast, one of the traditional stratification methods, ResponseX, yields higher CV values than the CV value in the observed sample, implying that the ASD based on this method does not necessarily lead to improved survey estimates.

Figure 2.

Coefficient of variation (CV) of estimated individual response propensities with respect to predicted target survey variables for optimal design solutions based on different stratification methods. The horizontal line indicates the CV in the observed sample.

Sensitivity to RRs and Budgets

We assess the robustness of the proposed stratification methods by varying the overall RR limit between 50% and 55% at 1% intervals and the budget limit between 80% and 100% of realized costs at 5% intervals. Varying the overall RR limit does not significantly alter the pattern. However, sudden changes in CV values occur when the budget limit varies, as shown in Figure 3. Specifically, when the budget limit is set at 95% and 100% of realized costs, the Response $\hat{Y}$ method may perform better than the ResponseX method, and the Visit $\hat{Y}$ method may perform better than VisitX. On the other hand, when the budget limit is set at 80% and 85% of realized costs, the Response $\hat{Y}$ method may perform better than the ResponseX method, but the Visit $\hat{Y}$ method may not perform better than VisitX; besides, the optimization does not find any allocations of strategies using the ResponseX method that satisfy the budget limit of 80% of realized costs. The CostX method performs similarly to the VisitX method; both methods yield lower CV values when the budget limit is set at 80% and 85% compared to other methods, implying that the cost consideration in the stratification becomes more important when the budget becomes stringent.

Figure 3.

Coefficient of variation (CV) of estimated individual response propensities for the response rate limit at 50% and the budget limit at 80%, 85%, 95%, and 100% of realized costs. The horizontal line indicates the CV in the observed sample. (a) Budget limit 80%; (b) budget limit 85%; (c) budget limit 95%; (d) budget limit 100%.

The results suggest that ASD based on the traditional stratification method, ResponseX, may not minimize nonresponse bias in survey estimates effectively. While the proposed method, Response $\hat{Y}$ , may be sensitive to budget constraints, another proposed method, Visit $\hat{Y}$ , is insensitive to varying constraints on the overall RR and the budget compared to other methods. We may conclude that the optimal stratification is the one oriented to explaining the number of in-person visits with predicted target survey variables—Visit $\hat{Y}$ –accounting for the variation in estimated response propensities, estimated costs, and, most importantly, predicted target survey variables.

Discussion

In this paper, we present a parsimonious stratification method for ASD, with the aim of minimizing nonresponse bias. This method uses auxiliary variables available prior to data collection to predict target survey variables and is therefore particularly suitable for multi-stage, mixed-strategy surveys conducted by statistical institutes with access to high quality administrative data. Stratification based on predicted target survey variables identifies subgroups defined by predicted target survey variables. Balancing responses across these subgroups minimizes the variation in estimated response propensities with respect to predicted target survey variables, thereby minimizing nonresponse bias but not inflating variances. We also establish a criterion for evaluating the effectiveness of the proposed method compared to other traditional methods that do not utilize target survey variables. We show through a case study that the optimal allocation based on the proposed method minimizes the variation in estimated response propensities with respect to predicted target survey variables, leading to more accurate survey estimates.

Similar to calibration methods (e.g., generalized regression estimator), the proposed stratification method relies on the auxiliary data available prior to data collection. This, obviously, creates a dependency on the amount and type of auxiliary data available. At Statistics Netherlands, these auxiliary data from the population registers are usually demographic and socio-economic and are usually weakly or moderately correlated with target survey variables. With more limited availability of auxiliary data, our proposed method looses efficacy but remains efficient. The proposed method can also be used to address nonresponse bias in longitudinal surveys where nonrespondents in subsequent waves may have responded in previous waves and these variables would more strongly predict the target survey variables than demographic variables. Perhaps, more importantly, the prediction of target survey variables must be based fully on historical survey response. This means that predictions are based on observed associations between target survey variables and auxiliary variables for respondents only. Doing so, it implicitly makes a MAR assumption. However, this assumption applies only to the maximal response that may be obtained by the set of data collection strategies under consideration. Sample units that would not respond to any of the strategies do not impact efficacy of our proposed method. It is, therefore, important to evaluate to what extent associations depend on the different strategies.

While we show the advantages of the proposed stratification method, we have not included some other stratification methods, such as methods based on partial R-indicators and regression diagnostic measures (Schouten and Shlomo 2017; Wagner 2014). Future research may evaluate these methods with the proposed criterion in this paper, i.e., CV of estimated individual response propensities with respect to predicted target survey variables. In addition, we primarily focus on bias reduction and assume potential variance reduction. Future research may explicitly examine whether the proposed method also protects against inflated variance in survey estimates.

The optimization in this paper is based on some simplifications for the sake of computational efficiency. We specify a deterministic allocation of strategies, i.e., for a given subgroup, a given strategy is either allocated or not. The ideal optimization allows the allocation probability to vary between 0 and 1. This avoids abrupt changes in strategy and smooths jumps in the values of quality and cost indicators. We also assume that all units in the same subgroup have the same estimated response propensities and costs. In other words, we assume that the response propensities, the number of in-person visits, and interview duration are relatively uniform within subgroups. While these assumptions are not realistic in general, the proposed stratification method is still motivated to effectively and efficiently minimize nonresponse bias. Future research may develop more flexible optimization techniques to accommodate a stochastic allocation of strategies and realistic assumptions about response propensities and costs.

Since subgroups are identified using historical survey data and carried over to future adaptive surveys, a critical but unrealistic assumption is that the stratification remains stable over time. We may update subgroups if a portion of the sample in future adaptive surveys is retained for randomized experiments using different data collection strategies. While Bayesian CART (Chipman, George and McCulloch 1998) is more flexible and allows for the incorporation of randomness in the tree structure and parameters, in practice, a single decision tree is often necessary for computational efficiency. We therefore use the conventional CART to generate a single tree for stratification. Future research may explore the use of Bayesian CART to improve the adaptability of stratification to changing circumstances.

Supplemental Material

sj-pdf-1-smr-10.1177_00491241251345463 - Supplemental material for An Optimal Stratification Method for Addressing Nonresponse Bias in Bayesian Adaptive Survey Design

Supplemental material, sj-pdf-1-smr-10.1177_00491241251345463 for An Optimal Stratification Method for Addressing Nonresponse Bias in Bayesian Adaptive Survey Design by Yongchao Ma, Nino Mushkudiani and Barry Schouten in Sociological Methods & Research

Footnotes

Acknowledgments

The authors thank the anonymous reviewers for their thoughtful and constructive feedback on earlier versions of the manuscript. Special appreciation is extended to one reviewer for recommending the simplification of several mathematical expressions, which significantly improved the clarity and accessibility of the text. An earlier version of this paper was presented at the 9th European Survey Research Association Conference and the 7th International Workshop on Advances in Adaptive and Responsive Survey Design in 2021.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Yongchao Ma

Nino Mushkudiani

Barry Schouten

Data and Code Availability Statement

The micro data collected by Statistics Netherlands are secure use files and can only be accessed through on-site visits. The computer program to reproduce the results and instructions to access the data are available at .

Supplemental Material

Supplemental materials and Appendices to this article are available online.

Author Biographies

Yongchao Ma completed his research master’s degree in Methodology and Statistics from Utrecht University and previously interned at Statistics Netherlands, where this research was conducted. He is currently a Ph.D. student in Survey and Data Science at the Survey Research Center, Institute for Social Research, University of Michigan. His research focuses on developing methods to address nonresponse bias in survey data collection.

Nino Mushkudiani is a researcher at the methodology department of Statistics Netherlands. She has several research areas of interest such as: Studying techniques for optimally applying adaptive survey designs, e.g. a Bayesian framework or random trees; macro-integration techniques for reconciliation of different data and estimation of measurement errors induced by these techniques; investigation of the measurement problems triggered by new economic phenomena such as globalization and digitization, free services, the value of data. Her research has been published in international journals and conference proceedings.

Barry Schouten is a senior methodologist at Statistics Netherlands and Professor at Utrecht University. His research focuses on methods for the reduction and correction of nonresponse and measurement error in surveys and the introduction of smart features in surveys. He has several publications in this field.

References

Bethlehem

. 1988. “Reduction of Nonresponse Bias Through Regression Estimation.” Journal of Official Statistics 4(3): 251–260.

Bethlehem

. 2009. Applied Survey Methods: A Statistical Perspective. 1 ed. Hoboken, NJ: Wiley. doi:https://doi.org/10.1002/9780470494998.

Bethlehem

Cobben

Schouten

. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, NJ: John Wiley & Sons.

Breiman

Friedman

J.H.

Olshen

R.A.

Stone

C.J.

. 1984. Classification And Regression Trees. 1 ed. New York: Taylor & Francis. doi:10.1201/9781315139470.

Chipman

H.A.

George

E.I.

McCulloch

R.E.

. 1998. “Bayesian CART Model Search.” Journal of the American Statistical Association 93(443): 935–948. doi:https://doi.org/10.1080/01621459.1998.10473750.

Coffey

S.M.

Elliott

M.R.

. 2024. “Optimizing Data Collection Interventions to Balance Cost and Quality in a Sequential Multimode Survey.” Journal of Survey Statistics and Methodology 12(3): 741–763. https://doi.org/10.1093/jssam/smad007

Coffey

Reist

Miller

P.V.

. 2020. “Interventions On-Call: Dynamic Adaptive Design in the 2015 National Survey of College Graduates.” Journal of Survey Statistics and Methodology 8(4): 726–747. doi:https://doi.org/10.1093/jssam/smz026.

Groves

R.M.

2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70(5): 646–675. doi:https://doi.org/10.1093/poq/nfl033.

Hedlin

. 2020. “Is there a ‘safe Area’ Where the Nonresponse Rate Has Only a Modest Effect on Bias Despite Non-ignorable Nonresponse?” International Statistical Review 88(3): 642–657. doi:https://doi.org/10.1111/insr.12359.

10.

Kish

. 1988. “Multipurpose Sample Designs.” Survey Methodology 14(1): 19–32.

11.

Little

R.J.

1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review 54(2): 139–157. doi:https://doi.org/10.2307/1403140.

12.

Little

R.J.

2022. “A Note About the Definition of Response Propensity for Survey Nonresponse.” Journal of Survey Statistics and Methodology 10(4): 1098–1106. doi:https://doi.org/10.1093/jssam/smab014.

13.

Little

R.J.

Vartivarian

. 2005. “Does Weighting for Nonresponse Increase the Variance of Survey Means?” Survey Methodology 31(2): 161–168.

14.

Lohr

Hsu

Montaquila

. (2015) “Using Classification and Regression Trees to Model Survey Nonresponse.” , Pp. 2071–2085 in JSM Proceedings, Survery Research Methods Section. Seattle, WA: American Statistical Association. http://www.asasrms.org/Proceedings/y2015/files/234054.pdf.

15.

Luiten

Hox

De Leeuw

. 2020. “Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of An International Study Across Countries and Surveys.” Journal of Official Statistics 36(3): 469–487. doi:https://doi.org/10.2478/jos-2020-0025.

16.

Luiten

Schouten

. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): 169–189. doi:https://doi.org/10.1111/j.1467-985X.2012.01080.x.

17.

Lundquist

Särndal

C.E.

. 2013. “Aspects of Responsive Design with Applications to the Swedish Living Conditions Survey.” Journal of Official Statistics 29(4): 557–582. doi:https://doi.org/10.2478/jos-2013-0040.

18.

Martin

A.D.

Quinn

K.M.

Park

J.H.

2011. “MCMCpack: Markov Chain Monte Carlo in R.” Journal of Statistical Software 42(9): 1–21. doi:https://doi.org/10.18637/jss.v042.i09.

19.

R Core Team. 2023. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

20.

Rubin

D.B.

1976. “Inference and Missing Data.” Biometrika 63(3): 581–592. doi:https://doi.org/10.1093/biomet/63.3.581.

21.

Särndal

C.E.

2011. “The 2010 Morris Hansen Lecture. Dealing with Survey Nonresponse in Data Collection, in Estimation.” Journal of Official Statistics 27(1): 1–21.

22.

Särndal

C.E.

Lundquist

. 2014. “Accuracy in Estimation with Nonresponse: A Function of Degree of Imbalance and Degree of Explanation.” Journal of Survey Statistics and Methodology 2(4): 361–387. doi:https://doi.org/10.1093/jssam/smu014.

23.

Schouten

Bethlehem

Beullens

Kleven

Ø.

Loosveldt

Luiten

Rutar

Shlomo

Skinner

. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80(3): 382–399. doi:https://doi.org/10.1111/j.1751-5823.2012.00189.x.

24.

Schouten

Cobben

Bethlehem

. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35(1): 101–113.

25.

Schouten

Cobben

Lundquist

Wagner

. 2016. “Does More Balanced Survey Response Imply Less Non-response Bias?.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 179(3): 727–748. doi:https://doi.org/10.1111/rssa.12152.

26.

Schouten

Mushkudiani

Shlomo

Durrant

Lundquist

Wagner

. 2018. “A Bayesian Analysis of Design Parameters in Survey Data Collection.” Journal of Survey Statistics and Methodology 6(4): 431–464. doi:https://doi.org/10.1093/jssam/smy012.

27.

Schouten

Peytchev

Wagner

. 2017. Adaptive Survey Design. Boca Raton, FL: CRC Press. ISBN 978-1-4987-6788-0.

28.

Schouten

Shlomo

. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85(1): 143–163. doi:https://doi.org/10.1111/insr.12159.

29.

Schouten

Shlomo

Skinner

. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27(2): 1–24.

30.

Statistics Netherlands. 2020. Health Survey as of 2014. https://www.cbs.nl/en-gb/our-services/methods/surveys/brief-survey-description/health-survey-as-of-2014.

31.

Therneau

Atkinson

. 2019. rpart: Recursive Partitioning and Regression Trees. https://CRAN.R-project.org/package=rpart.

32.

Van Berkel

Van der Doef

Schouten

. 2020. “Implementing Adaptive Survey Design with An Application to the Dutch Health Survey.” Journal of Official Statistics 36(3): 609–629. doi:https://doi.org/10.2478/jos-2020-0031.

33.

Vartivarian

Little

R.J.

. 2002. “On the Formation of Weighting Adjustment Cells for Unit Nonresponse.” Pp. 3553–3558 in JSM Proceedings, Survery Research Methods Section. American Statistical Association.

34.

Wagner

2014. “Limiting the Risk of Nonresponse Bias by using Regression Diagnostics as a Guide to Data Collection.” Pp. 28–38 in JSM Proceedings, Survery Research Methods Section. Boston, MA: American Statistical Association.

35.

Wagner

Zhang

Elliott

M.R.

West

B.T.

Coffey

S.M.

. 2023. “An Experimental Evaluation of a Stopping Rule Aimed At Maximizing Cost-quality Trade-offs in Surveys.” Journal of the Royal Statistical Society Series A: Statistics in Society 186(4): 788–810. doi:https://doi.org/10.1093/jrsssa/qnad059.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.74 MB