Calibration Weighting for Analyzing Non-Probability Samples

1. Introduction

Nonprobability samples (NPS) are increasingly popular in recent years due to their convenience and low cost (Baker et al. 2013). Unlike probability samples, NPS are subject to selection bias, which is difficult to control (Bethlehem 2016; Meng 2018). Because of this difficulty, Rohr et al. (2024) claimed that NPS should not be used for general purpose inferences. Nonetheless, NPS are widely used in many social surveys despite their potential drawbacks. Wu (2022) provides a comprehensive overview of the statistical methods for analyzing NPS survey data.

Mitigating selection bias in nonprobability samples (NPS) remains a critical challenge in survey research. This paper explores calibration weighting as a statistical method to address this bias. Weighting is one of the central estimation steps in survey sampling. Haziza and Beaumont (2017) presented a comprehensive review of the weighting methods in survey sampling practice. While many weighting methods for NPS rely on propensity score (PS) models (Chen et al. 2020; Elliott and Valliant 2017; Valliant and Dever 2011), these models often face significant limitations. Accurately modeling the complex factors influencing survey participation is inherently difficult. Survey participation involves multiple selection processes, including internet access, willingness to participate, and non-response, making it difficult to capture all relevant factors in a single model (Herzing et al. 2022). Moreover, even with a correctly specified PS model, the resulting estimator can exhibit high variability if the propensity score has a weak or no relationship with the study variables of interest (Park et al. 2019).

On the other hand, the proposed calibration weights leverage an outcome regression (OR) model. Thus, it allows for the incorporation of subject-matter knowledge from other surveys to build the OR model and subsequently develop the calibration estimator. The validity of the resulting estimator depends on the plausibility of the OR model assumption and the additional assumption of the ignorability of the sampling mechanism (Sugden and Smith 1984). Furthermore, model diagnostic tools can be used to validate the specified OR model. Thus, in practice, one can have more confidence on the OR model than on the PS model. Notably, this prediction model-based approach to calibration weighting has been considered by Kott and Liao (2012) in the context of handling unit nonresponse.

This paper presents the core concepts of calibration weighting incorporating an outcome regression (OR) model. We then introduce the generalized entropy calibration framework (Kwon et al. 2025) as a unified approach to calibration weighting for nonprobability sample (NPS) data analysis. We conclude by briefly discussing some implications of the algebraic properties of this proposed calibration method and some future research directions.

2. Basic Setup

Consider a finite population with index set $U=\{1,\dots ,N\}$ . We use ${\mathit{x}}_i\in {\mathbb{R}}^p$ and $y_i$ to denote the vector of auxiliary variables and the study variable associated with unit $i$ , respectively. We assume that a non-probability sample $S\subset U$ is selected from the finite population and observe $y_i$ in the sample. We further assume that the auxiliary variables are observed throughout the finite population.

We are interested in estimating $Y=\sum _{i=1}^N{y}_i$ from the sample. To incorporate the auxiliary variable ${\mathit{x}}_i$ observed in the finite population, we consider a linear regression model

$$y_i={\mathit{x}}_i^{\top }\mathit{\beta }+e_i$$

(1)

for some $\mathit{\beta }$ , where $e_i~(0,{\sigma }^2)$ and $e_i$ is independent of ${\mathit{x}}_i$ . Here, an intercept term is included in ${\mathit{x}}_i$ . We assume the sampling mechanism is ignorable in the sense that $\delta \perp Y\mid \mathit{x}$ under model Equation (1), where ${\delta }_i=1$ if $i\in S$ and ${\delta }_i=0$ otherwise. The ignorability assumption means that the regression model in Equation (1) also holds in the sample.

We consider a class of linear estimators defined as

$${\hat{Y}}_{\omega }=\sum _{i\in S}{\omega }_i{y}_i$$

for some ${\omega }_i>0$ that does not depend on $y$ -values. Linear estimators are attractive as they produce internally consistent estimates (Fuller 2009, page 99). We impose that the final weights satisfy the calibration constraint:

$$\sum _{i\in S}{\omega }_i{\mathbf{x}}_i=\sum _{i=1}^N{\mathbf{x}}_i.$$

(2)

Now, using the calibration weights $\{{\hat{\omega }}_i;i\in S\}$ , the weighted regression coefficients are computed as $\hat{\beta }={\left(\sum _{i\in S}{\hat{\omega }}_i{\mathit{x}}_i{\mathit{x}}_i^{\top }\right)}^{-1}\sum _{i\in S}{\hat{\omega }}_i{\mathit{x}}_i{y}_i,$ which implies that

$$\sum _{i\in S}{\hat{\omega }}_i(y_i-{\mathit{x}}_i^{\top }\hat{\beta })=0.$$

(3)

Therefore, we obtain

$$\sum _{i\in S}{\hat{\omega }}_i{y}_i=\sum _{i\in S}{\hat{\omega }}_i{\mathit{x}}_i^{\top }\hat{\beta }=\sum _{i=1}^N{\mathit{x}}_i^{\top }\hat{\beta },$$

(4)

where the first equality follows from Equation (3) and the second equality follows from Equation (2). Equation (4) shows the calibration estimator is indeed a projection estimator (Kim and Rao 2011) using the prediction model in Equation (1). Constraint Equation (3) guarantees that the resulting projection estimator is model-unbiased. The algebraic equivalence in Equation (4) clearly suggests that the calibration estimator is justified under the outcome regression model in Equation (1). That is, the outcome regression model is implicitly used in the calibration weighting, while it is explicitly used in the projection estimation.

Because of the algebraic equivalence in Equation (4), we can use the statistical tools for regression analysis when constructing calibration weights. For instance, sieve estimation method can be used to construct a nonparametric regression projection estimator as in Chan et al. (2015), with polynomial orders determined by minimizing mean squared prediction errors in the validation sample.

3. Generalized Entropy Calibration

If the dimension of the covariate vector is smaller than the sample size, there may exist infinitely many solutions to the calibration equation in Equation (2). To uniquely determine ${\omega }_i$ , Kwon et al. (2025) proposed minimizing the generalized entropy function $\sum _{i\in S}G\left({\omega }_i\right)$ subject to the calibration constraint in Equation (2), where $G\left(\nu \right)$ is a strictly convex and differentiable function. For example, the choice of

$$G\left(\omega \right)=\omega \{\log \left(\omega \right)-1\}$$

leads to the maximum entropy calibration of Hainmueller (2012).

Using the Lagrange multiplier method, the final weight can be expressed as

$${\hat{\omega }}_i\left(\hat{\lambda }\right)=g^{-1}\left({\mathit{x}}_i^{\top }\hat{\lambda }\right),$$

where $g\left(\omega \right)=\mathrm{dG}\left(\omega \right)/d\omega$ and $\hat{\lambda }$ is obtained by solving the calibration equation in Equation (2). For the choice of $G\left(\omega \right)=\omega \{\log \left(\omega \right)-1\}$ , we have ${\hat{\omega }}_i\left(\mathit{\lambda }\right)=\exp \left({\mathit{x}}_i^{\top }\mathit{\lambda }\right)$ . Thus, for a suitable choice of $G(·)$ , we can avoid negative weights.

In practice, we often wish to have the final weights bounded (Huang and Fuller 1978; Rao and Singh 2009). That is, we wish to achieve ${\hat{\omega }}_i\left(\mathit{\lambda }\right)\le M$ for some $M$ which is the upper bound of the final weight. To achieve the goal, one way is to use

$$G^{*}\left(\omega \right)=\{\begin{array}{cc}G\left(\omega \right)\hfill & \left|\omega \right|\le M\hfill \\ \multicolumn{1}{c}{\infty }& \left|\omega \right|>M\hfill \end{array}$$

(5)

as the objective function for calibration.

To determine $M$ to minimize the prediction error, we can treat $M$ as a tuning parameter in the weighted regression and choose an optimal $M$ in a data-driven way. For simplicity, suppose that we use $G\left(\omega \right)=\omega \{\log \left(\omega \right)-1\}$ . In this case, the final weights using Equation (5) can be expressed as

$${\hat{\omega }}_i^{*}(\mathit{\lambda },M)=\min \{\exp \left({\mathit{x}}_i^{\top }\mathit{\lambda }\right),M\},$$

(6)

which is essentially weight trimming. Now, similarly to Equation (4), we can express the final calibration estimator as a projection estimator

$$\sum _{i\in S}{\hat{\omega }}_i^{*}(\hat{\lambda },M)y_i=\sum _{i=1}^N{\mathit{x}}_i^{\top }{\hat{\beta }}_M,$$

where

$${\hat{\beta }}_M={\left\{\sum _{i\in S}{\hat{\omega }}_i^{*}(\hat{\lambda },M){\mathit{x}}_i{\mathit{x}}_i^{\top }\right\}}^{-1}\sum _{i\in S}{\hat{\omega }}_i^{*}(\hat{\lambda },M){\mathit{x}}_i{y}_i$$

(7)

and $\hat{\lambda }$ are determined from the calibration constraints in Equation (2). Thus, the regression coefficient in Equation (7) is computed using the trimmed weights in Equation (6), which depend on $M$ . A standard cross-validation method can be used to determine the optimal $M$ that minimizes the mean squared prediction errors using ${\hat{y}}_i={\mathit{x}}_i^{\top }{\hat{\beta }}_M$ .

4. Discussion

Calibration weighting presents a promising approach to enhancing the utility of nonprobability sample (NPS) data. Instead of relying on potentially unreliable propensity score models, we advocate for using an outcome regression model to construct calibration weights. When the study variable of interest is known, the outcome regression model can be informed by subject-matter knowledge or previous surveys and further validated using model diagnostic tools. This implied regression model framework proves valuable in developing model selection procedures and improving the statistical efficiency of the calibration weighting process. For instance, the weighted regression approach in Section 3 demonstrates how regression techniques can be employed to balance bias and variance trade-offs in weight trimming. Moreover, leveraging the dual relationship between calibration and regression allows for the adaptation of model selection techniques from regression analysis to guide variable selection for calibration constraints. When dealing with a large number of auxiliary variables, exact calibration on all auxiliary variables can lead to highly variable calibration weights and inflate the variance of the final estimator. In such cases, some of the calibration constraints can be relaxed to achieve “soft calibration” (Chambers 1996; Guggemos and Tillé 2010). The soft calibration estimator can be interpreted as a projection estimator using ridge regression. Thus, the unified approach presented in this paper offers a valuable tool for analyzing NPS data under the assumption of ignorable sampling mechanism. Further extensions and applications of this framework will be explored in future work.

Calibration weighting for NPS data, while promising, presents several critical limitations. Firstly, the assumption of an ignorable sampling mechanism, a cornerstone of many weighting methods, is inherently untestable. For a given NPS dataset, it is impossible to definitively ascertain whether this assumption holds true. Consequently, calibration weighting, even when implemented effectively, can only mitigate selection bias rather than completely eliminate it. Its success hinges on the ability of the calibration procedure to reduce bias compared to unweighted estimates. Secondly, calibration weighting may not effectively address bias stemming from undercoverage in the NPS sample. If a specific demographic group is entirely absent from the sample, either due to random chance or systematic undercoverage, the calibration equations may become unsolvable when the auxiliary variables for calibration include indicators for that group. This scenario highlights how calibration weights can provide insights into the data quality of the NPS sample. Thirdly, the curse of dimensionality poses a significant challenge when dealing with high-dimensional covariate spaces. In such cases, the estimation error associated with calibration weights can increase, potentially diminishing the benefits of the approach. While model selection techniques within the regression framework can be employed to reduce the number of covariates in the OR model and improve stability, this introduces the risk of omitted variable bias. This bias, in turn, can weaken the validity of the ignorability assumption of the sampling mechanism. Therefore, selecting the optimal set of calibration variables involves a delicate balance between bias and variance, akin to the trade-offs encountered in traditional regression modeling.

Addressing selection bias due to a potential nonignorable sampling mechanism calls for robust strategies. One such approach involves leveraging nonresponse instrumental variables (IVs) in the calibration process, as proposed by Lesage et al. (2019). These IVs can help mitigate bias when the standard ignorable sampling assumption is violated. However, a crucial limitation arises from the difficulty in verifying the validity of the nonresponse IV assumption. To enhance robustness, researchers can explore multiple nonresponse models and implement a multiply robust estimation procedure, as outlined by Cho et al. (2025). While this approach offers improved protection against model misspecification, it is computationally demanding.

In conclusion, while nonprobability samples (NPS) offer cost-effectiveness in data collection, they can significantly increase the complexity and cost associated with data analysis and dissemination. Applying naive analytical methods to NPS data without careful consideration of the inherent biases can jeopardize public trust in both the survey industry and statistical science. It is incumbent upon statisticians to develop and implement robust analytical strategies that maximize the utility of NPS data, even when they fall short of the ideal sampling framework. We believe that the calibration weighting approach presented in this paper constitutes a valuable step toward achieving this goal.