Measuring Risk of Re-Identification for a Nonprobability Sample Using a General Reference Sample

Abstract

Estimating the risk of re-identification probabilistically is well-developed for the case of a random representative sample drawn from the general population, such as large-scale government surveys conducted regularly at National Statistical Institutes. Recent work extended this procedure to assess the risk of re-identification in non-probability subpopulation registers such as a cancer register. In this paper, we extend this work further to the case of samples drawn from registers or more generally to non-probability samples, such as those used in opt-in panels at survey organizations. The assumption is that membership to the subpopulation register is not known and the sampling mechanism is also unknown. We show how to assess the risk of re-identification for these types of non-probability samples using a probability-based reference sample to infer population parameters under the probabilistic modeling framework. We demonstrate with a simulation study and a real application on the 2021 Survey of Doctoral Recipients drawn from a subpopulation register of all PhD recipients from an accredited US institution.

Keywords

disclosure risk registers non-probability samples propensity scores log-linear modeling

1. Introduction

Much of the literature on quantifiying the risk of re-identification for random sample microdata have used probabilistic modeling to estimate population uniqueness from among sample uniques on a set of indirect quasi-identifying variables that are typically categorical, such as: age, sex, place of residence, marital status, and occupation. See for example, early papers Bethlehem et al. (1990), Benedetti et al. (1998), Fienberg and Makov (1998), Skinner and Holmes (1998), and Skinner and Shlomo (2008). In this setting, the authors focus on quantifying the risk of re-identification for a random probability-based sample drawn from the general population, for example large-scale government surveys such as the Labour Force Survey. The disclosure risk scenario is that an intruder wishes to link an individual in the released sample microdata to the general population through a vector X of indirect quasi-identifying variables and there are no measurement errors in the way vector X is recorded.

As mentioned, the disclosure risk assessment for survey microdata is based on quantifying the risk of re-identification and not on attribute disclosures. This is because there are hundreds of attributes from collected target variables in the survey microdata, and rather than trying to provide a probability of predicting each attribute, the risk of re-identification provides a single interpretable measure of disclosure risk for all attributes. We assume here that identity disclosure is a pre-requisite for individual attribute disclosures and thus, protecting the survey dataset against identity disclosure will avoid attribute disclosures (Willenborg and De Waal 2001).

In any sample (both probability and non-probability surveys), an intruder knows that it is possible that an individual in the sample microdata can be linked to an individual in the population. In this scenario, direct identifying variables, such as name, address, or identification numbers, are removed. Nevertheless, the risk of re-identification can arise when there are small counts on a set of cross-classified indirect quasi-identifying variables. These indirect quasi-identifying variables can be used to identify an individual and further confidential information may be learned from the survey target variables.

Under a probabilistic modeling approach, disclosure risk is assessed on the contingency table of sample counts spanned by the indirect quasi-identifying variables. The assumption is that the sample microdata contain responding individuals from a survey and the population counts are unknown. The risk of re-identification is therefore a function of both the population and the sample and measured in terms of population uniqueness, for example, the probability that a sample unique in a cell of the contingency table is a population unique. Besides measuring the disclosure risk for the sample microdata, the sample is also used as a data source for estimating population parameters and the disclosure risk measures under the probabilistic modeling framework.

Shlomo and Skinner (2022) consider a new setting where there may be microdata available from a register that represents a particular subpopulation in the general population. The register may be publicly available but the membership of the subpopulation may not be known. The subpopulation is not representative of the general population as is the case for a random probability-based sample in the previous setting. A register is a type of administrative dataset, and is considered non-probability data when it is used for statistical purposes. This is because its selection is not based on random chance rather on some specific inclusion criteria, leading to potential biases when attempting to generalize to the general population. Unlike probability data, which requires a defined sampling frame to ensure each unit has a known non-zero chance of selection, a register contains individuals who have been identified and included for non-random reasons (Golini and Righi 2024).

Similar to the original approach, Shlomo and Skinner (2022) also assume the same disclosure risk scenario in that an intruder aims to link an invidiual in the subpopulation register to an individual in the population of which the subpopulation is a subset and the population counts are unknown. They also assume that there are categorical indirect quasi-identifying variables X that may allow for inference about population uniqueness and in addition, that there are no measurement errors in the way the data is recorded.

In order to allow inference about the general population uniqueness in the subpopulation, Shlomo and Skinner (2022) assume that there also exists survey microdata from a random sample to be used as a general probability-based reference sample. Thus, whereas in the original work, particularly in Elamir and Skinner (2006) and Skinner and Shlomo (2008), the sample microdata file served two purposes, one as the file about which disclosure risk is a concern and one for inference about population uniqueness, Shlomo and Skinner (2022) assume that we need to resort to using separate files for these two purposes.

In this paper, we demonstrate how the developed theory may be extended to assess disclosure risk in non-probability samples drawn from a subpopulation register or more generally to non-probability samples, such as those obtained from opt-in internet panel members within survey agencies where inclusion probabilities are unknown.

Section 2 reviews the original theory of assessing the risk of reidentification in probability-based random survey microdata and is followed by the case where we assess the risk of re-identification for a subpopulation register using a probability-based reference sample in Section 3. We extend the theory in Subsection 3.1 to include samples drawn from a register or more generally to non-probability samples where inclusion probabilities and the sample mechanism are not known. We demonstrate the theory in a simulation study in Section 4. Section 5 presents an application to assess the disclosure risk in the public-use microdata of the Survey of Doctoral Recipients (SDR) 2021 (National Center for Science and Engineering Statistics (NCSES) 2023). The application includes assessing the disclosure risk of re-identification according to the original method where the statistical agency would like to know the number of subpopulation uniques on the set of sample uniques according to the set of indirect quasi-identifying variables, assuming that the intruder would have access to the register. Moreover, we also show the more realistic intruder scenario where we assess population uniqueness with respect to the general population on the set of sample uniques. For this approach, we need a probability-based reference sample drawn from the general population to be used to correct for the selection bias in the non-probabilty sample. For the application, we use the American Community Survey (ACS) 2021 (available here: https://www.census.gov/programs-surveys/acs/microdata/access.html) as the probability-based reference sample. We conclude in Section 6 with a discussion.

2. Overview of Assessing Disclosure Risk for Probability-Based Sample Microdata

Individual per-record risk measures in the form of a probability of re-identification are estimated using a probabilistic model. These per-record risk measures are then aggregated to obtain global risk measures for the entire file. We denote F_k the population size in cell k of a table spanned by indirect quasi-identifying variables having K cells, f_k the sample size in cell k, $\sum_{k} F_{k} = N$ and $\sum_{k} f_{k} = n$ . The set of sample uniques is defined by: $SU = {k : f_{k} = 1}$ and these are the high-risk records with the potential to be population uniques. We focus on the global disclosure risk measure defined as the number of sample uniques that are population uniques τ₁ (where I is the indicator function):

\begin{matrix} τ_{1} = \sum_{k} I (f_{k} = 1, F_{k} = 1) . The risk measure is estimated by : \\ {\hat{τ}}_{1} = \sum_{k} I (f_{k} = 1) \hat{P} (F_{k} = 1 | f_{k} = 1) . \end{matrix}

Skinner and Holmes (1998), Elamir and Skinner (2006), and Skinner and Shlomo (2008) propose using a Poisson Distribution as the probabilistic modeling framework and a log-linear model to estimate population parameters to calculate τ₁. In this model, they assume that $F_{k} ~ Pois (λ_{k})$ for each cell k. A sample is drawn by Poisson or Bernoulli sampling with a sampling fraction π_k in cell k: $f_{k} | F_{k} ~ Bin (F_{k}, π_{k})$ . It follows that: $f_{k} ~ Pois (π_{k} λ_{k})$ and $F_{k} | f_{k} ~ Pois (λ_{k} (1 - π_{k}))$ where the population cell counts F_k are assumed independent given the sample cell counts f_k. The parameters λ_k are estimated using log-linear modeling.

The sample frequencies f_k are independent Poisson distributed with a mean of $μ_{k} = π_{k} λ_{k}$ . A log-linear model for the μ_k is expressed as: $\log (μ_{k}) = x_{k}' β$ where $x_{k}$ is a design vector which denotes the main effects and interactions of the model for the indirect quasi-identifying variables. The maximum likelihood (MLE) estimator $\hat{β}$ is obtained by solving the score equations:

\sum_{k} (f_{k} - π_{k} \exp (x_{k}^{'} β)) x_{k} = 0 .

(1)

The fitted values are then calculated by: ${\hat{μ}}_{k} = \exp (x_{k}^{'} \hat{β})$ and to obtain population parameters we define : ${\hat{λ}}_{k} = {\hat{μ}}_{k} / π_{k}$ . Individual disclosure risk measures for cell k are: $P (F_{k} = 1 | f_{k} = 1) = \exp (λ_{k} (1 - π_{k}))$ . Plugging ${\hat{λ}}_{k}$ for λ_k leads to the estimates $\hat{P} (F_{k} = 1 | f_{k} = 1)$ and then to ${\hat{τ}}_{1}$ .

Skinner and Shlomo (2008) developed goodness of fit criteria, termed the B-statistics, that can be used to assess the model fit of the log-linear model assuming the Normal distribution critical value of 1.96. The B-statistics represent the deviation from the assumptions of the first and second moments of the Poisson distribution, and when the value is small, we can expect a good fit under the log-linear modeling. They found that for the type of large-scale probability-based surveys conducted within National Statistical Institutes, such as the Labour Force Survey, an all 2-way interactions log-linear model generally provides accurate results with respect to estimated disclosure risk measures, as this model strikes a balance between structural and random zeroes in the contingency table spanned by the indirect quasi-identifying variables.

Skinner and Shlomo (2008) also consider the case of a complex sample design where differential weights are assigned to each unit in the random probability-based sample. The λ_k can be estimated consistently using a pseudo-maximum likelihood estimation where the estimating equation is modified to:

\sum_{k} ({\hat{F}}_{k} - \exp (x_{k}^{'} β)) x_{k} = 0

(2)

and ${\hat{F}}_{k}$ is obtained by summing the survey weights over individuals i, w_i, in cell k: ${\hat{F}}_{k} = \sum_{i \in k} w_{i}$ (Rao and Thomas 2003). In other words, instead of estimating ${\hat{λ}}_{k} = {\hat{μ}}_{k} / π_{k}$ as shown under score function (1), we can estimate $\hat{λ}$ directly from score function Equation (2). The fitted values are then calculated by ${\hat{λ}}_{k} = \exp (x_{k}^{'} \hat{β})$ . The resulting estimates ${\hat{λ}}_{k}$ are plugged into ${\hat{τ}}_{1}$ and π_k is replaced by the estimate ${\hat{π}}_{k} = f_{k} / {\hat{F}}_{k}$ . This approach is relevant as it will be used to assess the disclosure risk of re-identification for the simulation study in Section 4 and the application of SDR 2021 in Section 5 according to the framework described in Section 3.

3. Assessing Disclosure Risk for a Non-Probability Subpopulation Register

The theory for measuring the risk of re-identification for a non-probability subpopulation register was developed in Shlomo and Skinner (2022). Let U and U₁ denote the population and the subpopulation, respectively, with $U_{1} \subset U$ . Let R_i be the subpopulation (sample) indicator variable for individual i with R_i = 1 if $i \in U_{1}$ and R_i = 0 otherwise. As before, we label the possible combinations of the indirect quasi-identifying variables by k, k = 1, …, K and refer to each combination as a cell in a multi-way contingency table. In addition, as before, we denote the population frequency in cell k by F_k so that the cell is population unique if F_k= 1 and the population counts are unknown.

Denote the subpopulation count in cell k as $F_{k}^{1} .$ The most high-risk records are for those cells with a unique unit $F_{k}^{1} = 1$ and, analogous to the derivation presented above, an alternative risk measure for the case that we have a subpopulation is: $τ_{1}^{*} = \sum_{k} P (F_{k} = 1 | F_{k}^{1} = 1) I (F_{k}^{1} = 1)$ . The subpopulation microdata provides information about the $F_{k}^{1}$ but not about the F_k in U. As a non-probability subpopulation register, the distribution of X in U₁ may be very different to that in U so the subpopulation carries no direct information about the F_k. We assume that there is a random sample microdata file s drawn from the finite population U in which the values of X are recorded. We suppose that the sample microdata and the subpopulation register are not linked, for example by an ID number, so we do not know if there are any individuals in the subpopulation register that have also been drawn into the sample. Let f_k denote the frequency in cell k in s. The counts f_k and $F_{k}^{1}$ are observed, but the F_k are not. If the intruder has access to the sample microdata file, then it may be advantageous to restrict attention to cells with f_k= 1, leading to the following risk measures $τ_{1}^{* *} = \sum_{k} P (F_{k} = 1 | F_{k}^{1} = 1, f_{k} = 1) I (F_{k}^{1} = 1, f_{k} = 1)$ .

To estimate disclosure risk measures $τ_{1}^{*}$ and $τ_{1}^{* *}$ for a subpopulation register, we assume that F_k is Poisson distributed, $F_{k} ~ Pois (λ_{k})$ where the parameter λ_k follows the log-linear model $\log (λ_{k}) = x_{k}' β$ . We also assume that within cell k the unknown membership variable R_i takes the value 1 with probability p_k, independently for each of the F_k units, so that $F_{k}^{1} ~ Pois (ϕ_{k})$ where $ϕ_{k} = λ_{k} p_{k}$ , and the subpopulation $F_{k}^{1}$ are binomially distributed $F_{k}^{1} | F_{k} ~ Bin (F_{k}, p_{k})$ conditional on the F_k. Further, we assume that p_k follows the logistic model: $logit (p_{k}) = x_{k}' ξ$ . Shlomo and Skinner (2022) then estimate the individual disclosure risk measures under this setting using the estimates in Equations (3) and (4) where the focus is on the subpopulation unique cells $F_{k}^{1} = 1$ as follows:

P (F_{k} = 1 | F_{k}^{1} = 1) = P (F_{k} - F_{k}^{1} = 0) = \exp (- (λ_{k} - ϕ_{k}))

(3)

and:

\begin{matrix} P (F_{k} = 1 | F_{k}^{1} = 1, f_{k} = 1) = \frac{π_{k} ϕ_{k} \exp (- (1 - π_{k}) (λ_{k} - ϕ_{k}))}{π_{k} ϕ_{k} + (1 - π_{k}) π_{k} ϕ_{k} (λ_{k} - ϕ_{k})} \\ = \frac{\exp (- (1 - π_{k}) (λ_{k} - ϕ_{k}))}{1 + (1 - π_{k}) (λ_{k} - ϕ_{k})} \end{matrix}

(4)

3.1. Extending the Theory to Non-Probability Samples

From here, we wish to extend the theory from quantifying the disclosure risk of a non-probability subpopulation register to the case where we have a sample s₁ drawn from the non-probability subpopulation register U₁, or more generally a non-probabillity sample where membership to a sample frame and a sample mechanism is unknown. We do not observe the entire subpopulation, rather we observe a non-probability sample s₁ drawn from some hypothetical subpopulation $U_{1} .$ For this case, we need to estimate the disclosure risk measures in Equations (3) and (4) for the non-probability sample s₁. We denote the sample frequencies in the non-probability sample in cell k by $f_{k}^{1}$ and now define the individual risk measures as follows:

P (F_{k} = 1 | f_{k}^{1} = 1)

(5)

and

P (F_{k} = 1 | f_{k}^{1} = 1, f_{k} = 1)

(6)

with the estimation of parameter ϕ_k shown in Equations (3) and (4) to account for the non-probability sample counts $f_{k}^{1}$ as explained below.

We expect smaller disclosure risk measures for the non-probability sample s₁ in Equations (5) and (6) compared to the case of estimating the disclosure risk measures for the subpopulation U₁ as shown in Shlomo and Skinner (2022) in Equations (3) and (4) due to the extra variation and uncertainty arising from some (unknown) sampling procedure.

As in the original setting described in Section 2, we assume we have a random sample s drawn from a finite population U in which the values of X are recorded. We change the focus to cases that are unique in the population conditional on both sample s and the non-probability sample s₁.

In a theoretical sense, for a sample drawn from the non-probability subpopulation register, we can assume that: $f_{k}^{1} | F_{k}^{1} ~ Bin (F_{k}^{1}, π_{k}^{1})$ and $f_{k}^{1} ~ Pois (π_{k}^{1} ϕ_{k})$ , and therefore we would be able to estimate ϕ_k using the sample counts $f_{k}^{1}$ drawn from the subpopulation U₁. However, as mentioned, for a non-probability sample or a sample drawn from a subpopulation register, the sample fractions $π_{k}^{1}$ are not known.

If the F_k and $F_{k}^{1}$ were observed it would be straightforward to express the likelihood for estimating the unknown parameters as a function of two components, one dependent on $β$ via $F_{k} ~ Pois (λ_{k})$ and one dependent on $ξ$ via $F_{k}^{1} | F_{k} ~ Bin (F_{k}, p_{k})$ , and thus carry out the estimation in one step. However, we only observe f_k and $f_{k}^{1}$ and the conditional distribution $f_{k}^{1} | f_{k}$ cannot be expressed in general as a function of p_k. We therefore propose to carry out an estimation procedure in a two-step approach.

In the first step, we use the full microdata of the non-probability sample and estimate the propensity score: ${\tilde{p}}_{i}$ $i = 1, \dots, n_{1}$ for each individual, where n₁ is the number of individuals in the non-probability sample. To use this approach, Chen et al. (2020) list the assumptions needed to produce unbiased estimates: (1) The inclusion indicator of the non-probability sample is independent of any target variable given a set of covariates X ; (2) All units have a non-zero propensity score to be included in the non-probability sample; (3) Inclusion indicators for units i and j are independent given x_i and x_j for $i \neq j$ . The second assumption may be problematic given a specific subpopulation of interest although in theory we can assume a non-zero chance of individuals having membership in this subpopulation. We assume these assumptions are met in our setting. We calculate estimates for the population totals ${\hat{F}}_{k}$ by inverse probability weighted (IPW) estimation and then estimate λ_k on the non-probability sample using a log-linear model according to the pseudo-maximum likelihood estimation shown in Equation (2) as described in Section 2.

We set out the proposed estimation procedure in detail as follows:

Step 1: Estimate for each individual in the non-probability sample the propensity score ${\tilde{p}}_{i}$ , $i = 1, \dots, n_{1}$ where n₁ is the number of individuals in the sample. The estimation is carried out by using the random sample microdata file as the reference sample. We assume that the random sample microdata have survey weights defined for each individual j, w_j, $j = 1, \dots, n$ . We therefore use the approach proposed in Chen et al. (2020) to estimate ${\tilde{p}}_{i}$ assuming an unknown sampling mechaism and assigning a weight of 1 on the non-probability sample.

Step 2: Since the propensity score ${\tilde{p}}_{i}$ corrects for selection bias in the non-probability sample, we can now calculate estimates for population totals ${\hat{F}}_{k}$ by inverse probability weighted (IPW) estimation using the non-probability sample. Thus, we have an estimate of the population by ${\hat{F}}_{k} = \sum_{i \in k} 1 / \hat{\tilde{p_{i}}}$ where $\hat{\tilde{p_{i}}}$ is the estimate of ${\tilde{p}}_{i}$ . Note that we also calibrate ${\hat{F}}_{k}$ to known (weighted) distributions obtained from the reference sample to allow for more precise population totals within weighting classes. This is known as the doubly robust estimator for the finite population total as described in Chen et al. (2020).

Step 3: Using the population estimates ${\hat{F}}_{k}$ from Step 2, we calculate marginal distributions to be used for iterative proportional fitting (Bishop, et al. 1975) to obtain estimates λ_k under the log-linear modeling according to the pseudo-maximum likelihood approach with estimating Equation (2). The goodness-of-fit criteria (B statistics) mentioned in Section 2 can be used in this setting to ensure the fit of the log-linear model, for example, using all 2-way or all 3-way interactions model (or a combination of both). Typically, an all 2-way interactions model works well in realistic settings.

Step 4: Defining ${\hat{p}}_{k} = 1 / {\hat{F}}_{k}$ we estimate ${\hat{ϕ}}_{k} = {\hat{λ}}_{k} {\hat{p}}_{k}$ and calculate the risk measures $τ_{1}^{*}$ and $τ_{1}^{* *}$ according to the estimates shown in Equations (5) and (6).

4. Simulation Study

We use the simulation set-up from Shlomo and Skinner (2022) based on the UK Census 2001 synthetic data having the following variables (between brackets are the number of categories): Geography (6), Age group (14), Sex (2), Marital Status (6), Ethnicity (16), Economic Activity (10), and Ill Health (2) with N = 1,003,401. The subpopulation register contains individuals having ill health where N₁ = 179,699. We produce a multiway contingency table of size K = 161,280 cells defined by all variables except Ill Health. Shlomo and Skinner (2022) compared the distributions in the population and the subpopulation data for quasi-identifying variables Age Group, Sex and Economic Activity (see Table 1) and showed that the subpopulation mainly contained the elderly population.

Table 1.

Average of 200 Iterations (Simulation Standard Errors in Parenthesis).

Disclosure risk measures	True values	Estimate
$τ_{1}^{*} = \sum_{k} I (F_{k} = 1, f_{k}^{1} = 1)$	543.9 (1.51)	583.4 (1.28)
$\sum_{k} I (f_{k}^{1} = 1)$	3,150 (3.28)	3,150 (3.28)
Proportion $τ_{1}^{*}$	0.173	0.185
$τ_{1}^{* *} = \sum_{k} I (F_{k} = 1, f_{k}^{1} = 1, f_{k} = 1)$	26.7 (0.38)	32.4 (0.28)
$\sum_{k} I (f_{k}^{1} = 1, f_{k} = 1)$	589 (1.69)	589 (1.69)
Proportion τ₁**	0.045	0.055

The simulation steps were as follows:

Step 1: Draw 200 random samples without replacement from the population using Bernoulli sampling where π= 1/20 and resulting in a sample size of n = 50,171 on average. Draw 200 random samples without replacement from the sub-population using Bernoulli sampling where π¹= 1/5 and resulting in a sample size of n₁ = 35,940 on average.

Step 2: Use the samples as the reference sample and nonprobability sample, respectively, to estimate ${\tilde{p}}_{i}$ . We use the indirect quasi-identifying variables: Geography, Age group, Sex, Marital Status, Ethnicity, and Economic Activity and the main effects design matrix for the propensity score estimation.

Now we follow Steps 1 to Step 4 in Subsection 3.1 to calculate the risk measures $τ_{1}^{*}$ and $τ_{1}^{* *}$ based on estimates for Equations (5) and (6) using a combination of 2-way and 3-way interactions that provided a good fit as described in Section 2.

Table 1 shows the results of the estimation of the disclosure risk measures $τ_{1}^{*}$ and $τ_{1}^{* *}$ .

From Table 1 we see a slight bias in the estimates of the disclosure risk measures based on the selected log-linear model with a combination of 2-way and 3-way interactions. The measures are slightly overestimated and therefore are more conservative disclosure risk measures. As mentioned, we used a combination of 2-way and 3-way interactions because of the large sample sizes drawn compared to their populations: 1/20 for the probability sample and 1/5 for the non-probability sample from the subpopulation register. A more in-depth analysis of goodness-of-fit might have improved the fit of the models. For typical and more realistic sample fractions, we generally use the all 2-way interaction model as will be shown in the application in Section 5.

Furthermore, from the simulation set-up in Shlomo and Skinner (2022) we obtained that the true (average) value of population uniques and subpopulation U₁ uniques according to Equation (3) was 2,721 and the true (average) value of population uniques, subpopulation U₁ uniques, and random sample s uniques was 55.5. Given we are now assessing the disclosure risk in the non-probability sample s₁ drawn from some hypothetical subpopulation U₁, we obtain a true (average) value of 543.9 for the number of population uniques and non-probability sample s₁ uniques, and the true (average) value of 26.7 for the number of population uniques, non-probability sample s₁ uniques, and random sample s uniques. As expected, we obtain smaller values of disclosure risk measures when assessing the disclosure risk on a non-probability sample from a subpopulation versus the disclosure risk measures on the whole subpopulation.

5. Risk of Re-Identification in the Survey for Doctoral Recipients 2021

The dataset of the Survey for Doctoral Recipients (SDR) 2021 (National Center for Science and Engineering Statistics (NCSES) 2023) included 80,295 individuals in a sample drawn from a subpopulation register with the estimated size of approximately 1,186,000 individuals between the ages 25 to 75 having received a PhD at an accredited university in the United States (US) in the area of science, engineering, and health. As mentioned in Section 1, a subpopulation register is considered a non-probability data source because its selection is not based on random chance rather on the administration processes of a data collection authority. Hence, any sample from a register is also a non-probablity sample and we assume that the sampling mechanism to a general population is unknown.

The sample for SDR is drawn from a subpopulation register referring to all persons awarded a PhD from an accredited US institution in a given year, called the Survey of Earned Doctorates (SED). The SED is an annual census conducted since 1957 and is sponsored by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) and by three other federal agencies: the National Institutes of Health, Department of Education, and National Endowment for the Humanities.

We now turn to the disclosure risk assessment of SDR 2021. We start with the original framework described in Section 2 where we assume that an intruder would try and match unique cells derived from a set of indirect quasi-identifying variables in SDR 2021 to PhD recipients in the subpopulation register of the SED and therefore we calculate the probability that a sample unique is unique in the subpopulation register τ₁ based on the estimate $\hat{P} (F_{k}^{1} = 1 | f_{k}^{1} = 1) = \exp ({\hat{λ}}_{k} (1 - π_{k}))$ where π_k are the sampling fractions in SDR 2021. Note that this disclosure risk scenario assumes that an intruder would have access to labeled register data of the SED. We denote this approach as the “Original Approach” in Table 3. We use the following indirect quasi-identifying variables: NSDRMENTOD (Field of Study—26 categories); GENDER (2 categories); AGEGRP (10 categories); RACETHMP (Race\Ethnicity—5 categories); BTHUS (Born in the US—2 categories); CTZN_DRF (Type of Citizenship—5 categories).

In this original framework, we use the pseudo-maximum likelihood log-linear model to estimate the population parameters λ_k, k = 1, …K, where the marginals for the log-linear model are obtained by summing the weighted sample counts ${\hat{F}}_{k} = \sum_{i \in k} ω_{i}$ and ω_i are the original sample weights in SDR 2021 (these are calculated by the raking-ratio method to sub-population counts obtained from the SED). To carry out the log-linear modeling, we initialize each cell of the table spanned by the indirect quasi-identifying variables with “1” and use iterative proportional fitting to fit the distributions of the margins (Bishop et al. 1975). We also calculate the B-statistic to assess the fit of the log-linear model and the accuracy of the disclosure risk measure. It was difficult to identify a log-linear model that fit the non-probability sample SDR 2021 drawn from the SED register as there is selection bias compared to the general population of PhD recipients. The best fitting log-linear model was an all 4-way interactions model as described in Table 3.

We now turn to the new approach described in Subsection 3.1. Here, we assess the general population uniqueness for the SDR 2021 according to the general population of PhD recipients (and not the subpopulation register). This would be a more realistic disclosure risk assessment with respect to the statistical agency as the subpopulation of labeled PhD recipients in the SED would not be available to potential intruders. To assess the disclosure risk in the general population, we use a subset of the American Community Survey (ACS) 2021 as the probability-based reference sample. The subset included only those respondents declaring that they had a PhD and in the appropriate ages between 25 and 75. This led to a sample size of 35,755 individuals and the estimated population size 3,288,181.00 after summing the weights in the ACS 2021. The inconsistencies in the population sizes between SDR 2021 and ACS 2021 are due to the non-probability nature of SDR 2021 and in particular, the ACS includes respondents declaring a PhD degree in any subject and from any university, including universities abroad. With these inconsistencies, we expect that estimating population uniqueness to the general population of PhD degree recipients will have a lower risk of re-identification in SDR 2021 compared to estimating the subpopulation uniqueness based on the SED in the Original Approach.

We follow the procedure as outlined in Subsection 3.1 and denote this as the “New Approach” in Table 3. To estimate the propensity score ${\tilde{p}}_{i}$ $i = 1, \dots, n_{1}$ according to step 1 for SDR 2021, we use ACS 2021 as the reference sample and follow the logistic modeling approach described in Chen et al. (2020) where we assume a main effects model and utilize the weights in the ACS in the estimation procedure. The weights in SDR 2021 are set to 1. Only five common variables were identified between the two data sources to estimate the propensity score. The variables that were used in the logistic modeling approach and their distributions are shown in Table 2. After estimating the propensity score in SDR 2021 we obtain the sum of inverse propensities equal to: 3,378,447.54. We therefore benchmark the inverse propensities to the population total in ACS 2021 to 3,288,181, that is, we multiplied the inverse propensities by a common correction factor: C = 3,288,181/3,378,447.54.

Table 2.

Variables and Their Distributions Used to Estimate the Propensity Score in SDR 2021 According to ACS 2021.

Variable	ACS 2021		SDR 2021		SDR 2021 (weighted by inverse propensity and benchmark)
	Weighted count	Percent	Sample count	Percent	Weighted count	Percent
Age group
25–29	152,534	4.6	607	0.8	147,397	4.5
30–34	372,272	11.3	7,290	9.1	384,615	11.7
35–39	431,115	13.1	11,874	14.8	446,224	13.6
40–44	402,289	12.2	11,472	14.3	402,058	12.2
45–49	352,826	10.7	9,307	11.6	343,618	10.5
50–54	336,254	10.2	8,490	10.6	328,607	10.0
55–59	304,374	9.3	8,208	10.2	295,457	9.0
60–64	296,821	9.0	7,864	9.8	292,832	8.9
65–69	305,914	9.3	7,484	9.3	307,281	9.3
70–74	333,782	10.2	7,699	9.6	340,092	10.3
Total	3,288,181	100	80,295	100	3,288,181	100
Sex
Male	1,778,149	54.1	46,163	57.5	1,799,464	54.7
Female	1,510,032	45.9	34,132	42.5	1,488,717	45.3
Total	3,288,181	100	80,295	100	3,288,181	100
Citizenship
Born US/US territories	2,306,987	70.2	49,992	62.3	2,313,727	70.4
US citizen born abroad	41,775	1.3	488	0.6	45,203	1.4
Naturalization	544,471	16.6	14,516	18.1	544,232	16.6
Not a US citizen	394,948	12.0	15,299	19.1	385,019	11.7
Total	32,88,181	100	80,295	100	3,288,181	100
Employment
Employed	2,673,647	81.3	69,315	86.3	2,687,103	81.7
Unemployed	52,946	1.6	1,381	1.7	50,620	1.5
Not in Labor Force	561,588	17.1	9,599	12.0	550,459	16.7
Total	3,288,181	100	80,295	100	3,288,181	100
Race
Asian alone	555,499	16.9	16,701	20.8	543,654	16.5
Black/African American	227,238	6.9	4,814	6.0	214,803	6.5
Hispanic alone	230,948	7.0	7,701	9.6	221,799	6.7
White alone	2,131,548	64.8	48,742	60.7	2,153,260	65.5
All other	142,948	4.3	2,337	2.9	154,665	4.7
Total	3,288,181	100	80,295	100	3,288,181	100

From Table 2, we can see large differences between the probability-based reference sample ACS 2021 in columns 2 and 3, and the original unadjusted SDR 2021 in columns 4 and 5, for example, a larger percentage of males, employed, non-citizens, and Asians in SDR compared to ACS. The last 2 columns in Table 2 show the adjusted inverse propensity weighted (IPW) SDR 2021 distributions and it can be seen that these weighted SDR 2021 distribution are more similar to the ACS 2021 distributions.

We now turn to estimating the risk of re-identification for SDR 2021 according to population uniqueness in the general population of PhD recipients following steps 2 through 4 in Subsection 3.1. We use the same indirect quasi-identifying variables as shown for the Original Approach framework: NSDRMENTOD (Field of Study—26 categories); GENDER (2 categories); AGEGRP (10 categories); RACETHMP (Race\Ethnicity—5 categories); BTHUS (Born in the US—2 categories); CTZN_DRF (Type of Citizenship—5 categories). However, because these variables are not defined in the probability-based reference sample ACS 2021, this means that we do not have the sample uniques f_k= 1 on these variables and are not able to calculate $τ_{1}^{* *}$ according to the estimate based on Equation (6). We therefore only calculate $τ_{1}^{*}$ according to the estimate based on Equation (5). In step 2, we model the all 2-way interaction pseudo-likelihood log-linear model on SDR 2021. These marginal totals are obtained by summing the appropriate weighted bivariate counts from SDR 2021: ${\hat{F}}_{k} = \sum_{i \in k} 1 / \hat{\tilde{p_{i}}}$ where $\hat{\tilde{p_{i}}}$ is the estimate of ${\tilde{p}}_{i}$ that was obtained in the previous step 1. The log-linear model produces the estimates for the population parameters λ_k, k = 1, …K shown in step 3. Moving to step 4, we calculate ${\hat{p}}_{k} = 1 / {\hat{F}}_{k}$ and estimate ${\hat{ϕ}}_{k} = {\hat{λ}}_{k} {\hat{p}}_{k}$ and calculate the risk measure $τ_{1}^{*}$ according to the estimate in Equation (5).

Table 3 contains the two disclosure risk measures. Under the Original Approach, the estimate for the number of SDR 2021 unique cells that are also unique on the set of indirect quasi-identifying variables in the subpopulation register of doctoral recipients (SED) is approximately 251, although the B-statistic is 5.75 and above the expected level of 1.96. This informs of potential over-estimation. However, SDR 2021 contains only a subset of PhD recipients and is a non-probability dataset with distributions that differ from the general population of PhD recipients. We can use the weights in SDR 2021 to infer on population uniqueness to the SED, but these weights are very skewed ranging from 1 to 100 with a median of 10. We see that this non-probability sample of SDR 2021 is difficult to model according to the original log-linear modeling setting in Section 2, and we obtain unreliable goodness-of-fit (B-statistic) criteria as the dataset clearly deviates from a random probability-based sample of which the theory was developed. Moreover, from the perspective of the statistical agency, it is unrealistic to assume that an intruder would have access to the labeled subpopulation register of the SED to attempt to make a re-identification.

Table 3.

Final Risk Measures Under Original and New Approach.

Original approach	New approach
Survey weighted margins obtained from SDR 2021 (all 4-way interactions)	IPW weighted margins obtained from SDR 2021 (all 2-way interactions)
τ ₁= $\sum_{k} I (F_{k}^{1} = 1, f_{k}^{1} = 1)$ $\sum_{SU} \exp (λ_{k} (1 - π_{k}))$	$τ_{1}^{*}$ = $\sum_{k} I (F_{k} = 1, f_{k}^{1} = 1) =$ $\sum_{SU} \exp (- (λ_{k} - ϕ_{k}))$
250.72 (5.75)	58.84 (1.53)

Under the New Approach, the statistical agency releasing SDR 2021 data is interested in identifying population uniqueness according to the general population of PhD recipients. We therefore rely on a probability-based reference sample to estimate the risk of re-identification. The result for the New Approach in Table 3 is approximately 59 sample uniques that are general population uniques, with a B-statistic (under the pseudo-maximum likelihood approach) of 1.53. In this approach, the B-statistic is more reliable and the log-linear model using the inverse propensity weighted SDR 2021 is more stable.

6. Discussion

The original approach of assessing the risk of re-identification in surveys from non-probability registers or non-probability samples assumes that membership to a nonrepresentative subpopulation is known in order to make a re-identification from the survey microdata. This is currently the case for SDR 2021 where weighted margins from the survey are used to fit log-linear models to estimate population parameters for disclosure risk measures despite the problems of trying to fit models with data that have selection biases and skewed distributions, such as recipients of a PhD in science, engineering, and health. The original approach of measuring disclosure risk assumes the scenario that an intruder would have access to the labeled subpopulation register from which the SDR 2021 is drawn. Whether this is a viable disclosure risk scenario depends on the statistical agency perspective and the data environment on whether the information on the particular set of PhD recipients is accessible public knowledge.

If, however, the statistical agency wants to assess the risk of re-identification in SDR 2021 against the general population of PhD recipients, we can turn to using a probability-based reference sample to obtain propensity scores for the SDR 2021 where the inverse propensity scores provide weights that are a representation of this general population. As can be seen in the application, given the discrepancies in distributions between SDR 2021 and ACS 2021 shown in Table 2, the risk of re-identification under the new approach, where the ACS 2021 is used as the probability-based reference sample, is much reduced by almost 75%—from 251 estimated register-based population uniques that are sample uniques, to 59 estimated general population uniques that are sample uniques.

This application shows different estimates of the risk of re-identification under different disclosure risk scenarios. It is the first attempt to apply new theory shown in Section 3 and to assess the viability of this approach where non-probability subpopulations and non-probability samples are not representative to the general population for assessing disclosure risk.

Footnotes

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Natalie Shlomo

Received: July 11, 2024

Accepted: January 14, 2026

References

Benedetti

Capobianchi

Franconi

1998. “Individual Risk of Disclosure Using Sampling Design Information.” Contributi Istat. https://ebiblio.istat.it/digibib/Contributi%20ISTAT/2003_14-1.pdf (accessed July 9, 2024).

Bethlehem

Keller

Pannekoek

1990. “Disclosure Control of Microdata.” Journal of the American Statistical Association 85 (409): 38–45. DOI: https://doi.org/10.1080/01621459.1990.10475304.

Bishop

Fienberg

Holland

1975. Discrete Multivariate Analysis: Theory and Practice. MIT Press. DOI: https://doi.org/10.1007/978-0-387-72806-3.

Chen

2020. “Doubly Robust Inference with Non-Probability Survey Samples.” Journal of the American Statistical Association 115 (532): 2011–21. DOI: https://doi.org/10.1080/01621459.2019.1677241.

Elamir

Skinner

C. J.

2006. “Record-Level Measures of Disclosure Risk for Survey Microdata.” Journal of Official Statistics 22 (3): 525–39. https://media.proquest.com/media/hms/OBJ/mFveU?_s=KFrmhhdVckrYJwc96j1UZZf5JCQ%3D (accessed July 9, 2024).

Fienberg

S. E.

Makov

U. E.

1998. “Confidentiality, Uniqueness and Disclosure Limitation for Categorical Data.” Journal of Official Statistics 14 (4): 385–97. https://www.proquest.com/openview/7da3f33250f6da2ed71169bcc83d1046/1?pq-origsite=gscholar&cbl=105444 (accessed July 9, 2024).

Golini

Righi

2024. “Integrating Probability and Big Non-Probability Samples Data to Produce Official Statistics.” Statistical Methods and Applications 33: 555–80. DOI: 10.1007/s10260-023-00740-y.

National Center for Science and Engineering Statistics (NCSES). 2023. “Survey of Doctorate Recipients, 2021.” NSF 23-319, National Science Foundation, Alexandria, VA. https://ncses.nsf.gov/pubs/nsf23319 (accessed July 11, 2024).

Rao

J. N. K.

Thomas

D. R.

2003. “Analysis of Categorical Response Data from Complex Surveys: An Appraisal and Update.” In Analysis of Survey Data, edited by Chambers

R. L.

Skinner

C. J.

Wiley. DOI: https://doi.org/10.1002/0470867205.

10.

Shlomo

Skinner

C. J.

2022. “Measuring Risk of Re-Identification in Microdata: State-of-the Art and New Directions.” Journal of the Royal Statistical Society Series A: Statistics in Society 185 (4): 1644–62. DOI: https://doi.org/10.1111/rssa.12902.

11.

Skinner

C. J.

Holmes

1998. “Estimating the Re-Identification Risk per Record in Microdata.” Journal of Official Statistics 14 (4): 361–72. https://www.proquest.com/openview/5a1bd315f0e8673267a3645fa86a1349/1?pq-origsite=gscholar&cbl=105444 (accessed July 9, 2024).

12.

Skinner

C. J.

Shlomo

2008. “Assessing Identification Risk in Survey Micro-Data Using Log Linear Models.” Journal of the American Statistical Association 103 (483): 989–1001. DOI: https://doi.org/10.1198/016214507000001328.

13.

Willenborg

De Waal

2001. Elements of Statistical Disclosure Control in Practice. Lecture Notes in Statistics, 155. Springer-Verlag. DOI: https://doi.org/10.1007/978-1-4613-0121-9.