Comparing Regression Adjustment,Matching,and Inverse Probability Weights in Small Sub-Populations in the HIV SUCCESS Consortium: A Real-World Data Example

Abstract

Objectives

To understand the trade-offs between different statistical modeling approaches, using real world data with small sub-populations, with rare exposures and increasingly rare outcomes. In particular, to compare adjusted regression, inverse probability weighting, and matching.

Methods

Data for these analyses came from the RADAR (N=1,134) and combined CNICS/JHHCC (N=14,434) cohorts. We estimated prevalence ratios (PRs) for self-reported use of specific substances comparing subpopulations (SP), SP-1 vs SP-3 and SP-2 vs SP-3, where SP-1 was the largest proportion (92% in RADAR, 81% in CNICS/JHHCC), SP2 was moderate proportion (18% in CNICS/JHHCC) and SP-3 was the smallest proportion (8% in RADAR, 1% in CNICS/JHHCC) of the population. We calculated PRs using 1) unadjusted relative risk regression (RR); and adjusted estimates controlling for age, race/ethnicity, study site, and year of interview using: 2) standard adjustment in RR; 3) stabilized inverse probability of treatment weighting (IPTW); and 4) matching with up to 3 matches from SP-1 or SP-2 per SP-3 participant.

Results

For most substances, all methods yielded consistent estimates. There were large weights in some of the IPTW analyses and in three cases these resulted in substantially divergent estimates. For the comparison between SP-1 and SP-3, the estimate for smoking was 1.3-fold greater in the matched analysis (PR=1.33, 95% CI: 1.02-1.75) than in IPTW (PR=1.03, 95% CI: 0.78-1.37 ATE and PR=1.05, 95% CI: 0.88-1.26 ATT). Even more extreme divergence in estimates was observed for differences between SP-2 and SP-3 with respect to methamphetamine/amphetamine, (IPTW-ATE: PR=1.51, 95% CI: 0.79-2.89; IPTW-ATT: PR=2.36, 95% CI: 1.36-4.12); vs Matching: PR=2.71, 95%CI: 1.47-4.99) and cocaine (IPTW-ATE: PR=1.15, 95% CI: 0.55-2.38; IPTW-ATT: PR=1.81, 95% CI: 1.03-3.18); vs Matching: PR=1.38, 95%CI: 0.76-2.53).

Conclusion

The combination of a rare exposure and a rare outcome can produce challenges for commonly used confounding adjustment strategies, and it is often best to compare different modeling approaches to gain greater insight.

Keywords

matching inverse probability of treatment weighting (IPTW)positivity small samples small population sparse data bias

Introduction

Accurate comparisons of health burdens in small exposure groups is often limited by available sample size. Thus, dealing with small subpopulations is a substantial methodological challenge, but critical to epidemiology and population health. Specifically, analyses of small populations can be particularly vulnerable to violations of the positivity assumption,¹ sparse data bias,^2-4 and general modeling problems with sparse data.^5,6 While these biases are well established, there are limited practical examples of when to be concerned about these biases and how to decide which estimates to be concerned about, especially in the context of weighting where these effects can be enhanced.^7,8

Commonly used adjustment techniques in epidemiology include, but are not limited to, regression adjustment, inverse probability of treatment weighting (IPTW), and matching. Regression adjustment is one of the simplest and most frequently used approaches for addressing confounding in the association between a given predictor and outcome and generally performs well with multiple covariates given a large enough sample size, model assumptions are met, and collinearity between covariates is not an issue.⁹ While a popular approach due to ease-of-use, regression adjustment is often less desirable than causal inference techniques, such as IPTW. IPTW adjusts for confounding by creating a pseudo-population in which the covariate distribution is similar between groups by reweighting each individual in the study based on the inverse of the probability for being assigned to a given group.¹⁰ IPTW has gained popularity as a modelling technique for observational studies over the past few decades,¹¹ in part due to straightforward interpretability of adjustment for confounding, enabling direct estimation of causal effects, integration with modern causal frameworks, and ease of statistical application, among others.^12,13 However, IPTW validity depends on the correct specification of the propensity score model. If this model is misspecified,¹⁴ the resulting weights may fail to adequately balance covariates between groups leading to biased estimates. Adjusting for confounding at the study design stage by using matching is also a popular approach. However, only the individuals of interest and their matched counterparts are included,^15,16 potentially resulting in loss of some information, which may reduce generalizability to the study population. While IPTW^17,18 and matching¹⁹ have advantages, larger sample sizes have tended to make IPTW more popular. These methodological challenges are not merely statistical concerns, they directly affect clinical interpretation in small or vulnerable populations, where incorrect inference can lead to misleading estimates of risk and inappropriate targeting of interventions.

The purpose of this study was to better understand the properties and potential failure points of regression adjustment, IPTW, and matching in datasets with a rare subgroup. To do this, we used a practical example of a rare subgroup and increasingly rare outcomes (types of substance use) with complex distributions in the sample. We conducted diagnostics to determine straightforward ways to decide which method showed the best performance under these conditions.

Methods

Data, Participants & Ethics

The HIV and Substance Use Cohort Coordinating Center (CC) for Emerging and High Impact Scientific Cross Cohort Studies (HIV SUCCESS) is a collaboration of the National Institute of Drug Abuse (NIDA) funded substance use and HIV cohorts, which brings together data and people to address unanswered questions around the intertwining HIV and substance use epidemics and promotes knowledge to end the HIV epidemic in the United States. The collaborating HIV and substance use cohorts include participants from a wide range of demographically diverse people in different geographic locations who are at risk for or living with HIV.²⁰ Cohorts collect information on demographic characteristics and health-related behaviors, with some conducting medical record review for clinical information, as well as, testing of biological samples for important biomarkers (e.g., HIV status, HIV viral load, immune expression, etc.).

Data for this study came from two of the NIDA-funded HIV and substance use cohorts, the RADAR cohort and the Johns Hopkins HIV Clinical Cohort (JHHCC), as well as from the Centers for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS) cohort. RADAR, conducted at Northwestern University, is a longitudinal, interval, community cohort of English-speaking people assigned male at birth who were between 16 and 20 years of age at enrollment, had a sexual encounter with a man in the previous year, and who reside in the Chicago metropolitan area.^21,22 RADAR participants complete interviews on social relationships, health behaviors and outcomes, and syndemic service usage and barriers; HIV testing and/or biomarker testing; and biospecimens are collected for banking every six months. The JHHCC is a clinical cohort of adult (≥18 years) people with HIV (PWH) receiving primary care at Johns Hopkins.²³ PWH who attend HIV primary care practices at Johns Hopkins University-run clinics and have consented to participate in the cohort complete questionnaires on social characteristics, behaviors, and patient reported outcomes every ∼6 months. These data are combined with key clinical, laboratory, visit, and diagnosis data. CNICS is a 10 site, US-wide, clinical cohort of PWH receiving primary care.²⁴ Data in CNICS comes from electronic health records and other clinical sources, as well as from the CNICS clinical assessment of patient reported outcomes and measures (PROs), completed by PWH collected as part of their routine HIV care appointments every ∼6 months.²⁵ For this study, data from seven CNICS sites were included: Fenway Health, Boston; University of Alabama, Birmingham; University of North Carolina, Chapel Hill; Case Western Reserve University, Cleveland; University of California, San Diego; University of California, San Francisco; and University of Washington, Seattle. All participants from these studies with a patient reported measure of substance use were eligible for this study.

Protocols, data collection, and data use for each cohort received ethical approval from the institutional review boards (IRB) of all involved universities. All participants completed informed consent. Harmonization and use of cross-cohort data through HIV SUCCESS has received IRB exemption as it involves use of de-identified data that cannot be directly linked back to cohort participants. All methods for this study were carried out in accordance with relevant guidelines and regulations.

Measures

In order to illustrate examples of different methods of adjustment and explore potential corresponding issues, we selected a demographic variable with three categories in the CNICS and JHHCC cohorts and two categories within the RADAR cohort. In order to focus on methodological issues rather than associations, which have been previously reported for CNICS,²⁶ we refer to these subgroups as subpopulation (SP)-1, SP-2 and SP-3. CNICS and JHHCC include all three subgroups with most people identifying with SP-1 or SP-2 and only 1.3% identifying with SP-3. RADAR only includes SP-1 and SP-3, with 92.3% of people in SP-1 and 7.7% in SP-3. To further illustrate the point, there are many small subgroups of interest that might meet these criteria in epidemiological studies, including rare racial/ethnic groups in a population, such as Indigenous/Native American people; people who report uncommon behaviors, such as use of xylazine, or competing in ironman triathlons; and those with rare exposures such as rare solvents, medications, or Sin Nombre virus; among many others.

We included different types of substance use as our outcome variables, and included substances that are more commonly reported, such as cigarette smoking, alcohol, and cannabis, as well as substances that are reported by fewer people, such as cocaine/crack, methamphetamine/amphetamine, and illicit opioids. Cigarette smoking in the past 6 months was defined in RADAR using the question, “When was the last time you smoked a cigarette, even a puff?” which was dichotomized into “yes” for people who responded that they had smoked in the prior 6 months and “no” for everyone else including those who reported never smoking in their lifetime. In CNICS and JHHCC, current smoking was measured by response to the question, “Do you currently smoke tobacco cigarettes?” where respondents could answer “yes” or no”. Alcohol use was measured across all three cohorts using the Alcohol Use Disorders Identification Test-Concise (AUDIT-C), in which participants in RADAR were asked about alcohol use in the past 6 months and those in CNICS and JHHCCC were asked about the last year. Cannabis, cocaine/crack, methamphetamine/amphetamine, and illicit opioid use were collected across all three cohorts using a modified version of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) in which participants in RADAR were asked about frequency of use in the past 6 months and those in CNICS and JHHCC were asked about use in the past 3 months. Current drug use questions from the ASSIST (e.g., frequency of use) were dichotomized into any use in the past 6 months (RADAR) or 3 months (CNICS and JHHCC) by collapsing any use within the timeframe as “yes” and no use during that timeframe or over the lifetime as “no”. High agreement between 3 and 6 month drug recall has been previously observed in the JHHCC and other cohorts in the Collaborating Consortium of Cohorts Producing NIDA Opportunities, of which RADAR was a member.²⁷

As covariates we included age in years and race/ethnicity from a 4-level categorical variable, including White, Black/African American, Latine/Hispanic, and Another race/ethnicity. Additionally, we included year of participant interview, and geographic site for the CNICS and JHHCC cohort. Due to consistency in questions asked and data structure, CNICS and JHHCC were combined for the purposes of analyses.

Analyses

We applied three different common adjustment strategies: regression adjustment, IPTW, and individual-level matching (hereafter referred to as “matching”). The estimand of IPTW for average treatment effect (ATE) is the marginal average exposure effect in the full population, contrasting the effect of everyone being exposed versus nobody being exposed; however, the estimand for the average treatment effect of the treated (ATT) can also be calculated, which represents the average exposure effect of treatment among the treated. Matching and adjusted regression estimate a conditional effect that is defined as average effect, at the participant level, of moving a participant from subpopulation (SP)-1 (“unexposed”) to SP-3 (“exposed”) while holding all covariates constant.²⁸ Matching also has the further property of removing participants who do not have a match, which is often considered to be an advantage²⁹ when it identifies areas of non-overlap in propensity to be exposed. Regression methods provide a conditional estimate within the same target population as IPTW and do not require the positivity assumption because they extrapolate over areas lacking positivity and are included as a comparison due to their wide use. As participants could decline to answer questions, but few did, we used only those with complete data in our analyses.

Matching

In each cohort participants were matched 1:1 on race/ethnicity, +/-2 years of interview date and +/-5 years of age, basing our approach on previously published results in the CNICS portion of this study population.²⁶ CNICS/JHHCC participants were further matched 1:1 on study site, which was not necessary for RADAR as all participants live in the Chicago metropolitan area. Matches were randomly selected (with replacement) from individuals meeting the matching parameters using a uniform random number generator with a fixed seed for reproducibility in Stata. In the CNICS/JHHCC cohort, people from SP-3 were matched to a maximum of 3 people from SP-1 and 3 people from SP-2. In the RADAR cohort, people from SP-3 were matched to a maximum of 3 people from SP-1; SP-2 was not a subgroup in RADAR. Given the modest size of the “treatment” groups, 1:3 matching was used to improved precision and retain power while limiting the poorer-quality matches that can arise with larger matching ratios.³⁰ Matching with replacement is recommended when ratio of exposed to unexposed is high for better balance, while avoiding reliance on only a few units.³⁰

IPTW

We calculated both ATE and ATT, where ATE represents what would happen if everyone was treated vs untreated and ATT represents the effect of treatment among the treated; with “treatment” in our example corresponding to SP-3. While ATT and ATE are expected to be similar when treatment effects are homogeneous and the treated and untreated populations are comparable, ATT is most similar to the result provided from the matched analysis.^31-33 In our example, it would be very odd if ATT and the average effect in the untreated (ATU) varied for contrasts between SP1 and SP3, but it is not impossible that SP-2 and SP-3 could have effect heterogeneity for some outcomes.³⁴ To create stabilized inverse probability weights for ATE, we first estimated the individual probability of belonging to SP-3 and SP-1 (and separately, SP-3 and SP-2) using logistic regression models where age, race/ethnicity, and year of interview were the predictors, with study site as an additional predictor in CNICS/JHHCC; the same variables used for matching, following the procedures for stabilized inverse probability weights in Stata specified by Hernan and Robins¹² and code from Murray and Logan,³⁵ which calculate SW_i (ATE)= [A_i*P(A=1) + (1-A_i)*P(A=0)]/[A_i*P(A=1|L_i) + (1-A_i)*P(A=0|L_i)]. Nonparametric bootstrapping was used to produce robust 95% confidence intervals (CI) with the propensity score model refitted inside each bootstrap sample. For ATT, we use the same methods to calculate SW_i (ATT) = A_i + (1-A_i)*[P(A=1|L_i)/P(A=0|L_i)]. We also estimated inverse probabilities using Firth’s penalized logistic regression models using these same methods, but using the corrected probabilities.^36,37 In order to examine the potential for separation in our data we plotted Firth’s stabilized penalized inverse probability weights against those from the logistic regression. We examined the mean, standard deviation, and range of weights for each comparison to check for severe positivity deviations and extreme weights. We then used these weights in a regression model to balance exposure groups.^38-40 Sensitivity analyses were conducted by trimming the 0.5% and 1.0% of the most extreme high and low weights.

Standard Regression Adjustment

We estimated crude and adjusted prevalence ratios (PRs) for self-reported use of each specific substance comparing subgroups in pairs (SP-3 vs SP1 and SP-3 vs SP-2) using relative risk regression, the modified Poisson version⁴¹ where a Poisson family and log link are specified instead of binomial and logit, with a robust variance in Stata. The robust variance also accounted for the correlation caused from matched data.

Comparison of Results

In order to examine notable differences in IPTW and matching, we made violin plots of the stabilized inverse probabilities weights of IPTW results. Overlap plots were used to compare the propensity score distribution by exposure status. Additionally, we provide a “qualitative” table of the most extreme large and small weights. We also calculated the percent difference between estimated prevalence ratios of IPTW (both ATE and ATT) and matching as follows: ((Estimate/Ref)-1)*100, with ≥20% difference considered notable. Additionally, we estimated effective sample size (ESS) using the Kish estimator as a diagnostic to improve understanding the variability of the inverse probability weights, not for inferential purposes, by dividing the sum of weights squared by the sum of squared weights.⁴² Our inferences about precision relied on our estimates of robust standard errors. Analyses were performed in Stata version 19.0 (College Station, TX) and SAS 9.4 (Cary, NC).

Results

In the combined CNICS/JHHCC cohort, 193 PWH in SP-3 matched to three PWH in SP-1; one person from SP-3 had no matches to PWH in SP-1 (Table 1). In contrast, 10 PWH (5%) in SP-3 failed to match to any other person in SP-2, 17 people from SP-3 matched to only one person in SP-2, 20 matched to two people in SP-2 and only 147 (76%) matched to three other PWH in SP-2. In RADAR all 87 people from SP-3 matched to three other people in SP-1.

Table 1.

Results of Matching

Number matches to SP-3	CNICS/JHHCC		RADAR
Number matches to SP-3	SP-1	SP-2	SP-1
None	1	10	0
One	0	17	0
Two	0	20	0
Three	193	147	87

The combined CNICS/JHHCC cohort included 11,691 PWH in SP1, 2,549 PWH in SP-2, and 194 PWH in SP-3 (Table 2). Overall, people in SP-3 tended to be younger than those in either SP-1 or SP-2, although years when PWH were interviewed was similar across all three groups. Compared to SP-1, people in SP-3 tended to report White race/ethnicity at a lower proportion and Black/African American, Latine/Hispanic, or another race/ethnicity at higher proportions. However, when compared to SP-2, people in SP-3 reported White race/ethnicity at a similar proportion to those in SP-2 and Black/African American race/ethnicity at a lower proportion, but Latine/Hispanic and another race/ethnicity at higher proportions. There were also differences by site in these three subpopulation groups. Matched set A (SP-3 matched to SP-1) included 579 PWH in SP-1 and 193 PWH in SP-3. Matched set B (SP-3 matched to SP-2) included 498 PWH in SP-2 and 184 PWH in SP-3.

Table 2.

Summary of Participants by Total and Match Sets in CNICS/JHHCC

	All participants			Matched set A		Matched set B
	SP-1 (N=11,691) N (%)	SP-2 (N=2,549) N (%)	SP-3 (N=194) N (%)	SP-1 (N=579) N (%)	SP-3 (N=193) N (%)	SP-2 (N=498) N (%)	SP-3 (N=184) N (%)
Demographic characteristics
Age category
18-29 years	697 (5.9)	87 (3.4)	23 (21.9)	63 (10.9)	23 (11.9)	41 (8.2)	22 (12.0)
30-39 years	2,248 (19.2)	333 (13.1)	58 (29.9)	182 (31.4)	58 (30.1)	157 (31.5)	53 (28.8)
40-49 years	2,326 (19.9)	571 (22.4)	52 (26.8)	135 (23.3)	51 (26.4)	122 (24.5)	49 (26.6)
50-59 years	3,445 (29.5)	841 (33.0)	39 (20.1)	135 (23.3)	39 (20.2)	117 (23.5)	39 (21.2)
≥ 60 years	2,975 (25.5)	717 (28.1)	22 (11.3)	64 (11.1)	22 (11.4)	61 (12.3)	21 (11.4)
Last interview year
2015- 2017	1,689 (14.5)	288 (11.3)	25 (12.9)	68 (11.7)	24 (12.4)	64 (12.9)	24 (13.0)
2018 -2020	2,135 (18.3)	498 (19.5)	36 (18.6)	106 (18.3)	36 (18.7)	84 (16.9)	34 (18.5)
2021-2023	7,867 (67.3)	1,763 (69.2)	133 (68.6)	405 (70.0)	133 (68.9)	350 (70.3)	126 (68.5)
Race/Ethnicity
White	5,259 (45.0)	537 (21.1)	40 (20.6)	120 (20.7)	40 (20.7)	106 (21.3)	39 (21.2)
Black/African American	3,871 (33.1)	1,684 (66.1)	88 (45.4)	264 (45.6)	88 (45.6)	244 (49.0)	87 (47.3)
Latine/Hispanic	1,870 (16.0)	256 (10.0)	50 (25.8)	150 (25.9)	50 (25.9)	125 (25.1)	48 (26.1)
Another Race/Ethnicity	691 (5.9)	72 (2.8)	16 (8.3)	45 (7.8)	15 (7.8)	23 (4.6)	10 (5.4)
Site
1	722 (6.2)	205 (8.0)	7 (3.6)	21 (3.6)	7 (3.6)	21 (4.2)	7 (3.8)
2	1,172 (10.0)	31 (1.2)	22 (11.3)	66 (11.4)	22 (11.4)	26 (5.2)	16 (8.7)
3	869 (7.4)	494 (19.4)	8 (4.1)	24 (4.2)	8 (4.2)	24 (4.8)	8 (4.4)
4	2,297 (19.7)	663 (26.0)	43 (22.2)	126 (21.8)	42 (21.8)	122 (24.5)	42 (22.8)
5	2,919 (25.0)	365 (14.3)	41 (21.1)	123 (21.2)	41 (21.2)	114 (22.9)	41 (22.3)
6	964 (8.3)	103 (4.0)	32 (16.5)	96 (16.6)	32 (16.6)	74 (14.9)	29 (15.8)
7	1,197 (10.2)	454 (17.8)	23 (11.9)	69 (11.9)	23 (11.9)	66 (13.3)	23 (12.5)
8	1,551 (13.3)	234 (9.2)	18 (9.3)	54 (9.3)	18 (9.3)	51 (10.2)	18 (9.8)
Substance use past 3 months ^*
Smoking	2,733 (23.4)	646 (25.5)	57 (29.4)	126 (21.8)	56 (29.0)	105 (21.1)	52 (28.3)
At-risk Alcohol ^†	2,662 (26.6)	497 (22.2)	40 (25.3)	29.7 (14.8)	39 (24.8)	114 (27.8)	35 (23.3)
Heavy episodic Alcohol ^‡	3,399 (33.4)	563 (24.8)	58 (35.8)	202 (39.9)	57 (35.4)	146 (34.8)	52 (33.8)
Cannabis	3,499 (42.5)	513 (28.6)	72 (53.7)	189 (45.7)	71 (53.4)	122 (35.4)	69 (54.8)
Cocaine/Crack	674 (7.9)	148 (7.5)	15 (11.4)	40 (9.5)	14 (10.7)	29 (8.0)	14 (11.1)
Illicit opioids	315 (3.8)	85 (4.6)	6 (4.5)	17 (4.0)	6 (4.5)	22 (6.0)	6 (4.8)
Methamphetamine	1,102 (12.6)	57 (2.9)	19 (13.9)	69 (15.8)	19 (14.0)	18 (4.7)	18 (13.9)
Illicit Amphetamine	121 (1.4)	15 (0.7)	4 (2.8)	5 (1.1)	4 (2.8)	1 (0.3)	4 (3.0)
Methamphetamine/Amphetamine	1,158 (9.9)	68 (2.7)	20 (10.3)	71 (12.2)	20 (10.4)	19 (3.8)	19 (10.3)

*N varies by substance as substance types were added over time and participants could decline to answer.

†defined as AUDIT-C score of ≥4 for cisgender men and ≥3 for women.

‡defined as ≥5 drinks per sitting for women and ≥6 drinks per sitting for men.

There were 1,047 people in SP-1 and 87 in SP-3 in the RADAR cohort (Table 3). Most participants were 18-39 years old, with a greater proportion of those in SP-3 in the 30-39 age group. People in SP-3 also had a slightly higher proportion of people who were last interviewed between 2021-2023 and fewer between 2015-2017 than in SP-1. Additionally, participants in SP-3 reported Black/African American race/ethnicity at higher proportions than those in SP-1 and lower proportions in White and Latine/Hispanic racial/ethnic groups. Matching balanced participants by these characteristics; however, this resulted in bigger differences in proportions reporting some types of substance use over the past 6 months between people in SP-1 and those in SP-3.

Table 3.

Summary of Participants by Total and Match Sets in RADAR

	All participants		Matched set
	SP-1 (N=1,047) N (%)	SP-3 (N=87) N (%)	SP-1 (N=261) N (%)	SP-3 (N=87) N (%)
Demographic characteristics
Age Category
18-29 years	795 (75.9)	56 (64.4)	185 (70.9)	56 (64.4)
30-39 years	251 (24.0)	31 (35.6)	76 (29.1)	31 (35.6)
40-49 years	1 (0.1)	0	0	0
Last Year Interview
2015- 2017	116 (11.1)	7 (8.1)	18 (6.9)	7 (8.1)
2018 -2020	302 (28.8)	24 (27.6)	75 (28.7)	24 (27.6)
2021-2023	629 (60.1)	56 (64.4)	168 (64.4)	56 (64.4)
Race/Ethnicity
White	271 (25.9)	11 (12.6)	33 (12.6)	11 (12.6)
Black/African American	341 (32.6)	49 (56.3)	147 (56.3)	49 (56.3)
Latine/Hispanic	321 (30.7)	17 (19.5)	51 (19.5)	17 (19.5)
Another Race/Ethnicity	114 (10.7)	10 (11.5)	30 (11.5)	10 (11.5)
HIV positive
Substance use past 6 months ^*
Smoking	286 (27.3)	25 (28.7)	65 (24.9)	25 (28.7)
At-Risk Alcohol ^†	480 (45.9)	18 (20.7)	102 (39.1)	18 (20.7)
Heavy Episodic Alcohol ^‡	610 (58.3)	25 (28.7)	127 (48.7)	25 (28.7)
Cannabis	734 (70.1)	50 (57.5)	179 (68.6)	50 (57.5)
Cocaine/Crack	141 (13.5)	11 (12.6)	37 (14.2)	11 (12.6)
Illicit Opioids	23 (2.2)	1 (1.2)	6 (2.3)	1 (1.2)
Methamphetamine	31 (3.0)	4 (4.6)	6 (2.3)	4 (4.6)
Illicit Amphetamine	65 (33.5)	3 (20.0)	14 (37.8)	3 (20.0)
Methamphetamine/Amphetamine	90 (8.6)	7 (8.1)	18 (6.9)	7 (8.1)

*N varies by substance as substance types were added over time and participants could decline to answer.

†defined as AUDIT-C score of ≥4 for cisgender men and ≥3 for transgender and cisgender women.

‡defined as ≥5 drinks per sitting for women and ≥6 drinks per sitting for men.

Regression adjusted, IPTW (both ATE and ATT), and matched analyses in RADAR provided similarly interpreted results across all substance types (Table 4). Both heavy episodic alcohol and cocaine results differed by ≥20% in point estimates between matching and IPTW, however the inclusion of 1 in the 95% CI would result in the same interpretation. A greater number of discrepancies were noted for the combined CNICS/JHHCC cohort. When comparing SP-1 to SP-3, the point estimate for smoking in the matched analysis was 21% higher than ATT and 22% higher than ATE in the IPTW analysis; it also had a tighter 95% confidence interval (CI) and was statistically significant as opposed to either IPTW estimate which did not demonstrate a difference in smoking between the two groups. Again, comparing SP-1 to SP-3, the estimates for methamphetamine/amphetamine by ATT in IPTW and matched estimates were similar, but were 22% different between ATE and match estimates; however, all estimates had confidence intervals that overlapped and contained one. The SP-2 and SP-3 groups had more differences in point estimates for IPTW and matching than the other two comparisons and also demonstrated differences in ATE and ATT. The estimates for both heavy episodic alcohol and illicit opioids varied by >20% for matching and IPTW, with ATE and matching giving very similar results for heavy episodic alcohol use and ATT having a higher point estimate and all three providing >20% difference in point estimates for illicit opioid use. There was a 1.8-fold difference between ATE from IPTW and matching in the point estimate for recent methamphetamine/amphetamine use, but the difference was only 13% for ATT from IPTW and matching. With respect to recent cocaine use, ATT from IPTW had a 31% higher, and significant point estimated compared to match, which did not demonstrate significance; while ATE from IPTW was more similar to matching, it was also not significant. Firth’s penalized regression showed very similar results to those of standard region for IPTW estimates and trimming extreme IPTW weights resulted in similar results most of the time, with extreme changes observed in some instances, likely due to changing the population included in the analyses (see Supplemental Table 1). Comparison of standard logistic regression and Firth’s penalized weights indicated no obvious separation (see Supplemental Figure 1).

Table 4.

Prevalence Ratios (PR) of Substance Use by Subpopulation and Adjustment Method

Measure (N)	Model	SP-1 vs SP-3 CNICS/JHHCC	SP-1 vs SP-3 RADAR	SP-2 vs SP-3 CNICS/JHHCC
Measure (N)	Model	PR (95%CI)	PR (95%CI)	PR (95%CI)
Smoking	Crude	1.16 (0.97, 1.39)	1.05 (0.75, 1.49)	0.95 (0.79, 1.15)
	Adjusted	0.99 (0.83, 1.19)	1.01 (0.72, 1.44)	0.98 (0.81, 1.19)
	IPTW (ATE)	1.03 (0.78, 1.37)	1.09 (0.74, 1.60)	0.89 (0.60, 1.32)
	IPTW (ATT)	1.05 (0.88, 1.26)	1.01 (0.71, 1.44)	1.02 (0.82, 1.25)
	Matched	1.33 (1.02, 1.75)	1.15 (0.78, 1.71)	1.08 (0.86, 1.36)
At-risk Alcohol ^*	Crude	0.95 (0.73, 1.25)	0.45 (0.30, 0.69)	1.14 (0.86, 1.51)
	Adjusted	0.86 (0.65, 1.12)	0.53 (0.36, 0.80)	0.97 (0.72, 1.29)
	IPTW (ATE)	0.97 (0.67, 1.40)	0.64 (0.42, 0.97)	0.89 (0.56, 1.42)
	IPTW (ATT)	0.84 (0.64, 1.10)	0.53 (0.35, 0.80)	0.97 (0.72, 1.31)
	Matched	0.84 (0.62, 1.13)	0.53 (0.34, 0.82)	0.84 (0.60, 1.17)
Heavy episodic Alcohol	Crude	1.07 (0.87, 1.32)	0.49 (0.35, 0.69)	1.44 (1.16, 1.80)
	Adjusted	0.90 (0.73, 1.11)	0.57 (0.41, 0.79)	1.15 (0.91, 1.45)
	IPTW (ATE)	0.94 (0.68, 1.30)	0.65 (0.46, 0.91)	0.97 (0.65, 1.47)
	IPTW (ATT)	0.91 (0.73, 1.12)	0.57 (0.41, 0.80)	1.23 (0.97, 1.56)
	Matched	0.89 (0.70, 1.12)	0.59 (0.42, 0.84)	0.97 (0.75, 1.26)
Cannabis	Crude	1.27 (1.08, 1.48)	0.82 (0.68, 0.99)	1.88 (1.58, 2.23)
	Adjusted	1.13 (0.97, 1.32)	0.83 (0.69, 1.00)	1.65 (1.37, 1.99)
	IPTW (ATE)	1.18 (0.92, 1.52)	0.79 (0.63, 0.99)	1.42 (0.96, 2.09)
	IPTW (ATT)	1.15 (0.98, 1.35)	0.84 (0.69, 1.01)	1.62 (1.32, 2.00)
	Matched	1.16 (0.97, 1.42)	0.83 (0.69, 1.02)	1.55 (1.25, 1.92)
Cocaine/Crack	Crude	1.43 (0.89, 2.32)	0.93 (0.53, 1.67)	1.51 (0.92, 2.50)
	Adjusted	1.14 (0.70, 1.86)	1.22 (0.68, 2.17)	1.58 (0.91, 2.76)
	IPTW (ATE)	1.23 (0.60, 2.51)	1.29 (0.71, 2.37)	1.15 (0.55, 2.38)
	IPTW (ATT)	1.21 (0.75, 1.97)	1.11 (0.62, 2.01)	1.81 (1.03, 3.18)
	Matched	1.13 (0.64, 2.01)	0.89 (0.48, 1.67)	1.38 (0.76, 2.53)
Illicit opioids	Crude	1.76 (0.53, 2.59)	0.52 (0.07, 3.83)	0.98 (0.44, 2.21)
	Adjusted	1.06 (0.48, 2.36)	0.65 (0.08, 5.03)	0.91 (0.39, 2.11)
	IPTW (ATE)	1.22 (0.50, 2.93)	0.56 (0.08, 4.13)	0.62 (0.19, 1.94)
	IPTW (ATT)	1.13 (0.51, 2.51)	0.52 (0.07, 3.94)	0.96 (0.41, 2.27)
	Matched	1.12 (0.45, 2.78)	0.50 (0.06, 4.11)	0.78 (0.33, 1.89)
Methamphetamine/Amphetamine	Crude	1.04 (0.69, 1.58)	0.94 (0.45, 1.96)	3.87 (2.40, 6.23)
	Adjusted	0.86 (0.58, 1.28)	1.29 (0.62, 2.69)	2.33 (1.44, 3.77)
	IPTW (ATE)	0.66 (0.37, 1.18)	1.13 (0.50, 2.53)	1.51 (0.79, 2.89)
	IPTW (ATT)	0.90 (0.59, 1.37)	1.23 (0.57, 2.63)	2.36 (1.36, 4.12)
	Matched	0.85 (0.53, 1.35)	1.17 (0.50, 2.70)	2.71 (1.47, 4.99)

*At risk alcohol defined as AUDIT-C score of ≥4 for cisgender men and ≥3 for women.

Bold font indicates that point estimates differ by ≥20% and would be interpreted differently.

Bold italicized font indicates that point estimates differ by ≥20%, but are unlikely to be interpreted differently.

Violin plots of the stabilized inverse probability weights for ATE demonstrate large variation in weights, with an example shown in Figure 1 of methamphetamine/amphetamine use. All other substances are shown in Supplemental Figures 3-7. We also provide overlap plots (Figure 2) for each comparison “cohort” (CNICS/JHHCC: SP-1 vs SP-3, RADAR: SP-1 vs SP-3, and CNICS/JHHCC: SP-2 vs SP-3) comparing the exposed and unexposed, which demonstrate comparable patterns, suggesting that there are no obvious positivity violations.

Examining the summary statistics for the stabilized inverse probability weights for ATE by study and populations compared (Table 5), we observed similar means in stabilized inverse probability weights between SP-1 and SP-3 for RADAR, and while there are some differences in standard deviation and median values, the ranges for stabilized inverse probability weights in both SP-1 and SP-3 were modest. In contrast, for the CNICS/JHHCC combined cohorts we observed larger differences in means compared between SP-1 and SP-3 and SP-2 and SP-3. Furthermore, the range of stabilized inverse probability weights for SP-1 and SP-2 were both modest and reasonable, however, the ranges for SP-3 were wide, demonstrating both very low and very high stabilized inverse probability weights for some people in SP-3. These extremes were greater in IPTW analyses comparing SP-2 to SP-3 as opposed to SP-1 to SP-3. Examining the ATT IPTW weights for the treated group (SP-3) by definition are set to 1 for all individuals, and it is the ESS of the comparator group that changes. One concern with matching is that we had to drop 1 individual in the SP-1 vs SP-3 comparison and 10 individuals in the SP-2 vs SP-3 comparison for CNICS/JHHCC; however, that still left a larger sample size than the ESS for IPTW for our rare group (SP-3) when using ATE. While the comparator groups, either SP-1 or SP-2, in IPTW had extremely modest reduction in ESS, our rare group, SP-3, was reduced to 45% of the original sample in the SP-1 vs SP-3 comparison and 24% in the SP-2 vs SP-3 comparison. In contrast, matching left 99.5% of the original SP-3 sample in the SP-1 vs SP-3 analyses and 95% in the SP-2 vs SP-3 analysis. Similarly, even though RADAR demonstrated less variance in weights, ESS for SP-3 was 63, which is only 73% of the original sample size in this rare subgroup. For ATT, this is quite different, all individuals in the treatment group (SP-3) are retained and the comparator sample size is smaller.

Table 5.

Effective Sample Size (ESS) and Summary Statistics for Stabilized Inverse Probability Weights for IPTW Analyses by Study

	ESS IPTW		Stabilized inverse probability weights
	Comparator	SP-3	Mean (±SD) Comparator	Median (range) Comparator	Mean (±SD) SP-3	Median (range) SP-3
CNICS/JHHCC
Original SP-1 vs SP-3	11,691	194	-	-	-	-
IPTW SP-1 vs SP-3 (ATE)	11,688	86	0.99 (± 0.01)	0.99 (0.98-1.12)	1.48 (± 1.66)	0.93 (0.17-11.16)
IPTW SP-1 vs SP-3 (ATT)	6,832	194	0.02 (± 0.01)	0.01 (0.00-0.15)	1.00 (± 0.00)*	1.00 (1.00-1.00)
Original SP-2 vs SP-3	2,549	194	-	-	-	-
IPTW SP-2 vs SP-3 (ATE)	2,517	49	0.97 (± 0.11)	0.94 (0.90-2.71)	1.61 (± 2.76)	0.83 (0.13-25.40)
IPTW SP-2 vs SP-3 (ATT)	721	194	0.08 (± 0.12)	0.04 (0.00-2.01)	1.00 (± 0.00)*	1.00 (1.00-1.00)
RADAR
Original SP-1 vs SP-3	1,047	87	-	-	-	-
IPTW SP-1 vs SP-3 (ATE)	1,043	63	1.00 (± 0.06)	0.97 (0.93-1.17)	0.97 (± 0.61)	0.79 (0.37-2.54)
IPTW SP-1 vs SP-3 (ATT)	659	87	0.08 (±0.06)	0.05 (0.01-0.27)	1.00 (± 0.00)*	1.00 (1.00-1.00)

*Note. For ATT, SP-3 weights are fixed at 1 by construction, while the comparator group carries the weighting.

Examining the characteristics of the ten largest and ten smallest stabilized inverse probability weights (Table 6), in analyses comparing SP-1 to SP-3, 60% of the highest weights were among older, White people from two sites in the West of the US. Conversely 90% of the lowest weights were among middle-aged, Latine/Hispanic people from a third site in the West, resulting in lower impact of their responses to substance use questions on the prevalence ratio. When comparing, SP-2 to SP-3 we also observed that 70% of the highest weights were from people older than 60 years in SP-3, however, these people were more likely to identify with Black/African American race/ethnicity. Conversely, those who were weighted the lowest when comparing SP-2 to SP-3 were all younger than 40 years-old, 40% identified as Latine/Hispanic and all came from a second site in the Northeast. Additionally, while those who were dropped from matching were not represented in the highest weight group, for the SP-2 vs SP-3 comparison, four of the ten people who were dropped in matching were also essentially “dropped” by significantly low weighting in IPTW.

Table 6.

Characteristics of Participants by Current Substance Use and Study for People With the 10 Most Extreme Upper and Lower Stabilized Inverse Probability Weights

Standardized inverse probability weight	Age	Subpopulation	Year	Race/Ethnicity	Study	Site (Geographic region)	Dropped in matching
SP-1 vs SP-3 (Upper Extreme)
4.46	46	SP-3	2020	White	CNICS/JHHCC	6 (West)	X
5.09	57	SP-3	2023	White	CNICS/JHHCC	1 (West)	X
5.51	58	SP-3	2016	White	CNICS/JHHCC	6 (West)	X
5.67	70	SP-3	2023	White	CNICS/JHHCC	6 (West)	X
6.07	45	SP-3	2015	White	CNICS/JHHCC	1 (West)	X
7.00	60	SP-3	2020	White	CNICS/JHHCC	6 (West)	X
7.79	75	SP-3	2020	Black	CNICS/JHHCC	4 (Northeast)	X
7.84	54	SP-3	2023	White	CNICS/JHHCC	2 (Midwest)	X
10.65	73	SP-3	2020	White	CNICS/JHHCC	6 (West)	X
11.16	82	SP-3	2023	White	CNICS/JHHCC	8 (Southeast)	X
SP-1 vs SP-3 (Lower Extreme)
0.17	35	SP-3	2019	Black	CNICS/JHHCC	7 (West)	X
0.21	38	SP-3	2021	Latine	CNICS/JHHCC	7 (West)	X
0.22	40	SP-3	2021	Latine	CNICS/JHHCC	7 (West)	X
0.23	32	SP-3	2023	Latine	CNICS/JHHCC	7 (West)	X
0.23	35	SP-3	2022	Latine	CNICS/JHHCC	7 (West)	X
0.24	33	SP-3	2024	Latine	CNICS/JHHCC	3 (Northeast)	X
0.24	24	SP-3	2024	Black	CNICS/JHHCC	5 (Southeast)	X
0.25	35	SP-3	2023	Latine	CNICS/JHHCC	7 (West)	X
0.26	44	SP-3	2021	Latine	CNICS/JHHCC	7 (West)	X
0.26	37	SP-3	2021	Other	CNICS/JHHCC	7 (West)	X
SP-2 vs SP-3 (Upper Extreme)
5.14	54	SP-3	2023	Black	CNICS/JHHCC	2 (Midwest)	X
5.22	69	SP-3	2023	Black	CNICS/JHHCC	5 (Southeast)	X
5.64	73	SP-3	2020	White	CNICS/JHHCC	6 (West)	X
5.83	54	SP-3	2023	Black	CNICS/JHHCC	2 (Midwest)	X
8.23	57	SP-3	2023	Black	CNICS/JHHCC	4 (Northeast)	X
9.33	68	SP-3	2019	Black	CNICS/JHHCC	2 (Midwest)	X
10.16	61	SP-3	2023	Black	CNICS/JHHCC	4 (Northeast)	X
14.94	82	SP-3	2023	White	CNICS/JHHCC	8 (Southeast)	X
18.14	72	SP-3	2023	Black	CNICS/JHHCC	4 (Northeast)	X
25.40	75	SP-3	2020	Black	CNICS/JHHCC	4 (Northeast)	X
SP-2 vs SP-3 (Lower Extreme)
0.13	32	SP-3	2021	Other	CNICS/JHHCC	3 (Northeast)	✓
0.13	32	SP-3	2021	Other	CNICS/JHHCC	3 (Northeast)	X
0.14	33	SP-3	2024	Latine	CNICS/JHHCC	3 (Northeast)	X
0.15	23	SP-3	2017	Latine	CNICS/JHHCC	3 (Northeast)	✓
0.15	33	SP-3	2022	Other	CNICS/JHHCC	3 (Northeast)	✓
0.15	37	SP-3	2024	Latine	CNICS/JHHCC	3 (Northeast)	X
0.16	30	SP-3	2023	Latine	CNICS/JHHCC	3 (Northeast)	X
0.16	24	SP-3	2022	White	CNICS/JHHCC	3 (Northeast)	✓
0.17	33	SP-3	2024	White	CNICS/JHHCC	3 (Northeast)	X
0.17	32	SP-3	2021	White	CNICS/JHHCC	3 (Northeast)	X

X No ✓ Yes.

Discussion

Our study demonstrated clear divergence in point estimates of a given outcome when comparing IPTW to covariate adjustment and matching as methods of controlling for additional factors when at least one comparison group is small, with more marked differences when one of the outcome groups is less common. It is clear that some degree of positivity⁴³ (also known as the experimental treatment assignment⁴⁴) violation is present in more than one of our contrasts, given the large observed weights. No estimator can be trusted to perform well in the presence of large weights relative to sample size. Furthermore, assessment of the stabilized inverse probability weights demonstrated extreme variance in weights of those from the smaller comparison group (SP-3), while the larger comparison groups (SP-1 and SP-2) had much smaller and expected variance. While matching resulted in loss of some individuals from SP-3, and therefore loss of information for participants who did not have a well-defined comparison group, it still resulted in a greater number of individuals in the rare subgroup (SP-3) when compared to ESS in IPTW for ATE, which resulted in an effective loss of 55-76% of the population size for SP-3. Indeed, the RADAR cohort demonstrated less extreme inverse probability weights in the rare subgroup, but still had an ESS in this group of 72% of the original sample size, whereas matching resulted in a 1:3 match for every individual in SP-3. With respect to ATT, while the sample size in the smallest group is fully retained, we observed consistence with ATE in some instances and divergence from ATE estimates in others, mostly with respect to SP-2 vs SP-3, and these did not explain differences from matching. While ESS is an intuitive measure of the variance costs of matching, the primary evaluation of stability remains largely on variable inverse probability weights.^7,8

Examination of covariate distribution and extreme inverse probability weights revealed that age, race/ethnicity and site potentially contributed to inducing large and variable inverse probability weights. However, we only observed actual sparse data bias among three outcomes that would change our interpretation of the findings: methamphetamine/amphetamine and cocaine use in the comparison of SP-2 to SP-3 and smoking in the comparison of SP-1 to SP-3 for CNICS/JHHCC. This suggests that, even in cases of weak positivity violations, it is not inevitable that sparse data bias will be observed. As has been noted, the presence of large and variable inverse probability weights makes any estimator fragile,⁸ but does not guarantee failure. Interestingly, ATE and ATT in these three instances only differed for the SP-2 vs SP-3 “treatment” groups, and in the methamphetamine analysis the ATT estimate was closer to matching estimate. This may occur for a number of reasons but suggests heterogeneity in the treatment effects by one of the adjusted variables. Given the nature of SP-1, SP-2, and SP3,²⁶ deviations between SP-2 and SP-3 are more credible than for the other contrasts and have been reported for methamphetamine as an outcome in other research.³⁴ While ATT and ATE are similar in most instances, they do address different questions and it is important to consider these when estimating treatment effects.^31-33,45,46 These particular examples derive from actual data.²⁶ Theoretical work from Kang and Schafer⁴⁷ and Cole and Hernan¹ show these properties under extreme conditions. There are also other practical examples⁷ and extensive commentary^8,44,48,49 on the consequences of large and variable inverse probability weights on estimates. What this example adds is a clear illustration of how the sparse data bias that is a consequence of positivity can be dependent on the distribution of the outcome, as the outcomes of those participants with large and influential weights could, by chance alone, be reasonably representative of the population.

Guidance for what to do in these situations is complex and depends on the properties of the specific dataset. There are approaches to truncate or trim large inverse probability weights⁵⁰ (as well as more sophisticated methods⁴³) that can handle positivity issues with large and variable inverse probability weights. However, note that in our example the Firth’s penalized regression provided comparable results, and trimming had a variable effect with modest results in some instances and extreme changes in others that did not appear to improve estimate. As trimming can ultimately change the population from which the estimate is drawn, it should be approached with caution. The top weight for SP3 was 25.40 versus 2.71 for SP-2 in an estimate with sparse data bias, a meaningfully large weight with a total sample size of 194 exposed participants. Trimming and truncating may have the disadvantage of making the target population dependent on a statistical model, as the estimates no longer apply equally to the full sample.⁷ The extremely low rates of methamphetamine/amphetamine use among participants may make it difficult to provide any IPTW estimate with good statistical properties.^8,47,51 It may be better to accept the theoretical limitations of matching in these small and complex samples when weak positivity violations are present.

Loss of data due to failure to match can be a concern when choosing which method to use in adjustment, particularly when your “treatment/exposure” group is a rare subgroup of the population, as was the case for SP-3. In our analyses when comparing SP-2 to SP-3, we lost 5% of our SP-3 sample with failure to match. However, it is also important to remember that IPTW and other methods that use weighting for adjustment can substantially reduce the effective number of data points, which can result in unreliable estimates, especially with small sample sizes⁵²; noting that ATE results in loss of the rare group (SP-3), while ATT results in loss of the comparator (SP-2 and SP-3), which is also an important consideration when choosing analytical methods. Indeed, the loss of effective sample size when estimating ATE was significant for the SP-3 group in all of our comparisons, with a 28% reduction from the original sample size in the RADAR SP-1 vs SP-3 comparison, 56% reduction for CNICS/JHHCC SP-1 vs SP-3, and 75% reduction for CNICS/JHHCC SP-2 vs SP-3. This lower ESS results in an inflated variance,⁵³ as observed in the 95% confidence intervals for ATE. It can also result in small sample size vulnerability, leading to more pronounced impact of large weights, potentially resulting in unstable and/or unreliable inferences,^52,53 as was observed with the smoking outcome in the SP-1 vs SP-3 comparison and the methamphetamine/amphetamine use outcome in the SP-2 vs SP-3 comparison.

There are some limitations to this work. We do not know the “true” estimate for any of these outcomes and it is possible that there are additional sources of bias present that we were not able to control for. These data came from a rare subpopulation²⁶ of the overall population and some of the substance use measures^54,55 were relatively rare within this subpopulation – populations with more common exposure status are likely to have propensity score distributions with more congenial weights. Additionally, we did not conduct a priori sample size calculations as our objective was to demonstrate potential divergence in estimates using different modeling techniques and we deliberately picked a variable that would result in smaller subgroups. Furthermore, we see this as estimating a causal effect in an existing database, making the primary goal unbiased estimation.⁵⁶

This study demonstrated how sparse data bias can vary by specific outcome, in small and complex samples, and that, while high inverse probability weights alone do not indicate biased estimates, it increases the vulnerability of these estimators to sparse data concerns. It also highlights the importance of considering a broad range of estimates, including estimand differences such as ATE vs ATT when estimating associations to better understand the sensitivity of small data samples to analytic assumptions in contexts where large sample properties may not hold. ATE can also be calculated for matched samples when using propensity matching.¹³ Careful assessment of the patterns of covariates in the data is key to detecting this issue and ensuring that the best estimate is reported. In small and complex samples there can be an advantage to matching, as it focuses the estimation on the part of the population that has comparators, when data are sparse for a specific outcome. It is also important to calculate ESS under these conditions to help in understanding the generalizability in estimates derived from IPTW when assessing ATE. It also demonstrates that, when working with rare subgroups, it is important to carefully compare different approaches to better understand why estimates may vary between approaches. In general, matching does offer an attractive option, which is why it may be beneficial to consider when estimating outcomes differences in cases of very rare exposures²⁶; but it is best done in a context of carefully understanding the assumptions underlying different approaches for accounting for confounding.

Supplemental Material

Supplemental Material - Comparing Regression Adjustment, Matching, and Inverse Probability Weights in Small Sub-Populations in the HIV SUCCESS Consortium: A Real-World Data Example

Supplemental Material for Comparing Regression Adjustment, Matching, and Inverse Probability Weights in Small Sub-Populations in the HIV SUCCESS Consortium: A Real-World Data Example by Lydia N. Drumright, Ryan P. Kyle, Dominique Heinke, Sonia Napravnik, Geetanjali Chander, Catherine R. Lesko, Ross A. Baiers, Brian Mustanski, Edward Cachay, Michele Kipke, Richard D. Moore, Kenneth H. Mayer, Luis Parra, Rob J. Fredericksen, Sari Reisner, Katerina Christopoulos, George A. Yendewa, Marianna Baum, Laura Bamford, Mari M. Kitahata, Michael S. Saag, Heidi M. Crane, Joseph A. C. Delaney, Bridget M. Whitney in Substance Use: Research and Treatment

Footnotes

Acknowledgements

The authors would like to thank the study participants for generously sharing their data and the study and clinical staff for their work supporting participants and collecting and collating data used in this study from the Centers for AIDS Research Network of Integrated Clinical Systems, the Johns Hopkins HIV Clinical Cohort, and the RADAR cohort.

ORCID iDs

Lydia N. Drumright

Joseph A. C. Delaney

Ethical Considerations

This study used de-identified existing data; the protocol was reviewed by the University of Washington Human Subjects Division (REF: STUDY00016838) and received a determination that it does not involve human subjects; and it has received a certificate of confidentiality.

Author Contributions

LND: development of concept, data harmonization and cleaning, primary analyses, interpretation of findings, primary manuscript writing; RPK: data harmonization and cleaning support, interpretation of findings, manuscript editing, editing, final manuscript approval; DH: Initial exploratory analyses, data harmonization support, final manuscript approval; SN: data collection and preparation, statistical and methodological review, interpretation of findings, manuscript editing, final manuscript approval; GC: discussion and development of concept, interpretation of findings, manuscript editing, final manuscript approval; CRL: discussion and development of concept, interpretation of findings, manuscript editing, final manuscript approval; RAB: discussion and development of concept, data clarifications and support, interpretation of findings, manuscript editing, final manuscript approval; BM: data collection and preparation, data clarifications and support, manuscript editing, final manuscript approval; EC: data collection and preparation, interpretation of findings, manuscript editing, final manuscript approval; MK: project working group lead, discussion and development of concept, manuscript editing, final manuscript approval; RDM: data collection and preparation, interpretation of findings, manuscript editing, final manuscript approval; KHM: discussion and development of concept, data collection and preparation, interpretation of findings, manuscript editing, final manuscript approval; LP: discussion and development of concept, manuscript editing, final manuscript approval; RJF: original concept development, manuscript editing, final manuscript approval; SR: original concept development, manuscript editing, final manuscript approval; KC: data collection and preparation, manuscript editing, final manuscript approval; GAY: data collection and preparation, manuscript editing, final manuscript approval; MB: manuscript editing, final manuscript approval; LB: manuscript editing, final manuscript approval; MMK: data collection and preparation, manuscript editing, final manuscript approval; MSS: data collection and preparation, manuscript editing, final manuscript approval; HMC: discussion and development of concept, data collection and preparation, interpretation of findings, manuscript editing, final manuscript approval; JACD: development of concept, statistical and methodological review, joint manuscript writing with LND, manuscript editing, final manuscript approval; BMW: original concept development, preliminary analyses, manuscript preparation, editing and final approval.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by NIDA under U24DA058307, U01DA036935, and U01DA03693902. Further support was received from the National Institute on Alcohol Abuse and Alcoholism (NIAAA) under U24AA020801, U01AA020793, and U01AA020802; and the National Institute of Allergy and Infectious Diseases (NIAID) under R24AI067039, P30AI027757, P30AI050410, and P30AI27767.

Declaration of Conflicting Interests

None of the authors have competing interests to declare.

Data Availability Statement

Data that supported these findings are available from the CFAR Network of Integrated Clinical Systems (CNICS), see https://sites.uab.edu/cnics/, and for RADAR from HIV SUCCESS, see

Supplemental Material

Supplemental material for this article is available online

References

Cole

Hernan

. Constructing Inverse Probability Weights for Marginal Structural Models. Am J Epidemiol. 2008;168:656-664.

Greenland

Mansournia

Altman

. Sparse data bias: a problem hiding in plain sight. BMJ. 2016;352:i1981.

Burnell

Banerjee

Prescott

Sutton

Tishkovskaya

. Estimating real-world treatment effects in the presence of measurement error and sparse outcome data using propensity score methods. Front Pharmacol. 2026;17:1380586. doi:10.3389/fphar.2026.1380586.

Mansournia

Nazemipour

Etminan

. Balancing scores and causal diagrams. Int J Epidemiol. 2025;54:dyaf114.

Jewell

. Small-sample bias of point estimators of the odds ratio from matched sets. Biometrics. 1984;40:421-435.

Richardson

Cole

Ross

, et al. Meta-Analysis and Sparse-Data Bias. Am J Epidemiol. 2021;190:336-340.

Platt

Delaney

JAC

Suissa

. The positivity assumption and marginal structural models: the example of warfarin use and risk of bleeding. Eur J Epidemiol. 2012;27:77-83.

Robins

Sued

Lei-Gomez

, et al. Comment: Performance of Double-Robust Estimators When “Inverse Probability” Weights Are Highly Variable. Stat Sci. 2007;22:544-559. doi:10.1214/07-STS227D. Epub ahead of print 1 November 2007.

McNamee

. Regression modelling and other methods to control confounding. Occup Environ Med. 2005;62:500-506.

10.

Chesnaye

Stel

Tripepi

, et al. An introduction to inverse probability of treatment weighting in observational research. Clin Kidney J. 2022;15:14-20.

11.

Austin

Stuart

. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med. 2015;34:3661-3679.

12.

Hernan

Robins

. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC; 2020.

13.

Austin

. An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivar Behav Res. 2011;46:399-424.

14.

Lefebvre

Delaney

JAC

Platt

. Impact of mis‐specification of the treatment model on estimates from a marginal structural model. Stat Med. 2008;27:3629-3642.

15.

Schlesselman

. Case-Control studies: design, conduct, analysis. New York: Oxford Univ. Press; 1982.

16.

Rothman

Lash

VanderWeele

, et al. Modern epidemiology. Fourth edition Philadelphia: Wolters Kluwer / Lippincott Williams & Wilkins; 2021.

17.

Joffe

Ten Have

Feldman

Kimmel

. Model Selection, Confounder Control, and Marginal Structural Models: Review and New Applications. Am Stat. 2004;58:272-279.

18.

Robins

Hernán

MÁ

Brumback

. Marginal Structural Models and Causal Inference in Epidemiology. Epidemiology. 2000;11:550-560.

19.

Stuart

. Matching Methods for Causal Inference: A Review and a Look Forward. Stat Sci;25. doi:10.1214/09-STS313. Epub ahead of print 1 February 2010.

20.

Gorbach

Siminski

Ragsdale

C3PNO Investigators . Cohort Profile: The Collaborating Consortium of Cohorts Producing NIDA Opportunities (C3PNO). Int J Epidemiol. 2021;50:31-40.

21.

Mustanski

Ryan

Newcomb

D'Aquila

Matson

. Very High HIV Incidence and Associated Risk Factors in a Longitudinal Cohort Study of Diverse Adolescent and Young Adult Men Who Have Sex with Men and Transgender Women. AIDS Behav. 2020;24:1966-1975.

22.

Mustanski

Morgan

D’Aquila

Birkett

Janulis

Newcomb

. Individual and Network Factors Associated With Racial Disparities in HIV Among Young Men Who Have Sex With Men: Results From the RADAR Cohort Study. JAIDS J Acquir Immune Defic Syndr. 2019;80:24-30.

23.

Lesko

Fojo

Keruly

, et al. Cohort profile update: the Johns Hopkins HIV clinical cohort, 1989–2023. Eur J Epidemiol. 2024;39:1051-1062.

24.

Kitahata

Rodriguez

Haubrich

, et al. Cohort profile: the Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol. 2008;37:948-955.

25.

Crane

Lober

Webster

, et al. Routine Collection of Patient-Reported Outcomes in an HIV Clinic Setting:The First 100 Patients. Curr HIV Res. 2007;5:109-118.

26.

Reisner

Whitney

Crane

, et al. Clinical and Behavioral Outcomes for Transgender Women Engaged in HIV Care: Comparisons to Cisgender Men and Women in the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) Cohort. AIDS Behav. 2023;27:2113-2130.

27.

Tang

Schalet

Janulis

, et al. Evaluating the agreement between different substance use recall periods in multiple HIV cohorts. Drug Alcohol Depend. 2024;254:111043.

28.

Austin

. The performance of different propensity score methods for estimating marginal hazard ratios. Stat Med. 2013;32:2837-2849.

29.

Stürmer

Wyss

Glynn

Brookhart

. Propensity scores for confounder adjustment when assessing the effects of medical interventions using nonexperimental study designs. J Intern Med. 2014;275:570-580.

30.

Greifer

Stuart

. Matching Methods for Confounder Adjustment: An Addition to the Epidemiologist’s Toolbox. Epidemiol Rev. 2021;43:118-129.

31.

Fang

Brooks

Chrischilles

. Apples and Oranges? Interpretations of Risk Adjustment and Instrumental Variable Estimates of Intended Treatment Effects Using Observational Data. Am J Epidemiol. 2012;175:60-65.

32.

Cafri

. Why we should be prioritizing the average treatment effect on the treated over other estimands when evaluating drug and device safety. Am J Epidemiol. 2025;194:3602-3608.

33.

Debray

TPA

Nguyen

T-L

Platt

. Comparative Effectiveness and Personalized Medicine Research Using Real-World Data. 1st edn. Boca Raton: Chapman and Hall/CRC. 10.1201/9781003300809. Epub ahead of print 11 March 2026.

34.

Fredericksen

Nance

Drumright

, et al. Differences in Methamphetamine Use, Self-Reported Impact, and HIV-Related Outcomes by Gender and Sexual Orientation in a Multisite U.S. Cohort of People with HIV in Care. AIDS Behav. 2026;30:418-428.

35.

Murray

Logan

. The code used in the What If book: Stata. Miguel Hernan. https://miguelhernan.org/whatifbook

36.

Firth

. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80:27-38.

37.

Heinze

Schemper

. A solution to the problem of separation in logistic regression. Stat Med. 2002;21:2409-2419.

38.

Delaney

JAC

Platt

Suissa

. The impact of unmeasured baseline effect modification on estimates from an inverse probability of treatment weighted logistic model. Eur J Epidemiol. 2009;24:343-349.

39.

Austin

Stuart

40.

Kostouraki

Hajage

Rachet

, et al. On variance estimation of the inverse probability‐of‐treatment weighting estimator: A tutorial for different types of propensity score weights. Stat Med. 2024;43:2672-2694.

41.

Zou

. A Modified Poisson Regression Approach to Prospective Studies with Binary Data. Am J Epidemiol. 2004;159:702-706.

42.

Remiro‐Azócar

Heath

Baio

. Methods for population adjustment with limited access to individual patient data: A review and simulation study. Res Synth Methods. 2021;12:750-775.

43.

Zivich

Edwards

Lofgren

Cole

Shook-Sa

Lessler

. Transportability Without Positivity: A Synthesis of Statistical and Simulation Modeling. Epidemiology. 2024;35:23-31.

44.

Petersen

Porter

Gruber

Wang

van der Laan

. Diagnosing and responding to violations in the positivity assumption. Stat Methods Med Res. 2012;21:31-54.

45.

Austin

AYX

Vyas

Kapral

. Applying Propensity Score Methods in Clinical Research in Neurology. Neurology. 2021;97:856-863.

46.

Goetghebeur

Le Cessie

De Stavola

Moodie

Waernbaum

“on behalf of” the topic group Causal Inference TG7 of the STRATOS initiative . Formulating causal questions and principled statistical answers. Stat Med. 2020;39:4922-4948.

47.

Kang

JDY

Schafer

. Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data. Stat Sci. 2007;22:523-539. doi:10.1214/07-STS227. Epub ahead of print 1 November 2007.

48.

Westreich

Cole

. Invited Commentary: Positivity in Practice. Am J Epidemiol. 2010;171:674-677.

49.

Edwards

Cole

Lesko

, et al. An Illustration of Inverse Probability Weighting to Estimate Policy-Relevant Causal Effects. Am J Epidemiol. 2016;184:336-344.

50.

Cheng

Thomas

, et al. Addressing Extreme Propensity Scores in Estimating Counterfactual Survival Functions via the Overlap Weights. Am J Epidemiol. 2022;191:1140-1151.

51.

Bang

Robins

. Doubly Robust Estimation in Missing Data and Causal Inference Models. Biometrics. 2005;61:962-973.

52.

Raad

Cornelius

Chan

Williamson

Cro

. An evaluation of inverse probability weighting using the propensity score for baseline covariate adjustment in smaller population randomised controlled trials with a continuous outcome. BMC Med Res Methodol. 2020;20:70.

53.

Pirracchio

Resche-Rigon

Chevret

. Evaluation of the Propensity score methods for estimating marginal odds ratios in case of small sample size. BMC Med Res Methodol. 2012;12:70.

54.

Luu

Ruderman

, et al. Alcohol and drug use severity are independently associated with antiretroviral adherence in the current treatment era. AIDS Care. 2024;36:618-630.

55.

Crane

Drumright

. HIV, Aging, and Comorbidities Research in Clinical Cohorts: 3 Lessons Learned Using Examples From the CNICS Cohort. JAIDS J Acquir Immune Defic Syndr. 2022;89:S10-S14.

56.

Hernán

. Causal analyses of existing databases: no power calculations required. J Clin Epidemiol. 2022;144:203-205.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.75 MB