Performance of propensity score matching to estimate causal effects in small samples

Abstract

Propensity score (PS) matching is a very popular causal estimator usually used to estimate the average treatment effect on the treated (ATT) from observational data. However, opting for this estimator may raise some efficiency issues when the sample size is limited. Therefore, we aimed to evaluate the performance of propensity score matching in this context. We started with a motivating example based on a cohort of 66 children with sickle cell anemia who received either allogeneic bone-marrow transplant or chronic transfusion. We found substantial differences in the ATT estimate according to the model selected for propensity score estimation and subsequent matching. Then, we assessed the performance of the different propensity score matching methods and post-matching analyses to estimate the ATT using a simulation study. Although all selected propensity score matching methods were based of previous recommendations, we found important discrepancies in the estimation of treatment effect between them, underlining the importance of thorough sensitivity analyses when using propensity score matching in the context of small sample sizes.

Keywords

Propensity score matching small sample size

Get full access to this article

View all access options for this article.

References

Rosenbaum

PR.

Model-based direct adjustment.

J Am Stat Assoc 1987; 82: 387–394.

Austin

Grootendorst

Normand

SLT

, et al. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Stat Med 2007; 26: 754–768.

Pearl

Causality: models. Reasoning and inference. 2nd ed. New York, NY: Cambridge University Press, 2009.

D’Agostino

Jr.

Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group.

Stat Med 1998; 17: 2265–2281.

Austin

PC.

Grootendorst

Anderson

GM.

A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study.

Stat Med 2007; 26: 734–753.

Rubin DB. Estimating causal effects from large data sets using the propensity score. Ann Intern Med 1997; 127: 757–763.

Pirracchio

Yue

Manley

, et al. Collaborative targeted maximum likelihood estimation for variable importance measure: illustration for functional outcome prediction in mild traumatic brain injuries. Stat Methods Med Res 2018; 27: 286–297.

Lunt

Selecting an appropriate caliper can be essential for achieving good balance with propensity score matching.

Am J Epidemiol 2013; 179: 226–235.

Austin

PC.

Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations.

Biomed J 2009; 51: 171–184.

10.

Austin

PC.

Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies.

Pharm Stat 2010; 10: 150–161.

11.

Caliendo

Kopeinig

Some paractical guidance for the implementation of propensity score matching.

J Economic Survey 2008; 22: 31–72.

12.

Austin

PC.

A comparison of 12 algorithms for matching on the propensity score.

Stat Med 2013; 33: 1057–1069.

13.

Austin

Lee

DS.

The concept of the marginally matched subject in propensity-score matched analyses.

Pharmacoepidemiol Drug Saf 2009; 18: 469–482.

14.

Dehejia

Wahba

Propensity score-matching methods for nonexperimental causal studies.

Rev Econ Stat 2002; 84: 151–161.

15.

Franklin

Eddings

Glynn

, et al. Regularized regression versus the high-dimensional propensity score for confounding adjustment in secondary database analyses. Am J Epidemiol 2015; 182: 651–659.

16.

Stürmer

Joshi

Glynn

, et al. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods J Clin Epidemiol 2006; 59: 437–447.

17.

Chevret

Verlhac

Ducros-Miralles

, et al. Design of the DREPAGREFFE trial: a prospective controlled multicenter study evaluating the benefit of genoidentical hematopoietic stem cell transplantation over chronic transfusion in sickle cell anemia children detected to be at risk of stroke by transcranial Doppler (NCT 01340404). Contemp Clin Trials 2017; 62: 91–104.

18.

Franklin

Rassen

Ackermann

, et al. Metrics for covariate balance in cohort studies of causal effects. Stat Med 2014; 33: 1685–1699.

19.

Smith

Todd

Does matching overcome LaLonde’s critique of nonexperimental estimators?

J Econometrics 2005; 125: 305–353.

20.

Augurzky

Kluve

Assessing the performance of matching algorithms when selection into treatment is strong.

J Appl Econometrics 2000: 22: 533–557.

21.

Heckman

JJ.

Ichimura

Todd

PE.

Matching as an econometric evaluation estimator: evidence from evaluating a job training programme.

Rev Econ Stud 1997; 64: 605–654.

22.

Rubin

DB.

Estimating causal effects from large data sets using the propensity score.

Ann Intern Med 1997; 127: 757–763.

23.

Ali

Groenwold

Pestman

, et al. Propensity score balance measures in pharmacoepidemiology: a simulation study. Pharmacoepidemiol Drug Saf 2014; 23: 802–811.

24.

Stuart

Lee

Leacy

FP.

Prognostic score-based balance measures for propensity score methods in comparative effectiveness research.

J Clin Epidemiol 2013; 66: S84–S90.

25.

Brookhart

Rothman

SSKJ

, Glynn

, et al. Variable selection for propensity score models. Am J Epidemiol 2006; 163: 1149–1156.

26.

Caruana

Chevret

Resche-Rigon

, et al. A new weighted balance measure helped to select the variables to be included in a propensity score model. J Clin Epidemiol 2015; 68: 1415–1422.

27.

Belitser

Martens

Pestman

, et al. Measuring balance and model selection in propensity score methods. Pharmacoepidemiol Drug Saf 2011; 16: 1115–1129.

28.

Schmid

Nonparametric estimation of the coefficient of overlapping – theory and empirical application.

Computat Stat Data Analysis 2006; 50: 1583–1596.

29.

Stine

Heyse

JF.

Non-parametric estimates of overlap.

Stat Med 2001; 20: 215–236.

30.

Glynn

Schneeweiss

Stürmer

Indications for propensity scores and review of their use in pharmacoepidemiology.

Basic Clin Pharmacol Toxicol 2006; 98: 253–259.

31.

Rosenbaum

PR.

Observational studies . 2nd ed. New York, NY: Springer, 2002.

32.

Abdia

Kulasekera

Datta

, et al. Propesnsity scores based methods for estimating average treatment effect and average treatment effect among treated: a comparative study. Biometric J 2017; 5: 967–985.

33.

Belloni

Chernozhukov

Hansen

Inference on treatment effects after selection amongst high-dimensional controls.

Rev Economic Studies 2014; 81: 608–650.

34.

Stuart

EA.

Matching methods for causal inference: a review and a look forward.

Stat Sci 2010; 25: 1–21.

35.

Kreif

Grieve

Radice

, et al. Regression-adjusted matching and double-robust methods for estimating average treatment effects in health economic evaluation. Health Serv Outcomes Res Meth 2013; 13: 174–202.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB