The estimation of causal effects in nonrandomized studies should comprise two distinct phases: design, with no outcome data available; and analysis of the outcome data according to a specified protocol. Here, we review and compare point and interval estimates of common statistical procedures for estimating causal effects (i.e. matching, subclassification, weighting, and model-based adjustment) with a scalar continuous covariate and a scalar continuous outcome. We show, using an extensive simulation, that some highly advocated methods have poor operating characteristics. In many conditions, matching for the point estimate combined with within-group matching for sampling variance estimation, with or without covariance adjustment, appears to be the most efficient valid method of those evaluated. These results provide new conclusions and advice regarding the merits of currently used procedures.
RubinDB. Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol1974; 66: 688–688.
2.
RubinDB. Bayesian inference for causal effects: the role of randomization. Ann Stat1978; 6: 34–58.
3.
RubinDB. Comment on randomization analysis of experimental data: the fisher randomization test. J Am Stat Assoc1980; 75: 591–593.
4.
RubinDB. Formal modes of statistical inference for causal effects. J Stat Plan Infer1990; 25: 279–292.
5.
HollandPW. Statistics and causal inference (with discussion). J Am Stat Assoc1986; 81: 945–970.
6.
Rubin DB. Bayesian inference for causality: the importance of randomization. In: Proceedings of the Social Statistics Section of the American Statistical Association. Alexandria, VA: American Statistical Association, 1975, pp.233–239.
7.
RubinDB. Assignment to treatment group on the basis of a covariate. J Educ Behav Stat1977; 2: 1–26.
8.
NeymanJ. Sur les applications de la thar des probabilities aux experiences agaricales: essay de principle. English translation of excerpts by Dabrowska D and Speed T (1990). Stat Sci1923; 5: 465–472.
9.
RosenbaumPRRubinDB. The central role of the propensity score in observational studies for causal effects. Biometrika1983; 70: 41–55.
10.
CochranWG. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics1968; 24: 295–313.
11.
RubinDB. Matching to remove bias in observational studies. Biometrics1973; 29: 159–183.
12.
RubinDB. Using multivariate matched sampling and regression adjustment to control bias. J Am Stat Assoc1979; 74: 318–328.
13.
RubinDBThomasN. Combining propensity score matching with additional adjustments for prognostic covariates. J Am Stat Assoc2000; 95: 573–585.
14.
RubinDB. Matched sampling for causal effects, New York, NY: Cambridge University Press, 2006.
15.
RosenbaumPR. Design of observational studies, New York, NY: Springer Verlag, 2009.
16.
RubinDB. The use of matched sampling and regression adjustment to remove bias in observational studies. Biometrics1973; 29: 185–203.
17.
ShahBRLaupacisAHuxJE. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J Clin Epidemiol2005; 58: 550–559.
18.
LuncefordJKDavidianM. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Stat Med2004; 23: 2937–2960.
19.
RobinsJMRotnitzkyAZhaoLP. Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc1994; 89: 846–866.
20.
AustinPC. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Med Decis Making2009; 29: 661–677.
21.
WaernbaumI. Model misspecification and robustness in causal inference: comparing matching with doubly robust estimation. Stat Med2012; 31: 1572–1581.
22.
AustinPC. A comparison of 12 algorithms for matching on the propensity score. Stat Med2014; 33: 1057–1069.
23.
LehmannELRomanoJP. Testing statistical hypotheses, 3rd ed. New York, NY: Springer, 2005.
24.
ChambersCDJohnsonKADickLM. Birth outcomes in pregnant women taking fluoxetine. N Engl J Med1996; 335: 1010–1015.
25.
GastilJPierre DeessEWeiserP. Civic awakening in the jury room: a test of connection between jury deliberation and political participation. J Polit2002; 64: 585–595.
26.
O’DeaJAWilsonR. Socio-cognitive and nutritional factors associated with body mass index in children and adolescents: possibilities for childhood obesity prevention. Health Educ Res2006; 21: 796–805.
27.
FisherRA. Statistical methods for research workers, 4th ed. Edinburgh: Oliver & Boyed, 1932.
28.
CochranWGRubinDB. Controlling bias in observational studies: a review. Sankhyā Indian J Stat Ser A1973; 35: 417–466.
29.
McCandlessLCGustafsonPAustinPC. Bayesian propensity score analysis for observational data. Stat Med2009; 28: 94–112.
30.
MyersJALouisTA. Comparing treatments via the propensity score: stratification or modeling. Health Serv Outcomes Res Methodol2012; 12: 1–15.
EubankRL. Nonparametric regression and spline smoothing, 2nd ed. New York, NY: Marcel Dekker, 1999.
33.
BoorC. A practical guide to splines, New York, NY: Springer, 2001.
34.
StoneCJ. Generalized additive models: comment. Stat Sci1986; 1: 312–314.
35.
HarrellFE. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis, New York, NY: Springer-Verlag, 2001.
WoodSN. On confidence intervals for generalized additive models based on penalized regression splines. Aust NZ J Stat2003; 48: 445–464.
39.
RubinDBThomasN. Combining propensity score matching with additional adjustments for prognostic covariates. J Am Stat Assoc2000; 95: 573–585.
40.
StuartEA. Matching methods for causal inference. Stat Sci2010; 25: 1–21.
41.
ImbensGWWooldridgeJM. Recent developments in the econometrics of program evaluation. J Econ Literature2009; 47: 5–86.
42.
AustinPC. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. Int J Biostat2009; 5(1): Article 13–Article 13.
43.
AbadieADrukkerDHerrJL. Implementing matching estimators for average treatment effects in stata. Stata J2004; 4: 290–311.
44.
SekhonJS. Multivariate and propensity score matching software with automated balance optimization: the matching package for R. J Stat Softw2011; 42: 1–52.
45.
ImbensGRubinDB. Causal inference in statistics, and in the social and biomedical sciences, New York, NY: Cambridge University Press, 2015.
46.
RosenbaumPR. A characterization of optimal designs for observational studies. J Roy Stat Soc Ser B (Methodological)1991; 53: 597–610.
47.
HansenBB. Full matching in an observational study of coaching for the sat. J Am Stat Assoc2004; 99: 609–618.
48.
RosenbaumPR. Model-based direct adjustment. J Am Stat Assoc1987; 82: 387–394.
49.
RubinDBThomasN. Characterizing the effect of matching using linear propensity score methods with normal distributions. Biometrika1992; 79: 797–809.
50.
HiranoKImbensGRidderG. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica2003; 71: 1161–1189.
51.
KangJDYSchaferJL. Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Stat Sci2007; 4: 1–18.
52.
AbadieAImbensGW. Bias-corrected matching estimators for average treatment effects. J Bus Econ Stat2011; 29: 1–11.
53.
Espindle LP. Improving confidence coverage for the estimate of the treatment effect in a subclassification setting, Bachelor’s thesis, Department of Statistics, Harvard University, Cambridge, MA, May 2004.
54.
AzzaliniA. A class of distributions which includes the normal ones. Scand J Stat1985; 12: 171–178.
55.
RosenbaumPRRubinDB. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am Stat1985; 39: 33–38.
56.
CangulMZChretienYRGutmanR. Testing treatment effects in unconfounded studies under model misspecification: logistic regression, discretization, and their combination. Stat Med2009; 28: 2531–2551.
57.
LinJ. Divergence measures based on Shannon entropy. IEEE Trans Inform Theory1991; 37: 145–151.
58.
KullbackSLeiblerRA. On information and sufficiency. Ann Math Stat1951; 22: 79–86.
59.
R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2014, http://www.R-project.org (accessed August 2014).
60.
HansenBBKlopferSO. Optimal full matching and related designs via network flows. J Comput Graph Stat2006; 15: 609–627.
61.
ImaiKRatkovicM. Covariate balancing propensity score. J Roy Stat Soc Ser B (Stat Methodol)2014; 76(1): 243–263.
62.
GutmanRRubinDB. Robust estimation of causal effects of binary treatments in unconfounded studies with dichotomous outcomes. Stat Med2013; 32: 1795–1814.
