Although experiments are viewed as the gold standard for evaluation, some of their benefits may be lost when, as is common, outcomes are not defined for some sample members. In evaluations of marriage interventions, for example, a key outcome—relationship quality—is undefined when a couple splits up. This article shows how treatment-control differences in mean outcomes can be misleading when outcomes are not defined for everyone and discusses ways to identify the seriousness of the problem. Potential solutions to the problem are described, including approaches that rely on simple treatment-control differences-in-means as well as more complex modeling approaches.
Angrist, J.D., E. Bettinger, and M. Kremer.2006. Long-term educational consequences of secondary school vouchers: Evidence from administrative records in Colombia . The American Economic Review96 (3): 847—862.
2.
Angrist, J.D., G.W. Imbens, and D.B. Rubin.1996. Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association91:444—472.
3.
Cummings, E.M., M. Ballard, M. El-Sheikh, and M. Lake.1991. Resolution and children's responses to interadult anger. Developmental Psychology27 (3): 462—470.
4.
Dempster, A.P., N.M. Laird, and D.B. Rubin.1977. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society Series B, 39:1—38.
5.
Dominici, F., S.L. Zeger, G. Parmigiani, J. Katz, and P. Christian.2006. Estimating percentile-specific treatment effects in counterfactual models: A case-study of micronutrient supplementation, birth weight, and infant mortality. Applied Statistics55 (2): 261—280.
6.
Egleston, B.L., D.O. Scharfstein, E.E. Freeman, and S.K. West.2007. Causal inference for non-mortality outcomes in the presence of death. Biostatistics. 8(3): 526-545.
7.
Fairclough, D., H.F. Peterson, and V. Chang.1998. Why are missing quality of life data a problem in clinical trials of cancer therapy. Statistics in Medicine17:667—677.
8.
Foster, E.M., G.Y. Fang, and the Conduct Problems Prevention Research Group.2004. Alternative methods for handling attrition: An illustration using data from the Fast Track evaluation. Evaluation Review28 (5): 434—464.
9.
Frangakis, C.E., and D.B. Rubin.2002. Principal stratification in causal inference . Biometrics58:20—29.
10.
Frangakis, C.E., R.S. Brookmeyer, R. Varadhan, M. Safaeian, D. Vlahov, and S.A. Strathdee.2004. Methodology for evaluating a partially controlled longitudinal treatment using principal stratification, with application to a Needle Exchange Program. Journal of the American Statistical Association99:239—249.
11.
Gelman, A., J.B. Carlin, H.S. Stern, and D.B. Rubin.2004. Bayesian data analysis: Second edition. New York: Chapman & Hall/CRC.
12.
Glazerman, S., S. Senesky, N. Seftor, and A. Johnson.2006. Design of an impact evaluation of teacher induction programs. Report for the U.S. Department of Education, National Center for Education Evaluation. Washington, DC: Mathematica Policy Research, Inc., January 2006.
13.
Gottman, J.2004. Sound relationship house scale. Seattle, WA: University of Washington.
14.
Heckman, J.J.1979. Sample selection bias as a specification error . Econometrica47 (1): 153—161.
15.
Hirano, K., G.W. Imbens, D.B. Rubin, and X.-H. Zhou.2000. Assessing the effect of an influenza vaccine in an encouragement design with covariates. Biostatistics1:69—88.
16.
Horowitz, J.L., and C.F. Manski.2000. Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association95:77—84.
17.
Kodrzycki, Y.K.1997. Training programs for displaced workers: What do they accomplish? New England Economic Review3:39—59.
18.
Lee, D.S.2005. Training, wages, and sample selection: Estimating sharp bounds on treatment effects. National Bureau of Economic Research Working Paper 11721. Available at http://www.nber.org/papers/w11721. Cambridge, MA: NBER.
19.
Leeb, H., and B.M. Potscher.2006. Can one estimate the conditional distribution of post-model selection estimators? The Annals of Statistics34 (5): 2554—2591.
20.
Martinson, K., and R. Hendra.2006. The Employment Retention and Advancement Project. Results from the Texas ERA Site. New York, NY : MDRC report, February2006.
21.
McConnell, S., E. Stuart, K. Fortson, P. Decker, I. Perez-Johnson, B. Harris, and J. Salzman.2006. Managing customer's training choices:Findings from the individual Training Account Experiment. Final Report. Report for the U.S. Department of Labor, Employment and Training Administration . Washington, DC: Mathematica Policy Research, Inc., December 2007
22.
McGuigan, K.A., P.L. Ellickson, R.D. Hays, and R.M. Bell.1997. Adjusting for attrition in school-based samples: Bias, precision, and cost trade-offs of three methods. Evaluation Review21 (5): 554—567.
23.
Myers, D., R. Olsen, N. Seftor, J. Young, and C. Tuttle.2004. The impacts of regular Upward Bound: Results from the third follow-up data collection. Report for the U.S. Department of Education. Washington, DC: Mathematica Policy Research, Inc., April 2004 .
24.
O'Leary, C.L.1995. An impact analysis of employment programs in Hungary . Upjohn Institute Staff Working Paper.
25.
Robins, J.M., and S. Greenland.1992. Identifiability and exchangeability for direct and indirect effects. Epidemiology3 (2): 143—155.
26.
Rubin, D.B.2000. Discussion of “Causal Inference with Counterfactuals” by A.P. Dawid. Journal of the American Statistical Association95:435—437.
27.
Rubin, D.B.2006. Causal inference through potential outcomes and principal stratification: Application to studies with “censoring” due to death. Statistical Science21 (3): 299—309.
28.
Sartori, A.E.2003. An estimator for some binary outcome selection models without exclusion restrictions. Political Analysis11:111—138.
29.
Schochet, P., J. Burghardt, and S. Glazerman.2001. National Job Corps study: The impact of Job Corps on participants' employment and related outcomes. Washington, DC: Department of Labor.
30.
Shepherd, B.E., P.B. Gilbert, and T. Lumley.2007. Sensitivity analyses comparing time-to-event outcomes only existing in a subset selected post-randomization. Journal of the American Statistical Association102 (478): 573—582.
31.
Shepherd, B.E., P.B. Gilbert, and D.V. Metrova.2007. Eliciting a counterfactual sensitivity parameter. The American Statistician61:56—63.
32.
Stanley, S., and H. Markman.1997. Marriage in the 90s: A nationwide randomized phone survey. Denver, CO: PREP Inc .
33.
Stuart, E.A., and D.B. Rubin. (in press). Matching methods for causal inference: Designing observational studies. In Best practices in quantitative social science, ed. J. Osborne.New York: Russell Sage.
34.
Winship, C., and R.D. Mare.1992. Models for sample selection bias. Annual Review of Sociology18:327—350.
35.
Zhang, J.L., and D.B. Rubin.2003. Estimation of causal effects via principal stratification when some outcomes are truncated by “death.” Journal of Educational and Behavioral Statistics28 (4): 353—368.
36.
Zhang, J.L., D.B. Rubin, and F. Mealli.2005. Using the EM algorithm to estimate the effects of job training programs on wages. In Proceedings of the 55th Session of the International Statistical Institute .
37.
Zhang, J.L., D.B. Rubin, and F. Mealli.2006. Evaluating the effects of job training programs on wages through principal stratification. In Modelling and Evaluating Treatment Effects in Econometrics. eds. D. Millimet, J. Smith , and E. Vytlacil. Vol. 21 of Advances in Econometrics, Amsterdam : Elsevier .
