Researchers have become increasingly interested in programs’ main and interaction effects of two variables (A and B, e.g., two treatment variables or one treatment variable and one moderator) on outcomes. A challenge for estimating main and interaction effects is to eliminate selection bias across A-by-B groups. I introduce Rubin’s causal model to approximate factorial experimental designs for studies with partial randomization and nonrandomization. I apply a Monte Carlo simulation to evaluate several propensity score applications. The findings suggest the following two applications for reducing bias and mean square error of parameter estimates when analyzing the relationship of two variables and an outcome: (a) inverse of propensity score weighting based on one multinomial propensity score model and (b) subclassification based on two binary propensity score models. As a demonstration, I examine whether the effects of the Head Start program, compared to other center-based care, for improving children’s reading achievement vary by child care quality.
AustinP. C. (2007). The performance of different propensity score methods for estimating marginal odds ratios. Statistics in Medicine, 26, 3078–3094.
2.
BaronR. M.KennyD. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.
3.
BurtonA.AltmanD. G.RoystonP.HolderR. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25, 4279–4292. doi:10.1002/sim.2673
4.
D’AgostinoR. B.Jr (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17, 2265–2281.
5.
FrazierP. A.TixA. P.BarronK. E. (2004). Testing moderator and mediator effects in counseling psychology research. Journal of Counseling Psychology, 51, 115–134.
6.
HarmsT.CliffordR. M.CryerD. (1998). Early childhood environment rating scale-revised. New York, NY: Teachers College Press.
7.
HillJ. L.Brooks-GunnJ.WaldfogelJ. (2003). Sustained effects of high participation in an early intervention for low-birth-weight premature infants. Developmental Psychology, 39, 730–744.
8.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
9.
HongG.NomiT. (2012). Weighting methods for assessing policy effects mediated by peer change. Journal of Research on Educational Effectiveness, 5, 261–289.
10.
ImaiK.KeeleL.TingleyD. (2010). A general approach to causal mediation analysis. Psychological Methods, 15, 309–334.
11.
ImaiK.KingG.StuartE. A. (2008). Misunderstandings among experimentalists and observationalists in causal inference. Journal of the Royal Statistical Society, Series A, 171, 481–502.
12.
ImaiK.van DykD. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99, 854–866.
13.
ImbensG. W. (2000). The role of the propensity score in estimating dose-response functions. Biometrika, 87, 706–710.
14.
ImbensG. W.RubinD. (2009). Causal inference in statistics, and in the social and biomedical sciences. New York, NY: Cambridge University Press.
15.
KraemerH. C.KiernanM.EssexM.KupferD. J. (2008). How and why criteria defining moderators and mediators differ between the Baron & Kenny and MacArthur approaches. Health Psychology, 27. S101–S108.
16.
KraemerH. C.SticeE.KazdinA.OffordD.KupferD. (2001). How do risk factors work together to produce an outcome? Mediators, moderators, and independent, overlapping and proxy risk factors. American Journal of Psychiatry, 158, 848–856.
17.
LochmanJ. E.BoxmeyerC. L.PowellN. P.RothD.WindleM. (2006). Masked intervention effects: Analytic methods for addressing low dosage of intervention. New Directions for Evaluation, 110, 19–32.
18.
LuB.GreevyR.XuX.BeckC. (2011). Optimal nonbipartite matching and its statistical applications. The American Statistician, 65, 21–29.
19.
LuB.RosenbaumP. (2004). Optimal pair matching with two control groups. Journal of Computational and Graphical Statistics, 13, 422–434.
20.
LuncefordJ. K.DavidianM. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine, 23, 2937–2960.
21.
PeckL. R. (2003). Subgroup analysis in social experiments: Measuring program impacts based on post-treatment choice. American Journal of Evaluation, 24, 157–187.
22.
PeckL. R. (2005). Using cluster analysis in program evaluation. Evaluation Review, 29, 178–196.
23.
PeckL. R. (2013). On analysis of symmetrically predicted endogenous subgroups: Part one of a method note in three parts. American Journal of Evaluation, 34, 225–236. doi:10.1177/1098214013481666
24.
PeckL. R.BellS. H. (2012). Estimating the influence of Head Start quality on child development. Paper presented at the 2012 Annual Fall Research Conference of the Association for Public Policy Analysis and Management, Baltimore, MD.
25.
PeckL. R.CamilloF.D’AttomaI. (2009). A promising new approach to eliminating selection bias. Canadian Journal of Program Evaluation, 24, 31–56.
26.
PotterF. J. (1993). The effect of weight trimming on nonlinear survey estimates. In Proceedings of the Section on Survey Research Methods of American Statistical Association. San Francisco, CA: American Statistical Association, San Francisco.
RosenbaumP. R. (2002). Observational studies (2nd ed.). New York, NY: Springer-Verlag.
29.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
30.
RosenbaumP. R.RubinD. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524.
31.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
32.
RubinD. B. (1980). Comments on “Randomization analysis of experimental data: The fisher randomization test,” by D. Basu. Journal of the American Statistical Association, 75, 591–593.
33.
RubinD. B. (1990). Comments on “On the application of probability theory to agricultural experiments: Essay on principles. Section 9,” by J. Splawa-Neyman (D. M. Dabrowska & T. P. Speed, Trans.). Statistical Science, 5, 472–480.
34.
ScharfsteinD. O.RotnitzkyA.RobinsJ. M. (1999). Adjusting for non-ignorable drop-out using semiparametric non-response models. Journal of the American Statistical Association, 94, 1096–1120.
35.
SchochetP. Z.BurghardtJ. (2007). Using propensity scoring to estimate program-related subgroup impacts in experimental program evaluations. Evaluation Review, 31, 95–120.
36.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
37.
StuartE. A. (2007). Estimating causal effects using school-level data sets. Educational Researcher, 36, 187–198.
38.
StuartE. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
