When nonrandom treatments occur across sites, within-site matching (WM) is often desirable. This approach, however, can significantly reduce treatment group sample size and exclude substantively important subgroups. To limit these drawbacks, we extend a matching approach developed by Stuart and Rubin to a multisite study. We demonstrate the proposed method through a multisite analysis of algebra enrollment effects in 50 middle schools, where within each school students are assigned to algebra or pre-algebra and test the utility of the proposed method with a simulation study. The results document the method’s conceptual appeal and indicate that two-stage matching is a viable alternative to strict WM or matching that ignores the nested data structure (pooled matching).
ArpinoB.MealliF. (2011). The specification of the propensity score in multilevel observational studies. Computational Statistics & Data Analysis, 55, 1770–1780.
2.
AustinP. C. (2011). An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behavioral Research, 46, 399–424.
CochranW. G. (1965). The Planning of Observational Studies of Human Populations. Journal of the Royal Statistical Society. Series A (General), 128, 234–266.
5.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.
6.
CookT. D.SteinerP. M.PohlS. (2009). How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: results from two types of within-study comparisons. Multivariate Behavioral Research, 44, 828–847.
7.
FabeloT.ThompsonM. D.PlotkinM.CarmichaelD.MarchbanksM. P.BoothE. A. (2011). Breaking schools’ rules: A statewide study of how school discipline relates to students’ success and juvenile justice involvement (p. 124). New York, NY: Council of State Governments Justice Center.
8.
HoD. E.ImaiK.KingG.StuartE. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199–236. doi:10.1093/pan/mpl013
9.
HongG.RaudenbushS. W. (2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis, 27, 205–224.
10.
KelceyB. (2011a). Assessing the effects of teachers’ reading knowledge on students’ achievement using multilevel propensity score stratification. Educational Evaluation and Policy Analysis, 33, 458–482.
11.
KelceyB. (2011b). Propensity score matching within versus across schools. Paper presented at the American Educational Research Association Annual Meeting, New Orleans, LA.
12.
KimJ.SeltzerM. (2007). Causal inference in multilevel settings in which selection processes vary across schools (CSE Technical Report 708). Los Angeles: CRESST/University of California.
13.
LittleR. J.RubinD. B. (2002). Statistical analysis with missing data, second edition (2nd ed.). Hoboken, NJ: Wiley-Interscience.
14.
McNealR. B. (1995). Extracurricular activities and high school dropouts. Sociology of Education, 68, 62–80.
15.
MorganS. L.HardingD. J. (2006). Matching estimators of causal effects. Sociological Methods & Research, 35, 3–60.
16.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
17.
RaudenbushS. W.WillmsJ. D. (1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307–335.
18.
ReardonS. F.RaudenbushS. W. (2009). Assumptions of value-added models for estimating school effects. Education Finance and Policy, 4, 492–519.
19.
RicklesJ. H. (2011). Using Interviews to Understand the Assignment Mechanism in a Nonexperimental Study. Evaluation Review, 35, 490–522.
20.
RosenbaumP. R. (1986). Dropping out of high school in the United States: An Observational Study. Journal of Educational and Behavioral Statistics, 11, 207–224.
21.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
22.
RosenbaumP. R.RubinD. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39, 33–38.
23.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
24.
RubinD. B. (1991). Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism. Biometrics, 47, 1213–1234.
25.
RubinD. B. (2005). Causal inference using potential outcomes. Journal of the American Statistical Association, 100, 322–331.
26.
SchaferJ. L.KangJ. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological methods, 13, 279–313.
27.
SeltzerM. (2004). The use of hierarchical models in analyzing data from experiments and quasi-Experiments conducted in field settings. In KaplanD. (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 259–280). Thousand Oaks, CA: Sage.
28.
ShadishW. R.ClarkM. H.SteinerP. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334–1344.
29.
ShadishW. R.CookT. D. (2009). The renaissance of field experimentation in evaluating interventions. Annual Review of Psychology, 60, 607–629.
30.
SteinerP. M. (2011). Matching strategies for observational data with multilevel structure. Paper presentation presented at the Society for Research on Educational Effectiveness Spring Meeting, Washington, DC.
31.
StuartE. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
32.
StuartE. A.RubinD. B. (2008). Matching with multiple control groups with adjustment for group differences. Journal of Educational and Behavioral Statistics, 33, 279–306.
ThoemmesF.KimE. S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46, 90–118.
35.
ThoemmesF.WestS. G. (2011). The use of propensity scores for nonrandomized designs with clustered data. Multivariate Behavioral Research, 46, 514–543.
36.
WinshipC.MorganS. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659–706.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
