The present article provides a synthesis of the conceptual and statistical issues involved in using multisite randomized trials to learn about and from a distribution of heterogeneous program impacts across individuals and/or program sites. Learning about such a distribution involves estimating its mean value, detecting and quantifying its variation, and estimating site-specific impacts. Learning from such a distribution involves studying the factors that predict or explain impact variation. Part I of the article introduces the concepts and issues involved. Part II focuses on estimating the mean and variation of impacts of program assignment. Part III extends the discussion to variation in the impacts of program participation. Part IV considers how to use multisite trials to study moderators of program impacts (individual-level or site-level factors that influence these impacts) and mediators of program impacts (individual-level or site-level “mechanisms” that produce these impacts).
AngristJ. D.ImbensG.RubinD. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444–455.
3.
BloomH. S.HillC. J.RiccioJ. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22, 551–575.
4.
BloomH. S.RaudenbushS. W.WeissM.PorterK. (2014). Using multi-site evaluations to study variation in effects of program assignment. New York, NY: MDRC.
5.
BloomH. S.WeilandC. (2015, 3) Quantifying variation in Head Start effects on young children’s cognitive and socio-emotional skills using data from the National Head Start Impact Study. New York, NY: MDRC.
6.
BrykA. S.RaudenbushS. W. (1988). Heterogeneity of variance in experimental studies: A challenge to conventional interpretations. Psychological Bulletin, 104, 396–404.
7.
DempsterA. P.RubinD. B.TsutakawaR. K. (1981). Estimation in covariance components models. Journal of the American Statistical Association, 76, 341–353.
8.
FisherR. A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver and Boyd.
9.
FrangakisC. E.RubinD. B. (2002). Principal stratification in causal inference. Biometrics, 58, 21–29.
10.
GreenbergD. H.ShroderM. (2004). The digest of social experiments. Washington, DC: The Urban Institute.
11.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy. Journal of the American Statistical Association, 101, 901–910.
12.
HeckmanJ. J. (1979). Sample selection bias as a specification error. Econometrica: Journal of the Econometric Society, 47, 153–161.
13.
HeckmanJ. J.VytlacilE. (1998). Instrumental variables methods for the correlated random coefficient model: Estimating the average rate of return to schooling when the return is correlated with schooling. Journal of Human Resources, 33, 974–987.
14.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
15.
HollandP. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18, 449–484.
16.
HongG. (2015). Causal inference in a social world: Moderation, mediation, and spillover. Sussex, England: John Wiley.
17.
ImaiK.KeeleL.YamamotoT. (2010). Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science, 25, 51–71.
18.
KatzL. F.KlingJ. R.LiebmanJ. B. (2000). Moving to opportunity in Boston: Early results of a randomized mobility experiment(No. w7973). Cambridge, MA: National Bureau of Economic Research.
19.
KlingJ. R.LiebmanJ. B.KatzL. F. (2007). Experimental analysis of neighborhood effects. Econometrica, 75, 83–119.
20.
LindleyD. V.SmithA. F. (1972). Bayes estimates for the linear model. Journal of the Royal Statistical Society. Series B (Methodological), 34, 1–41.
21.
LouisT. A. (1984). Estimating a population of parameter values using Bayes and empirical Bayes methods. Journal of the American Statistical Association, 79, 393–398.
22.
MorrisC.N. (1983)Parametric empirical Bayes inference: Theory and applications. Journal of the American Statistical Association, 78, 47–65.
23.
NeymanJ. S. (1990). On the applicability of probability theory to agricultural experiments: Essay on principles, Section 9. In DabrowskaD. M.SpeedT. P. (Eds. & Trans.), Statistical Science (Vol. 5, pp. 465–480). (Original work published 1923)
24.
PearlJ. (2001, 8). Direct and indirect effects. In BreeseJ.KollerD. (Eds.), Proceedings of the seventeenth conference on uncertainty in artificial intelligence (pp. 411–420). San Francisco, CA: Morgan Kaufmann Publishers Inc.
25.
RaudenbushS. W. (2014). Random coefficient models for multi-site randomized trials with inverse probability of treatment weighting. Chicago, IL: Department of Sociology, University of Chicago.
26.
RaudenbushS. W.BrykA. (2002). Hierarchical linear models: Data analysis and applications (2nd ed.). Thousand Oaks, CA: Sage.
27.
RaudenbushS. W.ReardonS.NomiT. (2012). Statistical analysis for multi-site trials using instrumental variables. Journal of Research and Educational Effectiveness, 5, 303–332.
28.
ReardonS. F.RaudenbushS. W. (2013). Under what assumptions do multi-site instrumental identify average causal effects?Sociological Methods and Research, 42, 143–163.
29.
ReardonS. F.UnluF.ZhuP.BloomH. S. (2014). Bias and bias correction in multisite instrumental variables analysis of heterogeneous mediator effects. Journal of Educational and Behavioral Statistics, 39, 53–86. doi:1076998613512525
30.
RobinsJ. M.GreenlandS. (1992). Identifiability and exchangeability for direct and indirect effects. Epidemiology, 3, 143–155.
31.
RoyA. D. (1951). Some thoughts on the distribution of earnings. Oxford Economic Papers, 3, 135–146.
32.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
33.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Eeducational Psychology, 66, 688.
34.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 7, 34–58.
35.
RubinD. B. (1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81, 961–962.
36.
RumbergerR. W. (1995). Dropping out of middle school: A multilevel analysis of students and schools. American Educational Research Journal, 32, 583–625.
37.
SpybrookJ. (2013). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82, 334–357. doi:10.1080/00220973.2013.813364
38.
VanderWeeleT. J. (2015). Explanation in causal inference: Methods for mediation and interaction. Oxford, England: Oxford University Press.
39.
WeissM. J.BloomH. S.BrockT. (2014). A conceptual framework for studying sources of variation in program effects. Journal of Policy Analysis and Management, 33, 778–808
