By design, randomized experiments (XPs) rule out bias from confounded selection of participants into conditions. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from XPs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset by confounded selection into locations. This work shows that this tradeoff can lead to situations where estimates from QEs are less-biased from selection than are estimates from uncompromised XPs when drawing causal generalizations. This work establishes the conditions theoretically, demonstrates the idea empirically, and discusses the implications of the results.
AlbrightL.MalloyT. E. (2000). Experimental validity: Brunswik, Campbell, Cronbach and enduring issues. Review of General Psychology, 4(4), 337–353. https://doi.org/10.1037/1089-2680.4.4.337
2.
BellS. H.OrrL. L.BlomquistJ. D.CainG. G. (1995). Program applicants as a comparison group in evaluating training programs. Upjohn Institute for Employment Research.
3.
BloomH. S.HillC. J.RiccioJ. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154
4.
BloomH. S.MichalopoulosC.HillC. J. (2005). Using experiments to assess nonexperimental comparison -group methods for measuring program effect. In BloomH. S. (Ed.), Learning more from social experiments (pp. 173–235). New York: Russell Sage Foundation.
5.
CampbellD. T.StanleyJ. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
6.
ColeS. R.StuartE. A. (2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG-320 trial. American Journal of Epidemiology, 172(1), 107–115. https://doi.org/10.1093/aje/kwq084
7.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750. https://doi.org/10.1002/pam.20375
8.
CronbachL. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116–127. https://doi.org/10.1037/h0076829
9.
CronbachL. J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
10.
DehejiaR.Pop-ElechesC.SamiiC. (2021). From local to global: External validity in a fertility natural experiment. Journal of Business and Economic Statistics, 39(1), 217–243. https://doi.org/10.1080/07350015.2019.1639407
11.
FinnJ. D.AchillesC. M. (1990). Answers and questions about class size: A statewide experiment. American Educational Research Journal, 27(3), 557–577. https://doi.org/10.3102/00028312027003557
12.
FrakerT.MaynardR. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. The Journal of Human Resources, 22(2), 194–227. https://doi.org/10.2307/145902
GlazermanS.LevyD.MyersD. (2003). Nonexperimental versus experimental estimates of earnings impacts. American Academy of Political and Social Science, 589, 63–93. https://doi.org/10.1177/0002716203254879
15.
HeckmanJ. J.IchimuraK.ToddP. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64(4), 605–654. https://doi.org/10.2307/2971733
16.
HillC. J.BloomH. S.BlackA. R.LipseyM. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x
17.
HotzV. J.ImbensG. W.KlermanJ. A. (2006). Evaluating the differential effects of alternative welfare-to-work training components: A reanalysis of the California GAIN Program. Journal of Labor Economics, 24(3), 521–566. https://doi.org/10.1086/505050
18.
HotzV. J.ImbensG. W.MortimerJ. H. (2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125(1–2), 241–270. https://doi.org/10.1016/j.jeconom.2004.04.009
19.
HullemanC. S.CordrayD. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2(1), 88–110. https://doi.org/10.1080/19345740802539325
20.
ImaiK.KongG.StuartE. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A, 171(Part 2), 481–502. https://doi.org/10.1111/j.1467-985X.2007.00527.x
21.
JaciwA. P. (2016). Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology. Evaluation Review, 40(3), 199–240. https://doi.org/10.1177/0193841X16664456
22.
JaciwA. P. (2023). Do social programs help some beneficiaries more than others? Evaluating the potential for comparison group designs to yield low-bias estimates of differential impact. American Journal of Evaluation. https://doi.org/10.1177/10982140231160561
23.
JaciwA. P.UnluF.NguyenT. (2022). A within-study approach to evaluating the role of moderators of impact in limiting generalizations from “large to small.”. American Journal of Evaluation, 43(1), 108–131. https://doi.org/10.1177/10982140211030552
24.
KernH. L.StuartE. A.HillJ.GreenD. P. (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 19(1), 103–127. https://doi.org/10.1080/19345747.2015.1060282
25.
KruegerA. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532. https://doi.org/10.1162/003355399556052
26.
KruegerA. (2000). The class size policy debate. Economic Policy Institute Working Paper No.121.
27.
LalondeR. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620.
28.
MichalopolousC.BloomH. S.HillC. J. (2004). Can propensity-score methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs?The Review of Economics and Statistics, 86(1), 156–179. https://doi.org/10.1162/003465304323023732
29.
MiratrixL. W.WeissM. J.HendersonB. (2021). An applied researcher’s guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates. Journal of Research on Educational Effectiveness, 14(1), 270–308. https://doi.org/10.1080/19345747.2020.1831115
30.
MostellerF. (1995). The Tennessee study of class size in the early school grades. The Future of Children, 5, 113–127.
31.
NyeB.HedgesL. V.KonstantopoulosS. (2000). The effects of small classes on academic achievement: The results of the Tennessee class size experiment. American Educational Research Journal, 37(1), 123–151. https://doi.org/10.3102/00028312037001123
32.
NyeB.HedgesL. V.KonstantopoulosS. (2001). Are effects of small classes cumulative? Evidence from a Tennessee experiment. The Journal of Educational Research, 94(6), 336–345. https://doi.org/10.1080/00220670109598771
33.
NyeN.HedgesL. V.KonstantopoulosS. (1999). The long-term effects of small classes: A five-year follow-up of the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 21(2), 127–142. https://doi.org/10.3102/01623737021002127
34.
NyeN.HedgesL. V.KonstantopoulosS. (2002). Do low-achieving students benefit more from small classes? Evidence from the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 24(3), 201–217. https://doi.org/10.3102/01623737024003201
35.
OrrL. L.OlsenR. B.BellS. H.SchmidI.ShivjiA.StuartE. A. (2019). Using the results from rigorous multi-site evaluations to inform local policy decisions. Journal of Policy Analysis and Management, 38(4), 978–1003. https://doi.org/10.1002/pam.22154
36.
RaudenbushS. W.BrykA. S. (2002). Hierarchical Linear Models (2nd ed). Thousand Oaks, CA: Sage.
37.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
38.
SchochetP. Z.PumaM.DekeJ. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. Retrieved from http://ies.ed.gov/ncee/edlabs
39.
ScrivenM. (2008). A summative evaluation of XP methodology: & an alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11–24. https://doi.org/10.56645/jmde.v5i9.160
40.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
41.
ShadishW. R.LuellenJ. K.ClarkM. H. (2006). Propensity scores and quasi-experiments: A testimony to the practical side of lee sechrest. In BootzinR. R.McKnightP. E. (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 143–157). Washington, DC: American Psychological Association..
42.
SmithJ. A.ToddP. E. (2005). Does matching overcome Lalonde’s critique of non-experimental estimators?Journal of Econometrics, 125(1-2), 305–353. https://doi.org/10.1016/j.jeconom.2004.04.011
43.
SteinerP. M.WongV. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–247. https://doi.org/10.1177/0193841X18773807
44.
SteinerP. M.WongV. C.AnglinK. (2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift fur Psychologie, 227(4), 280–292. https://doi.org/10.1027/2151-2604/a000385
45.
TiptonE. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266. https://doi.org/10.3102/1076998612441947
46.
UnluF.LauenD.FullerS. C.BerglundT.EsteraE. (2021). Can quasi-experimental evaluations that rely on state longitudinal data systems replicate experimental results: Findings from a within-study comparison. Forthcoming atJournal of Policy Analysis and Management, 40(2), 572–613. https://doi.org/10.1002/pam.22295
47.
WeidmannB.MiratrixL. (2021). Lurking inferential monsters? Quantifying selection bias in evaluations of school programs. Journal of Policy Analysis and Management, 40(3), 964–986. https://doi.org/10.1002/pam.22236
48.
WeissM. J.BloomH. S.BrockT. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760
49.
WildeE. T.HollisterR. (2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477. https://doi.org/10.1002/pam.20262
50.
WongV. C.SteinerP. M.AnglinK. L. (2018). What can be learned from empirical evaluations of nonexperimental methods?Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
