Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact
Restricted accessResearch articleFirst published online September, 2024
Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact
In the current socio-political climate, there is an extra urgency to evaluate whether program impacts are distributed fairly across important student groups in education. Both experimental and quasi-experimental designs (QEDs) can contribute to answering this question. This work demonstrates that QEDs that compare outcomes across higher-level implementation units, such as schools, are especially well-suited to contributing evidence on differential program effects across student groups. Such designs, by differencing away site-level (macro) effects, on average produce estimates of the differential impact that are closer to experimental benchmark results than are estimates of average impact based on the same design. This work argues for the importance of routine evaluation of moderated impacts, describes the differencing procedure, and empirically tests the methodology with seven impact evaluations in education. The hope is to encourage broader use of this design type to more-efficiently develop the evidence base for differential program effects, particularly for underserved students.
Allen-PlattC.GerstnerC.BoruchR.RubyA. (2021). Toward a science of failure analysis: A narrative review. Review of Research in Education, 45(1), 223–252.
2.
AtheyS.ImbensG. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences of the United States of America, 113, 7353–7360.
3.
BledsoeK. L. (2018). Introduction to the section on race and evaluation. American Journal of Evaluation, 39, 511–513.
4.
BloomH. S. (2005). Randomizing groups to evaluate place-based programs. In BloomH. S. (Ed.), Learning more from social experiments (pp. 115–172). Russell Sage Foundation.
5.
BloomH. S.HillC. J.BlackA. R.LipseyM. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions [MDRC Working Papers on Research Methodology]. MDRC.
6.
BloomH. S.MichalopoulosC.HillC. J. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effect. In BloomH. S. (Ed.), Learning more from social experiments (pp. 173–235). Russell Sage Foundation.
7.
ChavezA.JaciwA.YiP.MaB. (2012). Comparative effectiveness of McGraw Hill’s Jamestown Reading Navigator: A report of a randomized experiment in public schools (Empirical Education Report Empirical_JRN-6023-FR1-Y2-O.1). Empirical Education, Inc.
8.
ChavezA.JaciwA. P.YiP.MaB. (2010). Comparative effectiveness of McGraw-Hill’s Jamestown Reading Navigator Program: A report of a randomized experiment in Texas schools (Empirical Education Report). Empirical Education, Inc.
9.
CookT. D. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the education evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175–199.
10.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.
11.
CronbachL. J.RajaratnamN.GleserG. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137–163.
12.
FrakerT.MaynardR. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. The Journal of Human Resources, 22, 194–227.
13.
GarciaN. M.LópezN.VélezV. N. (2018). Quantcrit: Rectifying quantitative methods through critical race theory. Race Ethnicity and Education, 21, 149–157. https://doi.org/10.1080/13613324.2017.1377675
14.
GelmanA.SternH. (2006). The difference between ‘‘significant’’ and ‘‘not significant’’ is not itself statistically significant. The American Statistician, 60, 328–331.
15.
GlazermanS.LevyD.MyersD. (2003). Nonexperimental versus experimental estimates of earnings impacts. American Academy of Political and Social Science, 589, 63–93.
16.
HeckmanJ. J.IchimuraK.ToddP. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64, 605–654.
17.
HedgesL. V.HedbergE. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37, 445–489.
18.
HotzV. J.ImbensG. W.MortimerJ. H. (2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125, 241–270.
19.
JaciwA.LiaoC.MaB.SterlingL. (2011). Comparative effectiveness of the Rosetta Stone Dynamic Immersion Program: Report of a randomized experiment in the school district of Osceola County (Empirical Education Rep. No. RS-6026-FR1-YR1-O.2). Empirical Education Inc.
20.
JaciwA. P. (2016b). Applications of a within-study comparison approach for evaluating bias in generalized causal inferences from comparison group studies. Evaluation Review, 40(3), 241–276.
21.
JaciwA. P. (2016a). Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology. Evaluation Review, 40(3), 199–240.
22.
JaciwA. P.HegsethW.LinL.TobyM.NewmanD.MaB.ZacamyJ. (2016). Assessing impacts of Math in Focus, a ‘Singapore math’ program. Journal of Research on Educational Effectiveness, 9(4), 473–502.
23.
JaciwA. P.LinL.MaB. (2016). An empirical study of design parameters for assessing differential impacts for students in group randomized trials. Evaluation Review, 40(5), 410–443.
24.
JaciwA. P.LinL.SchellingerA. (2018). Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities. Methodological Innovations, 11(2), 1–18. https://journals.sagepub.com/doi/pdf/10.1177/2059799118787751
25.
JaciwA. P.NguyenT.LinL.ZacamyJ.KwongC.LauS. (2020). Final report of the i3 impact study of Making Sense of SCIENCE, 2016–17 through 2017–18 (Empirical Education Report No. Empirical_MSS-7030-FR1-2020-O.1). Empirical Education, Inc. https://empiricaleducation.com/mss
26.
KruegerA. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532.
27.
Krueger, A., & Hanushek, E. (2000). The class size policy debate, Economic Policy Institute, Working Paper, 121. http://epinet.org
28.
LalondeR. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620.
29.
NguyenT.JaciwA. P. (2021). Three-armed trials to evaluate the effectiveness of market and improvement versions of programs in education (An Empirical Education Report). Empirical Education, Inc.
30.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models (2nd ed.). Sage.
31.
RosenbaumP. R.RubinB. D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
32.
SchochetP. Z.PumaM.DekeJ. (2014). Understanding variation in treatment effects in education impact evaluations: An overview of quantitative methods (NCEE 2014–4017). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. http://ies.ed.gov/ncee/edlabs
33.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi experimental designs for generalized causal inference. Houghton Mifflin.
34.
ShavelsonR. J.WebbN. M. (2008). Generalizability theory and its contributions to the discussion of the generalizability of research findings. In ErcikanK.RothW. M. (Eds.), Generalizing from educational research (pp. 13–32). Routledge.
35.
SingerJ. D. (1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 23(4), 323–355.
36.
SingerJ. D.WillettJ. B. (2003). Applied longitudinal data analysis. Oxford University Press.
37.
SpybrookJ. (2013). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82(3), 334–357.
38.
SteinerP. M.WongV. C.AnglinK. (2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift fur Psychologie, 227(4), 280–292.
39.
SterlingL.PingY.MaB.JaciwA. (2011). Effectiveness of Achieve3000’s TeenBiz3000 program (Empirical Education Report, Empirical_A3K-6031-FR-Y1-O.1). Empirical Education, Inc.
40.
TiptonE. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.
41.
TobyM.MaB.LaiG.LinL.JaciwA. (2012, March). Comparative effectiveness of Houghton Mifflin Harcourt Fuse: Algebra 1: A report of a randomized experiment in four California school districts (Empirical Education Report No. Empirical_HMH-6038-FR1-YR1_O.3). Empirical Education Inc.
42.
UnluF.LauenD.FullerS. C.BerglundT.EsteraE. (2021). Can quasi-experimental evaluations that rely on state longitudinal data systems replicate experimental results: Findings from a within-study comparison. Journal of Policy Analysis and Management, 40(2), 572–613.
43.
WeidmannB.MiratrixL. (2021). Lurking inferential monsters? Quantifying selection bias in evaluations of school programs. JPAM, 40(3), 964–986.
44.
WeissM. J.BloomH. S.Verbitsky-SavitzN.GuptaH.VigilA.CullinanD. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876.
45.
WildeE. T.HollisterR. (2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477.
46.
WongV. C.SteinerP. M.AnglinK. L. (2018). What can be learned from empirical evaluations of nonexperimental methods?Evaluation Review, 42(2), 147–175.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
