How Three-Arm Random Assignment Within Sites can Improve Non-Experimental Cross-Site Estimates of the Relationship Between Program Characteristics and Impact
Restricted accessResearch articleFirst published online 2022
How Three-Arm Random Assignment Within Sites can Improve Non-Experimental Cross-Site Estimates of the Relationship Between Program Characteristics and Impact
This article considers a new method, called Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings (abbreviated as “CAMIC”), which seeks to reduce bias in analyses of the contributions of a program’s various design, implementation, and contextual characteristics to its overall impacts. It requires a multi-site experiment where some (or all) sites randomize individuals to one of three arms: a standard treatment group, an enhanced treatment group (that receives the standard treatment plus a program “enhancement”), or a control group (that has no access to the program). A recent evaluation—that of the Health Profession Opportunity Grants (HPOG) program—provides a motivating example of this design and its potential for both methodological and substantive lessons; and the article considers other evaluations that would be fitting for deploying and learning from CAMIC. We conclude that the promise for CAMIC lies in situations where the correlations between the selected program enhancement and alternative program characteristics of interest are relatively high, implying that producing an experimental estimate of the enhancement can reduce bias in the estimation of other non-randomized program characteristics.
BellS. H.HarvillE. L.MoultonS. R.PeckL. (2017). Using within-site experimental evidence to reduce cross-site attributional bias in connecting program components to program impacts, OPRE report #2017-13. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
2.
BloomH.HillC. J.RiccioJ. A. (2005). Modeling cross-site experimental differences to find out why program effectiveness varies. In BloomH. S. (Ed.), Learning more from social experiments (pp. 37–74). Russell Sage Foundation.
3.
BloomH. S.HillC. J.RiccioJ. A. (2001). Modeling the performance of welfare-to-work programs: The effects of program management and services, economic environment, and client characteristics. Manpower Demonstration Research Corporation.
4.
BloomH. S.HillC. J.RiccioJ. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154
5.
BosJ.GrangerR. (2000). Estimating the effects of day care use on children’s school readiness: Evidence from the new chance demonstration. Manpower Demonstration Research Corporation.
6.
BrykA.RaudenbushS. (1992). Hierarchical linear models: Applications and data analysis methods. Sage.
7.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 725–750. https://doi.org/10.1002/pam.20375
8.
DeMarcoD.FioreN.MoultonS.BocianD. G.PeckL.BallingerS.CuencaK.RothschildL.StephenW. (2017). The first-time homebuyer education and counseling demonstration baseline report: Study design and implementation. U.S. Department of Housing and Urban Development, Office of Policy Development and Research.
9.
DorsettR.RobinsP. K. (2014). A multilevel analysis of the impacts of services provided by the U.K. employment retention and advancement demonstration. Evaluation Review, 37(2), 63–108. https://doi.org/10.1177/0193841X13517383
10.
GennetianL. A.BosJ. M.MorrisP. A. (2002). Using instrumental variables analysis to learn more from social policy experiments. Manpower Demonstration Research Corporation.
11.
GlazermanS.LevyD. M.MyersD. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589(1), 63–93. https://doi.org/10.1177/0002716203254879
GreenS. B. (1991). How many subjects does it take to do a regression analysis?Multivariate Behavioral Research, 26(3), 499–510. https://doi.org/10.1207/s15327906mbr2603_7
14.
GreenbergD.MeyerR. H.WisemanM. (1993). Prying the lid from the black box: Plotting evaluation strategy for welfare employment and training programs. University of Wisconsin-Madison, Institute for Research on Poverty.
15.
GreenbergD.MeyerR. H.WisemanM. (1994). Multi-site employment and training evaluations: A tale of three studies. Industrial and Labor Relations Review, 47(4), 679–691. https://doi.org/10.1177/001979399404700411
16.
GreenbergD.RobinsP. K. (2011). Have welfare-to-work programs improved over time in putting welfare recipients to work?Industrial and Labor Relations Review, 64(5), 910–920. https://doi.org/10.2307/41343706
17.
GubitsD.LinW.BellS. H.JudkinsD. (2014). BOND implementation and evaluation: First- and second-year snapshot of earnings and benefit impacts for stage 2. (Deliverable 24c.5) U.S. Social Security Administration.
18.
HamiltonG.FreedmanS.GennetianL.MichalopoulosC.WalterJ.Adams-CiardulloD.Gassman-PinesA. (2001). National evaluation of welfare-to-work Strategies: How effective are different welfare-to-work approaches? Five-year adult and child impacts for eleven programs. U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning and Evaluation and U.S. Department of Education, Office of the Deputy Secretary, Planning and Evaluation Service, Office of Vocational and Adult Education.
LaLondeR. (1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76(4), 604–620.
21.
MagnussonK. A.McGroderS. (2002). The effects of increasing welfare mothers’ education on their young children’s academic problems and school readiness. Northwestern University.
22.
MoultonS.BellS. H.PeckL. R. (2014). Social policy impact pathfinder (SPI-path) analytic suite: SPI-Path|Site user guide. Abt Associates Inc.
23.
PeckL. RHilaryB.NicoleC. (2022). Insights from the health profession opportunity grant program’s three-armed, multi-site experiment for policy learning and evaluation practice. Evaluation Review. Advance online publication. https://doi.org/10.1177/0193841X221133420
24.
PeckL. RWernerA.HarvillE.LitwokD.MoultonS.FountainA. R.LockeG. (2018). Health profession opportunity grants (HPOG 1.0) impact study interim report: Program implementation and short-term impacts, OPRE report #2018–16a. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
25.
ReardonS. F.UnluF.ZhuP.BloomH. S. (2013). Bias and bias correction in multi-site instrumental variables analysis of heterogeneous mediator effects. Journal of Educational and Behavioral Statistics, 36(1), 53–86. https://doi.org/10.3102/1076998613512525
26.
SteinerP. M.KimY. (2016). The mechanics if omitted variable bias: Bias amplification and cancellation of offsetting biases. Journal of Causal Inference, 4(2), 20160009. https://doi.org/10.1515/jci-2016-0009
27.
SteinerP. M.WongV. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–247. https://doi.org/10.1177/0193841X18773807
28.
VanVoorhisC. R. W.MorganB. L. (2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(27), 43–50. https://doi.org/10.20982/tqmp.03.2.p043
29.
WongV. C.SteinerP. M.AnglinK. L. (2018). What can be learned from empirical evaluations of nonexperimental methods?Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870
30.
WooldridgeJ. M. (2002). Econometric analysis of cross section and panel data. The MIT Press.
