Administered by the Office of Family Assistance in the U.S. Department of Health and Human Services’ Administration for Children and Families (ACF), the Health Profession Opportunity Grants (HPOG) Program provided education and training to Temporary Assistance for Needy Families (TANF) recipients and other adults with low incomes for occupations in the healthcare field. The impact evaluation for the first cohort of grantees (HPOG 1.0) leveraged the program’s implementation across many locations, using a three-armed evaluation design (including a second treatment arm) in some places, as a way to examine whether any of three selected program components, or enhancements, contributed to the program’s overall impact. This article tells the story of the evaluation and draws lessons from that experience, discussing implications for future implementation of multi-armed experiments in a multi-site evaluation.
BellS. H.PeckL. R. (2016). On the “how” of social experiments: Experimental designs for getting inside the black box. New Directions for Evaluation, 152, 97–107. https://doi.org/10.1002/ev.20210
2.
BloomH. S.HillC. J.RiccioJ. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1177/10.1002/pam.10154
3.
FeinD. J. (2012). Career pathways as a framework for program design and evaluation: A working paper from the pathways for advancing careers and education (PACE) project (OPRE report 2012-30). Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Grant-ValloneE.ReidK.UmaliC.PohlertE. (2004). An analysis of the effects of self-esteem, social support, and participation in student support services on students’ adjustment and commitment to college. Journal of College Student Retention: Research, Theory & Practice, 5(3), 255–274. https://doi.org/10.1177/10.2190/C0T7-YX50-F71V-00CW
6.
GreenbergD.MeyerR. H.WisemanM. (1994). Multi-site employment and training evaluations: A tale of three studies. Industrial and Labor Relations Review, 47(4), 679–691.
7.
HarvillE.D.MoultonS.Rulf FountainA.PeckL.R. (2018). Technical supplement to the health profession opportunity grants (HPOG) impact study interim report: Report appendices (OPRE report 2018-16b). Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
8.
MaguireS.FreelyJ.ClymerC.ConwayM. (2009). Job training that works: Findings from the sectoral employment impact study. Public/Private Ventures.
PeckLaura R. (2020). Experimental Evaluation Design for Program Improvement. Thousand Oaks, CA: SAGE Publications.
11.
PeckL. R.WernerA.HarvillE.LitwokD.MoultonS.Rulf FountainA.LockeG. (2018). Health profession opportunity grants (HPOG 1.0) impact study interim report: Program implementation and short-term impacts (OPRE report 2018-16a). Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
12.
RiccioJ.FriedlanderD.FreedmanS. (with Farrell, M. E., Fellerath, V., Fox, S., & Lehman, D. J. (1994). GAIN: Benefits, costs, and three-year impacts of a welfare-to-work program. Manpower Demonstration Research Corporation.
13.
SolmeyerA. R.ConstanceN. (2015). Unpacking the “black box” of social programs and policies: Introduction. American Journal of Evaluation, 36(4), 470–474. https://doi.org/10.1177/1098214015600786
14.
WaltonD.HarvillE. L.PeckL. R. (2019). Which program characteristics are linked to program impacts? Lessons from the HPOG 1.0 evaluation. OPRE report 2019-51. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
15.
WernerA.LoprestP.SchwartzD.KoralekR.SickN. (2018). Final report: National implementation evaluation of the first round health profession opportunity grants (HPOG 1.0), OPRE report 2018-09. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
