This study explores the extent to which differences in research design explain variation in Head Start program impacts. We employ meta-analytic techniques to predict effect sizes for cognitive and achievement outcomes as a function of the type and rigor of research design, quality and type of outcome measure, activity level of control group, and attrition. Across program evaluations, the average program-level effect size was 0.27 standard deviations. About 41% of the variation in estimates across evaluations can be explained by research design features, including the extent to which the control group experienced other forms of early care or education, and 11% of the variation within programs can be explained by the quality and type of the outcome measures.
CamilliG.VargasS.RyanS.BarnettW. S. (2010). Meta-analysis of the effects of early education interventions on cognitive and social development. Teachers College Record, 112(3), 579–620.
4.
ChristianK.MorrisonF. J.FrazierJ. A.MassettiG. (2000). Specificity in the nature and timing of cognitive growth in kindergarten and first grade. Journal of Cognition and Development, 1(4), 429–448.
5.
CookT. (2006). What works in publicly funded pre-kindergarten education?Presented at the Children's Achievement: What the Evidence Says about Teachers, Pre-K Programs and Economic Policies Policy Briefing, Washington, D.C.
6.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.
7.
CookT. D.WongV. C. (2007). The warrant for universal pre-K: Can several thin reeds make a strong policy boat?Social Policy Report, 21(3), 14–15.
8.
CooperH.HedgesL. V. (2009). Research synthesis as a scientific process. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 3–17). New York, NY: Russell Sage.
9.
CurrieJ.ThomasD. (1995). Does Head Start make a difference?American Economic Review, 85, 341–364.
10.
de la TorreJ.CamilliG.VargasS.VernonR. F. (2007). Illustration of a multilevel model for meta-analysis. Measurement and Evaluation in Counseling and Development, 40, 169–180.
11.
GibbsC.LudwigJ.MillerD. (2012). Does Head Start do any lasting good? (Working Paper 17452). Cambridge, MA: National Bureau of Economic Research.
12.
GoreyK. M. (2001). Early childhood education: A meta-analytic affirmation of the short- and long-term benefits of educational opportunity. School Psychology Quarterly, 16(1), 9–30.
13.
GormleyW. T.Jr.PhillipsD.GayerT. (2008). Preschool programs can boost school readiness. Science, 320, 1723–1724.
14.
HenryG. T.GordonC. S.RickmanD. K. (2006). Early education policy alternatives: Comparing quality and outcomes of Head Start and state prekindergarten. Educational Evaluation and Policy Analysis, 28, 77–99.
15.
HoytW. T.KernsM.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4(4), 403–424.
16.
JacobR. T.CrepsC. L.BoulayB. (2004). Meta-analysis of research and evaluation studies in early childhood education. Cambridge, MA: Abt Associates.
17.
LayzerJ. I.GoodsonB. D.BernsteinL.PriceC. (2001). National evaluation of family support programs, volume A: The meta-analysis, final report. Cambridge, MA: Abt Associates.
18.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
19.
LudwigJ.PhillipsD. (2007). The benefits and costs of Head Start. Social Policy Report, 21(3), 3–18.
20.
MagnusonK.ShagerH. (2010). Early education: Progress and promise for children from low-income families. Children and Youth Services Review, 32(9), 1186-1198.
21.
McKeyR. H.CondelliL.GansonH.BarrettB. J.McConkeyC.PlantzM. C. (1985). The impact of Head Start on children, families and communities: Final report of the Head Start Evaluation, Synthesis and Utilization Project. Washington, DC: CSR.
22.
MoherD.PhamB.JonesA.CookD. J.JadadA. R.MoherM. . . . KlassenT. P. (1998). Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?Lancet, 352, 609–613.
23.
National Forum on Early Childhood Policy and Programs. (2010). Understanding the Head Start Impact Study. Retrieved from http://www.developingchild.harvard.edu/
24.
NelsonG.WesthuesA.MacLeodJ. (2003). A meta-analysis of longitudinal research on preschool prevention programs for children. Prevention and Treatment, 6, 1–34.
25.
RosenshineB. (2010, March). Researcher-developed tests and standardized tests: A review of ten meta-analyses. Paper presented at the Society for Research on Educational Effectiveness conference, Washington, DC.
26.
RothsteinH. R.HopewellS. (2009). Grey literature. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103–125). New York, NY: Russell Sage.
27.
SchulzK. F.ChalmersI.HayesR. J.AltmanD. G. (1995). Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273(5), 408–412.
28.
SchneiderB.CarnoyM.KilpatrickJ.SchmidtW. H.ShavelsonR. J. (2007). Estimating causal effects using experimental and observational designs (report from the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association.
29.
ShadishW. R.ClarkM. H.SteinerP. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334–1344.
30.
ShadishW. R.GalindoR.WongV. C.SteinerP. M.CookT. D. (2011). A randomized experiment comparing random and cutoff-based assignment. Psychological Methods, 16(2), 179–191.
31.
SweetM. A.AppelbaumM. I. (2004). Is home visiting an effective strategy? A meta-analytic review of home visiting programs for families with young children. Child Development, 75(5), 1435–1456.
32.
U.S. Department of Health and Human Services, Administration for Children and Families. (2005, May). Head Start impact study: First year findings. Washington, DC: Author.
33.
U.S. Department of Health and Human Services, Administration for Children and Families. (2010, January). Head Start impact study: Final report. Washington, DC: Author.
34.
U.S. Department of Health and Human Services, Administration for Children and Families, Office of Head Start. (2010a). Head Start performance standards & other regulations. Retrieved from http://www.acf.hhs.gov/programs/ohs/legislation/index.html
35.
U.S. Department of Health and Human Services, Administration for Children and Families, Office of Head Start. (2010b). Head Start program fact sheet. Retrieved from http://www.acf.hhs.gov/programs/ohs/about/fy2010.html
36.
Westinghouse Learning Corporation. (1969). The impact of Head Start: An evaluation of the effects of Head Start on children’s cognitive and affective development. In A report presented to the Office of Economic Opportunity (PB 184328). Washington, DC: Distributed by Clearinghouse for Federal Scientific and Technical Information, U.S. Department of Commerce, National Bureau of Standards, Institute for Applied Technology.
37.
WongV. C.CookT. D.BarnettW. S.JungK. (2008). An effectiveness-based evaluation of five state pre-kindergarten programs. Journal of Policy Analysis and Management, 27, 122–154.
38.
ZhaiF.Brooks-GunnJ.WaldfogelJ. (2010, March). Head Start and urban children’s school readiness: A birth cohort study in 18 cities. Paper presented at the Society for Research on Educational Effectiveness conference, Washington, DC.
39.
ZiglerE.ValentineJ. (Eds.). (1979). Project Head Start: A legacy of the war on poverty. New York, NY: Free Press.
