Performance-based accountability along with budget tightening has increased pressure on publicly funded organizations to develop and deliver programs that produce meaningful social benefits. As a result, there is increasing need to undertake formative evaluations that estimate preliminary program outcomes and identify promising program components based on their effectiveness during implementation. By combining longitudinal administrative data, multiple comparison group designs, and a progressive series of analyses that test rival explanations, evaluators can strengthen causal arguments and provide actionable program information for key stakeholders to improve program outcomes. In this article, we illustrate the application of rigorous methods to estimate preliminary program effects and rule out alternative explanations for preliminary effects, including site selection bias, individual selection bias, and resentful demoralization through the evaluation of the Collaborative Project, a North Carolina educational improvement project that incorporated multiple components aimed at boosting student achievement.
AngristJ. D.ImbensG. W.RubinD. B. (1996). Identification of causal effects using instrumental variables (with comments). Journal of the American Statistical Association, 91, 444–472.
2.
BerkR. A.ShermanL. W. (1988). Police responses to family violence incidents: An analysis of an experimental design with incomplete randomization. Journal of the American Statistical Association, 83, 70–76.
3.
BifulcoR. (2012). Can non-experimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 24, 113–132.
4.
BoydD.GrossmanP.LankfordH.LoebS.WyckoffJ. (2006). How changes in entry requirements alter the teacher workforce and affect student achievement. Education Finance and Policy, 1, 176–216.
5.
BradenR. M. (1992). Formative evaluation: A revised descriptive theory and a prescriptive model. Paper presented at the Annual Meeting of the Association for Educational Communications and Technology, Washington, DC.
6.
BuddinR.ZamarroG. (2009). Teacher qualifications and student achievement in urban elementary schools. Journal of Urban Economics, 66, 103–115.
7.
ChenH. T. (1996). A comprehensive typology of program evaluation. Evaluation Practice, 17, 121–130.
8.
ClotfelterC. T.LaddH. F.VigdorJ. (2007). How and why do teacher credentials matter for student achievement?National Bureau of Economic Research Working Paper No. 12828, National Bureau of Economic Research, Cambridge, MA.
9.
ClotfelterC. T.LaddH. F.VigdorJ. (2010). Teacher credentials and student achievement in high school: A cross-subject analysis with student fixed effects. Journal of Human Resources, 45, 655–681.
10.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.
11.
CorynC. L. S. (2007). Using hierarchical linear modeling for proformative evaluation: A case example. Journal of MultiDisciplinary Evaluation, 4, 53–60.
12.
DesimoneL. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualization and measures. Educational Researcher, 38, 181–199.
13.
DesimoneL. M.PorterA. C.GaretM.YoonK. S.BirmanB. (2002). Does professional development change teachers’ instruction? Results from a three-year study. Educational Evaluation and Policy Analysis, 24, 81–112.
14.
DiazJ. J.HandaS. (2005). An assessment of propensity score matching as a nonexperimental impact estimator: Evidence from Mexico’s PROGRESA program. Journal of Human Resources, 41, 319–345.
15.
FrangakisC.RubinD. (2002). Principal stratification in causal inference. Biometrics, 58, 21–29.
16.
GlazermanS.LevyD. M.MyersD. (2003). Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy of Political and Social Science, 589, 63–93.
17.
GoldhaberD.AnthonyE. (2007). Can teacher quality be effectively assessed? National Board certification as a signal of effective teaching. The Review of Economics and Statistics, 89, 134–150.
18.
GoldhaberD.BrewerD. J. (2000). Does teacher certification matter? High school teacher certification status and student achievement. Educational Evaluation and Policy Analysis22, 129–145.
19.
GormleyW. T.WeimerD. L. (1999). Organizational report cards. Cambridge, MA: Harvard University Press.
20.
HanushekE. (1997). Assessing the effects of school resources on student performance: An update.
Educational Evaluation and Policy Analysis19, 141–164.
21.
HarrisD.SassT. (2011). Teacher training, teacher quality, and student achievement. Journal of Public Economics, 95, 798–812.
22.
HeckmanJ. J.IchimuraH.ToddP. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64, 605–654.
23.
HenryG. T. (2010). Comparison group designs. In WholeyJoseph S.HatryHarry P.NewcomerKathryn E. (Eds.), Handbook of practical program evaluation (3rd ed.). San Francisco, CA: Jossey-Bass.
24.
HenryG. T.BastianK. C.FortnerC. K. (2011). Stayers and leavers: Early-career teacher effectiveness and attrition. Educational Researcher, 40, 271–280.
25.
HenryG. T.BastianK. C.FortnerC. K.KershawD. C.PurtellK. M.ThompsonC. L.ZulliR. A. (2014). Teacher preparation policies and their effects on student achievement. Education Finance and Policy.
26.
HenryG. T.FortnerC. K.BastianK. C. (2012). The effects of experience and attrition for novice high school science and mathematics teachers. Science, 335, 1118–1121.
27.
HenryG. T.ThompsonC. L.FortnerC. K.ZulliR. A.KershawD. C. (2010). The impact of teacher preparation on student learning in North Carolina public schools. Chapel Hill: Carolina Institute for Public Policy, University of North Carolina at Chapel Hill.
28.
Jonsson FunkM. L.WestreichD. (2008). Doubly robust estimation under realistic conditions of model misspecification. Pharmacoepidemiology and Drug Safety, 17, S106.
29.
KrumholzH.HerrinJ. (2000). Quality improvement: the need is there but so are the challenges. American Journal of Medicine, 109, 501–3.
30.
PattonM. Q. (2011). Developmental evaluation. New York, NY: Guilford.
31.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–45.
32.
RubinD. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100, 322–331.
33.
ScrivenM. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation, 17, 151–161.
34.
ShadishW. R.ClarkM. H.SteinerP. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103, 1334–1343.
35.
StetlerC. B.LegroM. W.WallaceC. M.BowmanC.GuihanM.HagedornH.KimmelB.SharpN. D.SmithJ. L. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine, 21, S1–S8.
36.
TessmerM. (1993). Planning and conducting formative evaluation. London, England: Taylor & Francis.
37.
ThompsonC. L.CunninghamE. K.SmithA.PhillipsJ. C.ZulliR. A. (2009). The collaborative project: The first two years. Chapel Hill: Carolina Institute for Public Policy, University of North Carolina at Chapel Hill.
