Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on a variety of assumptions and design decisions that are not clearly outlined in the literature. This article aims to serve as a practice guide for applied researchers when deciding how and whether to use this approach. We begin by providing an overview of the assumptions needed to estimate causal effects using school-level data, common threats to validity faced in practice and what effects can and cannot be estimated using school-level data. We then examine two analytic decisions researchers face in practice when implementing the design: correctly modeling the pretreatment functional form, which is modeling the preintervention trend, and selecting comparison cases. We then illustrate the use of this design in practice drawing on data from the implementation of the school improvement grant (SIG) program in Ohio. We conclude with advice for applied researchers implementing this design.
AbadieA.DiamondA.HainmuellerJ. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505.
2.
AndersonP. M.ButcherK. F.SchanzenbachD. W. (2017). Adequate (or adipose?) yearly progress: Assessing the effect of “No Child Left Behind” on children’s obesity. Education Finance and Policy, 121, 54–76.
3.
AshenfelterO. (1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 60, 47–57.
4.
AshenfelterO.CardD. (1985). Using the longitudinal structure of earnings to estimate the effect of training programs. Review of Economics and Statistics, 67, 648–660.
5.
AtheyS.ImbensG. W. (2006). Identification and inference in non-linear difference-in-difference models. Econometrica, 74(2), 431–497.
6.
BellS. H.OrrL. L.BlomquistJ. D.CainG. G. (1995). Program applicants as a comparison group in evaluating training programs: Theory and a test. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. https://doi.org/10.17848/9780585284545
7.
BettsJ.LevinJ.MirandaA. P.ChristensonB.EatonM.BosH. (2010). An evaluation of alternate matching techniques for use in comparative interrupted time series analyses: An application to elementary education. American Institutes for Research Working Paper.
8.
BifulcoR. (2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31(3), 729–751.
9.
BloomH. (2003). Using “short” interrupted time-series analysis to measure the impacts of whole schools reforms: With applications to a study of accelerated schools. Evaluation Review, 27, 3–49.
10.
BloomH. S.MichalopoulosC.HillC. J. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In BloomH. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 173–235). New York, NY: Russell Sage Foundation.
11.
ChingosM. M. (2012). The impact of a universal class-size reduction policy: Evidence from Florida’s statewide mandate. Economics of Education Review, 31(5), 543–562. https://doi.org/10.1016/j.econedurev.2012.03.002.
12.
CookT. D.CampbellD. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin.
13.
CookT. D.ShadishW. J.WongV. C. (2008). Three conditions under which observational studies produce the same results as experiments. Journal of Policy Analysis and Management, 27(4), 724–750.
14.
CurranF. C. (2016). Estimating the effect of state zero tolerance laws on exclusionary discipline, racial discipline gaps, and student behavior. Educational Evaluation and Policy Analysis, 39(4), 647–668.
15.
DeeT. S.JacobB.SchwartzN. (2013). The effects of NCLB on school resources and practices. Educational Evaluation and Policy Analysis, 35, 252–279.
16.
DelaneyJ. A.KearneyT. D. (2015). The impact of guaranteed tuition policies on postsecondary tuition levels: A difference-in-difference approach. Economics of Education Review, 47, 80–99. https://doi.org/10.1016/j.econedurev.2015.04.003.
FreedmanD. A. (2001). Ecological inference and the ecological fallacy. In SmelserN. J.BaltesP. B. (Eds.), International encyclopedia for the social and behavioral sciences (Vol. 6, pp. 4027–4030). New York, NY: Elsevier.
19.
FretheimA.SoumeraiS. B.ZhangF.OxmanA. D.Ross-DegnanD. (2013) Interrupted time-series analysis yielded an effect estimate concordant with the cluster randomized controlled-trial result. Journal of Clinical Epidemiology, 66(8), 883–887.
20.
FriedlanderD.RobinsP. K. (1995). Evaluating program evaluations: New evidence on commonly used nonexperimental methods. The American Economic Review, 85(4), 923–937.
21.
GlazermanS.LevyD.MyersD. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589, 63–91.
22.
HallbergK.WilliamsR. T.SwanlundA. (March2017). Examining the internal validity of school-level comparative interrupted time series designs using randomized experiment causal benchmarks. Paper presented at the Annual Meeting of the Society for Research on Educational Effectiveness, Washington, DC.
23.
HallbergK.WongV. C.CookT. D. (2017). Evaluating methods for selecting school-level comparisons in quasi-experimental designs: Results from a within-study comparison. Manuscript submitted for publication.
24.
HeckmanJ. J. (1979). Sample selection bias as a specification error. Econometrika, 47, 153–161.
25.
HeckmanJ.IchimuraH.SmithJ.ToddP. (1998). Characterizing selection bias using experimental data (No. w6699). National Bureau of Economic Research.
26.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.
27.
HollandP. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
28.
ImbensG. W. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47(1), 5–86.
29.
JacobR. T.GoddardR. D.KimE. S. (2014). Assessment of the use of aggregate data in the evaluation of school-based interventions: Implications for evaluation research and state policy regarding public-use data. Educational Evaluation and Policy Analysis, 36(1), 44–66.
30.
JacobR.SomersM.-A.ZhuP.BloomH. (2016). The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Evaluation Review, 40(3), 167–198. https://doi.org/10.1177/0193841X16663414
31.
LaLondeR. (1986). Evaluating the econometric evaluations of training programs with experimental data. Annual Economic Review, 76, 604–620.
32.
LecherM. (2010). The estimation of causal effects by difference-in-difference models. University of St. Gallen Discussion Paper no. 2010–28.
33.
MillerL. C.MittlemanJ. (2012). High Schools That Work and college preparedness: Measuring the model’s impact on mathematics and science pipeline progression. Economics of Education Review, 31(6), 1116–1135. https://doi.org/10.1016/j.econedurev.2012.07.014.
34.
NeymanJ. (1935). Statistical problems in agricultural experimentation (with discussion). Supplement to the Journal of the Royal Statistical Society, 2, 107–108.
35.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
36.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.
37.
SchneeweissS.MaclureM.CarletonB.GlynnR. J.AvornJ. (2004) Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: Direct comparison of randomised and observational evaluations. BMJ, 328, 560.
38.
ShadishW. R.CookT.CampbellD. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
39.
SomersM.ZhuP.JacobR.BloomH. (2013). The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation. MDRC working paper in research methodology. New York, NY.
40.
St. ClairT.CookT. D.HallbergK. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35(3), 1–17.
41.
St. ClairT.HallbergK.CookT. D. (2016). The validity and precision of the comparative interrupted time series design: Three within-study comparisons. Journal of Educational and Behavioral Statistics, 41(3), 269–299.
42.
SteinerP.M.CookT.D.ShadishW.R. (2011). On the importance reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics, 36(2), 213-236.
43.
StuartE. A. (2007). Estimating causal effects using school-level data sets. Educational Researcher, 36(4): 187–198.
44.
StuartE. A.RubinD. B. (2008). Matching with multiple control groups and adjusting for group differences. Journal of Educational and Behavioral Statistics, 33(3): 279–306.
WongM.CookT. D.SteinerP. M. (2015). Adding design elements to improve time series designs: No Child Left Behind as an example of causal pattern matching. Journal of Research on Educational Effectiveness, 8(2), 245–279.
48.
WongV. C.WingC.SteinerP. M.WongM.CookT. D. (2012). Research designs for program evaluation. In VelicerW.SchinkaJ. (Eds.), Handbook of psychology: research methods in psychology (2nd ed.). Hoboken, NJ: Wiley and Sons.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
