Short Comparative Interrupted Time Series Using Aggregate School-Level Data in Education Research

Abstract

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on a variety of assumptions and design decisions that are not clearly outlined in the literature. This article aims to serve as a practice guide for applied researchers when deciding how and whether to use this approach. We begin by providing an overview of the assumptions needed to estimate causal effects using school-level data, common threats to validity faced in practice and what effects can and cannot be estimated using school-level data. We then examine two analytic decisions researchers face in practice when implementing the design: correctly modeling the pretreatment functional form, which is modeling the preintervention trend, and selecting comparison cases. We then illustrate the use of this design in practice drawing on data from the implementation of the school improvement grant (SIG) program in Ohio. We conclude with advice for applied researchers implementing this design.

Keywords

achievement evaluation policy quasi-experimental analysis research methodology school/teacher effectiveness

Get full access to this article

View all access options for this article.

References

Abadie

Diamond

Hainmueller

(2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505.

Anderson

P. M.

Butcher

K. F.

Schanzenbach

D. W.

(2017). Adequate (or adipose?) yearly progress: Assessing the effect of “No Child Left Behind” on children’s obesity. Education Finance and Policy, 121, 54–76.

Ashenfelter

(1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 60, 47–57.

Ashenfelter

Card

(1985). Using the longitudinal structure of earnings to estimate the effect of training programs. Review of Economics and Statistics, 67, 648–660.

Athey

Imbens

G. W.

(2006). Identification and inference in non-linear difference-in-difference models. Econometrica, 74(2), 431–497.

Bell

S. H.

Orr

L. L.

Blomquist

J. D.

Cain

G. G.

(1995). Program applicants as a comparison group in evaluating training programs: Theory and a test. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. https://doi.org/10.17848/9780585284545

Betts

Levin

Miranda

A. P.

Christenson

Eaton

Bos

(2010). An evaluation of alternate matching techniques for use in comparative interrupted time series analyses: An application to elementary education. American Institutes for Research Working Paper.

Bifulco

(2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31(3), 729–751.

Bloom

(2003). Using “short” interrupted time-series analysis to measure the impacts of whole schools reforms: With applications to a study of accelerated schools. Evaluation Review, 27, 3–49.

10.

Bloom

H. S.

Michalopoulos

Hill

C. J.

(2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In Bloom

H. S.

(Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 173–235). New York, NY: Russell Sage Foundation.

11.

Chingos

M. M.

(2012). The impact of a universal class-size reduction policy: Evidence from Florida’s statewide mandate. Economics of Education Review, 31(5), 543–562. https://doi.org/10.1016/j.econedurev.2012.03.002.

12.

Cook

T. D.

Campbell

D. T.

(1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin.

13.

Cook

T. D.

Shadish

W. J.

Wong

V. C.

(2008). Three conditions under which observational studies produce the same results as experiments. Journal of Policy Analysis and Management, 27(4), 724–750.

14.

Curran

F. C.

(2016). Estimating the effect of state zero tolerance laws on exclusionary discipline, racial discipline gaps, and student behavior. Educational Evaluation and Policy Analysis, 39(4), 647–668.

15.

Dee

T. S.

Jacob

Schwartz

(2013). The effects of NCLB on school resources and practices. Educational Evaluation and Policy Analysis, 35, 252–279.

16.

Delaney

J. A.

Kearney

T. D.

(2015). The impact of guaranteed tuition policies on postsecondary tuition levels: A difference-in-difference approach. Economics of Education Review, 47, 80–99. https://doi.org/10.1016/j.econedurev.2015.04.003.

17.

de la Torre

Allensworth

Jagesic

Sebastian

Salmonowicz

Meyers

Gerdeman

R. D.

(2012). Turning around low-performing schools in Chicago: Summary report. Chicago, IL: University of Chicago Consortium on Chicago School Research. Retrieved from https://consortium.uchicago.edu/sites/default/files/publications/12CCSRTurnAround-3.pdf.

18.

Freedman

D. A.

(2001). Ecological inference and the ecological fallacy. In Smelser

N. J.

Baltes

P. B.

(Eds.), International encyclopedia for the social and behavioral sciences (Vol. 6, pp. 4027–4030). New York, NY: Elsevier.

19.

Fretheim

Soumerai

S. B.

Zhang

Oxman

A. D.

Ross-Degnan

(2013) Interrupted time-series analysis yielded an effect estimate concordant with the cluster randomized controlled-trial result. Journal of Clinical Epidemiology, 66(8), 883–887.

20.

Friedlander

Robins

P. K.

(1995). Evaluating program evaluations: New evidence on commonly used nonexperimental methods. The American Economic Review, 85(4), 923–937.

21.

Glazerman

Levy

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589, 63–91.

22.

Hallberg

Williams

R. T.

Swanlund

(March 2017). Examining the internal validity of school-level comparative interrupted time series designs using randomized experiment causal benchmarks. Paper presented at the Annual Meeting of the Society for Research on Educational Effectiveness, Washington, DC.

23.

Hallberg

Wong

V. C.

Cook

T. D.

(2017). Evaluating methods for selecting school-level comparisons in quasi-experimental designs: Results from a within-study comparison. Manuscript submitted for publication.

24.

Heckman

J. J.

(1979). Sample selection bias as a specification error. Econometrika, 47, 153–161.

25.

Heckman

Ichimura

Smith

Todd

(1998). Characterizing selection bias using experimental data (No. w6699). National Bureau of Economic Research.

26.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.

27.

Holland

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.

28.

Imbens

G. W.

(2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47(1), 5–86.

29.

Jacob

R. T.

Goddard

R. D.

Kim

E. S.

(2014). Assessment of the use of aggregate data in the evaluation of school-based interventions: Implications for evaluation research and state policy regarding public-use data. Educational Evaluation and Policy Analysis, 36(1), 44–66.

30.

Jacob

Somers

M.-A.

Zhu

Bloom

(2016). The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Evaluation Review, 40(3), 167–198. https://doi.org/10.1177/0193841X16663414

31.

LaLonde

(1986). Evaluating the econometric evaluations of training programs with experimental data. Annual Economic Review, 76, 604–620.

32.

Lecher

(2010). The estimation of causal effects by difference-in-difference models. University of St. Gallen Discussion Paper no. 2010–28.

33.

Miller

L. C.

Mittleman

(2012). High Schools That Work and college preparedness: Measuring the model’s impact on mathematics and science pipeline progression. Economics of Education Review, 31(6), 1116–1135. https://doi.org/10.1016/j.econedurev.2012.07.014.

34.

Neyman

(1935). Statistical problems in agricultural experimentation (with discussion). Supplement to the Journal of the Royal Statistical Society, 2, 107–108.

35.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

36.

Rubin

D. B.

(1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.

37.

Schneeweiss

Maclure

Carleton

Glynn

R. J.

Avorn

(2004) Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: Direct comparison of randomised and observational evaluations. BMJ, 328, 560.

38.

Shadish

W. R.

Cook

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

39.

Somers

Zhu

Jacob

Bloom

(2013). The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation. MDRC working paper in research methodology. New York, NY.

40.

St. Clair

Cook

T. D.

Hallberg

. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35(3), 1–17.

41.

St. Clair

Hallberg

Cook

T. D

. (2016). The validity and precision of the comparative interrupted time series design: Three within-study comparisons. Journal of Educational and Behavioral Statistics, 41(3), 269–299.

42.

Steiner

P.M.

Cook

T.D.

Shadish

W.R.

(2011). On the importance reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics, 36(2), 213-236.

43.

Stuart

E. A.

(2007). Estimating causal effects using school-level data sets. Educational Researcher, 36(4): 187–198.

44.

Stuart

E. A.

Rubin

D. B.

(2008). Matching with multiple control groups and adjusting for group differences. Journal of Educational and Behavioral Statistics, 33(3): 279–306.

45.

U.S. Department of Education. (2013). Investing in Innovation Fund (I3) awards [Webpage]. Retrieved from http://www2.ed.gov/programs/innovation/awards.html.

46.

What Works Clearinghouse. (2018). Procedures and standards handbook (Version 4.0). Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf

47.

Wong

Cook

T. D.

Steiner

P. M.

(2015). Adding design elements to improve time series designs: No Child Left Behind as an example of causal pattern matching. Journal of Research on Educational Effectiveness, 8(2), 245–279.

48.

Wong

V. C.

Wing

Steiner

P. M.

Wong

Cook

T. D.

(2012). Research designs for program evaluation. In Velicer

Schinka

(Eds.), Handbook of psychology: research methods in psychology (2nd ed.). Hoboken, NJ: Wiley and Sons.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB