Matched Comparison Group Design Standards in Systematic Reviews of Early Childhood Interventions

Abstract

Background:

Systematic reviews assess the quality of research on program effectiveness to help decision makers faced with many intervention options. Study quality standards specify criteria that studies must meet, including accounting for baseline differences between intervention and comparison groups. We explore two issues related to systematic review standards: covariate choice and choice of estimation method.

Objective:

To help systematic reviews develop/refine quality standards and support researchers in using nonexperimental designs to estimate program effects, we address two questions: (1) How well do variables that systematic reviews typically require studies to account for explain variation in key child and family outcomes? (2) What methods should studies use to account for preexisting differences between intervention and comparison groups?

Methods:

We examined correlations between baseline characteristics and key outcomes using Early Childhood Longitudinal Study—Birth Cohort data to address Question 1. For Question 2, we used simulations to compare two methods—matching and regression adjustment—to account for preexisting differences between intervention and comparison groups.

Results:

A broad range of potential baseline variables explained relatively little of the variation in child and family outcomes. This suggests the potential for bias even after accounting for these variables, highlighting the need for systematic reviews to provide appropriate cautions about interpreting the results of moderately rated, nonexperimental studies. Our simulations showed that regression adjustment can yield unbiased estimates if all relevant covariates are used, even when the model is misspecified, and preexisting differences between the intervention and the comparison groups exist.

Keywords

systematic review nonexperimental research methods early childhood home visiting

Get full access to this article

View all access options for this article.

References

Angrist

J. D.

(1998). Estimating the labor market impact of voluntary military service using social security data on military applicants. Econometrica, 66, 249–288.

Bang

Robins

J. M.

(2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61, 962–973.

Bifulco

(2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31, 729–751.

Bloom

H. J.

Bos

Lee

(1999). Using cluster random assignment to measure program impacts: Statistical implications for evaluation of education programs. Evaluation Review, 23, 445–469.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.

Cook

T. D.

Steiner

P. M.

(2010). Case matching and the reduction of selection bias in quasi-experiments: The relative importance of pretest measures of outcome, of unreliable measurement, and of mode of data analysis. Psychological Methods, 15, 56.

Clearinghouse of Labor Evaluation and Research. (2017). Study profile: The impact of earnings disregards on the behavior of low-income families. Retrieved January 31, 2017, from http://clear.dol.gov/study/impact-earnings-disregards-behavior-low%E2%80%90income-families-matsudaira-blank-2014

Deke

Chiang

(2017). The WWC attrition standard: Sensitivity to assumptions and opportunities for refining and adapting to new contexts. Evaluation Review, 41, 130–154.

Demirtas

Amatya

Doganay

(2014). BinNor: An R package for concurrent generation of binary and normal data. Communications in Statistics-Simulation and Computation, 43, 569–579.

10.

Demirtas

Doganay

(2012). Simultaneous generation of binary and normal data with specified marginal and association structures. Journal of Biopharmaceutical Statistics, 22, 223–236.

11.

Fortson

Gleason

Kopa

Verbitsky-Savitz

(2015). Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased. Economics of Education Review, 44, 100–113.

12.

Foster

E. M.

(2003). Propensity score matching: An illustrative analysis of dose response. Medical Care, 41, 1183–1192.

13.

Gelman

Hill

(2007). Data analysis using regression and multilevel/hierarchical models (Vol. 1). New York, NY: Cambridge University Press.

14.

Hallberg

Steiner

P. M.

Cook

T. D.

(2011). The role of pretest and proxy pretest measures of the outcome in removing selection bias in observational studies. Paper presented at the Society for Research on Educational Effectiveness Annual Conference, Washington, DC. Retrieved May 16, 2014, from https://www.sree.org/conferences/2011/program/downloads/slides/167-2.pdf

15.

Harder

V. S.

Stuart

E. A.

Anthony

J. C.

(2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15, 234.

16.

Hill

C. J.

Gormley

W. T.

Adelstein

(2015). Do the short-term effects of a high-quality preschool program persist? Early Childhood Research Quarterly, 32, 60–79.

17.

Imbens

Wooldridge

(2007). What’s new in econometrics? NBER Summer Institute. Retrieved June 20, 2016, from http://www.nber.org/minicourse3.html

18.

Kang

J. D.

Schafer

J. L.

(2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22, 523–539.

19.

Kazdin

A. E.

(2011). Single-case research designs: Methods for clinical and applied settings. New York, NY: Oxford University Press.

20.

Luscombe

M. D.

Owens

B. D.

Burke

(2011). Weight estimation in paediatrics: A comparison of the APLS formula and the formula “weight = 3 (age) + 7.” Emergency Medicine Journal, 28, 590–593.

21.

Peikes

D. N.

Moreno

Orzol

S. M.

(2008). Propensity score matching. The American Statistician, 62, 222–231.

22.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688.

23.

Rubin

D. B.

(1977). Assignment to treatment group on the basis of a covariate. Journal of Educational and Behavioral Statistics, 2, 1–26.

24.

Rubin

D. B.

(1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 66, 34–58.

25.

Rubin

D. B.

(1980). Bias reduction using Mahalanobis-metric matching. Biometrics, 36, 293–298.

26.

Rubin

D. B.

(1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine, 127, 757–763.

27.

Schafer

J. L.

Kang

(2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13, 279–313.

28.

Schochet

P. Z.

(2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.

29.

Seftor

(2016). Raising the bar. Evaluation Review. doi:10.1177/0193841X16665023

30.

Steiner

P. M.

Cook

T. D.

Shadish

W. R.

Clark

M. H.

(2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–267.

31.

Stuart

E. A.

(2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.

32.

Westbrook

T. P.

Avellar

S. A.

Seftor

(2016). Reviewing the reviews: Examining similarities and differences between federally funded evidence reviews. Evaluation Review. doi:10.1177/0193841X16666463

33.

What Works Clearinghouse. (2014a). Procedures and standards handbook (Version 3.0).

34.

What Works Clearinghouse. (2014b). Review protocol for early childhood education interventions (Version 3.0).

35.

Wooldridge

J. M.

(2015). Introductory econometrics: A modern approach. Boston, MA: Cengage Learning.

36.

Zhao

(2008). Sensitivity of propensity score methods to the specifications. Economics Letters, 98, 309–319.