Combining a Local Comparison Group,a Pretest Measure,and Rich Covariates: How Well Do They Collectively Reduce Bias in Nonequivalent Comparison Group Designs?

Abstract

This study examined bias reduction in the eight nonequivalent comparison group designs (NECGDs) that result from combining (a) choice of a local versus non-local comparison group, and analytic use or not of (b) a pretest measure of the study outcome and (c) a rich set of other covariates. Bias was estimated as the difference in causal estimate between each NECGD and a carefully appraised randomized experiment with the same intervention, outcome, and estimand. Results indicated that bias generally declined with the number of design elements in an NECGD, that combining all three sufficed to eliminate bias but was not necessary for it, and that this pattern of results was largely replicated across five different replication factors.

Keywords

within-study comparison design features selection bias reduction

Get full access to this article

View all access options for this article.

References

Aiken

L. S.

West

S. G.

Schwalm

D. E.

Carroll

J. L.

Hsiung

(1998). Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review, 22(2), 207–244.

Angrist

J. D.

Pischke

J. S.

(2008). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Chaplin

D. D.

Cook

T. D.

Zurovac

Coopersmith

J. S.

Finucane

M. M.

Vollmer

L. N.

Morris

R. E.

(2018). The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study comparisons. Journal of Policy Analysis and Management, 37(2), 403–429.

Cook

T. D.

(2008). “Waiting for life to arrive”: A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142(2), 636–654.

Bifulco

(2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31(3), 729–751.

Bloom

H. S.

Michalopoulos

Hill

C. J.

(2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In Bloom

H. S.

(Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 173–235). Russell Sage Foundation.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Houghton Mifflin.

Cook

T. D.

Campbell

D. T.

(1979). Quasi-experimentation: Design and analysis for field settings. Rand McNally.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiment and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.

10.

Cook

T. D.

Zhu

Klein

Starkey

Thomas

(2020). How much bias is there if quasi-experimental designs combine local comparison groups, pretest outcomes, and other covariates? A within study comparison of preschool effects. Psychological Methods, 25(6), 726–746.

11.

Coopersmith

Cook

T. D.

Chaplin

D. D.

Zuravec

Vollmer

L. N.

Morris

(2022). Internal and external validity of the comparative interrupted time-series design: A meta-analysis. Journal of Policy Analysis and Management, 41(1), 252–277.

12.

Cronbach

L. J.

(1982). Designing evaluations of educational and social programs. Jossey-Bass.

13.

Diaz

J. J.

Handa

(2006). An assessment of propensity score matching as a nonexperimental impact estimator evidence from Mexico’s PROGRESA program. Journal of Human Resources, 41(2), 319–345.

14.

Ding

Miratrix

L. W.

(2015). To adjust or not to adjust? Sensitivity analysis of M-bias and butterfly-bias. Journal of Causal Inference, 3(1), 41–57.

15.

Eckles

Bakshy

(2021). Bias and high-dimensional adjustment in observational studies of peer effects. Journal of the American Statistical Association, 116(534), 507–517.

16.

Elwert

Winship

(2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53.

17.

Ferraro

P. J.

Miranda

J. J.

(2014). The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark. Journal of Economic Behavior & Organization, 107, 344–365.

18.

Glazerman

Levy

D. M.

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. The ANNALS of the American Academy of Political and Social Science, 589, 63–93.

19.

Hallberg

(2013). Identifying conditions that support causal inference in observational studies in education: Empirical evidence from within-study comparisons [Unpublished doctoral dissertation]. Northwestern University.

20.

Hallberg

Cook

T. D.

Steiner

P. M.

Clark

M. H.

(2018). Pretest measures of the study outcome and the elimination of selection bias: Evidence from three within study comparisons, Prevention Science, 19, 274–283.

21.

Heckman

J. J.

Ichimura

Todd

P. E.

(1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64(4), 605–654.

22.

Heckman

J. J.

Ichimura

Smith

Todd

P. E.

(1998). Characterizing selection bias using experimental data. Econometrica, 66(5), 1017–1098.

23.

Heckman

J. J.

Smith

J. A.

(1999). The pre-programme earnings dip and the determinants of participation in a social programme. Implications for simple programme evaluation strategies. Economic Journal, 109(457), 313–348.

24.

Hong

Raudenbush

S. W.

(2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis, 27(3), 205–224.

25.

Imbens

G. W.

Rubin

D. B.

(2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.

26.

Keele

(2015). The statistics of causal inference: A view from political methodology. Political Analysis, 23(3), 313–335.

27.

LaLonde

(1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76, 604–620.

28.

Lechner

Wunsch

(2013). Sensitivity of matching-based program evaluations to the availability of control variables. Labour Economics, 21(C): 111–121.

29.

Little

R. J. A.

Rubin

D. B.

(2002). Statistical analysis with missing data (2nd ed.). Wiley-Interscience.

30.

Pearl

(2010). Causal inference. JMLR Workshop and Conference Proceedings, 6, 39–58.

31.

Pearl

Glymour

Jewell

N. P.

(2016). Causal inference in statistics: A primer. John Wiley & Sons.

32.

Raghunathan

T. E.

Lepkowski

J. M.

Van Hoewyk

Solenberger

(2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85–95.

33.

Rogers

J. L.

Howard

K. I.

Vessey

J. T.

(1993). Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin, 113(3), 553–556.

34.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

35.

Rubin

D. B.

(2006). Causal inference through potential outcomes and principal stratification: Application to studies with “censoring” due to death. Statistical Science, 21(3), 299–309.

36.

Rubin

D. B.

(2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine, 26(1), 20–36.

37.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.

38.

Shadish

W. R.

Clark

M. H.

Steiner

P. M.

(2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103(484), 1334–1344.

39.

Song

Wayne

A. J.

Garet

M. S.

Brown

Rickles

(2021). Impact of providing teachers and principals with performance feedback on their practice and student achievement: Evidence from a large-scale randomized experiment. Journal of Research on Educational Effectiveness, 14(2), 353–378.

40.

Stecher

B. M.

Holtzman

D. J.

Garet

M. S.

Hamilton

L. S.

Engberg

Steiner

E. D.

Robyn

Baird

M. D.

Gutierrez

I. A.

Peet

E. D.

Brodziak de los Reyes

Fronberg

Weinberger

Hunter

G. P.

Chambers

(2018). Improving teaching effectiveness: Final report: The Intensive Partnerships for Effective Teaching through 2015–2016. RAND Corporation.

41.

Steiner

P. M.

Cook

T. D.

Clark

M. H.

(2015). Bias reduction in quasi-experiments with little selection theory but many covariates. Journal of Research on Educational Effectiveness, 8(4), 552–576.

42.

Steiner

P. M.

Cook

T. D.

Shadish

W. R.

Clark

M. H.

(2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250–267.

43.

Steiner

P. M.

Wong

V.C.

(2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–247.

44.

Stuart

E. A.

Rubin

D. B.

(2008). Matching with multiple control groups with adjustment for group differences. Journal of Educational and Behavioral Statistics, 33(3), 279–306.

45.

Wellek

(2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). CRC Press.

46.

Wilde

E. T.

Hollister

(2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26(3), 455–477.

47.

Wong

C. V.

Steiner

P. M.

(2018). Designs of empirical evaluations of non-experimental methods in field settings. Evaluation Review, 42(2), 176–213.

48.

Wong

Cook

T. D.

Steiner

P. M.

(2015). Adding design elements to short interrupted time series when evaluating national programs: No Child Left Behind as an example of pattern-matching. Journal of Research on Educational Effectiveness, 8(2), 245–279.