Assessing the Accuracy of Generalized Inferences From Comparison Group Studies Using a Within-Study Comparison Approach

Abstract

Background:

Various studies have examined bias in impact estimates from comparison group studies (CGSs) of job training programs, and in education, where results are benchmarked against experimental results. Such within-study comparison (WSC) approaches investigate levels of bias in CGS-based impact estimates, as well as the success of various design and analytic strategies for reducing bias.

Objectives:

This article reviews past literature and summarizes conditions under which CGSs replicate experimental benchmark results. It extends the framework to, and develops the methodology for, situations where results from CGSs are generalized to untreated inference populations.

Research design:

Past research is summarized; methods are developed to examine bias in program impact estimates based on cross-site comparisons in a multisite trial that are evaluated against site-specific experimental benchmarks.

Subjects:

Students in Grades K–3 in 79 schools in Tennessee; students in Grades 4–8 in 82 schools in Alabama.

Measures:

Grades K–3 Stanford Achievement Test (SAT) in reading and math scores; Grades 4–8 SAT10 reading scores.

Results:

Past studies show that bias in CGS-based estimates can be limited through strong design, with local matching, and appropriate analysis involving pretest covariates and variables that represent selection processes. Extension of the methodology to investigate accuracy of generalized estimates from CGSs shows bias from confounders and effect moderators.

Conclusion:

CGS results, when extrapolated to untreated inference populations, may be biased due to variation in outcomes and impact. Accounting for effects of confounders or moderators may reduce bias.