Sage Journals: Discover world-class research

Abstract

The primary technique that many researchers use to analyze data from randomized control trials (RCTs)—detecting the average treatment effect (ATE)—imposes assumptions upon the data that often are not correct. Both theory and past research suggest that treatments may have significant impacts on subgroups even when showing no overall effect. Giving primacy to ATEs thus may lead to the rejection of treatments that in fact are helpful to some people. Using simulations, I examine the power of ATEs to detect treatment impacts when treatment impacts vary. Models that allow for varying impacts accurately measure the treatment effect in the simulation and are robust in a variety of circumstances. However, a focus on ATEs often fails to find the known treatment effects.

Keywords

experimental research program evaluation educational policy regression analyses

Get full access to this article

View all access options for this article.

References

Food and Drug Administration. (2014). FDA approves Zykadia for late-stage lung cancer. FDA News Release. Retrieved from http://www.fda.gov/newsevents/newsroom/pressannouncements/ucm395299.htm

Glazerman

Dolfin

Bleeker

Johnson

Isenberg

Lugo-Gil

Grider

Britton

(2008). Impacts of comprehensive teacher induction: Results from the first year of a randomized control study. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Miladinovic

Kumar

Hozo

Djulbegovic

(2011). Instrumental variable meta-analysis of individual patient data: Application to adjust for treatment non-compliance. BMC Medical Research Methodology. Retrieved from http://www.biomedcentral.com/content/pdf/1471-2288-11-55.pdf

Office of Management and Budget. (2004). What constitutes strong evidence of a program’s effectiveness? Retrieved from http://www.whitehouse.gov/sites/default/files/omb/part/2004_program_eval.pdf

Pool

(2010). Field evaluation in the intelligence and counterintelligence context: Workshop summary. Washington, DC: National Research Council.

Shadish

Cook

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. (2009). The impacts of regular Upward Bound on postsecondary outcomes seven to nine years after scheduled high school graduation. Washington, DC: Author.

Viadero

(2009, 4 1). ‘No Effects’ studies raising eyebrows. Education Week. Retrieved from http://www.edweek.org/ew/articles/2009/04/01/27rct_ep.h28.html

Reconsidering Findings of “No Effects” in Randomized Control Trials

Abstract

Keywords

Get full access to this article

References