Contemporary Thinking About Causation in Evaluation: A Dialogue With Tom Cook and Michael Scriven

Abstract

Legitimate knowledge claims about causation have been a central concern among evaluators and applied researchers for several decades and often have been the subject of heated debates. In recent years these debates have resurfaced with a renewed intensity, due in part to the priority currently being given to randomized experiments by many funders of evaluation studies, such as the Institute for Educational Sciences. In this dialogue, which took place at Western Michigan University in October 2008, two of the field’s leading theorists and methodologists, Thomas D. Cook and Michael Scriven, described their current thinking and views about causation and causal inference in evaluation. They also discussed recent methodological developments for cause-probing investigations that sometimes produce results comparable to those produced by randomized experiments. Both Cook and Scriven prepared clarifying postscripts after reading the edited transcript.

Keywords

causal inference causation randomized controlled trials methodology

Get full access to this article

View all access options for this article.

References

Cook, T.D. ( 1991). Clarifying the warrant for generalized causal inferences in quasi-experimentation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century (pp. 115-144). Chicago: National Society for the Study of Education .

Cook, T.D. , Habib, F. , Phillips, M. , Settersten, R.A. , Shagle, S.C. , & Degirmencioglu, S.M. (1999). Comer’s school development program in Prince George’s County: A theory-based evaluation . American Educational Research Journal, 36, 543-597.

Cook, T.D. , & Hirschfield, P.J. (2008). Comer’s School Development Program in Chicago: Effects on involvement with the juvenile justice system from the late elementary through the high school years. American Educational Research Journal, 45, 38-67.

Cook, T.D. , Hunt, H.D. , & Murphy, R.F. ( 2000). Comer’s School Development Program in Chicago: A theory-based evaluation. American Educational Research Journal , 37, 535-597.

Cook, T.D. , Shadish, W.R. , & Wong, V.C. ( 2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724-750.

Cook, T.D. , & Wong, V.C. (in press). Empirical tests of the validity of the regression discontinuity design. Annales d’Economie et de Statistique.

Coryn, C.L.S. , SchrÖter, D.C. , & Hanssen, C.E. ( 2009). Adding a time-series design element to the Success Case Method to improve methodological rigor: An application for non-profit program evaluation. American Journal of Evaluation, 30(1), 80-92.

Donaldson, S. I. , Christie, C. A. , & Mark, M. M. ( 2009). (Eds.). What counts are credible evidence in applied research and evaluation practice? Thousand Oaks, CA: SAGE.

Fleming, J.E. , Cook, T.D. , & Stone, C.A. ( 2002). Interactive influences of perceived social contexts on the reading achievement of urban middle schoolers with learning disabilities . Learning Disabilities Research & Practice, 17, 47-64.

10.

LaLonde, R. ( 1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76, 604-620.

11.

Scriven, M. ( 1993). Defects of the necessary condition analysis of causation . In E. Sosa & M. Tooley (Eds.), Causation (pp. 56-59). Oxford, NY: Oxford University Press.

12.

Scriven, M. ( 1976). Maximizing the power of causal investigations: The modus operandi method. In G. V. Glass (Ed.), Evaluation studies review annual (Vol. 1, pp. 108-118). Beverly Hills, CA: SAGE.

13.

Scriven, M. ( 2008). A summative evaluation of RCT methodology: & an alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5, 11-24.

14.

Shadish, W.R. , Cook, T.D. , & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

15.

Wong, M. , Cook, T.D. , & Steiner, P.M. ( 2009). An evaluation of No Child Left Behind using short interrupted time-series analyses with non-equivalent comparison groups. Unpublished manuscript, Northwestern University at Evanston, Illinois.