This paper discusses current issues in theory-driven evaluation from the perspective of the evaluation practice literature. Applications of theory-driven approaches are not easily found, in part, because there are few procedures for conducting them. I offer confirmatory program evaluation as one way to use theory, in combination with quantitative analytical techniques, to assess the effects of social and educational programs. In contrast to many other approaches, theory-driven evaluation generally emphasizes the explication and testing of a priori program theories in determining effectiveness. Confirmatory program evaluation is an impact assessment that examines the pattern of empirical findings against several causal criteria, including temporality, size, gradient (dosage/response), specificity, consistency, and coherence of the program-outcome relationship. A special emphasis is given to identifying causal mechanisms or active ingredients of programs that yield effects. An illustration of confirmatory program evaluation is provided for a child development intervention called the Child Parent Center Program. The limitations of this method are discussed, as well as the conditions under which it is most useful.
Get full access to this article
View all access options for this article.
References
1.
Anderson, S., Auquier, A., Hauck, W. W., Oakes, D., Vandaele, W., & Weisberg, H. I. (1980). Statistical methods for comparative studies: Techniques for bias reduction. New York: Wiley.
2.
Berrueta-Clement, J. R., Schweinhart, L. J., Barnett, W. S., Epstein, A. S., & Weikart, D. P. (1984). Changed lives: The effects of the Perry Preschool Program on youths through age 19. Ypsilanti, MI: High/Scope.
3.
Bickman, L. (1987). The functions of program theory. In L. Bickman, (Ed.), Using program theory in evaluation: No. 33 (pp. 5-18). San Francisco: Jossey-Bass.
4.
Bickman, L. (1996). A continuum of care: More is not always better. American Psychologist, 51, 689-701.
5.
Bross, I. D. J. (1967). Pertinency of an extraneous variable. Journal of Chronic Disease, 20, 487-495.
6.
Bronfenbrenner, U. (1975). Is early intervention effective? In M. Guttentag & E. Struening (Eds.), Handbook of evaluation research (Vol. 2, pp. 519-603). Beverly Hills: Sage.
7.
Campbell, D. T. (1994). Retrospective and prospective on program impact assessment. Evaluation Practice, 15, 291-298.
8.
Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
9.
Chen, H., & Rossi, P. H. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7, 283-302.
10.
Chen H., & Rossi, P. H. (Eds.). (1992). Using theory to improve program and policy evaluations, New York: Greenwood Press.
11.
Chen, H. T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
12.
Chicago Public Schools. (1987). ECIA Chapter I Evaluation of the Child Parent Centers. Department of Research and Evaluation: Author.
13.
Consortium for Longitudinal Studies (1983). As the twig is bent...Lasting effects of preschool programs. Hillsdale, NJ: Erlbaum.
14.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
15.
Cook, T. D., Anson, A. R., & Walchli, S. B. (1993). From causal description to causal explanation: Improving three already good evaluations of adolescent health programs. In S. G. Millstein, A. C. Peterson, & E. O. Nightengale (Eds.), Promoting the health of adolescents: New directions for the twenty-first century (pp. 339-374). New York: Oxford University Press.
16.
Cook, T. D., & Shadish, W. R. (1994). Social experiments: Some developments over the past fifteen years. Annual Review of Psychology, 45, 545-580.
17.
Cronbach, L. J. (1982). Designing evaluations for educational and social programs. San Francisco: Jossey-Bass.
18.
Haskins, R. (1989). Beyond metaphor: The efficacy of early childhood education. American Psychologist, 44, 274-282.
19.
Lipsey, M. W. (1993). Theory as method: Small theories as treatments. In L. Sechrest & A. Scott (Eds.), Understanding causes and generalizing about them. New directions for program evaluation (No. 57, pp. 5-38). San Francisco: Jossey-Bass.
20.
Lipsey, M. W., Crosse, S., Dunkle, J., Pollard, J., & Stobart, G. (1985). Evaluation: The state of the art and the sorry state of science. In D. S. Cordray (Ed.), Utilizing prior research in evaluation planning. New Directions for Program Evaluation (No. 27). San Francisco: Jossey-Bass.
21.
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment. American Psychologist, 48, 1181-1209.
22.
Mark, M. M., Hoffman, D. A., & Reichardt, C. S. (1992). Testing theories in theory-driven evaluations: (Tests of) moderation in all things. In H. T. Chen & P. H. Rossi (Eds.), Using theory to improve program and policy evaluations (pp. 71-84). New York: Greenwood Press.
23.
McKey, R. H., Condelli, L., Ganson, H., Barrett, B. J., McConkey, C., & Plantz, M. C. (1985). The impact of Head Start on children, families, and communities (DHHS Publication No. OHDS 85-31193). Washington, DC: U. S. Government Printing Office.
24.
Naisbitt, N. (1968). Child-Parent Education Centers, ESEA Title I, Activity I. Unpublished report, Chicago, IL.
25.
Overman, E. S. (Ed.). (1988). Methodology and epistemology for social science: Selected papers of Donald T. Campbell. Chicago: University of Chicago Press.
26.
Reynolds, A. J. (1992). Mediated effects of preschool intervention. Early Education and Development, 3, 139-164.
27.
Reynolds, A. J. (1994). Effects of a preschool plus follow-on intervention for children at risk. Developmental Psychology, 30, 787-804.
28.
Reynolds, A. J. (1995). One year of preschool intervention or two: Does it matter?Early Childhood Research Quarterly, 10, 1-31.
29.
Reynolds, A. J. (1998). The Chicago Child-Parent Centers: A longitudinal study of extended early childhood intervention, In J. Crane (Ed.), Social programs that work. New York: Russell Sage Foundation.
30.
Reynolds, A. J., Mavrogenes, N. A., Bezruczko, N., & Hagemann, M. (1996). Cognitive and family-support mediators of preschool effectiveness: A confirmatory analysis. Child Development, 67, 1119-1140.
31.
Reynolds, A. J., & Temple, J. A. (1995). Quasi-experimental estimates of the effects of a preschool intervention: Psychometric and econometric comparisons. Evaluation Review, 19.347-373.
32.
Reynolds, A. J., & Walberg, H. J. (1990). Program theory in evaluation. In H. Walberg & G. Haertel (Eds.), International encyclopedia of educational evaluation. NY: Pergamon.
33.
Reynolds, A. J., & Walberg, H. J. (1994). Theory-based evaluation. In T. Husen & T. N. Postlethwaite (Eds.), International Encyclopedia of Education (pp. 6378-6384). New York: Pergamon.
34.
Rosenbaum, P. (1984). From association to causation in observational studies: The role of tests of strongly ignorable treatment assignment. Journal of the American Statistical Association, 79, 41-48.
35.
Rosenbaum, P. (1995). Observational studies. New York: Springer-Verlag.
36.
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies for causal effects. Journal of Educational Psychology, 66, 688-701.
37.
Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Beverly Hills: Sage.
38.
Rubin, D. B. (1986). Which ifs have causal answers?Journal of the American Statistical Association, 81, 961-962.
39.
Schweinhart, L. J., Barnes, H. V., & Weikart, D. P. (1993). Significant benefits: The High/Scope Perry Preschool study through age 27. Ypsilanti, Ml: High/Scope.
40.
Schweinhart, L. J., & Weikart, D. P. (1988). The High/Scope Perry Preschool Program. In R. H. Price, E. L. Cowen, R. P. Lotion, & J. Ramos-McKay (Eds.), 14 ounces of prevention: A casebook for practitioners (pp. 53-65). Washington, DC: American Psychological Association.
41.
Striven, M. (1994). The tine line between evaluation and explanation. Evaluation Practice, 15.75-77.
42.
Seitz, V. (1990). Intervention programs for impoverished children: A comparison of educational and family support models. Annals of Child Development, 7, 73-103.
43.
Suchman, E. A. (1967). Evaluation research: Principles and practice in public service and social action programs. New York: Russell Sage Foundation.
44.
Susser, M. (1973). Causal thinking in the health sciences, concepts and strategies of epidemology. New York: Oxford University Press.
45.
Trochim, W. M. K. (1985). Pattern matching, construct validity, and conceptualization in program evaluation. Evaluation Review, 9, 575-604.
46.
Trochim, W. M. K. (1989). Concept mapping: Soft science or hard art?Evaluation and Program Planning, 12, 87-110.
47.
U. S. Department of Health, Education, and Welfare (1964). Smoking and health: Report of the advisory committee to the surgeon-general of the Public Health Service. Washington, DC: Public Health Service Publication No. 1103).
48.
Walberg, H. J. (1986). Synthesis of research on teaching. In M. Wittrock (Ed.), Handbook of research on teaching (pp. 214-229). Washington, DC: American Educational Research Association.
49.
Wang, M. C., & Walberg, H. J. (1983). Evaluating educational programs: An integrative causal-modeling approach. Educational Evaluation and Policy Analysis, 5, 347-366.
50.
Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness. Inglewood Cliffs, NJ: Prentice-Hall.
51.
Wholey, J. S. (1979). Evaluation: Promise and performance. Washington, DC: Urban Institute.
52.
White, K. R. (1985). Efficacy of early intervention. Journal of Special Education, 19, 401-416.
53.
Worthen, B. R. (1996). Editor’s note: The origins of theory-based evaluation. Evaluation Practice, 17, 169-171.
