How Can Theory-Based Evaluation Make Greater Headway?

Abstract

The idea of theory-based evaluation (TBE) is plausible and cogent, and it promises to bring greater explanatory power to evaluation. However, problems beset its use, including inadequate theories about pathways to desired outcomes in many program areas, confusion between theories of implementation and theories of programmatic action, difficulties in eliciting or constructing usable theories, measurement error, complexities in analysis, and others. This article explores the problems, describes the nature of potential benefits, and suggests that the benefits are significant enough to warrant continued effort to overcome the obstacles and advance the feasibility of TBE.

Get full access to this article

View all access options for this article.

References

Argyris, Chris , and D.A. Schön. 1978. Organizational learning: A theory of action perspective Reading, MA: Addison-Wesley.

—. 1996. Organizational learning II: Theory, method, and practice Reading, MA: Addison-Wesley.

Bickman, Leonard, ed. 1987. Using program theory in evaluation New Directionsfor Program Evaluation 33.

—, ed. 1990. Advances in program theory New Directions for Program Evaluation 47.

Campbell, Donald T. 1966. Pattern matching as an essential in distal knowing In The psychology of Egon Brunswik, edited by K. R. Hammond, 81-106. New York: Holt, Rinehart & Winston.

Campbell, Donald T. , and J.C. Stanley. 1963. Experimental and quasi-experimental designs for research Chicago: Rand McNally.

Chen, Huey-tsyh. 1990. Theory-driven evaluations Newbury Park, CA: Sage.

Chen, Huey-tsyh, and Peter H. Rossi. 1983. Evaluating with sense: The theory-driven approach Evaluation Review 7(3):283-302

. —, eds. 1992. Using theory to improve program and policy evaluations. Westport, CT: Greenwood .

10.

Conrad, K.J. , and J.R. Buelow. 1990. Developing and testing program classification and function theories Advances in Program Theory: New Directions for Program Evaluation, 47:73-91

11.

. Cronbach, Lee J. , S.R. Ambron , S.M. Dornbusch , R.D. Hess , R.C. Hornik , D.C. Phillips , D.F. Walker , and S.S. Weiner. 1980. Toward reform of program evaluation San Francisco: Jossey-Bass.

12.

Earp, Jo Anne , and S.T. Ennett. 1991. Conceptual models for health education research and practice Health Education Research 6 (2):163-71

13.

. Goodman, Robert M. , and A. Wandersman. 1994. FORECAST: A formative approach to evaluating community coalitions and community-based initiatives Journal of Community Psychology 6-25

14.

. Holder, H.D. , R.F. Saltz , A.J. Treno , J.W. Grube , and R.B. Voas. 1997. Evaluation design for a community prevention trial Evaluation Review 21(2):140-66

15.

. Judd, Charles M. , and D.A. Kenny. 1981. Process analysis: Estimating mediation in treatment evaluations Evaluation Review 5 (5):602-19

16.

. Keyfitz, Nathan. 1977. Introduction to the mathematics of population (with revisions Reading, MA: Addison-Wesley.

17.

Knapp, Michael. 1995. How shall we study comprehensive, collaborative services for children and families? Educational Researcher 24 (4):5-16.

18.

Lipsey, Mark W. , and J.A. Pollard. 1989. Driving toward theory in program evaluation: More models to choose from Evaluation and Program Planning 12:317-28

19.

. March, James G. 1973. Model bias in social action Review of Educational Research 42:413-29

20.

. March, James G. , and J.P. Olsen. 1976. Ambiguity and choice in organizations Bergen, Norway: Universitetsforlaget.

21.

Mark, Melvin. 1990. From program theory to tests of program theory Advances in Program Theory: New Directions for Program Evaluation 47:37-51

22.

. Marquart, Jules M. 1990. A pattern-matching approach to link program theory and evaluation data Advances in Program Theory: New Directionsfor Program Evaluation 47:93-107

23.

. McGraw, S.A. , D.E. Sellers , E.J. Stone , J. Bebchuk , E.W. Edmundson , C.C. Johnson , K.J. Bachman , and R.V. Luepker. 1996. Using process data to explain outcomes: An illustration from the Child and Adolescent Trial for Cardiovascular Health (CATCH Evaluation Review 20 (3):291-312

24.

. Murray, S.L. , and N.L. Smith. 1979. Causal research on teacher training In Educational environments and effects: Evaluation, policy, and productivity, edited by H. J. Walberg. Berkeley, CA: McCutchan.

25.

Olds, David. 1988. Common design and methodological problems encountered in evaluating family support services In Evaluating family programs, edited by H. B. Weiss and F. H. Jacobs, 239-65. Hawthorne, NY: Aldine.

26.

Patterson, F.R. 1986. Performance models for antisocial boys American Psychologist 4 (4):432-44

27.

. Patton, Michael Q. 1989. A context and boundaries for a theory-driven approach to validity Evaluation and Program Planning 12:375-77

28.

. Quane, James. 1994. Back from the future: Can evaluation survive dissension in the ranks?

Paper presented at the annual meeting of the American Evaluation Association, Boston, MA, November 2-5

29.

Riggin, L.J.C. 1990. Linking program theory and social science theory Advances in Program Theory: New Directions for Program Evaluation , 47:109-20

30.

. Sandler, I.N. , S.G. West , L. Baca , K.D. Reynolds , J.C. Gersten , F. Rogosch , L. Virdin , J. Beals , K.D. Reynolds , C. Kallgren , J.Y. Tein , G. Kriege , E. Cole , and R. Ramirez. 1992. Linking empirically based theory and evaluation: The Family Bereavement Program American Journal of Community Psychology 20 (4):491-521

31.

. Sechrest, Lee and Aurelio Figueredo. 1993. Program evaluation In Annual review of psychology. Vol. 44, edited by L. W. Porter and M. R. Rosenzweig, 645-74. Palo Alto, CA : Annual Reviews, Inc.

32.

Scheirer, Mary Ann. 1987. Program theory and implementation theory: Implications for evaluators Using Program Theory in Evaluation: New Directions for Program Evaluation 33:59-76

33.

. Schön, Donald A. , W.D. Drake , and R.I. Miller. 1984. Social experimentation as reflection-inaction : Community-level nutrition intervention revisited Knowledge: Creation, Diffusion, Utilization 6 (1).5-36

34.

. Shadish, W.R., Jr. 1987. Program micro- and macrotheories: A guide for social change Using Program Theory in Evaluation: New Directionsfor Program Evaluation, 33:93-109

35.

. Smith, Nick L. 1990

Using path analysis to develop and evaluate program theory

Advances in Program Theory. New Directions for Program Evaluation 47:53-57

36.

. Trochim, W.M.K. 1985. Pattern matching, validity, and conceptualization in program evaluation Evaluation Review 9:575-604

37.

. Van Ryzin, G.G. 1996. The impact of resident management on residents' satisfaction with public housing: A process analysis of quasi-experimental data Evaluation Review 20 (4):485-506.

38.

Weick, Karl E. 1979. The social psychology of organizing 2d ed. Reading, MA: Addison-Wesley.

39.

Weiss, Carol H. 1972. Evaluation research: Methods for assessing program effectiveness Englewood Cliffs, NJ: Prentice Hall.

40.

—. 1973. Where politics and evaluation research meet Evaluation 1(3):37-45

41.

. —. 1995. Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families In New approaches to evaluating community initiatives: Concepts, methods, and contexts, edited by J. P. Connell, A. C. Kubisch, L. B. Schorr, and C. H. Weiss, 65-92. Washington, DC : Aspen Institute.

42.

Wholey, Joseph S. 1994. Assessing the feasibility and likely usefulness of evaluation In Handbook of practical program evaluation , edited by J. S. Wholey, H. P. Hatry, and K. E. Newcomer, 15-39. San Francisco: Jossey-Bass.