Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justification, persuasion, and policy diffusion, from the social science research literature, and organize them into a framework that has three levels: individual, interpersonal, and collective. We illustrate how these change processes can be linked together to form “pathways” or working hypotheses that link evaluation processes to outcomes that move us along the road toward the goal of social betterment. In addition, we join with Kirkhart (2000) in moving beyond use, to focus our thinking on evaluation influence. Influence, combined with the set of mechanisms and interim outcomes presented here, offers a better way for thinking about, communicating, and adding to the evidence base about the consequences of evaluation and the relationship of evaluation to social betterment.
Get full access to this article
View all access options for this article.
References
1.
Ajzen, I. (2001). Nature and operation of attitudes. In S. T. Fiske, D. L. Schacter, & C. Zahn-Wexler (Eds.), Annual Review of Psychology, 52, 27-58.
2.
Alkin, M. C. (Ed.). (1990). Debates on evaluation. Newbury Park, CA: Sage.
3.
Alkin, M. C. (2003). Introduction (to section on evaluation use). In T. Kelligan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 189-196). Dordrecht: Kluwer Academic Publishers.
4.
Alkin, M. C., Daillak, R., & White, P. (1979). Using evaluations: Does evaluation make a difference?Beverly Hills, CA: Sage.
5.
Berry, F. S., & Berry, W. D. (1990). State lottery adoptions as policy innovations: An event history analysis. American Political Science Review, 84, 395-415.
6.
Caplan, N. (1977). A minimal set of condition necessary for the utilization of social science knowledge in the policy formulation at the national level. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 183-197). Lexington, MA: Lexington Books.
7.
Caracelli, V. J. (2000). Evaluation use at the threshold of the twenty-first century. In V. J. Caracelli & H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation, No. 88 (pp. 99-111). San Francisco, CA: Jossey-Bass.
8.
Caracelli, V., & Preskill, H. (eds). The expanding scope of evaluation use. New Directions for Evaluation, No. 88, San Francisco: Jossey-Bass.
9.
Cook, F. L., et al. (1983). Media and agenda setting: Effects on the public, interest group leaders, policy makers, and policy. Public Opinion Quarterly, 47, 16-35.
10.
Cousins, J. B. (2003). Utilization effects of participatory evaluation. In T. Kelligan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245-266). Dordrecht: Kluwer Academic Publishers.
11.
Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56, 331-364.
12.
Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.
13.
Fuller, B., Kagan, S. L., Caspary, G. L., & Gauthier, C. A. (2002). Welfare reform and child care options for low-income families. The Future of Children, 12, 97-119.
14.
Glazer, N. (1994). How social problems are born. The Public Interest, 115, 31-44.
15.
Greenberg, D., Mandell, M., & Onstott, M. (2000). The dissemination and utilization of welfare-to-work experiments in state policymaking. Journal of Policy Analysis & Management, 19, 367-382.
16.
Henry, G. T. (2002). Transparency, stakeholder involvement, and explanation in contemporary evaluation: A review essay stimulated by Success in Early Intervention: The Chicago Child-Parent Centers by Arthur J. Reynolds. American Journal of Evaluation, 23, 235-244.
17.
Henry, G.T. (2000). Whynot use? In V.J. Caracelli & H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation, No. 88, (pp. 85-98). San Francisco, CA: Jossey-Bass.
18.
Henry, G. T., & Gordon, C. S. (2001). Tracking issue attention: Specifying the dynamics of the public agenda. Public Opinion Quarterly, 65, 157-177.
19.
Henry, G. T., & Gordon, C. S. (2003). Driving less for better air: Impacts of a public information campaign. Journal of Policy Analysis and Management, 22, 45-63.
20.
Henry, G. T., & Julnes, G. (1998). Values and realist evaluation. In G. Henry, G. Julnos, & M. M. Mark (Eds.), Realist evaluation. New Directions for Evaluation, No. 78 (pp. 53-72). San Francisco, CA: Jossey-Bass.
21.
Henry, G. T., & Mark, M. M. (2003). Toward an agenda for research on evaluation. In C. A. Christie (Ed.), The practice-theory relationship in evaluation. New Directions for Evaluation, No. 97 (pp. 69-80). San Francisco, CA: Jossey-Bass.
22.
Henry, G. T., & Rog, D. J. (1998). A realist theory and analysis of utilization. In G. Henry, G. Julnes, & M. M. Mark (Eds.), Realist evaluation. New Directions for Evaluation, No. 78 (pp. 89-102). San Francisco, CA: Jossey-Bass.
23.
Higgins, E. T., Rholes, W., & Jones, C. R. (1977). Category accessibility and impression formation. Journal of Experimental Social Psychology, 13, 141-154.
24.
Hofstetter, C. H., & Alkin, M. C. (2003). Evaluation use revisited. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 197-222). Dordrecht: Kluwer Academic Publishers.
25.
House, E. (1995). Principled evaluation: A critique of the AEA guiding principles. Guiding principles for evaluators. New Directions for Evaluation, No. 66 (pp. 27-34). San Francisco, CA: Jossey-Bass.
26.
King, J. A. (2002). Building the evaluation capacity of a school district. In D. W. Compton, M. Baizerman, & S. H. Stackdill (Eds.), The art, craft, and science of evaluation capacity building. New Directions for Evaluation, No. 93 (pp. 63-80). San Francisco, CA: Jossey-Bass.
27.
King, J., Thompson, R., & Pechman, E. (1981). Improving evaluation use in local school settings. Washington, DC: National Institute of Education.
28.
Kirkhart, K. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. Caracelli & H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation, No. 88 (pp. 5-24). San Francisco, CA: Jossey-Bass.
29.
Knorr, K. D. (1977). Policymakers’ use of social science knowledge: Symbolic or instrumental? In C. H. Weiss (Ed.), Using social research in public policy making (pp. 165-182). Lexington, MA: Lexington Books.
30.
Krosnick, J. A., & Brannon, L. A. (1993). The impact of the gulf war on the ingredients of presidential evaluations: Multidimensional effects of political involvement. American Political Science Review, 87, 963-975.
31.
Krosnick, J. A., & Kinder, D. R. (1990). Altering the foundations of support for the president through priming. American Political Science Review, 84, 497-512.
32.
Majone, G. (1989). Evidence, arguments, and persuasion in the policy process. New Haven, CT: Yale University Press.
33.
Mark, M. M. (2001). Evaluation’s future: Furor, futile, or fertile?American Journal of Evaluation, 22, 457-479.
34.
Mark, M. M. (2003). Toward a comprehensive view of the theory and practice of program and policy evaluation. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 183-204). Hillsdale, NJ: Lawrence Erlbaum.
35.
Mark, M. M., & Henry, G. T. (in press). The mechanisms and outcomes of evaluation influence.
36.
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass.
37.
McCombs, M. E., & Zhu, J. (1995). Capacity, diversity, and volatility of the public agenda: Trends from 1954 to 1994. Public Opinion Quarterly, 59, 495-525.
38.
Mintrom, M. (1997). Policy entrepreneurs and the diffusion of innovation. American Journal of Political Science, 41, 738-770.
39.
Monroe, A. (1998). Public opinion and public policy: 1980-1993. Public Opinion Quarterly, 62, 6-27.
40.
Moscovici, S., Lage, E., & Naffrechoux, M. (1969). Influence of a consistent minority on the responses of a majority to a color-perception task. Sociometry, 32, 365-380.
41.
Nelson, T., Clawson, R. A., & Oxley, Z. M. (1997). Media framing of a civil liberties conflict and its effect on tolerance. American Political Science Review, 91, 1002-1009.
42.
Neuman, R. W. (1990). The threshold of public attention. Public Opinion Quarterly, 54, 159-176.
43.
Page, B. I., & Shapiro, R. Y. (1992). The rational public. Chicago, IL: The University of Chicago Press.
44.
Patton, M. Q. (1988). The evaluator’s responsibility for utilization. Evaluation Practice, 9, 5-24.
45.
Patton, M. Q. (1997). Utilization-focused evaluation: The new-century text. Thousand Oaks, CA: Sage.
46.
Patton, M. Q., Grimes, P. S., Guthrie, K. M., Brennan, N. J., French, B. D., & Blyth, D. A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 141-164). Lexington, MA: Lexington Books.
47.
Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude. New York, NY: Springer-Verlag.
48.
Preskill, H., & Caracelli, V. J. (1997). Current and developing conceptions of use: Evaluation Use Topical Interest Group survey results. Evaluation Practice, 18, 209-225.
49.
Reynolds, A. J., Temple, J. A., Roberston, D. L., & Mann, E. A. (2001). Long-term effects of an early childhood intervention on educational achievement and juvenile arrest. Journal of the American Medical Association, 285, 2339-2346.
50.
Rog, D. J. (1985). A methodological analysis of evaluability assessment. Doctoral dissertation, Vanderbilt University, Nashville, TN.
51.
Russ-Eft, D., Atwood, R., & Egherman, T. (2002). Use and non-use of evaluation results: Case study of environmental influences in the private sector. American Journal of Evaluation, 23, 33-44.
52.
Sabatier, P. A., & Jenkins-Smith, H. C. (1993). Policy change and learning: An advocacy coalition approach. Boulder, CO: Westview Press.
53.
Sabatier, P. A., & Jenkins-Smith, H. C. (1998). The advocacy coalition framework. In P. A. Sabatier (Ed.), Theories of policy change (pp. 117-166). Boulder, CO: Westview Press.
54.
Shulha, L. M., & Cousins, J. B. (1997). Evaluation use: Theory, research, and practice since 1986. Evaluation Practice, 18, 195-208.
55.
Shulock, N. (1999). The paradox of policy analysis: If it is not used, why do we produce so much of it?Journal of Policy Analysis and Management, 18, 226-244.
56.
Sinclair, R. C., Mark, M. M., & Shotland, R. L. (1987). Construct accessibility and generalizability across response categories. Personality and Social Psychology Bulletin, 13, 239-252.
57.
Suchman, E. A. (1967). Evaluation research: Principles and practices in public service and social action programs. New York, NY: Russell Sage Foundation.
58.
Weick, K. E., & Quinn, R. E. (1999). Organizational change and development. Annual Review of Psychology, 50, 361-386.
59.
Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39, 426-431.
60.
Weiss, C. H. (1988a). Evaluation for decisions: Is anybody there? Does anybody care?Evaluation Practice, 9, 5-19.
61.
Weiss, C. H. (1988b). If program decision hinged only on information: a response to Patton. Evaluation Practice, 9, 15-28.
62.
Weiss, C. H. (1998). Improving the use of evaluations: Whose job is it anyway?Advances in Educational Productivity, 7, 263-276.
63.
Weiss, C. H., & Bucuvalas, M. J. (1977). The challenge of social research to decision making. In C. H. Weiss (Ed), Using social research in public policy making (pp. 213-234). Lexington, MA: Lexington Books.
64.
Zaslow, M. J., Moore, K. A., Brooks, J. L., Morris, P. A., Tout, K., Redd, Z. A., & Emig, C. A. (2002). Experimental studies of welfare reform and children. The Future of Children, 12, 79-95.
65.
Zhu, J. (1992). Issue competition and attention distraction: A zero-sum theory of agenda setting. Journalism Quarterly, 69, 825-836.
