Considerable research has been conducted on the use of the findings of program evaluation, but little, if any, attention has been paid to the soundness of the methods of this research. If the methods are not sound or not well described in the research, the strength of the conclusions of the research is unknown. The authors examine the empirical studies reported in five widely cited reviews of the literature on program evaluation use and summarize the types of methods used, the professions for which they were conducted, and the extent to which they met two basic criteria for soundness. It was concluded that, by and large, the body of studies does not adequately address the criteria. With care, the findings of the research literature on the use of evaluations might provide guidance for evaluators, but they lack sufficient scientific credibility for other uses.
Alkin, A., & Daillak, R.H. (1979). A study of evaluation utilization. Educational Evaluation and Policy Analysis, 1, 41-49.
2.
Alkin, M.C., Kosecoff, J., Fitz-Gibbon, C., & Seligman, R. (1974). Evaluation and decision making: The Title VII experience. Los Angeles: Center for the Study of Evaluation.
3.
Alkin, M.C., & Stecher, B. (1983). Evaluation in context: Information use in elementary school decision making. Studies in Educational Evaluation , 9, 23-32.
4.
Andrews, J.V. (1979). Reactions to program evaluation: A qualitative analysis (Doctoral dissertation, Cornell University, 1979). Dissertation Abstracts International, 39, 6717A (University Microfilms No. 6779-6710, 6731).
5.
Ayers, T.D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program Planning, 10, 263-271.
6.
Barnette, J.J., & Thompson, J.C. (1979). A descriptive assessment of the effect of evaluations on instructions. Studies in Educational Evaluation, 5, 77-86.
Bigelow, D.A., & Ciarlo, J.A. (1976). The impact of therapeutic effectiveness data on community mental health center management. In G. V. Glass (Ed.), Evaluation studies review annual ((Vol. 1, pp. 371-382). Beverly Hills, CA: Sage.
9.
Brandon, P.R. (2002). Two versions of the contrasting-groups standard-setting method: A review. Measurement and Evaluation in Counseling and Development, 35, 167-181.
10.
Brandon, P.R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education , 17, 59-88.
11.
Braskamp, L.A., Brown, R.D., & Newman, D.L. (1978). The credibility of a local educational program evaluation report: Author source and client audience characteristics. American Educational Research Journal, 15, 441-450.
12.
Brown, R.D., & Newman, D.L. (1982). An investigation of the effect of different data presentation formats and order of arguments in a stimulated adversary evaluation . Educational Evaluation and Policy Analysis, 4, 197-203.
13.
Brown, R.D., Newman, D.L., & Rivers, L.S. (1985). Does the superintendent's opinion affect school boards' evaluation information needs? An empirical investigation. Urban Education, 20, 204-221.
14.
Carter, R.K. (1971). Client's resistance to negative feelings and the latent conservative function of evaluation studies. The American Sociologist, 6, 118-124.
15.
Clandinin, D.J., & Connelley, F.M. (1994). Narrative inquiry: experience and story in qualitative research. San Francisco: Jossey-Bass.
16.
Cousins, J.B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3-27.
17.
Cousins, J.B. (2003). Utilization effects of participatory evaluation . In T. Kellaghan, & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245-266). Dordrecht, Netherlands: Kluwer.
18.
Cousins, J. B. (Ed.). (2007). Process use in theory, research, and practice. New Directions for Evaluation , No. 116. San Francisco : Jossey-Bass.
19.
Cousins, J.B., Donohue, J.J., & Bloom, G.A. (1996). Collaborative evaluation in North America: Evaluators' self-reported opinions, practices and consequences. Evaluation Practice, 17, 207-226.
20.
Cousins, J.B., & Leithwood, K.A. (1986). Current empirical research in evaluation utilization. Review of Educational Research , 56, 331-364.
21.
Cousins, J.B., & Shulha, L.M. (2006). A comparative analysis of evaluation utilization and its cognate fields of inquiry: Current trends and issues. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The Sage handbook of evaluation (pp. 266-291). Thousand Oaks, CA: Sage.
22.
Daillak, R.H. (1983). Evaluators in an urban school district: Organizational constraints upon evaluation influence. Studies in Educational Evaluation, 9, 33-46.
23.
David, J.L. (1978). Local uses of Title I evaluation (Report prepared by the Office of the Assistant Secretary for Planning and Evaluation, Department of Health, Education and Welfare). Menlo Park, CA: SRI International.
24.
Dawson, J.A., & D'Amico, J.J. (1985). Involving program staff in evaluation studies: A strategy for increasing information use and enriching the data base. Evaluation Review, 9, 173-188.
25.
Dickey, B. (1980). Utilization of evaluations of small-scale innovative educational projects. Educational Evaluation and Policy Analysis , 2, 65-77.
26.
Dickman, F.B. (1981). Work activities, settings, methodologies and perceptions: Correlates of evaluative research utilization. Knowledge: Creation, Diffusion, Utilization, 2, 375-387.
27.
Dillman, D.A. (2000). Mail and Internet surveys: The tailored design method. New York: Wiley.
28.
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2004). Program evaluation: Alternative approaches and practical guidelines. Boston: Pearson .
29.
Folkman, D.V., & Rai, K. (1997). Reflection on facilitating a participatory community self-evaluation. Evaluation and Program Planning, 20, 455-465.
30.
Forss, K., Cracknell, B., & Samset, K. (1994). Can evaluation help an organization to learn? Evaluation Review, 18, 574-591.
31.
Goldberg, M. (1978, April). New York City administrators' perceptions of the helpfulness of outside mandated evaluations on the administrative decision-making in ESEA programs. Paper presented at the American Educational Research Association, Toronto, Ontario (ERIC Document Reproduction Service No. ED 156 720).
32.
Granville, A.C. (1977, April). An experimental study of factors that influence school decisions: Implications for evaluation. Paper presented at the American Educational Research Association, New York (ERIC Document Reproduction Service No. ED 139-824).
33.
Greene, J.C. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12, 91-116.
34.
Guilford, J.P. (1936). Psychometric methods (1sted.). New York: McGraw-Hill.
35.
Haenn, J.F. (1980). Reasons why evaluations and testing don't inform . Paper presented at the annual meeting of the American Educational Research Association, Boston (ERIC Document Reproduction Service No. ED 187 733).
36.
Henry, G.T., & Mark, M.M. (2003). Beyond use: Understanding evaluation's influence on attitudes and actions. American Journal of Evaluation, 24, 293-314.
37.
Hofstetter, C.H., & Alkin, M.C. (2003). Evaluation use revisited. In T. Kellaghan, & D. L. Stuifflebeam (Eds.), International handbook of educational evaluation (pp. 189-196). Dordrecht, Netherlands: Kluwer.
38.
Hollingworth, H.L. (1911). Judgments of persuasiveness. Journal of Abnormal and Social Psychology, 18, 43-49.
39.
House, E.R. (1972). School evaluation: The politics and process. Berkeley, CA: McCutchan.
40.
Johnson, K.W. (1980). Stimulating evaluation use by integrating academia and practice. Knowledge: Creation, Diffusion, Utilization , 2, 237-262.
41.
Johnson, R.L., Willeke, M.J., & Steiner, D.J. (1998). Stakeholder collaboration in the design and implementation of a family literacy portfolio assessment. American Journal of Evaluation, 19, 339-353.
42.
King, J.A., & Pechman, E.M. (1984). Pinning a wave to the shore: Conceptualizing evaluation use in school systems. Educational Evaluation and Policy Analysis , 6, 241-251.
43.
Kirkhart, K.E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In C. Caracelli, & H. Preskill (Eds.), The expanding scope of evaluation use. New directions for evaluation, No. 88(pp. 5-24). San Francisco: Jossey-Bass.
44.
Labin, S. (2008). Research synthesis: Toward broad-based evidence . In N. L. Smith, & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 89-110). New York: Guilford .
45.
Lau, G., & LeMahieu, P. (1997). Changing roles: Evaluator and teacher collaborating in school change. Evaluation and Program Planning, 20, 7-15.
46.
Levin, B. (1987). The uses of research: A case study in research and policy. Canadian Journal of Program Evaluation, 2, 43-55.
47.
Leviton, L.C., & Boruch, R.F. (1983). Contributions of evaluation to education programs and policy. Evaluation Review, 7, 563-598.
48.
Leviton, L.C., & Hughes, E.F.X. (1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review , 5, 525-548.
49.
Lissitz, R., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36, 437-448.
50.
Lorenzen, G.L., & Braskamp, L.A. (1978). Comparative influence of political, cost/benefit, and statistical information on administrative decision making. Evaluation and Program Planning, 1, 235-238.
51.
Maher, C.A. (1982). Utilization of program evaluation information in public schools: Perspectives and guidelines for school psychologists. Journal of School Psychology, 20, 113-121.
52.
Marshall, C. (1993). Goodness criteria: Are they objective or judgment calls? In E. G. Guba (Ed.), The paradigm dialog (pp. 188-197). Newbury Park, CA: Sage.
53.
Mathison, S. (1994). Rethinking the evaluator role: Partnerships between organizations and evaluators. Evaluation and Program Planning , 17, 299-304.
Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.
56.
Newman, D.L., Brown, R.D., & Rivers, L.S. (1983). Locus of control and evaluation use: Does sense of control affect information needs and decision making? Studies in Education Evaluation, 9, 77-88.
57.
Newman, D.L., Brown, R.D., Rivers, L.S., & Glock, R.F. (1983). School boards' and administrators' use of evaluation information: Influencing factors. Evaluation Review, 7, 110-125.
58.
Newman, D.L., & Cai, M. (1995). Teacher involvement in program evaluation and evaluation use: An empirical examination of the participatory evaluation approach . Paper presented at the annual meeting of the American Evaluation Association in conjunction with the International Evaluation Conference, Vancouver, B.C., Canada.
59.
Nielsen, V.G. (1975). Why evaluation does not improve program effectiveness . Policy Studies Journal, 3, 385-389.
60.
Osterlind, S.J. (1979). What makes a school evaluation work?Educational Evaluation and Policy Analysis, 1, 48-51.
61.
Owen, J.M., Lambert, F.C., & Stringer, W.N. (1994). Acquiring knowledge of implementation and change: Essential for program evaluators? Knowledge: Creation , Diffusion, Utilization, 15, 273-284.
62.
Patton, M.Q., Grimes, P.S., Guthrie, K.M., Brennan, N.J., French, B.D., & Blyth, D.A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 141-163). New York: Heath.
63.
Preskill, H., & Caracelli, V. (1997). Current and developing conceptions of use: Evaluation use TIG survey results. Evaluation Practice, 18, 209-225.
64.
Rossman, B.B., Hober, D.J., & Ciarlo, J.A. (1979). Awareness, use, and consequences of evaluation data in a community mental health center. Community Mental Health Journal, 15, 7-16.
65.
Schaeffer, N.C., & Presser, S. (2003). The science of asking questions. Annual Review of Sociology, 29, 65-88.
66.
Shulha, L., & Cousins, J.B. (1997). Evaluation utilization: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
67.
Siegal, K., & Tuckel, P. (1985). The utilization of evaluation research: A case analysis. Evaluation Review, 9, 307-328.
68.
Smith, N.L., & Chircop, S. (1989). The Weiss-Patton debate: Illumination of the fundamental concerns. Evaluation Practice, 10, 5-13.
69.
Stahler, G.J. (1995). Improving the quality of federal human services national demonstration programs. Evaluation and Program Planning , 18, 129-141.
70.
Stake, R., & Trumbull, D. (1982). Naturalistic generalizations. Review Journal of Philosophy and Social Science, 7, 1-12.
71.
Sudman, S.B., & Bradburn, N.M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.
72.
Sudman, S., Bradburn, N.M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
73.
Tash, W.R., & Stahler, G.J. (1982). Enhancing the utilization of evaluation findings . Community Mental Health Journal, 18, 180-189.
74.
The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Newbury Park, CA: Sage.
75.
Thompson, B., & King, J.A. (1981). Evaluation utilization: A literature review and research agenda. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles (ERIC Document Reproduction Service No. ED 199 271).
76.
Thompson, P.A., Brown, R.D., & Ferguson, J. (1981). Jargon and data do make a difference: The impact of report styles on lay and professional evaluation audiences. Evaluation Review, 5, 269-279.
77.
Toulmin, S. (1958). The uses of argument. New York : Cambridge University.
78.
Turnbull, B. (1999). The mediating effect of participatory efficacy on evaluation use. Evaluation and Program Planning, 22, 131-140.
79.
Vroom, P.I., Colombo, M., & Nahan, N. (1994). Confronting ideology and self-interest: Avoiding misuse of evaluation. In C. J. Stevens & M. Dial (Eds.), New directions for program evaluation, No. 64(pp. 49-60). San Francisco: Jossey-Bass.
80.
Wanke, M. (2002). Conversational norms and the interpretation of vague quantifiers. Applied Cognitive Psychology, 16, 301-307.
81.
Weeks, E.C. (1979). The managerial use of evaluation findings. In H. C. Schulberg, & J. M. Jerrell (Eds.), The evaluator and management (pp. 137-155). Beverly Hills, CA: Sage.
82.
Williams, W., & Evans, J.W. (1969). The politics of evaluation: The case of Head Start . Annals of the American Academy of Political and Social Science , 385, 118-132.
83.
Windle, C., & Bates, P. (1974). Evaluating program evaluation: A suggested approach . In P. O. Davidson (Ed.), Evaluation of behavioral programs. Champaign, IL: Research Press.
84.
Worthen, B.R., & Sanders, J.R. (1973). Educational evaluation: Theory and practice. Worthington, Ohio: Jones.
85.
Wright, D.B., Gaskel, G.D., & O'Muircheartaigh, C.A. (1994). How much is `Quite a bit'? Mapping between numerical values and vague quantifiers. Applied Cognitive Psychology, 8, 479-496.
