This article reviews two approaches to encouraging program managers to conduct
evaluations and to improve the quality of their evaluations. The two approaches are
Sunset reviews and evaluation plans. Examples of the application of each approach to
energy conservation programs are discussed. The strengths and weaknesses of both
approaches are considered and recommendations about how to improve the guidance
process are presented. A guidance process that combines the best features of both
approaches is needed. The essential problem is to strike the proper balance between
universal requirements and specific designs, methods, and procedures that are feasible to
implement. Sunset review requirements ensure that some evaluation is conducted by all
program managers, but do little to improve the quality of specific evaluation efforts. The
evaluation plan approach, in contrast, leads to good resultsfor specific programs, but does
little to ensure that comparisons across programs can be made. If generic evaluation plans
matched to appropriate program types were combined with Sunset mandates for
evaluation, significant improvements in the quality, comparability, and agencywide
coverage of evaluation efforts should result.
Get full access to this article
View all access options for this article.
References
1.
Alkin, M.C., R. Daillak, and P. White (1979) Using Evaluations: Does Evaluation Make a Difference ? Beverly Hills, CA: Sage.
2.
Bardach, E. (1980) "On designing implementable programs," in G. Majone and E. S. Quade (eds.) Pitfalls of Analysis. New York: John Wiley.
3.
Berman, P. and M. McLaughlin (1974) Federal Programs Supporting Educational Change , Vol. 1: A Model of Educational Change. Report R-1589/I-HEW. Santa Monica, CA: Rand Corporation.
4.
———( 1975) Federal Programs Supporting Educational Change, Vol. II: Factors Affecting Change Agent Projects. R-1589/2-HEW. Santa Monica, CA: Rand Corporation.
5.
———( 1976) Federal Programs Supporting Educational Change, Vol. IV: The Findings in Review. R-1589/4-HEW . Santa Monica, CA: Rand Corporation .
6.
———( 1978) Rethinking the Federal Role in Education. P-6114. Santa Monica, CA: Rand Corporation.
7.
———( 1979) An Exploratory Study of School District Adaptation. R-2010-NIE . Santa Monica, CA: Rand Corporation .
8.
Berry, L.G. (1983) "Residential conservation program impacts: methods of reducing self-selection bias." Evaluation Rev.7, 6: 753-775.
9.
———( 1986) The Role of Evaluation Results in the Bonneville Power Administration's Conservation Assessment and Demand Forecasting Models: Present Uses and Future Directions. ORNL/CON-197. Oak Ridge, TN : Oak Ridge National Laboratory.
10.
Beyer, J.M. and H.M. Trice (1982) "The utilization process: a conceptual framework and synthesis of empirical findings." Admin. Sci. Q.27: 591-622.
11.
Box, G.E.P., W.G. Hunter, and J.S. Hunter (1978) Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. New York: John Wiley.
12.
Eveland, J.D. and E.M. Rogers (1980) Measuring the Innovation Process in Public Organizations . Washington, DC: National Science Foundation.
13.
Guba, E. and H. Brickell (1974) Conceptual Strategies for Utilizing R&D Development Products in Education. Occasional Paper #2, Ohio State University, Columbus.
14.
Hirst, E., B. Bronfman, R. Goeltz, J. Trimble, and D. Lerman (1983) Trimble (1982) Evaluation Plan for the Bonneville Power Administration Residential Energy Conservation Programs. ORNL/CON-94. Oak Ridge, TN: Oak Ridge National Laboratory.
15.
———B. Bronfman, R. Goeltz, J. Trimble, and D. Lerman (1983) Evaluation of the BPA Residential Weatherization Pilot Program. ORNL/ CON-124. Oak Ridge, TN: Oak Ridge National Laboratory.
16.
Jolly, J.A., J.W. Creighton, and P.A. George (1978) Technology Transfer Process Model and Annotated Bibliography. Monterey, CA: Naval Postgraduate School.
17.
Keating, K.M. (1984) "The integral role of conservation program evaluation in the Bonneville Power Administration." Presented at the ACEEE Summer Study on Energy Efficiency in Buildings, Santa Cruz, CA.
18.
Leviton, L.C. and E.F.X. Hughes (1981) "Research on the utilization of evaluations: a review and synthesis." Evaluation Rev.5, 4: 525-548.
19.
National Science Foundation (1973) Science, Technology, and Innovation. Report on Contract NSF-C667. Columbus, OH: Batelle Columbus Laboratories.
20.
Patton, M.Q. (1978) Utilization-Focused Evaluation. Beverly Hills, CA: Sage.
21.
Pressman, J.L. and A. Wildavsky (1973) ImplementationBerkeley: Univ. of California Press.
Salasin, J. (1980) The Evaluation of Federal Research Programs. McLean, VA: MITRE Corporation.
24.
Soderstrom, E.J., E. Hirst, D.L. Greene, and J. Trimble (1984) "Have Department of Energy programs saved energy ?" Evaluation Rev.8, 1: 93-112.
25.
Spak, G.T. and E.F. Shelley (1978) How to Operate an Energy Advisory Service, Vol. I: Report and Recommendations. Final Report to the Department of Education. Old Westbury, NY: New York Institute of Technology.
26.
Tornatzky, L.G., E.O. Fergus, J.W. Avellar, and G.W. Fairweather , with M. Fleischer (1980) Innovation and Social Process. New York: Pergamon Press .
27.
U.S. Department of Energy (1979) Evaluation of the Energy Extension Service Pilot Program: Report to Congress . [Prepared by ICF, Inc. and Westat, Inc.] Washington, DC: Author.
28.
———( 1980a) An Evaluation of the State Energy Conservation Program from Program Implementation to September 1978. [Prepared by Energy and Environmental Analysis, Inc.] Washington, DC: Author.
29.
———( 1980b) Energy Extension Service Pilot Program Evaluation Report after Two Years, Vol I: Evaluation Summary. [Prepared by ICF, Inc. and Westat, Inc.] Washington, DC: Author.
30.
———( 1980c) Study of State Energy Conservation Program 1979 Energy Savings Indicators. [Prepared by Price Waterhouse.] Washington, DC: Author.
31.
———( 1981) Study of State Energy Conservation Program 1980 Energy Savings Indicators. [Prepared by Price Waterhouse.] Washington, DC: Author.
32.
———Office of Policy, Planning and Analysis (1982 a) Sunset Review: Report to the Congress, Department of Energy Organization Act, Title X, Vols. 1-3. Washington, DC: Author.
33.
———( 1982b) Study of State Energy Conservation Program 1981 Energy Savings Indicators. [Prepared by Price Waterhouse.] Washington, DC: Author.
34.
———( 1982c) Final Report: Evaluation of the Energy Extension Service . [Prepared by Mark Battle Associates.] Washington, DC : Author.
35.
Weiss, C. (1981) "Measuring the use of evaluation," in J. A. Ciarlo (ed.) Utilizing Evaluation: Concepts and Measurement Techniques. Beverly Hills, CA: Sage.
36.
Willette, J.L. and H.L. Fleischman (1982) "Federal legislation accountability: designing an evaluation system." Evaluation Rev.6, 2: 155-171.
37.
Witner, J.B. (1978) A Comparative Evaluation of Solar Alternatives: Implications for Federal R&D, Vol. 1. [Prepared for Solar Working Group of the U.S. Department of Energy by SRI International.] Washington, DC: U.S. Department of Energy.
38.
Yin, R.K. (1978) Changing Urban Bureaucracies. How New Practices Become Routinized. R-2277-NSF (Final Report to NSF. Santa Monica, CA: Rand Corporation.
39.
———K.A. Heald, M.E. Vogel, P.D. Fleischauer, and B.C. Vladeck (1976) A Review of Case Studies of Technological Innovations in State and Local Services. R-1870-NSF (Final Report to NSF. Santa Monica, CA: Rand Corporation.
