In response to an increasing demand for public sector accountability, many government agencies have sought to develop their internal evaluation capabilities. Often these efforts have focused on increasing the capacity to supply credible evaluations, yet addressing demand is just as important. This article focuses on a government agency and tracks its five-year journey towards developing such a capability. It documents contextual matters, drivers for change, the actions taken by the agency, and its response to emergent challenges during four phases. Based on feedback from project staff and managers and those involved in the capability development project, it offers seven recommendations. These are: start small and grow evaluation; address both supply and demand; work top-down and bottom-up simultaneously; use a theory of change behaviour; develop a common evaluation framework, including a generic programme theory; build knowledge of what works within the agency's context; and systematically and visibly evaluate each stage.
Get full access to this article
View all access options for this article.
References
1.
Bennett, C. F. (1977) Analysing Impacts of Extension Programs. Washington, DC: USDA Extension Service .
Bickman, L., ed. (1990) Advances in Program Theory, New Directions for Program Evaluation 47. San Francisco, CA: Jossey-Bass .
4.
Blalock, A. (1999) `Evaluation Research and the Performance Management Movement: From Estrangement to Useful Integration?' , Evaluation5(2): 117-149 .
5.
Boomsma, J., B. McDonald and M. Walton (1996) Target 10: The First Three Years. Melbourne: Department of Natural Resources and Environment, Government of Victoria .
6.
Dart, J. (2000) Stories for Change: A New Model for Agricultural Extension Projects in Australia. Melbourne: PhD thesis, Institute of Land and Food Resources, Melbourne University.
7.
Dart, J. (2002) Evaluation of the Evaluation Support Team. Unpublished report.
8.
Davidson, E. J. (2001) `Mainstreaming Evaluation into an Organization's “Learning Culture” ' , Paper presented at the Meeting of the American Evaluation Association Mainstreaming Evaluation. St Louis, 8 November. Available from: http://unix.cc.wmich.edu/~jdavidso/olcult.ppt
9.
Davies, I. (1999) `Evaluation and Performance Management in Government' , Evaluation5(2): 150-159 .
10.
Department of Natural Resources and Environment (2001) Innovation: Sustaining Export Growth in the Food and Agriculture Sector. Melbourne: Government of Victoria .
11.
Fetterman, D. M. (1996) `Empowerment Evaluation: An Introduction to Theory and Practice', in D. M. Fetterman, S. J. Kaftarian and A. Wandersman (eds) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability, pp. 3-46. Thousand Oaks, CA: Sage .
12.
Julnes, G., M. Mark and G. Henry (1998) `Promoting Realism in Evaluation: Realistic Evaluation and the Broader Context' , Evaluation4(4): 483-504 .
13.
Kiresuk, T., A. Smith and J. Cardillo, eds (1994) Goal Attainment Scaling: Applications, Theory and Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates .
14.
Leeuw, F. L., J. Toulemonde and A. Brouwers (1999) `Evaluation Activities in Europe: A Quick Scan of the Market in 1998' , Evaluation5(4): 487-496 .
15.
McAllister, E. (1998) `Overview and Conclusions' , in K. Mackay (ed.) Selected Proceedings of the World Bank Seminar: Public Sector Performance - The Critical Role of Evaluation, pp. 121-125 . Washington, DC: World Bank.
16.
McDonald, B. (1997) Improving Dairy Cow Nutrition Using a Marketing Approach. Melbourne: Department of Natural Resources and Environment, Government of Victoria .
17.
McDonald, B. and P. Rogers (1999) `Differentiated Program Theory Evaluation as an Analogy to Market Segmentation' , Paper presented at the Annual Meeting of the American Evaluation Association, Orlando, FL.
18.
McDonald, B., J. Winston and J. Boomsma (1997) `Getting Started in Evaluation Extension: A Case Study of Practices and Principles Used in Target 10' , in Proceedings of the 2nd Australasian Pacific Extension Conference `Managing Change - Building Knowledge and Skills'. Wodonga, Australia: APEN.
19.
Mark, M. (2002) What Works and How Can We Tell? Melbourne: Department of Natural Resources and Environment, Government of Victoria .
20.
Mark, M., G. Henry and G. Julnes (2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. San Francisco, CA: Jossey-Bass .
21.
Mayne, J. (2001) Performance Measurement. Melbourne: Department of Natural Resources and Environment, Government of Victoria .
22.
Mayne, J., S. Divorski and D. Lemaire (1999) `Locating Evaluation: Anchoring Evaluation in the Executive or the Legislature or Both or Elsewhere?', in R. Boyle and D. Lemaire (eds) Building Effective Evaluation Capacity, pp. 23-52. New Brunswick, NJ: Transaction Publishers .
23.
Milstein, B. and D. Cotton (2000) `Defining Concepts for the Presidential Strand on Building Evaluation Capacity' , Evaluation 2000 Conference Information. Available from: http://www.eval.org/eval2000/public/presstrand.pdf
24.
Milstein, B., T. Chapel, S. Wetterhall and D. Cotton (2002) `Building Capacity for Program Evaluation at the Centers for Disease Control and Prevention', in D. Compton, M. Baizerman and S. Stockdill (eds) The Art, Craft and Science of Evaluation Capacity Building, New Directions for Evaluation 93, pp. 27-46. San Francisco, CA: Jossey-Bass .
25.
O'Brien, G. and G. Hepworth (1994) Evaluation of a Pasture Management Program for Dairy Farmers in West and South Gippsland. Melbourne: Department of Agriculture, Government of Victoria .
Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage .
28.
Preskill, H. and R. T. Torres (1999a) Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage .
29.
Preskill, H. and R. T. Torres (1999b) `Building Capacity for Organizational Learning Through Evaluative Inquiry' , Evaluation5(1): 42-60 .
30.
Rogers, P. and B. McDonald (1999) `The Three Achilles Heels of Program Theory' , Paper presented at the Australasian Evaluation Association Conference, Evaluation: Challenging Boundaries. Perth, WA; 6-8 October.
31.
Rogers, P. J., A. J. Petrosino, T. Hacsi and T.A. Huebner, eds (2000) Program Theory Evaluation: Challenges and Opportunities, New Directions for Evaluation 87. San Francisco, CA: Jossey-Bass .
32.
Russon, C. and M. Patel (1999) `Evaluation Capacity in Eastern and Southern Africa' , Evaluation News and Comment8(1): 14-15 .
33.
Scriven, M. (1997) `Book Review: Empowerment. Evaluation: Knowledge and Tools for Self-Assessment and Accountability' , Evaluation Practice18(2): 165-175 .
34.
Stanfield, B. (ed.) (1997) The Art of Focused Conversation. Toronto: The Canadian Institute of Cultural Affairs .
35.
Weiss, C. H. (1998) Evaluation: Methods for Studying Programs and Policies. Upper Saddle River, NJ: Prentice-Hall .
