Evaluation is increasingly taken as a public good, as an activity aimed at development and betterment. At the same time, evaluation is increasingly integrated in accountability movements and in institutionalized systems for governance. It is also embedded in a consumer-oriented New Public Management ideology. Against this background, it is high time to start critically examining evaluation itself as a phenomenon and practice. A conceptual framework for carrying out critical studies of evaluations in national (or state) settings is provided, and examples from previous such studies are given to illustrate possible findings. The framework is meant to capture forces that shape an evaluation process and the knowledge claims that result. These forces can restrict evaluation or enable it to provide a critical examination meant to be part of a public debate. By scrutinizing the premises for, the implementation of, and the impact of public-sector evaluations from a critical perspective, evaluators should gain a more thorough understanding of the various power plays of which evaluation is a part. Further discussions of and additional research on public-sector evaluation, to increase our knowledge and understanding of evaluation as a practice, are encouraged.
Get full access to this article
View all access options for this article.
References
1.
Andersson, I. M., Karlsson, O., & Lundin, A. (2002). Metautvärdering av Högskoleverkets modell för kvalitetsbedömning av högre studier. Hur har lärosäten och bedömare uppfattat modellen?(Meta-evaluation of The National Agency for Higher Education’s model for judging quality in higher education in Sweden. How did the higher education institutions and the external reviewers look upon the model?). Stockholm: Högskoleverket (in Swedish).
2.
Burbules, N. C. (2000). Meta-evaluation of the Milwaukee Teacher Education Center. In M. Chandler, et al. (Eds.), Evaluation of the MTEC Alternative Teacher Education Program, Appendix A. Urbana-Champaign, IL: University of Illinois, CIRCE.
3.
Checklists. (March 1, 2002). On www.wmich.edu/evalcrtr/checklists/view-download.page.htm.
4.
Dahler-Larsen, P. (1998). Den Rituelle Reflektion—om evaluering i organisationer (The ritual reflection—On evaluations in organizations). Odense: Odense Universitetsforlag (in Danish).
5.
Dale, R. (1989). The State and Education Policy. Milton Keynes: Open University Press.
6.
Datta, L.-e. (1999). CIRCE’s demonstration of a close to ideal evaluation in a less than ideal world. American Journal of Evaluation, 20, 345-354.
7.
Draze, D. (2001). A study of the relationship between epistemic style and evaluation practice. Paper presented at the annual meeting of the AERA, Seattle, April 2001. Kalamazoo: Western Michigan University, SAMPI.
8.
Forss, K., & Samset, K. (1998). Square pegs and round holes: Evaluation, uncertainty and risk management. Paper presented at the European Evaluation Society’s Annual Conference, Rome, October 29-31, 1998.
9.
Franke-Wikberg, S. (1990). Utvärdering i och av gymnasieskolan—en principskiss (Evaluation in and of secondary education—An outline of principles). Arbetsrapport 86. Umeå: Department of Education, Umeå University (in Swedish).
10.
Franke-Wikberg, S. (1992). Utvärderingens mångfald—några ledtrådar för vilsna utbildare (The many faces of evaluation—Some clues for lost educators). Projektrapport 1992:4. Stockholm: UHÄ, FoU-enheten (in Swedish).
11.
Franke-Wikberg, S., & Johansson, M. (1975). Utvärdering av undervisning. En problemanalys och några empiriska studier på universitetsnivå (Evaluation of education. An analysis of problems and some empirical studies in higher education). Umeå: Department of Education, Umeå University (In Swedish).
12.
Franke-Wikberg, S., & Lundgren, U. P. (1980). Att värdera utbildning. Del 1 (To evaluate education. Part 1). Stockholm: Wahlström & Widstrand (in Swedish).
13.
Franke-Wikberg, S., & Lundgren, U. P. (Eds.). (1981). Att värdera utbildning. Del 2. En antologi om pedagogisk utvärdering (To evaluate education. Part 2. An anthology of educational evaluation). Stockholm: Wahlström & Widstrand (in Swedish).
14.
Franke-Wikberg, S., & Frykholm, C.-U. (1987). Utveckling och utvärdering—två sidor av samma mynt? (Development and evaluation—Two sides of the same coin). In L. Johansson (Ed.), Utvärdering—möte med verkligheten (Evaluation—A confrontation with reality) (pp. 25-26). Stockholm: Utbildningsförlaget (in Swedish).
15.
Franke-Wikberg, S., & Nitzler, R. (1991). Utvärdering och lokal utveckling inom barnomsorgen enligt “Umeå-modellen” (Evaluation and local development in pre-school according to the “Umeå-model”). Arbetsrapport Nr 91. Umeå: Department of Education, Umeå University (in Swedish).
16.
Furubo, J.-E. (1987). Utvärderingar—till vad och hur mycket (Evaluations—For what reason and how much). Stockholm: Riksrevisionsverket (in Swedish).
17.
Grasso, P. G. (1999). Meta-evaluation of an evaluation of reader-focused writing for the Veterans Benefits Administration. American Journal of Evaluation, 20, 355-370.
18.
Greene, J. C. (2000). Meta-evaluation: Evaluation of VBA Appeals Training Module Certify a Case to the Board of Veteran Appeals. In R. Davis, et al. (Eds.), VBA Appeals training Module: Certification of Appeals to the Board of Veterans Appeals (Chap. 12). Urbana-Champaign, IL: University of Illinois, CIRCE.
19.
Greene, J. C. (2001). Dialogue in evaluation. A relational perspective. Evaluation, 7, 181-187.
20.
Ham, C., & Hill, M. (1993). The policy process in the modern capitalists state (2nd ed.). New York, NY: Harvester Wheatsheaf.
21.
Hansen, H. F. (2001a). Evaluering: Teori og praksis i designprocessen (Evaluation: Theory and practice in the process of design). Nordiske Organisasjonsstudier, 3(3), 43-62 (in Danish).
22.
Hansen, H. F. (2001b). Rejsende praksis: Hvilke forhold påvirker udviklingen af evalueringspraksis på forskellige politikområder? (Traveling practice: What circumstances influence the development of evaluation practice in different policy areas?). In P. Dahler-Laresn & K. Krogstrup (Eds.), Tendenser i evaluering (Tendences in evaluation) (pp. 109-129). Odense: Odense Universitetsforlag (in Danish).
23.
Haug, P. (1996). Evaluation of government reforms. Evaluation, 2, 417-430.
24.
Henkel, M. (1991). Government, evaluation and change. London: Jessica Kingsley Publishers.
25.
Henry, G. T. (2001). How modern democracies are shaping evaluation and the emerging challenges for evaluation. American Journal of Evaluation, 22, 419-429.
26.
Henry, G. T., & Mark, M. M. (2003). Toward an agenda for research on evaluation. In C. A. Christie (Ed.), The Practice-Theory Relationship in Evaluation, New Directions for Evaluation, No. 97 (pp. 69-80). San Francisco, CA: Jossey-Bass.
27.
Hogwood, B. W., & Gunn, L. A. (1984). Policy analysis for the real world. Oxford: Oxford University Press.
28.
Holmlund, K., & Rönnerman, K. (1995). Kvalitetssäkra förskolan (To develop quality in pre-school). Lund: Studentlitteratur (in Swedish).
29.
House, E. R. (1988). Jesse Jackson and the politics of charisma: The rise and fall of the PUSH/Excel Program. Boulder, CO & London: Westview Press.
30.
House, E. R. (1994). The future perfect of evaluation. Evaluation Practice, 15, 239-247.
31.
House, E. R. (1997). Evaluation in the government marketplace. Evaluation Practice, 18, 37-48.
32.
House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage Publications.
33.
House, E. R., Haug, C., & Norris, N. (1995). Evaluation policies and issues in the National Science Foundation. Boulder, CO: School of Education, University of Colorado.
34.
House, E. R., Haug, C., & Norris, N. (1996). Producing evaluations in a large bureaucracy. Evaluation, 2, 135-150.
35.
Karlsson, O. (1995). Att utvärdera-mot vad? Om kriterieproblemet vid intressentutvärdering (To evaluate—To what purpose? On the problem of criteria in stakeholder evaluation. Ph.D. Dissertation. Stockholm: HLS Förlag (in Swedish).
36.
Karlsson, O. (1999). Utvärdering-mer än en metod. En översikt (Evaluation—More than methods. A review). Stockholm: Kommentus Förlag (in Swedish).
37.
Khakee, A. (1998). Planering och utvärdering-två oskiljaktiga begrepp (Planning and evaluation—Two inseparable concepts). Rapport Nr 1998:3. Linköping: Linköping University, Centrum för kommunstrategiska studier (in Swedish).
38.
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. J. Caracelli & H. Preskill (Eds.), The expanding scope of evaluation use, New Directions for Evaluation, No. 88 (pp. 5-23). San Francisco, CA: Jossey-Bass.
39.
Leeuw, F. L., & Toulemonde, J. (Conveners) Hegarty, D., Fitzpatrick, J., Reale, M., & France, J. M. (1998). Evaluation as a business. Session A3 at The European Evaluation Society Conference, Rome, September 29-31, 1998.
40.
Lindensjö, B., & Lundgren, U. P. (1986). Politisk styrning och utbildningsreformer (Political governance and educational reforms). Stockholm: Liber Utbildningsförlaget (in Swedish).
41.
Lipsky, M. (1980). Street-level bureaucracy: Dilemmas of the individual in public services. New York, NY: Russell Sage Foundation.
42.
Lundgren, U. P. (1990). Educational policymaking, decentralisation and evaluation. In M. Granheim, M. Kogan, & U. P. Lundgren (Eds.), Evaluation as policymaking: Introducing evaluation into a national decentralised education system (pp. 23-41). London: Jessica Kingsley Publishers.
43.
Mark, M. M. (2001). Evaluation’s future: Furor, futile, or fertile?American Journal of Evaluation, 22, 457-479.
44.
Mark, M., Henry, G. T., & Julnes, G. (1999). Towards an integrative framework for evaluation practice. American Journal of Evaluation, 20, 177-198.
45.
McEldony, J. J. (1997). Policy evaluation and the concepts of deadweight and additionality: A commentary. Evaluation, 3, 175-188.
46.
Molander, B. (1992). Tacit knowledge and silenced knowledge: Fundamental problems and controversies. In B. Göranzon & M. Florin (Eds.), Skill and education: Reflection and experience (pp. xx-xx). Berlin; New York, NY: Springer-Verlag.
47.
Naustdalslid, J., & Reitan, M. (1992). Kunnskap og styring. Om forskningens rolle i politikk og forvaltning (Knowledge and governance. The role of research in politics and administration). NIBR-rapport 1992:15. Oslo: Norsk institutt for by og regionforskning (in Norwegian).
48.
Norris, N. (1993). The contractual control of policy and programme evaluation. Norwich, UK: Centre for Applied Research in Education, University of East Anglia.
49.
Norris, N. (1995). Contracts, control and evaluation. Journal of Education Policy, 10, 271-285.
50.
Norris, N. (1997). The contractual control of applied research. Norwich, UK: Centre for Applied Research in Education, University of East Anglia.
51.
Owen, J. M., & Rogers, P. J. (1999). Programe evaluation: Forms and approaches. London: Sage Publications.
52.
Ozga, J. (1994). Frameworks for policy analysis in education. In D. Kallós & S. Lindblad (Eds.), New policy contexts for education: Sweden and United Kingdom (pp. 205-233). Pedagogiska rapporter Nr 42/1994. Umeå: Department of Education, Umeå University.
53.
Patton, M. Q. (1998). Discovering process use. Evaluation, 4, 225-233.
54.
Pettersson, O., & Wallin, E. (1995). Utvärderingsmakt (Evaluation-power). In B. Rombach & K. Sahlin-Andersson (Eds.), Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sector (From a search for truth to a governing instrument. Modern evaluations in public sectors) (pp. 93-108). Stockholm: Nerenius & Santérus Förlag (in Swedish).
55.
Pettigrew, M. (1993). Coming to terms with research: The contract business. Norwich, UK: Centre for Applied Research in Education, University of East Anglia.
56.
Pollitt, C. (1995). Justification by works or by faith? Evaluating the New Public Management. Evaluation, 1, 133-154.
57.
Pollitt, C. (1998). Evaluation in Europe: Boom or bubble?Evaluation, 4, 214-224.
58.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.
59.
Power, M. (1997). The audit society: Rituals of verification. Oxford: Oxford University Press.
60.
Premfors, R. (1989). Policyanalys (Policyanalysis). Lund: Studentlitteratur (in Swedish).
61.
Pressman, J. L., & Wildavsky, A. B. (1984). Implementation: How great expectations in Washington are dashed in Oakland: Or, why it’s amazing that federal programs work at all, this being a saga of the Economic Development Administration as told by two sympathetic observers who seek to build morals on a foundation of ruined hopes. Berkeley, CA: University of California Press.
62.
Rabo, A. (1995). Utvärdering som moderna ritualer (Evaluations as modern rituals). In B. Rombach & K. Sahlin-Andersson (Eds.), Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sector (From a search for truth to a governing instrument. Modern evaluations in public sectors) (pp. 171-192). Stockholm: Nerenius & Santérus Förlag (in Swedish).
63.
Radaelli, C. M., & Dente, B. (1996). Evaluation strategies and analysis of the policy process. Evaluation, 2, 51-66.
64.
Rogers, P. J. (2001). The whole world is evaluating half-full glasses. American Journal of Evaluation, 22, 431-435.
65.
Rombach, B., & Sahlin-Andersson, K. (Eds.). (1995). Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sektor (From a search for truth to a governing instrument. Modern evaluations in public sectors). Stockholm: Nerenius & Santérus Förlag (in Swedish).
66.
Rothstein, B. (1986). Den socialdemokratiska staten. Reformer och förvaltning inom svensk arbetsmarknads- och skolpolitik (The social democratic state. Reforms and administration in the Swedish politics of the labor-market and school). Ph.D. Dissertation. Lund: Arkiv avhandlingsserie (in Swedish).
67.
(English edition: 1998, Just institutions matter: The moral and political logic of the universal welfare state). Cambridge, MA: Cambridge University Press.
68.
Rothstein, B. (1996). The socialdemocratic state: The Swedish model and the bureaucratic problem of social reform. Pittsburgh, PA: University of Pittsburgh Press.
69.
Sahlin-Andersson, K. (1995). Utvärderingars styrsignaler (The signals of governance in evaluations). In B. Rombach & K. Sahlin-Andersson (Eds.), Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sector (From a search for truth to a governing instrument. Modern evaluations in public sectors) (pp. 71-92). Stockholm: Nerenius & Santérus Förlag (in Swedish).
70.
Sanders, J. R. (1999). A meta-evaluation of “The effectiveness of comprehensive, case management interventions: Evidence from the national evaluation of the Comprehensive Child Development Program”. American Journal of Evaluation, 20, 577-582.
71.
Schwandt, T. A. (2002). Democratization, globalization, and performativity: Implications for learning in and from evaluation. Keynote address given at the 5th biannual meeting of the European Evaluation Society, October 12, 2002, Seville, Spain.
72.
Schwandt, T. A. (2003). Linking evaluation and education: Enlightenment and engagement. In P. Haug & T. A. Schwandt (Eds.), Evaluating educational reform: Scandinavian perspectives (pp. 169-188). Greenwich, CT: Information Age Publishing.
73.
Scriven, M. (1980). The logic of evaluation. Inverness, CA: Edgepress.
74.
Segerholm, C. (1998a). Will changed evaluations restrict knowledge? Paper presented at The European Evaluation Society’s Conference, RomeOctober 29-31, 1998.
75.
Segerholm, C. (1998b). Att förändra barnomsorgen. En analys av en statlig satsning på lokalt utvecklingsarbete (To change the public child care system. An analysis of a national program for local development work). Ph.D. Dissertation. Umeå: Department of Education, Umeå University (in Swedish).
76.
Segerholm, C. (1999). Utvärderingspraxis och kunskap i statlig politik och förvaltning (Evaluation practice and knowledge in state politics and administration). Contribution to Nordic Political Science Association’s conference, workshop 14, in Uppsala, August 19-21, 1999 (in Swedish).
77.
Segerholm, C. (2000). Utvärdering—att styra och styras. En granskning av Högskoleverkets utvärdering 1996 av de medellånga vårdutbildningarna (Evaluation—To govern and be governed. A study of the national evaluation of care and nursing programs in 1996 carried out by the Swedish National Agency for Higher Education). Evaluation reports No. 3. Umeå: Umeå Centre for Evaluation Research, Umeå University (in Swedish).
78.
Segerholm, C. (2001a). National evaluations as governing instruments: How do they govern?Evaluation, 7, 427-438.
79.
Segerholm, C. (2001b). Rapport beskrivande utlandsvistelsen och forskningsprojektet “Evaluation praxis and knowledge in national (state) politics and administration” bekostade genom ett postdoktorsstipendium från Humanistiska Samhällsvetenskapliga Forskningsrådet år 2000 (Report on the post-doctoral project “Evaluation praxis and knowledge in national (state) politics and administration” funded by The National Council for Research in the Humanities and Social Sciences). Umeå: Department of Education, Umeå University (in Swedish).
80.
SFS1992:1528. Lagen om offentlig upphandling. Reprinted in 1997 as 1997:1068. The Public Procurement Act (in Swedish).
81.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1995). Foundations of program evaluation: Theories of practice. London: Sage Publications.
82.
SOU1997:09. Flexibel förvaltning. Förändring och verksamhetsanpassning av statsförvaltningens struktur (Flexible administration. Change and adaptation of the structure of the state administration). A Public Commission. Stockholm: Fritzes (in Swedish).
83.
Stake, R. E. (1986). Quieting reform: Social science and social action in an urban youth program. Chicago, IL: University of Illinois Press.
84.
Stake, R. E., & Trumball, D. J. (1982). Naturalistic generalizations. Review Journal of Philosophy & Social Science, VII(1/2), 1-12.
85.
Stronach, I., Halsall, R., & Hustler, D. (2002). Future imperfect: Evaluation in dystopian times. In K. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 167-192). Greenwich, CT: Information Age Publishing.
86.
Stufflebeam, D. L. (2000). Lessons in contracting for evaluations. American Journal of Evaluation, 21, 293-314.
87.
Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 22, 183-209.
88.
Sundin, E. (1995). Utvärdering och legitimitet (Evaluation and legitimacy). In Rombach, B. & Sahlin-Andersson, K. (Eds.), Frånsanningssökandetillstyrmedel.Modernautvärderingarioffentlig sector (From a search for truth to a governing instrument. Modern evaluations in public sectors) (pp. 149-170). Stockholm: Nerenius & Santérus Förlag (in Swedish).
89.
Vedung, E. (1991). Utvärdering i politik och förvaltning (Evaluations in politics and administrations). Lund: Studentlitteratur (in Swedish).
90.
Walker, R. (1997). Public policy evaluation in a centralized state: A case-study of social assistance in the United Kingdom. Evaluation, 3, 261-279.
91.
Weiss, C. H. (1972). Utilization of evaluation: Toward comparative study. In C. H. Weiss (Ed.), Evaluating action programs: Readings in social action and education. Boston, MA: Allyn & Bacon.
92.
Weiss, C. H. (1988). Evaluation for decisions: Is anybody there? Does anybody care?Evaluation Practice, 9, 5-19.
93.
Weiss, C. H. (1991). Evaluation research in the political context: Sixteen years and four administrations later. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century (pp. 211-231). Chicago, IL: National Society for the Study of Education.
94.
Wildavsky, A. B. (1979). Speaking truth to power: The art and craft of policy analysis. Boston, Toronto: Little Brown and Company.
95.
Worthen, B. R., Sanders, J. R., & Fitzpartick, J. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd ed.). New York, NY: Longman.
96.
Åström, E. (1996). Evaluation—The panopticon of pedagogy. In Lärarutbildning och forskning i Umeå (Vol. 4, pp. 61-76).
