The burgeoning performance management movement, with its emphasis on social program ‘results’ measured typically by a limited set of quantitative indicators, has developed a life of its own largely apart from the evaluation research movement. Reflecting the differences in the professional history, interests and training underlying the two movements, the relationship between these disparate approaches to establishing public accountability has lacked coordination and defied integration. This article discusses the basic concepts guiding the evolution of these movements in the context of the goals of information production, and explores the major conceptual, measurement and methodological problems resulting from the lack of accommodation between them. It also provides suggestions about how these two important approaches can be better integrated, both professionally and organizationally, for the purpose of enhancing the reliability and validity of social program assessments, and therefore for improving policy development and program management.
Get full access to this article
View all access options for this article.
References
1.
Blalock, A. B. (ed.) (1990) Evaluating Social Programs.Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.
2.
Blalock, A. B. (1997) ‘The European Commission's Effort to More Effectively Structure Evaluation Activities’, Evaluation Forum12.
3.
Bloom, H. (1993) ‘The National JTPA Study: Origins, Objectives and Issues of Interpretation’, Evaluation Forum9.
4.
Chelimsky, E. (1985) ‘Comparing and Contrasting Auditing and Evaluation’, Evaluation Review9.
5.
Decker, P. (1989) Systematic Bias in Earnings Data Derived from Unemployment Insurance Wage Records and Implications for Evaluating the Impact of Unemployment Insurance Policy on Earnings. Princeton, NJ: Mathematica Policy Research.
6.
Demings, W. E. (1982) Out of the Crisis.Cambridge, MA: Massachusetts Institute of Technology's Center for Advanced Engineering Study.
7.
Duran, P. , E. Monnier and A. Smith (1995) ‘Evaluation a la Française’, Evaluation1(1): 45–63.
8.
European Commission (1995a) Common Guide for Monitoring and Interim Evaluation.Brussels: EC Structural Funds.
9.
European Commission (1995b) MEANS Handbook No.1: Organising Intermediate Evaluation in the Context of Partnerships.Brussels: European Commission.
10.
European Commission (1997) Evaluating European Expenditure Programs: A Guide. Brussels: European Commission.
11.
Friedlander, D. and G. Burtless (1994) Five Years After: The Long-Term Effects of Welfare-to-Work Programs.New York: Russell Sage Foundation.
12.
Funnel, Sue (1997) ‘Program Logic: An Adaptable Tool for Designing and Evaluating Programs’, Evaluation News and Comment5(July).
13.
Gueron, J. and E. Pauly (1991) From Welfare to Work. New York: Russell Sage Foundation.
14.
Hatry, H. and D. Fountain (1990) Service Efforts and Accomplishments Information: Its Time Has Come.Norwalk: Governmental Accounting Standards Board.
15.
Hatry, H. and J. Wholey (1994) Toward Useful Performance Measurement: Lessons Learned from Initial Pilot Performance Plans Prepared Under the Government Performance and Results Act.Washington, DC: National Academy of Public Administration.
16.
Heckman, J. (1993) ‘Randomization and Social Program Evaluation’, in C. Manski and G. Garfinkel (eds) Evaluating Welfare and Training Programs. Cambridge, MA: Harvard University Press.
17.
Heckman, J. (1999) Performance Standards in a Government Bureaucracy. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.
18.
Heckman, J. and V. J. Hotz (1989) ‘Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs’, Journal of the American Statistical Association84.
19.
Hotz, V. J. (1992) ‘Designing an Evaluation of the Job Training Partnership Act’, in C. Manski and G. Garfinkel (eds) Evaluating Welfare and Training Programs.Cambridge, MA: Harvard University Press.
20.
Kettl, D. (1997) The Global Revolution in Public Management.Washington, DC: The Brookings Institution.
21.
Kettl, D. , P. Ingraham, R. Sanders and C. Horner (1996) Civil Service Reform: Building a Government That Works.Washington, DC: The Brookings Institution.
22.
King, C. T. and R. E. McPherson (1997) Evaluation Plan for the Texas Workforce Development System. Austin, TX: Center for the Study of Human Resources, University of Texas.
23.
Leeuw, F. L. and P. J. Rozendal (1994) ‘Policy Evaluation and the Netherlands Government: Scope, Utilization and Organizational Learning’, in F. L. Leeuw (ed.) Can Governments Learn?New Brunswick, NJ: Transaction Publishers.
24.
Moran, William C. (1990) ‘Evaluation Within the Federal Offices of Inspectors General’, New Directions in Program Evaluation48.
25.
Muller-Clemm, W. J. (1995) ‘A Historical Perspective on Federal Program Evaluation in Canada’. Paper given at the International Evaluation Research conference in Vancouver, British Columbia.
26.
National Academy of Public Administration (1997) Improving Performance/Improving Government.Washington, DC: National Academy of Public Administration.
27.
National Governors' Association (1994) Building State Workforce Development Systems: The Critical Roles of Policy Coordination and Quality Assurance.Washington, DC.
28.
National Performance Review (1994/1995) From Red Tape to Results: Creating a Government that Works Better and Costs Less; Putting Customers First: Standards for Serving the American People; and Common Sense Government. Washington, DC.
29.
Office of the Governor of North Carolina (1998) Report of the Governor's Commission on Workforce Preparedness. Raleigh, NC: Office of the Governor of North Carolina.
30.
Office of the Governor of Oregon (1998) Report of the Governor's Office of Education and Workforce Policy. Salem, OR: Office of the Governor of Oregon.
31.
Organization for Economic Cooperation and Development (1991) Evaluating Labour Market and Social Programmes: The State of the Art.Paris: OECD.
32.
Organization for Economic Cooperation and Development (1995) Governance in Transition: Public Management Reforms in OECD Countries.Paris: OECD.
33.
Organization for Economic Cooperation and Development (1997) Benchmarking, Evaluation and Strategic Management in the Public Sector. Paris: OECD.
34.
Osborne, D. and T. Gaebler (1992) Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector.Reading, MA: Addison-Wesley Press.
35.
Patton, M. (1990) Qualitative Evaluation and Research Methods.Thousand Oaks, CA: Sage Publications.
36.
Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage Publications.
37.
Pollit, C. (1998) ‘Evaluation in Europe’, Evaluation4(2): 214–224.
38.
Rahn, M. , E. Hoachlander and K. Levesque (1992) State Systems for Accountability in Vocational Education.Washington, DC: National Center for Research in Vocational Education, US Department of Education.
39.
Rossi, P. and H. Freeman (1994) Evaluation: A Systematic Approach.Thousand Oaks, CA: Sage Publications.
40.
Sanders, J. (1994) The Program Evaluation Standards.Thousand Oaks, CA: Sage Publications.
41.
Schmid, Gunther (1997) ‘The Evaluation of Labour Market Policy’, Evaluation3(4): 409–434.
42.
Scott, G. , I. Bell and T. Dale (1997) ‘New Zealand's Public Sector Management Reform: Implications for the US’, Journal of Policy Analysis and Management16(3).
43.
Smith, P. (1995) ‘On the Unintended Consequences of Publishing Performance Data in the Public Sector’, International Journal of Public Administration18.
44.
Stevens, D. (1989) Using State Unemployment Insurance Wage Records to Trace the Subsequent Labor Market Experiences of Vocational Education Program Leavers. National Assessment of Vocational Education. Washington, DC: US Department of Education.
45.
Toulemonde, J. , C. Fontaine, E. Landren and P. Vinche (1998) ‘Evaluation in Partnership’, Evaluation4(2): 171–188.
46.
US Congress (1994) Performance Standards for the Food Stamp Employment and Training Program.Washington, DC: Office of Technology Assessment.
47.
US Department of Labor (1997) Core Data Elements and Common Definitions for Employment and Training Programs, Washington, DC: US Department of Labor.
48.
US General Accounting Office (1992) Program Performance Measures.Washington, DC: US Congress.
49.
US General Accounting Office (1993a) Using Performance Measures in the Federal Budget Process.Washington, DC: US Congress.
50.
US General Accounting Office (1993b) Measuring Performance and Acting on Proposals for Change.Washington, DC: US Congress.
51.
US General Accounting Office (1994) Managing for Results: State Experiences Provide Insights for Federal Management Reforms. Washington, DC: US Congress.
52.
US General Accounting Office (1995a) Government Reform: Goal-Setting and Performance.Washington, DC: US Congress.
53.
US General Accounting Office (1995b) Managing for Results: Experiences Abroad Suggest Insights for Federal Management Reforms.Washington, DC: US Congress.
54.
US General Accounting Office (1997a) Measuring Performance: Strengths and Limitations of Research Indicators. Washington, DC: US Congress.
55.
US General Accounting Office : (1997b) GPRA – Managerial Accountability and Flexibility Pilot Did Not Work As Intended. Washington, DC: US Congress.
56.
US General Accounting Office (1997c) Managing for Results: Analytic Challenges in Measuring Performance.Washington, DC: US Congress.
57.
US General Accounting Office : (1997d) Agencies' Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review.Washington, DC: US Congress.
58.
Volcker, P. (1989) Leadership for America: Rebuilding the Public Service.Washington, DC: The Federal Reserve.
59.
Wholey, J. K. (1989) Improving Government Performance.San Francisco, CA: Jossey-Bass.
60.
Wirt, J. G. (1995) Performance Assessment Systems: Implications for a National System of Skill Standards.Washington, DC: National Governors' Association.
61.
Zornitsky, J. and M. Rubin (1995) Establishing a Performance Management System for Targeted Welfare Programs.Washington, DC: National Commission for Employment Policy.
