Evaluating the evaluation: Understanding the utility and limitations of evaluation as a tool for organizational learning

Abstract

Objective Organizational learning, underpinned by evidence-based health care, and greater user involvement in planning and delivery were key objectives of the NHS Plan. Evaluation, and specifically participatory evaluation, offers the potential to address these elements of the NHS modernization agenda. We discuss the strengths and limitations of evaluation in delivering the modernization agenda through use of a case study — the evaluation of a Healthy Living Centre project, catchon2us!

Setting A Healthy Living Centre project in Merseyside.

Results There was evidence that collaborative research can promote opportunities which foster significant learning and change, thus making the effort and time involved worthwhile. However, the two-way flow of information necessary for development of shared goals and learning at strategic, as well as provider, levels is not easily achieved. Barriers include the rigidity of organizational structures within large agencies such as the NHS, with priorities imposed from national levels overriding local priorities.

Conclusions The inherent contradictions in current strategic drivers in the NHS need to be addressed if services can ever deliver the goal of true organizational learning.

Keywords

evaluation Healthy Living Centres NHS modernization organizational learning

Get full access to this article

View all access options for this article.

References

Department of Health. The NHS Plan. A. Plan for Investment. A Plan for Reform. London: The Stationary Office, 2000.

Nutley SM , Davies Hto. Developing organisational learning in the NHS. Medical Education , 2001: 35: 35—42.

Chief Medical Officer. An Organisation with a Memory. London: The Stationery Office, 2000.

Dash P. , Gowman N. , Traynor M. Increasing the impact of health services research. British Medical Journal, 2003: 327: 1339—41.

Department of Health. Good Practice Guide: Learning Lessons from Post-Project Evaluation. London: Department of Health, 2002. Online. Available: http://www.dh.gov.uk/assetRoot/04/02/10/59/04021059.pdf (accessed 14 December 2005).

Frewer L. , Rowe G. , Marsh R. , Reynolds C. Public Participation Methods: Evolving and Operationalising an Evaluation Framework. London: Department of Health, 2000. Online. Available: http://www.dh.gov.uk/assetRoot/04/07/61/92/04076192.pdf (accessed 14 December 2005).

St Leger AS , Walsworth-Bell JP Change Promoting Research for Health Services. Buckingham : Open University Press, 1999.

Stufflebeam DL Evaluation models. New Directions for Evaluation, 2001: 89: 7—98.

Connell JP , Kubisch AC , Schorr LB , Weiss CH New Approaches to Evaluating Community Initiatives Volume 1: Concepts, Methods and Contexts. The Aspen Institute, 1995 .

10.

Fetterman DM In response. Evaluation Practice , 1995: 16(2): 179—99.

11.

Brunner I. , Guzman A. Participatory evaluation: A tool to assess projects and empower people. International Innovations in Evaluation Methodology, 1989: 42: 9—17.

12.

Folkman DV , Kalyani R. Reflections on facilitating a participatory community self-evaluation. Evaluation and Program Planning, 1997: 20(4): 445—65.

13.

Springett J. Practical Guidance on Evaluating Health Promotion. On behalf of the WHO-Euro Working Group on the Evaluation of Health Promotion. WHO Regional Office for Europe, 1999.

14.

Kolb DA Experiential Learning: Experience as a Source of Learning and Development . Englewood Cliffs, NJ: Prentice Hall, 1984.

15.

Springett J. Appropriate approaches to the evaluation of health promotion. Critical Public Health, 2001: 11 (2): 139—51.

16.

Nusbaumer MR

The reification problem in evaluation research: The case of the public health model of alcohol problems prevention. Journal of Applied

Sociology, 1988: 5: 41—9.

17.

Stufflebeam DL Empowerment evaluation, objectivist evaluation and evaluation standards: Where the future of evaluation should not go and where it needs to go. Evaluation Practice, 1994: 15(3): 321—38.

18.

Ashton JR The need for a strategic underview. Journal of Epidemiology and Community Health, 2003: 57: 808.

19.

Brunner I. , Guzman A. Participatory evaluation: A tool to assess projects and empower people. International Innovations in Evaluation Methodology, 1989: 42: 9—18.

20.

NHS Executive. Health Services Circular 1998/008. Healthy Living Centres. London: Department of Health, 1999.

21.

New Opportunities Fund. Healthy Living Centres Programme Evaluation

New Opportunities Fund, 2001. Online. Available: http://www.nof.org.uk/default.aspx?tc=264&tct=1 (accessed 14 December 2005).

22.

The World Bank Group. Social Capital for Development. Online. Available: http://www.1.worldbank.org/prem/poverty/scapital/whatsc.htm (accessed 14 December 2005).

23.

Bruner J. The Culture of Education. Cambridge, MA: Harvard University Press, 1996.

24.

Winch C. Quality and Education. Oxford: Blackwell Publishers and The Journal of the Philosophy of Education Society of Great Britain, 1996.

25.

Silverman D. Doing Qualitative Research. A Practical Handbook. London : Sage, 2000.

26.

26 Reeve J. Evaluation report for the Healthy Living Centre catchon2us! November 2002 (available from the authors).

27.

Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards. Thousand Oaks, CA: Sage, 1994.

28.

Berkeley D. , Springett J. From rhetoric to reality: Barriers faced by Health For All Initiatives. Social Science & Medicine, 2006: 63(1): 179—88.

29.

Taut S. , Brauns D. Resistance to evaluation. A psychological perspective. Evaluation , 2003: 9(3): 247—64.

30.

Al Yafai F. Failing Health. The Guardian, 23 July 2003 . Online.Available: http://politics.guardian.co.uk/publicservices/story/0,,1003789,00.html (accessed 14 December 2005).

31.

Gosling P. Unhealthy futures for Healthy Living Centres. Public Health News, 10 March 2006.

32.

Denis JL , Lomas J. Convergent evolution: The academic and policy roots of collaborative research . Journal of Health Services Research and Policy, 2003: 8(2): 1—6.