Making the most of evaluation: A mixed methods study in the English NHS

Abstract

Objectives:

To increase understanding about how evaluations of health policy initiatives are commissioned, conducted and used.

Methods:

A mapping exercise was conducted to identify evaluation of initiatives promoted by the White Paper ‘Our health, our care, our say’ in the English NHS. All evaluations were subjected to critical appraisal and 21 were purposively selected as case studies, involving documentary analysis and 60 interviews with those commissioning, conducting and affected by the evaluation.

Results:

Variation in the types of evaluation being undertaken did not reflect the importance of the initiatives being evaluated. Most evaluations collected evidence about uptake, processes of care and users’ perceptions. While some provided useful information about how initiatives could be improved, few provided robust evidence about the benefits or costs of the initiatives. Those who commissioned evaluations had similar concerns to those who conducted them. There was a commitment to the concept of evaluation but little clarity about how findings would be used. Evaluation was often commissioned too late to influence decisions about implementation. Compromises over research design and difficulties collecting data limited the potential to provide robust evidence about benefits. There were tensions between the desire of evaluators for methodological rigour and the needs of service providers for swift, contextually relevant findings. There were concerns about the transparency of methods and results.

Conclusions:

Considerable public resources are committed to evaluation, but this investment is less productive than it could be. This article specifies several ways in which the use of evaluation of initiatives in health and social care could be improved.