The NHS reforms in England: four challenges to evaluating success and failure

Introduction

Since 1948, the National Health Service (NHS) has undergone repeated organisational changes, ¹ but none so large or complex as the 2012 Health and Social Care Act (HSCA). Its stated purpose was to liberate the NHS from direct ministerial control, transferring responsibility to the independent body, NHS England, to deliver on an annual mandate from ministers. NHS England oversees local Clinical Commissioning Groups (CCGs), which were intended to allow general practitioners to decide on the commissioning of certain services from ‘any qualified provider’. An extensive network of new entities was created to facilitate this, such as Clinical Senates and Health and Wellbeing Boards, while some existing bodies had their powers enhanced, such as the economic regulator Monitor.

The resulting NHS system is extremely complex, with many uncertainties about how it should work and examples of a reality which differs from stated intentions. For example, the newly ‘liberated’ head of NHS England must now meet with the Secretary of State weekly, so that direct ministerial control continues.

Whenever large-scale reforms take place, they should be accompanied by robust evaluation to inform future policy. The latest change to the English NHS has been described as ‘so large you could probably see it from space’, ² but despite this we may never know its true impact on population health.

Previous NHS reforms illustrate the challenges involved in the HSCA’s evaluation.^3,4 A first challenge is political. Politicians resisted proposals to evaluate the introduction of the ‘internal market’ in the 1990s, so its impacts remain debated today. The second challenge is operational. New Labour committed funding to policy evaluations that improved understanding of processes by which change occurred, but failed to incorporate evaluation into actual policy design and implementation, so limiting the validity of their conclusions. This was again the case with the HSCA, as piloting CCGs or other elements of the reform was excluded, so precluding randomised designs which could permit causal inferences.

In 2013, the Department of Health (DH) called for research to evaluate the HSCA’s impact, issuing a second call when DH found submitted bids left ‘significant gaps’ and would not allow ‘an assessment of impact across the whole suite of reforms’. ⁵ Both calls asked for evidence of improved outcomes, with the first call stating: ‘All projects are expected to propose an appropriate explanatory framework or counterfactual, where this is feasible, against which the effects of the reforms can be evaluated and attributed’. ⁶

In this paper we ask, how would we know if the NHS reforms are working or not? We evaluate characteristics of the HSCA that fulfil ‘natural policy experiment’ criteria and propose alternative feasible designs.

The HSCA: a natural policy experiment?

Where a randomised trial is not practicable, as in the case of the HSCA, it may be possible to treat the intervention as a ‘natural experiment’. ⁷ The crucial characteristic is that intervention exposure is determined by an event outside researchers’ control. ⁷ Recent MRC guidance highlights how natural experiments can be robustly evaluated but success is more likely in specific circumstances – a rapidly implemented intervention which leads directly to substantial health changes that are, therefore, unlikely to be the result of other confounding factors. ⁷ Examples include the impacts of legislation for smoke-free public places on pregnancy-related complications or pesticide regulations on suicides.^8,9 This is clearly not the case with the HSCA given its complexity. There are at least four substantial methodological challenges that must be overcome to assess whether the HSCA succeeded in meeting its objectives.

Challenge 1: defining the intervention – what are the reforms to be evaluated?

There is a gap between stated policy and its intended consequences, sometimes referred to as ‘law on the books’ and ‘law on the streets’. ¹⁰ Here the differences are especially large due to confusion about what the HSCA actually requires in particular circumstances and the need for pragmatic solutions when its requirements suggest a course of action contrary to its stated goals. A further complication is that commentators have portrayed the HSCA as both a major health system change and a continuation of New Labour policies. The Private Finance Initiative (PFI), Foundation Trusts and Payment by Results were all introduced prior to the coalition government, reflecting increasing use of market mechanisms within the NHS. ¹ Indeed, some problems arising since the HSCA’s implementation, such as the finding that the Secretary of State had acted unlawfully in proposing service reductions at Lewisham Hospital, are primarily a consequence of unaffordable debts incurred in unwise PFI schemes. ¹¹ These gradual steps towards increasing market forces mean the most recent reforms are less likely to result in sudden measurable changes.

Challenge 2: defining the outcomes – what are the outcomes that are likely to change?

The goals of the NHS have been prespecified by the government. The HSCA places a duty on the Secretary of State, the NHS Commissioning Board and CCGs to ensure continuous improvement in quality of NHS services. Its Outcomes Framework operationalises this, categorising outcomes within five domains (Table 1). ¹² The Framework recognises the importance of organisations other than the NHS in contributing to health, with some indicators complementing those within equivalent Outcomes Frameworks for Public Health and Adult Social Care. It would seem intuitive that an evaluation of the HSCA should include achievement of these goals. Yet, despite the professed focus on outcomes, the DH research call has prioritised processes arising from the HSCA, such as how commissioning occurs (Table 2), with no clear theory of how they link to the Outcomes Framework.

Table 1.

The NHS Outcomes Framework.

	Domains	Overarching indicators	Examples of ‘improvement areas’
1	Preventing people from dying prematurely	Potential Years of Life Lost (PYLL) from causes considered amenable to healthcare Life expectancy at 75 years	• Reducing premature mortality from the major causes of death • Reducing premature death in people with serious mental illness
2	Enhancing quality of life for people with long-term conditions	Health-related quality of life for people with long-term conditions	• Ensuring people feel supported to manage their condition • Improving functional ability in people with long-term conditions
3	Helping people to recover from episodes of ill health or following injury	Emergency admissions for acute conditions that should not usually require hospital admission Emergency readmissions within 30 days of discharge from hospital	• Improving outcomes from planned treatments • Preventing lower respiratory tract infections (LRTI) in children from becoming serious
4	Ensuring that people have a positive experience of care	Patient experience of primary care Patient experience of hospital care Friends and family test	• Improving people’s experience of outpatient care • Improving hospitals’ responsiveness to personal needs • Improving access to primary care services
5	Treating and caring for people in a safe environment; and protecting them from avoidable harm	Patient safety incidents reported Safety incidents involving severe harm or death Hospital deaths attributable to problems in care	• Reducing the incidence of avoidable harm • Improving the safety of maternity services • Delivering safe care to children in acute setting

Adapted from the NHS Outcomes Framework 2013/2014. ¹²

Table 2.

Key priority areas for research, as identified by the Department of Health.

Thematic research area	Key research questions identified by the Department of Health
The new commissioning system	Has the quality of commissioning improved?
	What is the role and effects of each part of the new commissioning system, and how do they work together?
	To what extent are the changes in the commissioning system resulting in changes in services and care pathways that benefit patients?
The provider landscape and its regulation	What effects do the changes in the system have on providers’ behaviour?
	Are existing providers becoming more sustainable?
	To what extent are providers using their freedoms, and introducing innovation?
	What is the role of competition, failure, market entry and market exit in the provider landscape?
Mechanisms for improving the integration of care	Is the new system joining up services effectively, within and across health and care organisations and sectors, and looking more widely at other organisations that affect health and care outcomes?
	How do the organisations in the new system plan and commission together to deliver more integrated care, and with what results?
Patient empowerment	To what extent are patients taking more control over decisions about their care, and what are the effects of this?
	Is self-directed support happening?
	To what extent are patients using health information about themselves to take more control over decisions about their care?
Patient and public involvement	To what extent are organisations across the system taking account of views and feedback from the public, and how far is this influencing their decisions?
	What methods are adopted for ensuring public involvement is effective and beneficial?
Information, quality and outcomes	How do different organisations in the new system focus on outcomes, especially with respect to the outcomes frameworks?
	Are outcomes being used to measure performance and to drive improvement, and how is this happening?
	How are other sources of information about quality, particularly effectiveness, safety and patient experience, being used?
Autonomy and accountability	To what extent are organisations at each level of the new system free to make their own decisions without interference from elsewhere?
	Where do power and accountability lie in the new system?

Adapted from the Department of Health’s invitation for research tenders. ⁶

The Outcomes Framework is problematic for at least four reasons. First, any changes may be confounded by pre-existing trends occurring as a result of previous policy changes. Second, data quality may vary as a direct consequence of the reforms. For example, an increase in private sector service delivery may lead to spurious problems with existing NHS datasets (such as hospital episode statistics), as happened under the last government when Independent Sector Treatment Centres failed to meet contractual obligations to supply patient data. ¹³ Third, many prespecified outcomes that are most likely to be sensitive to change, such as patient satisfaction with general practice, were not collected routinely and reliably pre-HSCA. Fourth, of the objective outcomes available in routine statistics covering pre and post HSCA, several have well-known limitations. Mortality amenable to healthcare is simply an indicator of the need for further investigation. ¹⁴ The meaning of admissions theoretically avoidable (e.g. for diabetic ketoacidosis) as an indicator of the quality of primary care is similarly debated. ¹⁵

Even if the Outcomes Framework is set aside and the focus remains on processes listed in the revised call, challenges remain. It is unclear how to measure commissioning quality; nor is it evident where baseline data could be obtained. Indeed, the explicit request for before and after comparison demands many data elements which are unlikely to exist at baseline. The question of whether ‘existing providers [are] becoming more sustainable?’ is unknowable except with the benefit of hindsight. The fact that they continue to exist shows they have been sustained until now but whether that will continue can only be speculation. The answer to the question ‘is self-directed support happening’ is almost certainly yes, but this simply begs the question of what the answer actually means.

Challenge 3: defining the lag time – when will the reforms have an effect?

Establishing the intervention’s start is challenging. Some aspects of the HSCA were implemented before legislation was enacted while others have been delayed by those seeking to minimise the scale of change.

The HSCA is anticipated by both advocates and critics to impact on NHS performance but the time taken to see changes is unclear and will differ by outcome. It is important to distinguish short- and long-term effects. Major organisational changes, such as hospital mergers, are known to set back performance by several years, leading to short-term adverse effects. ¹⁶ On the other hand, some outcomes may take several years to become apparent. Moreover, any changes may not be uni-directional. There are already several examples of large outsourcing corporations pulling out of contracts as they recognise they are unable to make adequate profits while meeting agreed standards. A comprehensive evaluation should capture all of these effects, including those associated with the transition, as they represent a genuine impact of the HSCA’s introduction.

Challenge 4: defining the counterfactual – what would have happened if the reforms were not introduced?

The fundamental question is ‘what would have happened without the HSCA?’ To answer it, we need to find a ‘control’ group, unexposed for a period of time due to lags in implementation or limitations in the HSCA’s coverage.

One possibility is a pre-/post-study design comparing outcomes. Yet measures intended to capture healthcare quality (like amenable mortality) are closely correlated to overall health measures, so trends may reflect overall changes in determinants of health, rather than health policy. For example, the coalition government responded to the economic crisis with severe austerity, introducing changes to areas such as welfare and regional development – all with major health consequences. ¹⁷

Another possibility would be to compare English regions based on differential implementation timing. This is problematic because it may be impossible to measure HSCA implementation robustly, and because outcomes may systematically vary by local context, for example between rural areas and large urban centres.

These problems could be mitigated by comparing England with Scotland or Wales. This would allow outcomes for those unexposed to the HSCA to be observed while also taking some account of secular trends, arising from broader public policy and macro-economic changes. The analysis of individual-level data, incorporating adjustment for baseline health outcomes and theoretically informed confounding variables for geographically matched areas, exemplifies how such an analysis might work. ¹⁸

Yet, in the DH’s second research call, they specifically exclude the one comparative approach most likely to shed light on the HSCA’s causal effects:

Consideration has been given to the value of comparisons with the devolved administrations. A number of proposals in the first call intended to include comparisons with Scotland and Wales. However, given the concomitant reforms in Scotland and Wales, policy officials are not convinced that any significant differences in outcomes between England and the other devolved administrations could be attributed to a specific reform (such as changes to commissioning) in England. Instead, the preferred approach would be to focus is [sic] on whether the reforms have had an impact in England over time, including baseline measurement. ⁵

Although a recent comparison of health systems in the four countries of the UK shows this approach is not a panacea, with many practical problems including non-compatibility of data, ¹⁹ it seems perverse to exclude one of the most promising methods, and the only one that allows monitoring of long-term divergence between UK health systems.

Development of such methods could focus on identifying the impacts of specific components of the reforms to yield partial assessments of the HSCA’s impacts. Numerous challenges remain including multiple divergences in policy, differing populations and difficulties in obtaining comparable data (which will likely increase for both political and practical reasons over time).

Conclusions

The NHS reforms in England are expected to have far-reaching and long-term consequences for population health and equity. Despite this, we may never know their true impacts. This is disappointing, given the uncertainty about impacts of earlier reforms and the repeated lesson that evaluation must be integrated into policy design and implementation. The HSCA’s introduction has not been straightforward, and it is conceptually difficult to characterise. High-quality data with outcomes that are sensitive to change are lacking. The evaluation research being commissioned by DH is unlikely to allow the HSCA’s impacts to be assessed robustly, despite that being their stated aim. There are alternatives, such as a step-wise roll out across regions, with those included at each step selected at random, as was done with Seguridad Popular in Mexico. ²⁰ Given where we are now, comparing specific components of health system performance across countries of the UK, based on an explicit theory of change, seems the most promising way forward yet is explicitly rejected.