Abstract
Objectives
This study assessed whether the 14 National Health Service (NHS) acute trusts reviewed by Sir Bruce Keogh in July 2013 were performance outliers on the Acute Trust Quality Dashboard (ATQD) and examined whether high mortality indices results are associated with increased numbers of quality risk alerts and alarms.
Design
The proportion of dashboard indicators rated amber or red in the Keogh Review trust group was compared with that for all trusts. Quality performance at primary care level was assessed for lead Clinical Commissioning Groups (CCG) of Keogh Review trusts through the national General Practice High Level Indicators.
Setting
England.
Participants
The study examined the performance of all acute NHS trusts in England and the performance of the lead CCGs for all acute NHS trusts included in the Keogh Review.
Main outcome measure
The primary outcome was the proportion of alerts and alarms identified in the summer 2013 ATQD.
Results
The mean proportion of indicators scored as alert or alarm across all trusts was 14.1% (13.5–14.7%). For Keogh Review trusts, the mean was 14.5% (12.6–16.4%). There was no significant difference between Keogh Review trusts and all acute trusts in numbers of alerts or alarms or for lead CCG quality performance.
Conclusion
This study finds no evidence to suggest the trusts reviewed by Sir Bruce Keogh in spring 2013 were outliers for overall quality of care as defined by the ATQD. The use of mortality indices alone to identify potential quality outliers in secondary care may not be sufficient.
Introduction
Contemporary UK healthcare providers operate in an environment characterised by systematised scrutiny and target-based performance management. The current hegemony of central performance management of National Health Service (NHS) bodies dates back to the Thatcher Government of the 1980s. The Griffiths Report of 1983 recommended a system of general management in NHS hospitals and systematic performance monitoring of NHS hospitals be introduced in the hope that comparison between hospital units would drive managers to improve services. 1 The 1980s and 1990s saw the introduction of outcome measurement as a key element of performance monitoring and these outcomes were viewed as the ‘ultimate measure of quality of care’. 2 The White Paper ‘Equity and Excellence: Liberating the NHS’, which underpins the current organisation of the NHS, is explicit in setting a ‘relentless focus on clinical outcomes’ as the prime measurement of success. 3
In July 2013, the Department of Health published a review by the National Medical Director for the NHS, Sir Bruce Keogh, into the quality of care provided by 14 acute NHS trusts in England. 4 These trusts had been identified as having persistently high mortality rates; defined through being consistent statistical outliers for the previous two years on either the Hospital Standardised Mortality Ratio (HSMR) or the Summary Hospital-level Mortality Indicator (SHMI). Recent review (January 2014) identified six further instances of trusts being mortality ‘outliers’ using more recent data. 5 The Francis Report into failings of care at NHS Mid Staffordshire NHS Foundation Trust 6 had previously identified high mortality rates as being associated with other negative outcomes for clinical effectiveness, patient experience and safety. The Keogh Report found that ‘understanding mortality … is much more complex than studying a single hospital-level indicator’(4,p16) and that causes of high mortality are likely to be found in a combination of factors rather than in a single rogue surgeon or specialty.
The capability of hospital-wide mortality rates to identify avoidable deaths has been questioned elsewhere. In a retrospective case record review of 10 English acute hospitals, Hogan et al. 7 found the proportion of preventable deaths to be much lower than previous estimates and concluded that hospital deaths may not be a good indicator of hospital quality. There are also methodological issues with hospital mortality as a quality measure. From the United States, Shahian et al. 8 reported that when various methods to calculate hospital mortality were employed, they produced very different results. The authors question the utility of hospital deaths on their own as a quality measure.
In October 2013, the Care Quality Commission (CQC) announced the introduction of a new multi-factorial system of ‘intelligent monitoring’ of acute hospitals based on the analysis of more than 150 different performance indicators to inform prioritisation of inspection. 9 The results of this monitoring have enabled the CQC to group trusts into six quality risk bandings, with Band 1 containing those at highest risk of negative quality outcomes and Band 6 trusts having the lowest risk.
Prior to this recent development, in 2012 the East Midlands Quality Observatory developed an Acute Trust Quality Dashboard (ATQD) 10 in order to provide benchmarking information for trusts on a range of care quality indicators. The quality indicators use data from a wide variety of sources and use the most recent available data for each indicator. Appendix 1 provides a list of indicators included in the quality dashboard. Trust-specific reports are produced quarterly with results for each indicator presented in comparison to all other acute trusts with time trend analysis covering the previous seven quarters included. Statistical Process Control principles are used to indicate adverse and favourable variation from the mean for each indicator. 9 The dashboards are currently produced and published by Methods Insight Analytics. 11
The ATQD indicators are grouped into six domains in line with the NHS Outcomes Framework:
Preventing people from dying prematurely; Enhancing quality of life for people with long-term conditions; Helping people to recover from episodes of ill health or following injury; Ensuring that people have a positive experience of care; Treating and caring for people in a safe environment and protecting them from avoidable harm; Organisational context.
The ATQD include both an overall SHMI measure and SHMI measures for eight specific conditions to measure mortality and these are included in Domain 1.
Further, primary care dashboards have been developed by NHS England12,13 to assess the performance of general practices through a set of General Practice High Level Indicators (GPHLIs). Performance can be reviewed at both individual practice level and Clinical Commissioning Group (CCG) level. We accessed this through the Primary Care Web Tool 12 for approved users hosted by NHS England. Appendix 2 provides a list of available indicators.
This study aims to identify whether the 14 trusts reviewed by Professor Keogh are performance outliers on the ATQD and to examine whether high mortality indices results are associated with high numbers of alerts and alarms in the dashboard.
Method
The primary outcomes addressed in this study are the number and proportion of alerts and alarms identified in the Summer 2013 ATQD. Alerts are defined as any result showing an adverse difference of between 2 and 3 standard deviations from the mean (presented as amber results in the dashboard). Alarms are defined as any result showing an adverse difference of over 3 standard deviations from the mean (presented as red results). The number and proportion of green rated (favourable) results were also calculated for each domain.
The summer 2013 dashboard results for 130 acute NHS trusts in England were accessed from the Methods Insight Analytics website. 11 This dashboard was selected so as to be coterminous with the period of Keogh Review, with the majority of indicators covering either January to March 2013 or being an annual figure for April 2012 to March 2013. Results were collated for each of the 92 indicators, where data were analysed on a quality judgement basis and where trusts had made data available for the dashboards. Data completeness for all trusts ranged from 94.6% to 100% for this specific set of indicators.
For each trust, indicator results were grouped into four categories: better than expected (green), as expected (grey), alert (amber) and alarm (red). The proportions of green, grey, amber and red results were calculated for each trust with trusts being ranked by proportion of amber and red results. These rankings were calculated both for the total dashboard indicator set and for specific dashboard domains. Results for Domain 2 (Enhancing quality of life for people with long-term conditions) were excluded from analysis as there were only three indicators against which quality judgements are made.
The proportion of all indicators rated amber or red in the Keogh Review trust group was compared with the proportion for all trusts with 95% confidence intervals calculated using
To assess primary care performance for the locality of each Keogh Review acute trust, lead CCGs were identified through the NHS Choices website. 14 Review of the 2013 GPHLI results enabled calculation of the proportion of general practices with five or more adverse outlier results within each CCG. Outlier results were defined as being adverse scores of at least 2 standard deviations from the national mean.11,12 The results for this performance indicator were then compared for each Keogh Review trust CCG lead with the national mean.
Results
The results for the 130 acute trusts analysed through the ATQD for summer 2013 are shown in Figure 1. They are ranked by the proportion of amber or red indicators in that quarter's dashboard, with the highest proportion (and theoretically worst performing trusts) on the right hand side of the chart. Results for the 14 trusts reviewed by Sir Bruce Keogh are highlighted in the chart. The results show no clear association between inclusion in the Keogh review sample and negative comparative performance in the ATQD at the time of the Keogh review. Results for the Keogh Review trusts are distributed between the second and tenth tertiles for proportion of alerts and alarms. In every case, trusts achieved green rated results for at least seven of the 88 to 92 indicators where data were provided and rated.
Ranked results for all acute trusts in England for the proportion of combined amber and red, and green rated results in the Summer 2013 Acute Trust Quality Dashboard.
Sum and proportion of red or amber or green results across all domains of the Acute Trust Quality Dashboard for Keogh Review trusts compared with all acute trusts in England, Summer 2013.
Source: Acute Trust Quality Dashboards (Summer 2013), Methods Analytics.
The mean proportion of indicators scored either amber or red across all trusts was 14.1% (95% CI 13.5–14.7%). For the Keogh Review trusts, the mean proportion was 14.5% (95% CI 12.6–16.4%). There was no significant difference between Keogh Review trusts and all acute trusts in proportion of measures showing alert or alarm. When red ratings (indicating alarm) were analysed in isolation, there was likewise no statistically significant difference between the two groups, the proportions being 10.3% (95% CI 9.7–10.9%) for all acute trusts and 10.9% (95% CI 9.2–12.6%) for Keogh Review trusts.
Similarly, there is no significant difference in the proportion of green rated indicators scored for Keogh Review Trusts and that achieved by all acute trusts. For all acute trusts, the proportion of green results was 22.3% (95% CI 21.5–23.1%), whereas for the Keogh Review trusts, it was 21.1% (95% CI 18.9–23.3%).
Sum and proportion of red or amber or green results in specific domains of the Acute Trust Quality Dashboard for Keogh Review trusts compared with all acute trusts in England, Summer 2013.
Source: Acute Trust Quality Dashboards (Summer 2013), Methods Analytics.
For primary care performance, there was no statistically significant difference between the lead CCGs for Keogh Review trusts and the national CCG mean for the proportion of general practices with five or more adverse outlier results in the 2013 GPHLI dashboards. 11 The mean proportion of practices with five or more outlier quality scores was 4.6% (27 of 585) for Keogh Review trusts while the national mean was 6.6%.
Discussion
Statement of principal findings
When the clinical and organisational quality indicators included in the ATQD are analysed, the results suggest that adverse mortality results are not adequate markers of overall poor quality as defined in terms of the dashboard. Only two of the 14 Keogh Review trusts rank in the top quartile for proportion of indicators rated as alerts or alarms and six of the 14 have a proportion of favourably rated indicators above the median for all trusts.
Our analysis did not find any statistically significant differences between the Keogh Review group and the overall acute trust cohort for proportions of alert or alarm rated indicators for patient experience (Domain 4), safety (Domain 5) or workforce concerns (Domain 6).
There are numerous indicators of clinical and operational effectiveness distributed across multiple domains. As such, results for this key theme are more difficult to summarise within the ATQD. However, Domain 3 concentrates specifically on clinical and operational effectiveness, and it is noteworthy that the proportion of adverse scores was significantly lower in the Keogh Review trusts than the full trust cohort. The indicators contained in Domain 3 relate to emergency readmission, mean length of stay and patient reported outcome measures (PROMs). The results for these indicators suggest Keogh Review trusts were neither excessively risk averse nor premature in agreeing discharge dates for patients and patient reported outcomes were comparatively positive. Indeed, only one Keogh Review trust achieved an adverse score for a PROMs indicator. We are unable to identify a plausible reason for an association between a statistically significant favourable outcome for clinical and operational effectiveness and significantly adverse mortality beyond either chance or difference in data collection practices.
The results for primary care performance for the lead CCGs for Keogh Review trusts suggest that there is no significant difference in the quality of care provided at general practice level in the catchment areas for these trusts and that provided elsewhere. There was no preponderance of adverse outlier general practices in the Keogh Review trust localities, and this analysis provides no evidence of association between secondary care mortality indices and primary care quality as defined by this particular summary quality indicator.
The results of the CQC quality risk banding analysis published in October 2013 continue to show that 11 of the Keogh Review trusts are in Band 1 (indicating highest risk to care quality) with two more being in Band 2. 9 However, this could be viewed as inevitable as one criterion for inclusion in Band 1 is a trust being subject to ‘special measures’, an outcome associated with inclusion in the Keogh Review.
Strengths and weaknesses of the study
This study provides comparison for all NHS acute trusts in England. Data completeness for the ATQD in the study period was high (94.6–100%), and sparklines provided in the ATQD suggest that the results for the period were not markedly different than for the quarters directly preceding the period.
However, our study has limitations that need to be borne in mind while interpreting the results. The authors could not identify whether the ATQD or the Primary Care Web Tool have been rigorously evaluated. However, these tools have been developed by the NHS and are in accepted use as an acute trust quality management tool. The measurement of primary care quality and its association with secondary care providers has always been difficult, and we have used available and accessible indicators for our analysis.
We adopted the value judgement of the ATQD developers in classifying an indicator as amber or red (i.e. an indicator of poor performance). However, the developers have not provided their rationale for this value judgement.
There is a possibility that some of the variables may be over dispersed and could diminish the impact of any association. However, it is very unlikely that this would systematically vary between the two groups of trusts included in our analysis.
We were interested in special cause variation, and as a red result is categorised as being outside 99.8% of the distribution, we have not undertaken detailed analysis of the range of deviation from the mean within red scores. This is methodologically complex and could hinder interpretation of the results.
Strengths and weaknesses in relation to other studies
To our knowledge, there have been no previous studies comparing mortality rates of a trust and overall quality as measured by the ATQD.
Meaning of the study
The Keogh Review examined the quality of care in trusts with persistently high mortality rates. As such, it identified trusts for inclusion through the use of two indicators covering one specific outcome area. The rationale for the request to Sir Bruce Keogh was that the Francis Report had found an association between high mortality rates and wider failings in care quality.
The Keogh review found that the interpretation of current aggregated hospital mortality indices (HSMR and SHMI) is complex. The fact that different sets of outlier trusts are identified when each indicator is used is a testament to this. 4 Keogh notes that the CQC's development of its revised model for assessment should have broader inspection trigger mechanisms than mortality indices alone. 4 While mortality is a key outcome indicator for all NHS trusts, clinician rated recovery outcomes, patient reported outcomes and patient experience outcomes can all be significant measures of success and failure. 15
While there is a degree of overlap in the measures included in both the CQC banding calculations and the ATQD, the larger number of judgement enabled indicators (and the greater detail this brings) and the inclusion of assessment data from partner regulatory bodies may enable a more robust overall judgement to be made through the new CQC model. However, the allocation to Risk Band 1 of trusts subject to ‘special measures’ could be viewed as problematic when the primary criterion for inclusion in exceptional review is outlier status for mortality rate. There is a risk of negative performance for specific indicators (such as mortality indices) being double-counted in the overall analysis when including ‘special measures’ status in the summary risk assessment.
Unanswered questions and further research
This study used a proxy measure for the quality of primary care in patients prior to admission to the Keogh Review trusts. However, the majority of acute hospitals provide care to patients from more than one CCG and patients may not seek primary health care prior to secondary care admission. A more detailed analysis of primary and community care delivered to patients of the Keogh Review trusts in comparison to all other NHS acute trusts could be beneficial.
Conclusion
This study finds no evidence to suggest that the trusts reviewed by Sir Bruce Keogh in spring 2013 were outliers for overall quality of care as defined by the ATQD. While we agree that mortality is an important outcome indicator for care quality, the use of mortality indices in isolation as a means to identify potential quality outliers in secondary care may not be adequate. The Keogh Review identified serious quality concerns in each of the trusts and yet there is no discernible trend towards adverse results for these organisations in the ATQD. This study highlights the difficulties of trying to measure and monitor hospital quality using a small number of indicators and our limitations in surveillance of poorly performing hospitals. Any current or future quality monitoring systems in the NHS need to consider the methodology and include the breadth of coverage of Sir Bruce Keogh's review.