Abstract
Background
The 1845 UK Lunacy Act required physicians to rate patients in terms of whether they were dead, recovered, relieved or unrelieved, requiring regular collection of data and levying fines on physicians who failed to comply.
1
Although initially focused on psychiatry, this system was also applied to acute hospitals, in part because of the enthusiastic advocacy of Florence Nightingale. Emulating the 1845 legislation, she advocated the measurement of outcomes in terms of whether patients were dead, relieved or unrelieved. She declared:
I am fain to sum up with an urgent appeal for adopting this or some uniform system of publishing the statistical records of hospitals … I have applied everywhere for information, but in scarcely an instance have I been able to obtain hospital records fit for any purpose of comparison. If they could be obtained, they would … show subscribers how their money was being spent, what amount of good was really being done with it, or whether the money was doing mischief rather than good.
2
The system of recording ‘success’ in treating patients survived until 1948, and longer in some hospitals, but pressures on the early NHS were to meet the substantial backlog of unmet demand for healthcare. This created a long-lasting focus on funding, activity and waiting times, and the attention of policymakers to whether patients' health actually improved was unusual.
In recent postwar periods of economic crisis, there has often been questioning of the efficiency of healthcare financing and provision. For example, during the recession of the 1970s, the Department of Health and Social Security highlighted apparently inefficient variations in the NHS:
Within the national average for acute hospitals (9.8 days in 1972/73) there was a wide range between different areas for 14.7 days in the area with the longest average length of stay to 6.7 days in the shortest. If the average length of stay could be the reduced to the median, there would be a potential annual saving of £26 million of hotel costs. If the reduction was made to the lower quartile the potential annual savings would be £40 million – i.e. an amount equivalent to an increase in current expenditure on acute services of 2.5 percent.
3
More recent government declarations of the need to improve quality and reduce practice variations 4 are highly reminiscent of this document. Evidence from the 1970s through to the Darzi report 4 and beyond has not been translated into more efficient use of society's scarce resources for healthcare.
Parallels can also be found in overseas healthcare systems. For example, confronted by escalating Medicare costs in the early 1980s, US policymakers introduced a regulated fixed price prospective payment system using diagnosis related groups (DRGs) and began to publish hospital level mortality rates. This combination focused managerial attention on quantitative analysis of costs, to maximize income from the tariff system, and on differential success as measured by mortality rates. It took some decades before risk adjustment of mortality data at the clinical level began to influence providers' behaviour, and then not always in ways which were anticipated. 5
Mortality data at the clinical level, if suitably risk-adjusted, can be helpful in managing practice and informing patient choice, epitomised in the UK by publication of cardio-thoracic surgery survival rates. 6 But this is a partial picture, and despite long advocacy by academics, the application of quality-of-life measures to routine clinical practice has been slow. Generic health-related quality-of-life measures such as SF-36 7 and EQ-5D 8 have been translated into dozens of languages, used in thousands of clinical trials, used in national population health surveys (such as the Health Survey for England), 9 are recommended for use by the US Food and Drugs Administration, are required for use in submissions to the National Institute for Health and Clinical Excellence, and have been used routinely for performance management in the UK's independent healthcare sector. 10 Despite all this work, discussion of the use of quality-of-life instruments has only recently begun to penetrate policy environments.
PROMs instruments used in the NHS
The Department of Health intends to extend PROMs progressively across other conditions in the NHS over the next three years, to publish PROMs data on the NHS Choices website to inform patient choice and to encourage primary care trusts to use financial incentives to reward outcomes in their provider organizations. 13
So, centuries after the advocacy of Florence Nightingale and the wisdom of the Lunacy Act, and decades after the advocacy of academics and the widespread use of quality-of-life measures in clinical trials, the NHS is finally adopting them for the evaluation of routine clinical practice. This landmark decision is very welcome and could substantially improve our understanding of patient outcomes and care in the NHS. There are, however, a number of potential threats to the development of the use of PROMs, and these must not be allowed to undermine this key policy development. The policy must be implemented with caution, and appropriate levels of piloting and evaluation. With regard to routine use of PROMs in the NHS, we should learn to walk before we run.
Threats to PROMs
The first challenge to the implementation of PROMs is to ensure a response rate sufficient for robust statistical analysis. PROMs are part of the process of Commissioning for Quality and Innovation (CQUIN) which has introduced penalties for poor recording of specific performance activities. Hospitals are responsible for administering the preoperative PROMs questionnaire and achieving a response rate of at least 80%. After discharge a private contractor managing the PROMs programme is being paid a fee per questionnaire returned to incentivize response rates for the postoperative questionnaire. By early December 2009, 80,000 primary questionnaires and 7000 follow-up questionnaires had been received by analysts.
The pilot PROMs scheme carried out by the London School of Hygiene and Tropical Medicine recommended a response rate of 80% in the preoperative questionnaire to ensure a representative sample. 14 Patient compliance is of course voluntary, and this response rate is challenging – the pilot scheme recruited only centres who had demonstrated a previous enthusiasm for outcome measurement, and still participation rates varied from 33% to 100% between centres. 14 This variation has encouraged incentivization of response rates and publication of comparative performance data, which will hopefully encourage a high response rate.
The risk adjustment of PROMs data for publication is also a potential threat to further implementation. This will be carried out by a private sector data organization, but transparency and peer review of the methods planned are essential. Recommendations on risk adjustment were made in the evaluation study of the pilot PROMs scheme, 14 but these recommendations were based on analysis only of postoperative health status, not on the more important factor, as pointed out by Devlin et al., 12 which is the change in preoperative and postoperative health. The recommendations also focused on patient factors, such as age, sex and co-morbidities, and did not include other factors that are outside the control of the hospital, for example availability and quality of postoperative community care. There is a need for a thorough and probably multilevel approach to determining appropriate risk adjustment. If this process is not robust and transparent, PROMs data will be constantly challenged. There is also a risk that sophisticated risk adjustment may be impenetrable by managers and clinicians, which could generate suspicion and rejection of results even after careful and appropriate analysis.
Even if risk adjustment is robust, there is a need for caution in presentation, in particular to guide patients in interpreting the data. A burst of initial publicity with a media focus on poor performers could threaten the future use of these essential data. In particular, there are two related issues in respect of the interpretation of results: clinical responses to the publication of PROMs data and the problem of lagged benefits.
PROMs have been developed with limited explicit involvement with clinicians. Clinical commitment to this process is essential, as when PROMs data become available in 2010, poor performers will be identified and some practitioners may react to undermine this long-needed innovation. For example, if relatively ‘unsuccessful’ patient responses are evenly distributed, this should generate useful debate with all clinicians. But if they are concentrated within a small group of ‘poor performers’, how will this be managed? PCTs may decide, through processes like CQUIN, not to reimburse hospitals for patients who do not benefit from surgery, and if so, what will hospital managers do? Retraining practitioners will affect hospital activity rates and waiting times, and dismissal of practitioners would be highly disruptive even if it were possible. The effects of incentivizing PROMs through CQUIN could have significant effects on provider revenue with them having no easy remedies to poor performance in the short and medium term. In addition, the process of collecting and processing PROMs data, and then linking it with other patient episode data, is potentially time-consuming. Although inevitable, this restricts the use of PROMs in practice in terms of timely feedback to clinicians, and it places a heavy burden on the NHS Information Centre, who are responsible for timely data linkage.
Potential lags in patient health benefits also create difficulties in interpretation. For example, hernia repair patients may have few symptoms prior to surgery, and may report a worsening of their quality of life three months after treatment as pain levels are increased from the procedure. However, this is not necessarily any measure of failure, as the risk of strangulation is avoided. More generally, this highlights the problem of the unknown ‘counterfactual’ – we do not, and cannot, know what would have happened to these patients if they had not had any treatment.
Similar issues may arise from extending PROMs to other conditions. The quality of life of a breast cancer patient may be worse after a mastectomy than before, but hopefully the patient's length and quality of life are increased in the longer term. Recording 'success' after three or even six months may be incomplete. To go beyond this time period risks factors other than the procedure being evaluated affecting outcome. There is a risk that without adjustment for longer-term benefits, PROMs results will underestimate the value of many procedures. Such reporting may confuse patients and lead to clinical opposition. Methods of modelling longer-term benefits should be determined to help inform patients of the potential overall benefits of their treatment. Without due consideration of such effects the PROMs initiative may be undermined.
Another fundamental issue arising from investment in PROMs is evaluation of the scheme itself. In the pilot study, a unit cost of around £6.50 per patient was estimated, but this was in hospitals already collecting similar data, and may underestimate the costs of general use. There are also important questions of access to these data, particularly given the private sector input into data collection and risk adjustment. It is essential to cost the PROMs initiative fully and carefully so as to ensure value for money. Hopefully researchers' access to these fascinating data will be both prompt and full.
Overview
The Victorians were wise in their advocacy of a policy focus on the success of healthcare in making patients better. Policymakers in healthcare systems worldwide have nevertheless focused for decades on input and process issues such as the adequacy of funding and the problem of waiting times. The distinction between structure, process and outcome in healthcare is fundamental at every level of intervention, from clinical interventions through to policy change. Demonstrating that restructuring organizations and applying process-led clinical guidelines actually makes patients ‘better’ requires this late re-enactment of the 1845 Lunacy Act!
The creation of a system of PROMs, in conjunction with patient level costs and improved use of activity data, could transform patient care and inform patients and taxpayers, the modern equivalent of Nightingale's ‘subscribers’, whether they are getting value for money from the £100 billion spent on the NHS. Hopefully it has the potential to engage the competitive spirits of clinicians to improve patient safety and patient outcomes. Such engagement is essential if the PROMs investment is to prove its worth.
It is a cause for concern that despite the relatively long lead into the introduction of PROMs, from their announcement in December 2007 to partial introduction from April 2009, there has been inadequate consideration of the management of the ambitious system that has been introduced. To maximize the potential of PROMs to improve the effectiveness and efficiency of the NHS, it is essential to implement systems and report results with caution, making the most of lessons learnt from pilot studies and from each stage of implementation. In this exciting, innovative and potentially world-leading endeavour we must not forget to learn to walk before we run.
Footnotes
DECLARATIONS
Footnotes
Acknowledgements
None