Abstract
Introduction
New treatments tested in randomized trials are as likely to be inferior as they are to be superior to standard treatments. 1 The ethical implication of this empirical observation is that unevaluated new treatments should be assessed carefully in well-designed research before decisions are taken about whether or not they should become widely available. Indeed, the same principle applies when there are uncertainties about the effects of ‘standard’ treatments. The dangers of acquiescing when it is uncertain whether treatments do more good than harm has been illustrated dramatically by the results of the CRASH trial: 2 tens of thousands of iatrogenic deaths have occurred because of decades of failure to address uncertainties about the effects of systemic steroids for patients with acute traumatic brain injury
It is against this background that some clinicians believe that unproven treatments should only be offered within the context of appropriately designed research. In the perinatal field, for example, obstetricians decided that chorionic villus sampling should be used in low-risk pregnancies only within the context of a randomized comparison with amniocentesis. 3 Similarly, neonatologists restricted the use of neonatal extracorporeal membrane oxygenation (ECMO) to a randomized comparison with standard respiratory support,4,5 and they are currently restricting the use of prolonged whole-body cooling in term infants with perinatal asphyxial encephalopathy to a randomized comparison with standard treatment (the TOBY trial: Whole Body Hypothermia for the Treatment of Perinatal Asphyxial Encephalopathy)
Ethical considerations have been the main motivation for clinicians who have dealt with uncertainties about treatments in this way. But ethical considerations have also informed similar decisions made by agencies managing health service funds on behalf of defined populations to whom they are answerable for cost-effective use of resources. 6 For example, the Centre for Medicare and Medicaid Services (CMS) in the USA decided that, until more was known about its relative merits and demerits, lung reduction surgery should only be supported if given within the context of a randomized comparison with nonsurgical treatments. 7 The Centre is currently formalizing the criteria for applying this principle, 8 which has also been considered 9 and implemented on a single occasion so far (personal communication, Professor Wing) by the Pharmaceuticals Benefit Scheme in Australia. 10
The principle of recommending that when there are important uncertainties about a treatment's effects, that treatment should only be used in the context of research, was explicitly recognized by the National Institute for Health and Clinical Excellence (NICE) at its inception in 1999. NICE was set up to develop evidence-based guidance for the NHS and the broader public sector with the explicit objectives of promoting cost-effective use of resources while encouraging innovation, improving quality and reducing variation in practice. 11 To do so, NICE and its advisory bodies were given three decision options, set out by the Department of Health shortly before NICE's establishment in 1999. 12 When appraising health interventions (including health technologies, health promotion and disease prevention programmes, screening and diagnostic tests, surgical procedures, etc.), NICE would be able to recommend:
The routine use of an intervention in the NHS, either for all or specific licensed indications or patient subgroups; That the intervention is not used in the NHS because of inadequate evidence of effectiveness, or more frequently cost-effectiveness; or The use of the intervention in the NHS only in the context of appropriate research. This would be appropriate in the case of promising interventions not yet supported by sufficiently robust evidence to justify an unqualified recommendation. In those cases, NICE would ‘recommend that further research is carried out to see whether the potential promise of the intervention can be realized, indicate in broad terms the questions this research should address and advise clinicians that, in the meantime, they should only use the new intervention as part of a well-designed programme of research intended to answer these questions’ (paragraphs 33–36 in Faster Access to Modern Treatment: How NICE Appraisal Will Work
12
)
This paper discusses NICE's third option, its characteristics, use and implications for the evidence base, and its future prospects as a viable policy option for NICE and the NHS. This seems to us a good time to promote discussion on the issue, given NICE's recent launch of the single technology appraisal to assess new technologies closer to licensing, with clear implications on the available evidence base, and the reorganization of the medical research infrastructure in the UK following the Cooksey review. To make the most of the ‘Only In Research’ (OIR) option for patients and the Health Service, this discussion needs to start as early as possible
How Does Nice Deal with Uncertainty?
The evidence base on specific interventions or management strategies is rarely complete, so NICE and its advisory bodies are almost always faced with uncertainty when making their decisions. Despite this, NICE has to develop guidance on the clinical and cost-effectiveness of the interventions referred to it by the Department of Health, and it has to do so within specific timelines. Starting by explicitly acknowledging its existence, NICE has adopted a number of approaches to dealing with uncertainty. Throughout the topic selection process, NICE has always made a case against developing guidance in areas where the evidence is scarce; 13 instead, these topics would be forwarded to NHS Research and Development Programmes. Furthermore, in 2003, NICE's Board approved the establishment of an R&D Programme supported by the NICE R&D Advisory Committee, with the explicit objective of improving the quality of the evidence base underpinning its guidance through promoting appropriate research. 14 Over the years, NICE has encouraged the development of methodological tools to deal with uncertainty and has since adopted a number of them, including probabilistic sensitivity analysis, 15 now explicitly referred to in the NICE Technology Appraisal Methods Manual. 16 Consensus methods and elicited expert opinion are often used to supplement systematic reviews when developing clinical guidelines in areas where there is little good quality evidence, 17 based on NICE's methods manuals. 18 Finally, when important evidence gaps are identified during the guidance development process, NICE issues research recommendations, encouraging further research to help fill those gaps and inform future updates of its guidance. Calling for further research to inform clinical practice is an important NICE function, highlighted in both the NICE Framework document 19 and the WHO Review of Technology Appraisals. 20
Nevertheless, there will always be cases where the available evidence is insufficient to support a positive—or negative—recommendation. In these cases, NICE advisory bodies can consider issuing a recommendation for the use of the intervention only in the context of research.
Putting it into Practice: NICE Recommendations for use Only in the Context of Research
Sixteen (approximately 4%) of NICE's technology appraisal recommendations published between 1999 and early 2007 advised the use of a technology only in the context of research (Table 1). For example, in 2000, NICE recommended that laparoscopic resection for colorectal cancer should only be undertaken in the context of a clinical trial. 21 This decision encouraged recruitment to the then ongoing UK-based Medical Research Council CLASICC trial, which has since provided the necessary evidence to support a NICE recommendation that this intervention be introduced in routine NHS practice. 22 NICE's advice on the use of liquid based cytology is another situation in which NICE recommended further research before subsequently issuing a positive recommendation for the adoption of this new technology across the NHS. 23 Five of the technology appraisal OIR recommendations became supportive recommendations when the original guidance was reviewed in light of additional research evidence (Table 1).
Examples of NICE Only In Research recommendations from Technology Appraisals (1999-2007) for which the research has been or is currently being undertaken and has informed reviews of the guidance
This type of recommendation is now found not only in appraisals of specific technologies but also in clinical guidelines, which deal with broader disease management strategies, and (since April 2005) in public health guidance, which looks at specific interventions or broader programmes of prevention and health promotion. Indeed, in one of the first pieces of public health guidance published by NICE, three of the four interventions considered were recommended only in the context of research studies 24 (Table 2). Almost half of NICE's recommendations in its interventional procedures guidance (which advises on the safety and efficacy of surgical procedures) fall in the category of ‘cautionary guidance’ and are recommended only with additional requirements for consent, audit and, occasionally, for more formal research. 25
Examples of Only In Research recommendations in NICE clinical guidelines and public health guidance
Finally, NICE often recommends that interventions are used only if outcome data are assembled in appropriate registers or ‘databases’ to assess their safety as well as their clinical and cost-effectiveness (Table 3).
Examples of NICE recommendations for submitting data to clinical disease registers or databases
Supporting the Policy Option: Upstream and Downstream Measures
Implementing the OIR option in the NICE and broader NHS context has been challenging.
Issuing an OIR recommendation
NICE operates within an environment of inevitable uncertainty. 26 Over-exercising the OIR option in this setting could result in indecisiveness and inaction, hamper innovation, and negate NICE's very objective: to provide advice based on the best available evidence.
To ensure the OIR option is used in a pragmatic way, NICE is exploring the decision-making trail leading to and the criteria underpinning these decisions. These criteria should ideally reflect how well NICE and the NHS tolerate uncertainty. A number of questions need addressing:
How much uncertainty should there be before a recommendation is issued that a technology be used only in the context of research? How would this vary (if at all) when there is no ongoing or planned research? Should the focus be solely on the quality of the evidence or should the potential budgetary impacts and clinical importance of the practice be considered when issuing an OIR recommendation? How can these decisions be made in a transparent, consistent and methodologically sound way across NICE?
However challenging these questions may be, a methodological framework is already in place to help support the thinking in this area. Value of information principles already referred to in the NICE appraisals manual, supplemented by options theory, 27 could provide the methodological underpinnings for tackling some of these questions. At the same time, a qualitative checklist could also be developed to help guide the deliberations of the advisory bodies. For example, when further research is necessary and could help resolve uncertainty, advisory committee members could be asked to consider what the implications of making a positive recommendation would be:
On any ongoing research and thus the development of the evidence base; On total spending, particularly if the recommendation is reversed in the future but the NHS has already incurred irretrievable fixed costs.
Implementing OIR recommendations
No formal arrangements exist between NICE, industry and the clinical research community, including the NHS and patient organizations, for pursuing the research recommended by NICE in a timely manner, so that uncertainties are reduced to the extent that definitive guidance can be developed. This contrasts with the active arrangements for promoting the uptake of positive NICE recommendations. As a result, OIR recommendations are hard to implement, particularly when there is no ongoing trial addressing the question. Indeed, they are often perceived to be negative recommendations by patients, clinicians and industry alike. In a recent BMJ paper reviewing NICE recommendations between 1999 and 2005, Raftery classifies all recommendations for the use of a technology in the context of research as ‘no’ decisions. 28
The development of a co-ordinated approach with other interested parties could encourage NICE advisory bodies to consider OIR recommendations in a more systematic and consistent way. For example, in guidance on the use of beta interferon and glatiramer in the treatment of multiple sclerosis NICE recommended that the Department of Health and the National Assembly for Wales, along with the manufacturers, consider what action could be taken so that the NHS could obtain these drugs in a way that would be cost-effective. 29 A national data collecting system was established to ascertain which sub-groups of patients are most likely to benefit. However, the lack of a coordinated approach has made it difficult to ensure that the information collected will ever inform an update of the NICE guidance
Furthermore, as the burden of evaluation lies with NICE, mandating research from the technology sponsors is unrealistic and wishful thinking, particularly in cases where head-to-head trials are needed. The prospect of an appeal, and the pressure generated by lobbying groups keen to ensure access to the new technology, make the OIR option even less attractive to NICE advisory committees.
As a result, the danger is that OIR recommendations turn into a means of saying no politely, instead of generating the additional evidence needed to make more confident decisions.
Indeed, the recently published Cooksey Review 30 recommends that special arrangements are set up and funding earmarked to ‘… implement NICE recommendations calling on the NHS to use health interventions in a research context’. This is an exciting development, directly addressing the need for co-ordination between publicly funded research and evidence-based policy making, as reflected in the NICE process of appraising new technologies. Key stakeholders, including NICE, research funders, patient organizations, industry, government and the NHS, will have to work together to implement Cooksey's recommendations. In the meantime, some have hastened to interpret the Cooksey proposals as a call for conditional coverage of technologies, across the NHS, long before phase III trials are completed. 31 However important it is to accelerate access to new technologies, without the right processes in place, such moves could compromise patient safety and overall health outcomes in a financially restrained system.
Furthermore, the OIR option is now applied not only to new medical technologies with interested sponsors as originally anticipated, 12 but also to established clinical and public health strategies that almost always lack sponsorship. This has reinforced the implementation challenges described above but has also highlighted how important it is for NICE advisory bodies to be able to recommend data collection on existing, often well-established practices or interventions to inform recommendations for increased investment, or for disinvestment. The need to support the NHS by advising which ineffective, wasteful or even harmful practices should be discontinued 2 makes this evaluative dimension of the OIR option particularly relevant.
Linking research funding to NICE decision making is a pressing need; the recent single technology appraisal of Herceptin for early breast cancer is a case in point: in August 2006, only three months after the drug gained its license, NICE recommended the use of the drug for eligible women. The decision was supported by the available evidence at the time 32 and recently published studies further reinforce this decision. 33 However, a positive decision based on the best available evidence may not be enough. NICE-mandated, publicly-funded research into alternative therapeutic regimes beyond the single licensed indication determined by the manufacturer could inform future updates of the guidance and ensure the Health Service is obtaining value. 34 Carrying out health technology assessment earlier on in the development of a technology in the context of the single technology appraisal poses a significant challenge if the evidential standards are to be maintained. Alternative approaches to the yes/no dichotomy, which encourage the development of the evidence base, will need to be applied to avoid widespread use of unproven technologies or denying access to promising treatments (Table 4).
Implications of getting it wrong
The views of the NICE Citizens’ Council
NICE set up the Citizens’ Council in 2002 (http://www.nice.org.uk/citizenscouncil) to advise NICE and its advisory bodies on the social value judgments that should underpin its guidance. Its 30 members reflect the demography of England and Wales and meet twice a year to debate issues put to them by NICE. Their reports inform the Social Value Judgments principles which constitute official guidance to NICE's advisory bodies.
In January 2007 the Council met to discuss NICE's approach to uncertainty. In their draft report (currently under public consultation) they unanimously endorse OIR as a valuable decision option for NICE and its advisory bodies, and go on to discuss the circumstances under which NICE should consider issuing such a recommendation. A number of key considerations are identified:
The extent to which further research is likely to reduce current uncertainty; The value-for-money of the research; The implications of a positive recommendation on the evidence base and the NHS budget when this is made inappropriately in place of an OIR decision; The existence of an ongoing study or the feasibility setting one up study within a realistic time frame; Issues of patient access to the study across different geographical areas.
Having discussed the need for explicit criteria for OIR decisions and the difficulties of implementing the necessary research, the Council report concludes ‘… patients would be reassured to know that clinicians and the healthcare system in general could face up to uncertainty, and were confident enough to deal with it in a mature, scientific way, and avoid wasting money on unproven technologies.’ 35
Discussion
OIR recommendations have been statutory valid alternatives to yes and no decisions since NICE was set up by government in 1999. They represent the only rational way for addressing uncertainty, consistent with NICE's principles of transparency and methodological robustness. As demonstrated in a number of cases so far, OIR recommendations can be viable and workable decision options as long as there is some co-ordination between the research and decision-making communities.
Our study brings together all occurrences of such NICE decisions and describes how they have evolved over the years in response to the generation or absence of new evidence. We have not discussed here those cases where an OIR decision should have been made by NICE but was not, neither have we sought to identify any OIR that were made inappropriately given the circumstances. To do either of the above we would need to establish clear decision criteria as to when OIR recommendations are justified. To achieve this, additional work is underway to:
Develop a clear set of criteria for recommending interventions be used only ‘in the context of research’. These would reflect how much uncertainty NICE and the NHS are willing to accept without hampering innovation, restricting access or compromising patient safety and wasting valuable resources. Clear criteria will increase the transparency and consistency of such decisions in the future and allow evaluation of past OIR recommendations; Establish clear arrangements for implementing NICE OIR recommendations in a timely, responsive and practical manner. These cover issues around who would bear the burden of proof, the role of manufacturers and of the NHS in co-ordinating and/or financing the research, and what happens when a technology has no sponsor, as is the case for many public health or established interventions.
NICE guidance is based on the best available evidence; however, there are several situations in which there may be significant uncertainties about the intervention under consideration. In these cases, recommending the use of the intervention in the context of research can help fill the evidence gaps, while ensuring the efficient use of resources and encouraging innovation that adds value. Making OIR recommendations has been a policy option for NICE since it was established in 1999. However, setting out clear and consistent criteria leading to such decisions and ensuring that the recommended research is undertaken in a timely and responsive manner are challenging tasks. Through reviewing the cases where NICE has issued OIR recommendations, this paper reinforces the importance of acknowledging uncertainty and highlights the difficulties in implementing this policy option within the NHS. With the recent publication of the Cooksey review of UK medical research and NICE introducing new streams of work, including faster technology appraisals closer to licensing and reviews of established treatments to reduce wasteful activities, the OIR option needs to be revisited.
In light of the recently published Citizens’ Council report on OIR, we intent actively to engage with the broader public in this exercise so that any policy initiatives are not only consented upon but also driven by those of us who use the NHS.
Competing interests None declared.
Contributorship All authors have participated in data collection and analysis and the drafting of this article. All authors have seen and approved the final version.
Footnotes
Acknowledgments
We thank the NICE Research and Development Advisory Committee for encouraging NICE actively to consider the way it addresses uncertainty and the NICE Technical Forum for debating the role of OIR as a NICE decision option. We are grateful to Steven Pearson and Sean Tunis for valuable discussions and Kelly Stevens for her help with identifying ‘OIR recommendations’ from NICE guidance. Finally, we would like to acknowledge the contribution of Jeremy Wyatt and the input of the NICE Research and Development Advisory Committee.