Abstract
Make well-informed choices
What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms and costs.
The first step in making decisions about treatments is being clear about what your problem or goal is and what the options are. Only then can you make judgements about the relevance of the available evidence from treatment comparisons, including the relevance of:
the outcomes that were measured, the participants in the comparisons, the treatments that were compared and the circumstances in which the treatments were compared.
In this essay, we explain those considerations.
Once the trustworthiness of the available evidence has been assessed (as explained in essays 2.1–2.4) and its relevance (as explained in this essay), well-informed decisions about what to do depend on judgements about whether the expected advantages outweigh the expected disadvantages, including consideration of:
the benefits, savings, harms and costs of acting or not, the baseline risk or severity of the symptoms, the importance of each advantage and disadvantage, the certainty of the evidence and the need for further fair comparisons.
We explain these considerations in the next essay (3.2).
Be clear about what the problem or goal is and what the options are
Good decisions depend on correctly identifying the problem and considering an appropriate set of options to address the problem. For personal health choices, this means starting with a correct diagnosis (or assessment of risk) and then identifying the treatments that are available. For public health and health system policy decisions, this means describing the problem correctly and identifying the policy options relevant for that problem. Changing how a problem is framed can lead to different options for addressing it.
The following passage from Archie Cochrane’s autobiography is a striking illustration of the importance of correctly identifying the problem and considering appropriate options.
1
He recalls an event when he was a doctor in a German prisoner of war camp. The Germans dumped a young Soviet prisoner in my ward late one night. The ward was full, so I put him in my room as he was moribund and screaming and I did not want to wake the ward. I examined him. He had obvious gross bilateral cavitation and a severe pleural rub. I thought the latter was the cause of the pain and the screaming. I had no morphia, just aspirin, which had no effect. I felt desperate. I knew very little Russian then and there was no one in the ward who did. I finally instinctively sat down on the bed and took him in my arms, and the screaming stopped almost at once. He died peacefully in my arms a few hours later. It was not the pleurisy that caused the screaming but loneliness. It was a wonderful education about the care of the dying. I was ashamed of my misdiagnosis and kept the story secret.
Consider the relevance of the outcomes measured in the research
A fair comparison may not include all outcomes – short- and long-term – that are important. Patients, professionals and researchers may have different views about which outcomes are important. For example, studies often measure outcomes, such as heart rhythm irregularities, as surrogates for important outcomes, like death after heart attack. The effects of treatments on surrogate outcomes often do not provide a reliable indication of the effects on outcomes that are important. Similarly, short-term effects may not reflect long-term effects.
Despite dozens of randomised trials since the introduction of the first oral agent for treating type 2 diabetes, it has remained uncertain if any of those medicines favourably affects outcomes that are important to people, including morbidity, mortality and quality of life. 2 A key reason for this is that the trials have focused on glucose control measured with laboratory tests rather than on outcomes that are important to people with diabetes. Unfortunately, those laboratory tests (glycosylated hemoglobin, type A1C – glycosyl HbA1C) are not a reliable indicator of outcomes that are important to people with type 2 diabetes.
It is sometimes important to consider outcomes that are important to other people besides the person being treated. For example, the use of antibiotics may increase antibiotic resistance, and not being vaccinated for COVID-19 or not avoiding contact with other people may increase the risk of infection for others. Similarly, when decisions are made for a group of people rather than for individuals, the outcomes that are important to anyone who is affected should be considered.
Consider the relevance of fair comparisons in laboratories, animals or highly selected people
Studies that only include animals, or only a selected minority of people, may not provide results that are relevant to most people.
Table 1 shows some examples of misleading extrapolation from animals or a selected minority of people, found in news reports. 3
Examples of misleading extrapolation from animals or a selected minority of people found in news reports.a
aBased on Haneef et al. 3
Consider the relevance of the treatments that were compared
A fair comparison of the effects of a surgical procedure done in a specialised hospital or delivered by an experienced surgeon may not provide a reliable estimate of its effects and safety in other settings, or in the hands of less experienced surgeons.
For example, results from a large randomised trial showed that endarterectomy (surgical removal of part of the inner lining of an artery) for asymptomatic carotid stenosis (narrowing of the large arteries on each side of the neck) reduced the five-year absolute risk of stroke by about 5%. 4 However, the trial only accepted surgeons with a good safety record, rejecting 40% of applicants and subsequently barring those who had adverse operative outcomes in the trial from further participation. The benefit from surgery was largely attributable to the low operative risk. Operative mortality was eightfold higher outside of the trial and the risk of stroke and death was about threefold higher.
Similarly, comparing a new drug to a drug or dose that is not commonly used (and which may be less effective or safe than those in common use) would not provide a relevant estimate of how the new drug compares to what is commonly done.
For example, in randomised trials of atypical antipsychotics for schizophrenia, haloperidol (one of the most frequently prescribed ‘typical’ antipsychotics worldwide) was used as the comparison treatment. 5 However, the trials used haloperidol in doses that were higher than that recommended. In a meta-analysis of 52 randomised trials that controlled for the higher-than-recommended dose of comparator drugs, differences in effectiveness and overall tolerability between typical and atypical antipsychotics disappeared, suggesting that the perceived benefits of atypical antipsychotics were due to excessive doses of the comparison treatments, such as haloperidol. 6
Consider the relevance of the circumstances in which the treatments were compared
Some treatment comparisons are designed to find out if a treatment can work under ideal circumstances, for example, with people who are most likely to benefit and most likely to comply, and with highly trained practitioners who deliver the treatment exactly as intended. These comparisons, which are sometimes called ‘explanatory’ or ‘efficacy’ studies, may not reflect what happens under usual circumstances.
The North American Symptomatic Carotid Endarterectomy Trial is an example of an ‘explanatory randomised trial’.7–9 It demonstrated that a lower risk of stroke was experienced by highly selected patients with severe narrowing of the large arteries on either side of the neck (carotid stenosis). Those patients who had a high risk of stroke were most likely to respond to surgical removal of the inner layer of those arteries (endarterectomy). Participating patients were closely followed and were operated on by skilled surgeons in academic and specialist hospitals with adherence to a strict protocol. The trial showed that endarterectomy reduced the risk of stroke under those ideal circumstances, but it did not provide a reliable estimate of the beneficial and harmful effects of endarterectomy under more typical circumstances.
Implications
Make sure you are considering the correct diagnosis or problem, and appropriate options for addressing it. Always consider the possibility that important outcomes may not have been addressed in fair comparisons. Avoid being misled by surrogate outcomes. Results of systematic reviews of studies in animals, or highly selected groups of people, may be misleading. Be aware that treatments available to you may be sufficiently different from those in the available research studies that the results of those studies may not apply to you. Be aware that the results of studies with the aim of finding out if a treatment can work may overestimate the benefits of a treatment given under more usual circumstances.