Abstract
This is the fourth essay in this series explaining key concepts that can help you avoid being misled by claims that have an untrustworthy basis. The next four essays in the series will explain concepts that can help you recognise when evidence from comparisons (tests) of treatments is trustworthy and when it is not. In this essay, we will explain five assumptions about basing trust on the source of a claim alone.
This can be misleading as a result of assuming that:
personal experiences alone are sufficient, your beliefs are correct, opinions alone are sufficient, peer review and publication are sufficient, or there are no competing interests.
The basis for these concepts is described elsewhere. 1
Do not assume that personal experiences alone are sufficient
People can be led to believe that improvements in a health problem (for example, recovery from a disease) resulted from having received a treatment. Similarly, they might believe that an undesirable health outcome was due to having received a treatment. However, the fact that an individual recovered after receiving a treatment does not mean that the treatment caused the recovery, or that other people receiving the same treatment will also improve. The improvement (or the undesirable health outcome) might have occurred even without treatment.
One reason is that personal experiences – including a series of personal experiences – are sometimes misleading. This is because experiences, such as pain, fluctuate and tend to return to a more normal or average level. This is sometimes referred to as ‘regression to the mean’. For example, people often treat symptoms such as pain when they are very bad and would improve anyway without treatment. The same applies to a series of experiences. For example, if there is a spike in the number of traffic crashes someplace, traffic lights may be installed to reduce these. A subsequent reduction may leave the impression that the traffic lights caused this change. However, it is possible that the number of crashes would have returned to a more normal level without the traffic lights.
If you have a splinter that is causing pain and the pain goes away right after you pull out the splinter, you can be confident that pulling out the splinter (the treatment) caused the outcome (no more pain). This is because the outcome happened right after the treatment and without the treatment the pain was constant and would very likely continue. 2 However, few conditions are constant (unchanging without treatment) and respond quickly to treatment. So, for example, it is impossible to know, based on your personal experience, whether you did or did not have a stroke or cancer when you are 70 because of your diet when you were younger.
Unless an outcome rarely, if ever, occurs without treatment, it is not possible to know, based on personal experience, whether the treatment caused the outcome, even if the outcome occurs shortly after the treatment. For example, tension-type headaches are very common. In adults who have frequent headaches, about 5%, 20% and 44% are likely to be pain-free within 1, 2 and 4 h, respectively without taking paracetamol (acetaminophen). 3 So, if an individual with frequent tension-type headaches took paracetamol and the headache went away, it would not be possible for that individual to know whether it was because of the medicine or if it would have gone away just as quickly without the medicine.
Do not assume that your beliefs are correct
People often look for and use information to support their own beliefs, including beliefs about the effects of treatments. This is sometimes called ‘confirmation bias’. Confirmation bias can occur when people want a claim about treatment effects to be true. By focussing on evidence or arguments that support their existing beliefs and ignoring evidence or arguments that challenge these, people believe claims that confirm what they believe or wanted to be true without thinking critically about the basis for the claims.
When looking for health information, many people search the Internet. However, the information they select, and their perception of that information, may be biased based on their prior beliefs. For example, parents of young children are more likely to select information about vaccination that is consistent than information that is inconsistent with their prior beliefs, and they perceive information that is consistent with their prior beliefs as being more credible, useful and convincing. 4
Do not assume that opinions alone are sufficient
People – including doctors, researchers and patients – often disagree about the effects of treatments. This may be because their opinions are not always based on systematic reviews of fair comparisons of treatments. Who makes a treatment claim, how likable they are or how much experience and expertise they have do not provide a reliable basis for assessing how reliable their claim is. This does not mean that conflicting opinions should be given equal weight – or that the existence of conflicting opinions means that no conclusion can be reached. How much weight to give an opinion should be based on the strength of the evidence supporting it.
Experts, just like everyone else, do not always base what they say on systematic reviews. For example, experts did not begin to recommend aspirin after a heart attack until years after there was strong evidence supporting its use. 5 Conversely, experts continued to recommend medicines to reduce heart rhythm abnormalities years after there was strong evidence that they increased the risk of early death after a heart attack.
Do not assume that peer review and publication are sufficient
Even though a comparison of treatments – whether in a single study or in a review of similar studies – has been published in a prestigious journal, it may not be a fair comparison and the results may not be reliable. Peer review (assessment of a study by others working in the same field) does not guarantee that published studies are reliable. Assessments vary and may not be systematic. Similarly, just because a study is widely publicised does not mean that it is trustworthy.
Sometimes, research that has been peer reviewed and published is so untrustworthy that it is retracted. About half of all retractions involve misconduct, including fabrication or falsification.6,7 Perhaps the most widely known example of a widely publicised paper that was subsequently retracted was a small study published in The Lancet which suggested that measles, mumps and rubella vaccination might cause autism. 8 Publication of that paper contributed to vaccine scepticism and led to a decrease in vaccinated children, outbreaks of measles, serious illness and at least four deaths that could have been prevented.
Although a small proportion of published papers are retracted, many more are corrected or refuted by more reliable research. 9 Journals rely on peer review to ensure the quality of the research they publish. However, peer review is highly variable, inconsistent and flawed.10,11 For the most part, it is done by volunteers. Few peer reviewers have formal training and they commonly do not detect major errors. For example, the British Medical Journal sent three papers, each of which had nine major methodological errors inserted, to about 600 peer reviewers. 12 On average, the peer reviewers detected about one-third of the errors in each paper. Half of the peer reviewers were given brief training, which had only a slight impact on improving error detection.
Do not assume that there are no competing interests
People with an interest in promoting a treatment (in addition to wanting to help people) – for example, to make money – may promote treatments by exaggerating benefits, ignoring potential harmful effects, cherry picking which information is used or making false claims. Conversely, people may be opposed to a treatment for a range of reasons, such as cultural practices.
Tamiflu (oseltamivir) is an example of how financial conflicts of interest can result in misleading claims about the effects of a treatment.13,14 Tamiflu was approved for seasonal influenza by the U.S. Food and Drug Administration in 1999. Several randomised trials and systematic reviews emphasised the benefits and safety of Tamiflu. Most of them were funded by Roche, which also marketed and promoted Tamiflu. In 2005 and 2009, the fear of pandemic flu led to recommendations to stockpile Tamiflu and billions of dollars were spent on this. After battling with the company for over four years, a team of review authors finally accessed the complete data held by the company. After carefully reviewing all the documents, they found no compelling evidence to support claims that oseltamivir reduces the risk of complications of influenza, such as pneumonia and hospital admission, claims that had been used to justify international stockpiling of the drug. 15 Tamiflu was found to slightly reduce the time to alleviation of flu symptoms in adults and to slightly reduce the risk of flu symptoms in people exposed to the flu. It was also found to have adverse effects that potentially outweighed the benefits. As a result of biased reporting of the research and misinformed recommendations and decisions, billions of dollars were wasted.
Implications
If an individual improved after receiving a treatment it does not necessarily mean that the treatment caused the improvement, or that other people receiving the same treatment will also improve. Do not be misled by your own beliefs or rely on them unless they are based on the results of systematic reviews of fair comparisons of treatments. Do not rely on the opinions of experts or other authorities about the effects of treatments unless they have taken account of the results of systematic reviews of fair comparisons of treatments. Always consider whether a published comparison of the effects of treatments is fair and whether the results are reliable. Peer review is a poor indicator of reliability. Ask if people making claims that a treatment is effective have conflicting interests. If they do, be careful not to be misled by their claims about the effects of treatments.