Brutalized by Bias

The Netherlands is a densely populated country. The length and duration of traffic jams in the morning rush hour increased 20% in 2019. Therefore, theoretically I am at risk of being in one every day. Does that imply that I should not travel by car anymore out of fear of arriving late at work? No, it does not, because it happens less than once a month.

Even a “high risk of” is a “risk of” and not necessarily an “occurrence of.” Clearly “risk of” does not mean anything if the likelihood of occurrence and subsequently the probable effect size in case of occurrence are not described. It is interesting to know that human judgment, and especially decision-making under uncertainty, is very poor as demonstrated by Nobel laureate Daniel Kahneman. That holds true for everyday life, but—alas—also in the evaluation of medical studies.

In the judicial system of civilized countries, one cannot be punished for something that was not a crime in the code of law at the moment the alleged misdemeanor was performed. Likewise, applying newer methodological standards for evaluating risk-of-bias (ROB) to older studies needs to be done with great caution. Otherwise, due to ever improving methodological standards, older studies contain more and more biases. Consequently, almost every systematic review (SR) concludes that more trials are necessary because the outcome and conclusions distilled from the literature are “uncertain” or “inconclusive.” But SRs never state: the chance that this type of bias has occurred, and indeed influenced the trial result, is this large.

Naturally, the higher the ROB, the more careful we need to be when making inferences, the trend of completely disregarding a lot of potential evidence because of unsubstantiated ROB hinders medical practice.

Denying studies completely based on a renewed methodological system, seems a sort of punishment in conflict with the judicial “ne bis in idem”—notice: these studies were peer reviewed and already assessed as worth publishing— a harsh judgment, especially because the authors cannot defend themselves anymore.

Standardized methods are available for the classification of the evidential level. When using said methods, the reader can be left to him/herself to decide the value of the study. However, “all available evidence” constitutes far more than just trials.

Moreover, empirical evidence, experiential evidence and pathophysiological understanding differ in kind from one another, not in degree. ¹ Therefore, these cannot be placed in the same hierarchy and should be used next to each other.

In contrast to the judicial system in which corroborative evidence is used, SRs rarely use other evidential levels or other types of evidence in a corroborative manner.

Unfortunately, still no appropriate (corroborative) evidential level applies to logic, pathophysiologic rationale, and clinical experience. ¹ Although these are far more solid factors than unsubstantiated “risks-of.”

In favor of the use of retrospective studies, it is good to remember that Hill, arguably the most renowned medical statistician of all time, stated: "I would myself put a good deal of weight upon similar results reached in quite different ways, e.g. prospectively and retrospectively.” ²

The GRADE-group stated: “when methodologically strong observational studies yield large or very large and consistent estimates of the treatment effect, we may be confident about the results. In those situations, although the observational studies are likely to provide an overestimate of the true effect, the weak study design is unlikely to explain all of the apparent benefit.” ³

Recently, a SR on intramuscular steroid injections for hay fever was published in which the results were deemed inconclusive due to the fact that all ten trials were assessed as having too much ROB. ⁴ But if ten different studies have a positive outcome, it is unlikely that all this positivity is obtained by the bias only. Moreover, in the above-mentioned article, the positive outcome is supported with corroborative evidence from nine other studies with lower evidential level.

The Cochrane SR on antiviral treatment for Bell’s palsy neglects evidence from 10 out of 13 studies because of high risk-of-bias. ⁵ This whereas their full data set is clearly in favor of combination therapy. Pathophysiologic rationale and non-randomized studies could have been used as corroborative evidence to support the choice to use the entire data set. ⁶ A more recent SR by other authors indeed concludes that combination therapy remains the best regimen for good recovery without synkinesis in the case of Bell’s palsy. ⁷

To conclude, what is necessary is not performing more studies, but a more reasonable approach to dealing with the uncertain impact of ROB.