Recognizing,investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO

Why is incomplete reporting of research a problem?

Under-reporting of the results of research in any field of scientific enquiry is scientific misconduct because it delays discovery and understanding. In the field of clinical research, incomplete and biased reporting has resulted in patients suffering and dying unnecessarily. ¹ Reliance on an incomplete evidence base for decision-making can lead to imprecise or incorrect conclusions about an intervention's effects. Biased reporting of clinical research can result in overestimates of beneficial effects ² and suppression of harmful effects of treatments. Furthermore, planners of new research are unable to benefit from relevant past research.

Failure to publish is also unethical. Participants in clinical research are usually assured that their involvement will contribute to knowledge; but this does not happen if the research is not reported publicly and accessibly. Moreover, failure to publish is simply a waste of precious research and other resources. ³ Every year an estimated 12,000 clinical trials which should have been fully reported are not, wasting just under a million tonnes of carbon dioxide annually – the carbon emission equivalent of about 800,000 round-trip flights between London and New York. ⁴

In brief, failure to report research findings is not only unscientific but also unethical. ^5–8 How did this problem come to be recognized and investigated, and what steps are being taken today to deal with it?

Evidence of biased reporting of studies

‘Reporting bias’ occurs when the nature and direction of the results of research influences their dissemination. Research results that are not statistically significant (‘negative’) tend to be under-reported, ⁹ while results that are regarded as exciting or statistically significant (‘positive’) tend to be over-reported. ^10–12 The nature and direction of research results can influence whether or not research is reported at all, ^9,13 and if so, in which forms. ¹⁴ They can also influence the speed at which results are reported, ^15–17 the language in which they are published, ^18,19 and the likelihood that the research will be cited. ^20–25

Failure to publish research findings is pervasive. ^26,27 Studies demonstrating failure to publish have included research conducted in many countries, including Australia, France, Germany, Spain, Switzerland, the United Kingdom and the United States. For example, an analysis of follow-up studies based on 29,729 reports of research made available only in abstract form found that fewer than half of the studies went on to full publication, and that positive results were positively associated with full publication, regardless of whether ‘positive’ results had been defined as any ‘statistically significant’ result or as ‘a result favoring the experimental treatment’. ¹⁴

Recognition and investigation of biased reporting of research

The problem of reporting bias has been recognized for hundreds of years. In the 17th century, Francis Bacon noted that ‘The human intellect … is more moved by affirmatives than by negatives’; ²⁸ and Robert Boyle, the chemist, lamented the common tendency among scientists not to publish their results until they had a ‘system’ worked out, with the result that ‘many excellent notions or experiments are, by sober and modest men, suppressed’. ²⁹ Other scientists, across many fields, have also recognized the problem over the years. ^30–35

For example, the bronze statue of Albert Einstein outside the National Academy of Sciences in Washington, DC is inscribed with a quotation from a letter that he wrote on 3 March 1954, for a conference of the Emergency Civil Liberties Committee:

Academic freedom as I understand it means having the right to seek the truth and to publish and teach what is believed to be true. Naturally this right comes together with the duty not to withhold a part of what is believed to be true. It is clear that any restriction on academic freedom hinders the dissemination of knowledge in the population and therefore restrains rational judgement and action. ³⁶

In 1959, the father of medical statistics in Britain, Austin Bradford Hill, wrote:

A negative result may be dull but often it is no less important than the positive; and in view of that importance it must, surely, be established by adequate publication of the evidence. ³³

And in the same year, Seymour Kety, an American psychiatrist wrote:

A positive result is exciting and interesting and gets published quickly. A negative result, or one which is inconsistent with current opinion, is either unexciting or attributed to some error and is not published. So that at first in the case of a new therapy there is a clustering toward positive results with fewer negative results being published. Then some brave or naïve or nonconformist soul, like the little child who said that the emperor had no clothes, comes up with a negative result which he dares to publish. That starts the pendulum swinging in the other direction, and now negative results become popular and important. ³⁷

Although the importance of reporting biases had been recognized for centuries, it was not until the second half of the 20th century that researchers began to investigate the phenomenon. The impetus for these investigations came from the development of research synthesis, first by social scientists, then by health researchers. ^38–40 Unsurprisingly, researchers who have exposed reporting biases are often those who have also been involved in the application of methods for research synthesis.

Investigations of biased reporting of research began with surveys of journal articles, which revealed improbably high proportions of published studies showing statistically significant differences. ^41–43 Subsequent surveys of authors and peer reviewers showed that research that had yielded ‘negative’ results was less likely than other research to be submitted or recommended for publication. ^44–47 These findings were reinforced by the results of experimental studies, which showed that studies with no reported statistically significant differences were less likely to be accepted for publication. ^48–50

The most direct evidence of publication bias in the medical field has come from following up cohorts of studies identified at the time of funding, ⁵¹ ethics approval, ^52,53 submission for drug licences, ^54–56 or when they were reported in summary form, for example in conference abstracts. ^14,57 Systematic reviews of this body of evidence have shown that ‘positive findings’ are the principal factor associated with subsequent publication: a systematic review of data from five cohort studies following research projects from inception found that, overall, the odds of publication for studies with ‘positive’ findings was about two and a half times greater than the odds of publication of studies with ‘negative’ or ‘null’ results, and that study results were the principal factor explaining these differences in reporting. ^9,13,27,58

Even when studies are eventually reported in substantive publications, ‘negative’ findings take longer to appear in print: ^15,17,59,60 on average, clinical trials with ‘positive results’ are published about a year sooner than trials with ‘null or negative results’. There is also evidence that, compared to negative or null results, statistically significant results tend to be published in journals with higher impact factors, ⁵² and that publication in the mainstream (‘non-grey’) literature is associated with an overall 9% larger estimate of treatment effects compared to reports in the grey literature. ⁶¹ Articles reporting negative findings for efficacy, or reporting adverse events associated with an exposure, may be published but 'hidden' in harder to access sources. ⁶² Furthermore, even when studies initially published in abstract form are published in full, ‘negative’ results are less likely to be published in high impact journals than ‘positive’ results. ⁶³

Selective reporting of suspected or confirmed adverse treatment effects is an area for particular concern because of the potential for patient harm. In a study of adverse drug events submitted to Scandinavian drug licensing authorities, subsequently published studies were less likely than unpublished studies to have recorded adverse events. ⁵⁴ The lay and scientific media have drawn attention to failure to accurately report adverse events for drugs, for example, of selective serotonin uptake inhibitors for depression, ^64,65 rosiglitazone for diabetes, ⁶⁶ and rofecoxib for arthritis pain. ⁶⁷

Biased reporting of data within studies

Even when substantive reports of research are published, there may be biased reporting of outcome data within the reports. ^{13,56,68–71} Comparisons of published articles with the study protocols approved by an ethics committee in Denmark found that in nearly two-thirds of trial reports at least one planned outcome had been changed, introduced, or omitted in the published article. ⁷⁰ In a similar comparison of randomized trials funded by the Canadian Institutes of Health Research, primary outcomes differed between the protocol and published article 40% of the time. ⁶⁹ In both of these studies, outcomes that were statistically significant in favour of an experimental intervention had a higher chance of being published in full compared to those that were not statistically significant. Other analyses have shown important discrepancies between journal articles and information supplied for trial registration. ^72–75

Biased outcome reporting has also been shown in a comparison with subsequent publications of data about 12 antidepressant agents submitted for review to the Food and Drug Administration (FDA). ⁵⁶ Only 31% of the 74 FDA-registered studies had been published, and publication was associated with a ‘positive’ outcome (as determined by the FDA). Studies that the FDA had considered ‘negative’ or ‘questionable’ (n = 36) were either not published (22 studies), reported with a positive interpretation (11 studies), or reported in a manner consistent with the FDA interpretation (3 studies). In summary, evidence from the published literature suggested that 94% of studies had positive findings, while the FDA analysis concluded that only 51% had positive findings.

Who is responsible for biased reporting of clinical research?

Reporting bias can be due to researchers and sponsors failing to submit study findings for publication, or due to journal editors and others rejecting reports for publication. Numerous surveys of investigators have left little doubt that almost all failure to publish is due to the failure of investigators to submit reports for publication, ^63,76 with only a small proportion of studies remaining unpublished because of rejection by journals. ⁷⁷ Indeed, qualitative studies of editorial discussion indicate that a study's scientific rigour is the area of greatest concern. ⁷⁸ Researchers report that the reason they do not write up and submit reports of their research for publication is usually because they are ‘not interested’ in the results (‘editorial rejection by journals’ is only rarely given as a cause of failure to publish). Even those investigators who have initially published their results as (conference) abstracts are less likely to submit their findings for full publication unless the results are ‘significant’. ¹⁴

It is now also well-established that biased reporting of research studies is associated with the sources of funding. In particular, research funded by the pharmaceutical industry has been shown to be less likely to be published than research funded from other sources, ^79,80 and that studies sponsored by pharmaceutical companies are more likely to have outcomes favouring the sponsor than studies with other sponsors. ^81,82 There are several possible explanations for the association between industry support and failure to publish ‘negative’ results. Industry may selectively publish findings supporting a product's efficacy. It is also possible that industry is more likely to design studies with a high likelihood of a positive outcome, for example, by selecting a comparison population likely to yield results favouring the product. ^83,84 This is clearly ethical.

The practice of hiring a commercial firm to write up the results from a clinical trial is common in industry trials. ⁸⁵ It has been estimated that 75% of industry-initiated studies approved by two ethics committees in Denmark had ghost authors. ⁸⁶ In these cases, the named authors listed rarely included the hired writer. The World Association of Medical Editors has made it clear it considers such ghost authorship to be dishonest (see http://www.wame.org/resources/policies – accessed 1 August 2008). Unnamed, paid medical writers may be asked to address commercial interests in the way that research methods and results are presented. When the proportion of paid medical writers is sufficiently large, the literature, and thus opinion about the drug, may be influenced. ⁸⁷

Because industry is the main funder of clinical research, it must inevitably shoulder a high proportion of the blame for this unscientific and unethical behaviour. The responsibility for biased reporting of clinical research does not lie solely with industry, however. As long ago as 1998, the Ethics Committee of the Faculty of Pharmaceutical Medicine, which represents physicians working in industry in particular, declared that:

Pharmaceutical physicians … have a particular ethical responsibility to ensure that the evidence on which doctors should make their prescribing decisions is freely available … the outcome of all clinical trials on a medicine should be reported. ⁸⁸

Dealing with incomplete and biased reporting of research

Investigations of incomplete and biased reporting of clinical research conducted over the past three decades have made clear that this is a serious and extensive problem, which threatens the best interests of patients, undermines the scientific enterprise, and wastes resources.

Various attempts have been made to overcome the effects of reporting biases. These have included statistical adjustments of the results of published studies, ^89–91 surveys of investigators in attempts to locate unpublished studies, ⁹² editorial ‘amnesties’ for unpublished trials, ^93,94 and journals and journal sections ^95–97 specifically designated for reporting the misconceived notion of ‘negative results’. ⁵ None of these approaches has proved satisfactory, however.

In 1986, John Simes showed that analyses of treatments for ovarian cancer based on the results of trials that had been registered before their results were known showed no statistically significant differences, while analyses based on all published trials did. He postulated that these differences reflected biased under-reporting of trials, and suggested that this problem should be addressed by establishing an international registry of clinical trials. ⁹⁸ Over the following three decades pressure to register trials gradually increased. ^99–104

It took a public scandal in 2004 to provide the momentum needed to lead to a consensus that clinical trial registration, which had been called for repeatedly over the previous two decades, should become mandatory. In June of that year, Eliot Spitzer, the Attorney General of the State of New York, sued GlaxoSmithKline, makers of an antidepressant drug (paroxetine), for suppressing evidence of possible serious harmful effects, thus depriving physicians of the information needed to assess the drug's risks. ^64,65 A systematic review of the relevant published and unpublished data showed that the favourable impression created by the published studies was negated when unpublished data were included. ¹⁰⁵

The scandal prompted the International Committee of Medical Journal Editors to announce that their journals would require, as a condition of considering reports of clinical trials for publication, that the studies had been registered prior to enrolling participants. ⁶⁷ Furthermore, under the aegis of the World Health Organization (WHO), it was agreed that basic information about all clinical trials should be registered, at inception, and that this information should be publicly accessible through the WHO International Clinical Trials Registry Platform. ¹⁰⁶

Public availability of full study protocols, either at trial inception ^107,108 or at registration, ^71,109 or alongside reports of trials, ¹¹⁰ is also gaining momentum. ^74,111 This further development has been fuelled by evidence of biased reporting of outcomes within studies. ^{13,56,68–71,112} This has been reflected in the development of reporting guidelines for protocols. ¹¹³

It remains to be seen how well these measures will deal with a serious problem recognized nearly four centuries ago by Francis Bacon. ²⁸

DECLARATIONS

Competing interests

None declared

Funding

None

Ethical approval

Not applicable

Guarantor

KD

Contributorship

Both authors contributed equally

Acknowledgements

The authors are grateful to Doug Altman and Mike Clarke for drawing their attention to relevant historical material; and to Doug Altman, An-Wen Chan and Sally Hopewell for commenting on earlier drafts of this brief history of reporting biases. Additional material for this article is available from the James Lind Library website (www.jameslindlibrary.org), where it was originally published