Failure to publish negative results: A systematic bias in psychiatric literature

This month’s edition of ANZJP publishes an important negative study. Dean et al. (2016) report on the effects of N-acetylcysteine in children with autistic spectrum disorder. The trial was prospectively registered in 2010, the last participant was enrolled in 2013, there was a 6-month follow-up and the trial was published online in June 2016. Notable also is that the primary outcome measures are exactly as stated in the registration. This represents relatively rapid publication of important negative data, in line with the appropriate transparent processes, for which the authors are to be commended. However, such rigorous and rapid publication of negative results is relatively rare, and this represents an important problem in psychiatric literature.

Escalating and negative impact of publication bias

Publication bias has been highlighted as a profound problem in scientific research many times previously (Chan et al., 2014) with a particular emphasis on the potential clinical harm which may be caused, and the research funding which is wasted by the incomplete publication or incomplete availability of research data sets. In particular, failure to publish negative results not only has detrimental effects on the overall integrity of scientific knowledge but also leads to immense waste of time and resources, as other scientists in the field may repeat the same unsuccessful experiments. According to a large study that examined more than 4600 publications, publication bias has been escalating across multiple disciplines in recent decades, with the largest increases occurring in the fields of psychology and psychiatry (Fanelli, 2012). An important example from psychiatry is the non-publication of antidepressant trials. Turner et al. (2008) found that, of 74 registered Food and Drug Administration (FDA) studies of antidepressants, 31% were not published and these were almost entirely negative trials. In an individual case, Chan et al. (2014) showed that, for reboxetine, the evidence differs significantly between published and unpublished trials.

Driving forces behind publication bias

There are multiple systems issues which create this situation, in which there is a systematic bias to the publication of positive findings. Pressures exist in academic life and in the pharmaceutical industry to publish positive results from research. In industry, positive trials are required for registration of products and may increase the use of pharmaceuticals. In academic life, researchers’ careers depend upon ‘high impact’ publications, and these are generally more likely if positive results are achieved. Negative results, on the other hand, potentially detract from the use of pharmaceuticals, are generally less likely to be published in high impact journals and are less likely to be cited either academically or in the popular press. Negative trials as part of research programmes may detract from subsequent grant applications for further trials in the same area. There are therefore a number of reasons why negative trials might either deliberately be withheld or not prepared for publication.

The negative studies that are submitted for publication are more likely to be rejected than positive studies (Turner et al., 2008). This is because journals compete with each other based on impact factor and citations, and it is widely perceived, possibly quite correctly, that positive findings attract more citations. Studies with negative findings are more prone to criticism for methodological flaws and are more likely to be rejected on this basis. However, if rejected negative studies were compared with accepted positive studies based only on methodology, it is likely that there would be a very substantial overlap. In clinical trials, the distinction is made between a negative study and a failed study. A negative study is one in which it is deemed that the study results relate to the inactivity of a drug which has been examined. In failed studies, it is deemed that an investigational drug has not shown superiority over a placebo or another comparator because the design of the study was such that this led to a failure to show a benefit. However, if a similarly designed study happens to show a benefit, then this is deemed a positive study and is much more likely to be published.

An important related point is that many studies in the psychiatric and general medical literature are underpowered. For example, many studies of neurocognitive function, comparing moderately severe depression with healthy controls, show effect sizes of the difference between depressed patients and healthy controls between 0.4 and 0.6. Therefore, to show a difference between two groups, the groups should be somewhere above a number of 40 per group based on the larger possible effect size or 100 per group based on the smaller effect size. Many neuropsychological studies comparing depression with healthy controls have been published and have shown positive results despite extremely small groups. Had these studies failed to show a difference, then there is little doubt that they would have been rejected after peer-review, based on small sample sizes.

Therefore, it is likely that our clinical practice, which is based on randomised control trials and meta-analyses, and our understanding of the biology of mental illness is biased. Many positive results upon which we base our knowledge are probably chance findings and are balanced by a weight of negative evidence which has either been analysed and presented to a journal but rejected, deliberately suppressed, fully analysed and never presented or analysed in only a preliminary fashion and abandoned because the results did not appear positive.

Other ways in which negative findings are minimised

A more subtle but equally problematic way in which negative data may be ignored is that only positive outcomes are presented in a publication, leaving outcomes on other scales unpublished. As noted below, a system of prospective registration of clinical trials is in place to combat this issue. However, data suggest that the top-tier psychiatric journals still allow changes in registered primary outcomes without this being highlighted in the paper (Scott et al., 2015). Outside the area of randomised control trials, there is little in place to avoid this problem. The intended outcome measure is not pre-registered and indeed there is not the same expectation that a primary measure will be stated. While in some areas, it is standard to collect particular sets of data, the way in which that data is analysed can vary and data sets which are essentially negative may have been analysed in varying different ways before a positive result is found. Datasets for which no positive result is found on a particular set of measures may not be published, while small but secondary measures may be published simply because the results are positive.

A related issue is that of large data sets such as longitudinal studies with multiple measures over time. These data sets have almost infinite combinations of factors which can be examined in correlational or more complicated analyses. Multiple hypotheses may be tested but ultimately only those which result in positive findings may be published. For researchers and clinicians, there is no way of knowing that in fact a positive result was one of 20 hypotheses tested with 19 being rejected without any accompanying report. This raises the possibility that positive results from such large data sets are actually chance findings with many other equally plausible hypotheses having been rejected because of insignificant or inexplicable results. There is currently no central repository or system whereby such studies are able to register their hypotheses a priori, and access to the raw data is usually not possible.

Attempts to overcome the problem

There are various ways in which the field has tried to overcome the issue of non-publication of negative trials. Systematic reviews sometimes attempt to identify unpublished trials, and meta-analyses use sensitivity analyses or funnel plots in attempts to determine whether there has been a bias towards the publication of positive trials. However, particularly in situations in which there are relatively few trials in an area, the statistics of funnel plots or sensitivity analyses are not sensitive enough to determine this with any certainty.

In 2005, the International Committee of Medical Journal Editors mandated that clinical trials should be prospectively registered as a condition of publication. Many clinical trials are registered prior to commencement on Clinical trial registers. Potentially, this allows authors of reviews or meta-analyses to check a clinical trial database for trials which may have been conducted but not reported. However, firstly, registration of trials is not mandatory and smaller trials may not be covered. Secondly, as noted, it appears that journals do not always follow their own guidelines (Scott et al., 2015) regarding the accuracy of prospective registration. Thirdly, the ‘safeguard’ of prospective registration does not apply to anything other than clinical trials and therefore does not safeguard against publication bias in, for instance, biological or phenomenological studies in groups with various psychiatric illnesses. Such studies are not registered, the primary outcomes are often not specified a priori, and they are often small, making the publication of negative results less likely (Turner et al., 2008).

How can this situation be changed?

At least, increased awareness of this issue may lead to change and, for example, there are areas of medical and public health research in which negative studies are able to be and are regularly published. Epidemiological studies with large samples frequently publish null results, on the premise of replicating or extending previous findings, or in attempts to untangle possible causal factors underlying the associations between exposures and outcomes. Publications of genetic studies also feature null results, but to a lesser extent than epidemiological studies, and with increasing rarity. However, it is clear that these areas are the exceptions, rather than the rule.

The advent of multiple open access journals may certainly improve this situation. However, while this gives access to large numbers of people to published results, these journals generally also select papers, have an impact factor which they compete to increase and are likely to have the same selection biases.

The mandatory pre-registration of clinical trials could be extended to observational studies, and if the process was made simple enough, it would not require a great deal of additional work on the part of researchers. However, as noted, it has been shown that in fact many clinical trials published now have not either been pre-registered or kept to the outcomes which they stated in their registration (Scott et al., 2015), and this problem would need to be addressed in order for such a system to work. Funding bodies could require the official registration of research questions or hypotheses for large research projects and to provide information on results of hypothesis testing. One possible way of doing this would be to publish successful grant applications. While this may not be fully feasible in areas in which more exploratory work is being undertaken, it may provide a set of searchable databases for researchers to learn about null findings in a particular area. Some grant awarding bodies now make public access to databases of publicly funded studies mandatory. This will allow other researchers to check results in particular areas in these databases and produce negative results which might not otherwise have been published.

Meanwhile, Editors and reviewers should make a concerted effort to maintain the same standards of methodology and reporting for publication in journals regardless of whether the trial is positive or negative.