Regulatory Forum Opinion Piece*

Abstract

Historically it has been easier to publish positive scientific results than negative data not supporting the research hypothesis. This appears to be increasing, with fewer negative studies appearing in the literature across many disciplines. Failure to recognize the value of negative results has important implications for the toxicology community. Implications include perpetuating scientific fields based upon selective or occasionally erroneous, positive results. One example is decreased vaccination rates and increased measles infections that can lead to childhood mortality following one erroneous positive study linking vaccination to adverse effects despite multiple negative studies. Publication of negative data that challenges existing paradigms enhances progress by stopping further investment in scientifically barren topics, decreases the use of animals, and focuses research in more fruitful areas. The National Toxicology Program (NTP) publishes both positive and negative rodent data. Retrospective analysis of the NTP database has provided insights on the carcinogenic process and in the gradual acceptance of using fewer animals in safety studies. This article proposes that careful publication of both positive and negative data can enhance product safety assessment, add robustness to safety determinations in the regulatory decision-making process, and should be actively encouraged by those determining journal editorial policy.

Introduction

This opinion piece explores bias against publishing negative data, how the continuation of this bias adversely affects human health, and why fewer reports with negative data are appearing in the literature now than ever before. Scientists are often discouraged from publishing negative results (i.e., data that do not support the study initiating hypothesis), in part, by pressure from academic funding agencies looking for positive results (Matosin et al. 2014). However, negative data are the integral components of the “Scientific Method” and are crucial in refuting outdated paradigms, in creating new ideas, and in moving scientific understanding forward. We all may recall a bias for generating positive results, beginning our careers as graduate students, where it can be difficult to complete a PhD thesis with largely negative data. Many universities require at least one accepted publication prior to awarding a PhD degree. With the bias against publishing negative data, there is intense pressure to extract and possibly unduly focus on any positive signal, to the possible detriment of the discipline as a whole. To reinforce this point, one of the key eligibility criteria for entry into the diploma examination for the European College of Veterinary Pathologists is the publication of two papers in a veterinary pathology journal (ECVP 2015). This drive for publication acceptance may encourage less pioneering research, engendering the selection of safer projects likely to produce positive results that follow current practices of status quo, with a potential bias toward publicizing positive data from a study series while suppressing the negative, although potentially equally as valuable, negative results. For many of us, these early experiences can color and consequentially affect our subsequent decisions/behavior throughout our career (Table 1).

Table 1.

Barriers to publishing negative data.

Responsible Person/Area	Rationale	Effect
Graduate Students	Not acceptable for thesis	Valuable data left in file, others destined to try similar studies
Academia	Not worth time, need positive results for funding	Existing but wrong paradigms remain, slows progress in research
Product Development	Working on positive products, competitive advantage not to publish negative data	Slower progress in getting new products to market, unnecessary use of research animals
Reviewers	Not worthy for readers, was the study well done? If refuting a positive study, how well was the original replicated	Publishing bias, erroneous positive results remain in the literature
Journals	Will affect impact factor	Publishing bias, erroneous positive results remain in the literature

Pharmaceutical product safety submissions are one area where negative data are recorded and submitted to registration authorities throughout the world. Most of the data, including the animal studies (often negative data) used in support of product registration, is of limited availability to the scientific community as a whole. Decades of unsuccessful efforts, with unsuitable protein targets, or using unproductive chemical series to develop potential cancer therapeutics, for example, are mostly unpublished and difficult, or impossible, to locate. A registry, or the encouraged publication, of unsuccessful endeavors, would engender potentially enormous savings in resources and animals throughout the industry, while allowing a focus on untried compounds or undersubscribed biological targets. Negative studies are crucial to providing new products for improving human and animal health and in making the most efficient use of ever more limited resources. This will require a change in opinion since knowing that an area has proved negative after years of research does confer some competitive advantage knowing that others are expending resources in the same area. A cursory review of the literature shows clear bias against the submission, and publication, of negative data, with the very real potential to adversely affect the scientific community in many disciplines including that of toxicologic pathology (Scargle 2000).

There Is a Reluctance to Publish Negative Data

A recent analysis of scientific publications for positive, versus negative, conclusions showed that negative results are gradually disappearing from the literature in most disciplines and countries (Fanelli 2012). This is considered to be due in part to scientists not submitting negative data (Franco, Malhotra, and Simonovitz 2014), but responsibility also lies with the journals in their sometimes stated policies of publishing novel findings, of widespread interest (Young, Ioannidis, and Al-Ubaydii 2008).

Scientists have traditionally rushed to publish positive data, while negative data receives little if any attention, often remaining within a laboratory notebook, resigned to the individual’s drawer and lost forever to the scientific community (Scargle 2000). The National Science Foundation funded 221 similar time-sharing experiments in social science (Franco, Malhotra, and Simonovitz 2014). The investigators in these studies were known and those without publications were contacted. This review showed that 64.6% of the studies with a negative result were not submitted for publication (Franco, Malhotra, and Simonovitz 2014). Positive data more often receives a higher number of citations, which can be a competitive advantage for future research funding (Fanelli 2012). This creates an environment of publishing positive results, similarity among accepted published papers (mostly positive data), and domineering research hypotheses (Young, Ioannidis, and Al-Ubaydii 2008). Results that do not support the “currently acceptable hypotheses” encounter considerable resistance, including the unwillingness of journals to publish the negative results.

A literature review supports the view that journals are biased against publishing negative data (Dirnagl and Lauritzen 2010). Scientists and journals are intensely interested in impact factors, attracting what they perceive are increasingly high-quality articles and being successful economically. In a social context, where the news concept is based on the largest effects, and the strongest associations, the most exciting and unusually novel effects are preferentially published (Young, Ioannidis, and Al-Ubaydii 2008). Nature and Science actively publicize their low acceptance rate (8–10%), creating an illusion of high merit and great exclusivity. However, the initial judgment for acceptance is based upon topical interest to a broad readership (favoring positive novel data) and only these are subsequently submitted to the scientific peer review process (Young, Ioannidis, and Al-Ubaydii 2008).

Publication of both positive and negative data can result in considerable progress in a scientific field. A plethora of short-term genotoxicity assays existed in the 1970s, often with each assay suggested to be the definitive means of determining genotoxicity, and possible carcinogenicity, of untested chemicals (International Agency for Research on Cancer [IARC] 1980). Publications of correlative results were selectively biased in favor of strongly positive mutagenic carcinogens (Tennant et al. 1987). The National Institute of Environmental Health Sciences (NIEHS) conducted an evaluation of existing assays, where each laboratory evaluated chemicals in their own assay in a blinded fashion in an effort to minimize this positive bias (Tennant et al. 1987). Publication of both the positive and the negative results quickly showed that the respective genetic toxicity tests did not all complement each other and that combinations of the tests were no more predictive of carcinogenicity than the Salmonella typhimurium assay alone (Tennant et al. 1987; Zeiger 1998). This conclusion has resulted in a considerable saving of resources in targeting and limiting the recommended mutagenicity assays. Negative data are often useful wherein combination with positive data can lead to fresh hypotheses and newer and safer products (Ioannidis 2005). Meta-analyses of effects in the literature may result in an overestimation of an effect because of the lack of negative data available to the analyses (Scargle 2000) as was the case in genetic toxicity tests prior to publishing the negative data.

Negative Data Has Difficulty Correcting Published False Statements

Matosin et al. (2014), in an editorial addressing the difficulty of communicating negative results, discusses the tragedy that sprung from reporting a spurious association between autism and childhood vaccination (Wakefield et al. 1998). The Wakefield paper reported an association between the measles–mumps–rubella vaccination (MMR) and the onset of autism based on only 8 affected children without the inclusion of controls. These autism cases were brought to the attention of Wakefield, a London research gastroenterologist either by the parents of the children or by their family physician (Wakefield et al. 1998). The panic that ensued following publication of the Wakefield et al. (1998) report resulted in a decade long decrease in child immunization (Matosin et al. 2014) despite subsequent publication of a series of reports that refuted any association of vaccination with autism. These opposing studies were generally conducted under strict rules of epidemiological research and were considerably better powered and appreciably larger than the original report of 8 children. For example, a review of over 500,000 children born in Finland over a 6-year period showed no relationship between the date of vaccination and any subsequent development of autism (studies cited by Gerber and Offit 2009). Gerber and Offit (2009) also showed that approximately 50,000 British children per month receive the MMR vaccine between 1 and 2 years of age, an age at which autism first presents itself. Approximately 25 British children per month would have been diagnosed with autism, irrespective of receipt of the vaccine, given the prevalence of autism of 1 in 2,000 children in England at the time of the Wakefield publication (Gerber and Offit 2009). A Canadian study involving 27,749 schoolchildren found an inverse relationship with higher rates of autism when the MMR vaccination rates decreased (Fombonne et al. 2006). The Wakefield et al. (1998) article was ultimately retracted in 2010 but concerns still persist among the general public about the relationship of vaccination to the onset of autism. The consequence of the resulting decrease in MMR vaccine is the current rise in morbidity, and mortality, in developed countries from the preventable diseases of MMR (Matosin et al. 2014). The Centers for Disease Control and Prevention (CDC 2015) notes that as many as 1 of every 20 children with measles will end up with pneumonia, and for every 1,000 children with measles, 1 to 2 will die from it.

The public but also surprisingly scientists continue to cite withdrawn manuscripts (Van Noorden 2011) demonstrating the persistence of novel positive effects despite publications to the contrary.

In another example, Wertheimer and Leeper (1979) reported an association between electrical wiring configurations outside of the home and childhood cancer that led to widespread health concerns regarding exposure to electromagnetic fields (EMFs). In the Wertheimer and Leeper (1979) publication, EMFs were not measured, and “wire codes” were assigned in the knowledge of whether the home had a healthy child or a child with leukemia. A subsequent study showed poor association between wire codes and magnitude of magnetic fields measured within the home (National Research Council [NRC] 1996; Savitz and Poole 2001). The Wertheimer and Leeper (1979) report spawned a whole new field of research, initially supported by the U.S. Department of Energy (U.S. DOE) Biological Effects Program, and the U.S. Electric Power Research Institute (EPRI). Studies reporting that EMF-caused potentially adverse effects were considered topical and novel and often received accelerated approval for publication, increasing the public concern for EMF. A strictly controlled scientific program to address potential health concerns of human exposure to EMF was established in 1992 by the U.S. Energy Policy Act (Section 2118 for Public Law 102-486) utilizing the expertise of many U.S. agencies, with the U.S. DOE and the National Institutes of Health’s (NIH) NIEHS laboratories leading the program. More than 50 million U.S. dollars were allocated in the congressional mandate as well as additional financial contributions from multiple government agencies. The NIEHS research program involved short-term and long-term animal cancer studies, as well as EMF exposures of cellular systems, to investigate any possible health effects of EMF. Epidemiology studies were ruled out as being too costly and too long to complete within the time frame required of the U.S. Energy Policy Act. In vitro exposure of cells and cellular systems to EMFs, conducted in a blinded fashion, were essentially negative, refuting the positive effects reported in the literature of EMF exposure on gene expression, intracellular calcium, colony growth in soft agar, and induced ornithine decarboxylase activity (Boorman, Owen, et al. 2000; Morehouse and Owen 2000a, 2000b). Studies involving 1,000 rats (Boorman, McCormick, et al. 1999) and 1,000 mice (McCormick et al. 1999) exposed to EMF for 2 years failed to show an increase in leukemia or mammary cancer (Boorman, Rafferty, et al. 2000; Boorman, McCormick, et al. 2000). John E. Moulder, senior editor, Radiation Research, devoted a section of a journal issue to publication of this EMF research (Moulder 2000); the data presented was subject to the standard review process, with the caveat that no manuscript would be rejected simply on the grounds of it demonstrating a negative result. Seventeen articles were subsequently accepted that refuted many of the previously positive in vitro findings already present in the EMF literature, and readers can review the full extent of the research in Radiation Research, 153, 2000. NIEHS, in its report to the U.S. Congress, citing these articles among others, was able to state categorically about EMF exposure that “The weak epidemiological associations and lack of any laboratory support for these associations provide only marginal, scientific support that exposure to this agent [EMF] is causing any degree of harm” (Boorman, Bernheim, et al. 1999, p. 36).

Since the original publication by Wertheimer and Leeper (1979), more than 100 epidemiology studies have been published (NRC 1996), the majority of which found no association between EMF exposure and increased incidences of cancer. Major epidemiology studies in the United States (Linet et al. 1997), Canada (McBride et al. 1999), and the United Kingdom found few effects of exposure to EMFs (UK Childhood Cancer Study Investigators 1999). Following the DOE/NIEHS research effort, and the subsequent report to the U.S. Congress (Boorman, Bernheim, et al. 1999), a similar negative report in Japan (Takebe 2001), and negative reports elsewhere (NRC 1996), financial support for research on EMF has dramatically declined.

Publishing Negative Data Has a Positive Effect

The NTP at NIEHS publishes results of short- and long-term, generally 2-year, rodent toxicology and carcinogenesis studies, in NTP Technical Reports (NTP 2015) and/or, if appropriate, in peer-reviewed scientific journals. More than 500 long-term rodent carcinogenicity studies are available online in the NTP Technical Report series. The availability of this vast database, including both positive and negative studies, has been critical to the retrospective analysis of structure activity relationships for potential carcinogenic chemicals (Ashby and Tennant 1991), using prechronic lesions as predictors of tumorigenicity (Allen et al. 2004), and other studies too numerous to mention in this article. The public availability of both negative and positive data over a large database, coupled with the published data from the pharmaceutical industry, provided support for proposals to make carcinogenicity testing shorter, less expensive, and with the use of fewer animals (Sistare et al. 2011).

However, generally much of the (often negative) data used in support of the development of pharmaceuticals is essentially lost to the scientific community at large- While it is maintained in company archives, it is difficult to access or requires access fees. The aforementioned work by Sistare et al. (2011) presented a novel way of collecting and using negative data, with minimal, potentially adverse, competitive disadvantage, by placing 182 coded rat carcinogenicity studies, from marketed and nonmarketed pharmaceuticals, from 13 pharmaceutical firms, onto the World Wide Web database. Analyses of the data, including genotoxicity data, hormonal perturbation, and histopathology, suggested that many of the 2-year rat studies could be replaced by 6-month chronic rat studies and a 6-month transgenic mouse study, saving time and resources, and reducing dramatically the number of animals required to safely register a novel pharmaceutical (Sistare et al. 2011). This was a one-time effort and the data available were limited but exemplify that finding a mechanism to capture more studies could speed up drug discovery by preventing companies investing in chemical “dead ends” or intractable protein targets. Further, this could enhance compound development by focusing effort onto proven active chemical structures, reduce the number of animals used in research, and provide the regulatory agencies with greater confidence in the regulatory decision-making process as data from projects that never reached the submission stages would be available for comparable review with other novel drug submissions. A degree of initial reluctance to such an altruistic approach can be anticipated from pharmaceutical companies in this respect, especially if chemical structures were to be made freely available, but the absence of structural information, in regard to a given toxicity, would clearly limit the usefulness of the exercise. A detailed discussion of how this hurdle could be overcome is beyond the scope of this article. For the International Life Sciences Institute (ILSI) program on transgenic mouse models, initial steps involved a workshop made up of representatives gathered to see what data might be useful, how data could be made available, with the pros and cons clearly debated, and the potential dividends for the stakeholders involved (Robinson and MacDonald 2001).

Opportunities to Publish Negative Data

Access to negative data should save time, resources, and animals in avoiding efforts to repeat already failed approaches and therapies. On a practical level, the Journal of Negative Results in Biomedicine (JNRBM) was launched in 2002 on the premise that progress depends on collaboration, teamwork, and open communication of all results, both positive and negative (Anderson, Sprott, and Olsen 2013). However, JNRBM appears to have developed into an eclectic repository where relevant results may not receive adequate exposure (Dirnagl and Lauritzen 2010). The Journal of Cerebral Blood Flow and Metabolism, published by the Nature Publishing Group, introduced a specific negative results section (Dirnagl and Lauritzen 2010). The results are published as a one-page summary in the print addition, with the full paper appearing online. The studies have to meet the same rigorous scientific standards for publication of positive study data, and in addition, type II error (false negatives) considerations are also required to be addressed (Dirnagl and Lauritzen 2010). The Journal of Pharmaceutical Negative Results (http://www.journalonweb.com/jpnr/) is a peer-reviewed scientific journal that publishes theoretical, and empirical, negative findings, and research failures, in the pharmaceutical field. One goal of the journal is to prevent newer generations of researchers from wasting their time and money repeating the same studies and finding the same, unpublishable, results. The journal Neurobiology of Aging also includes a special section on negative data (Coleman 2004), while the Journal of Universal Computer Science also has a forum for negative results established in 1997 (Prechelt 2012). Furthermore, the Journal of Universal Computer Science website (http://page.mi.fu-berlin.de/prechelt/fnr/) lists other venues that specialize in publishing negative results in biology/medicine such as the Journal of Errology, All Results Journal: Biol, and Journal of Negative Results—Ecology and Evolutionary Biology.

Will Negative Data Add to Information Overload?

There is considerable effort in the scientific literature to avoid publishing duplicate information that simply adds to the bulk of data without advancing or providing anything significant or new to the chosen field of science. This is a laudable aim since almost every journal has a policy for acceptance, and rejection, of submitted manuscripts, and without strict editorial control on this aspect, useful manuscripts might not reach the scientific public simply by being outside of the stated limit for the journal. Saturating the literature with similar manuscripts makes scientific searching less efficient and more difficult, when thousands of papers are retrieved from particular areas for simple search terms. There is also intense pressure on researchers to publish; considering the effort to construct a manuscript, which may take many months in preparation, authors need to judge what they believe will have the best chance of acceptance. All of these issues tend to lead authors away from attempting to publish negative data, and therefore journals are less likely to be presented with negative data. On the other hand, it might be argued that replacing less than rigorously conducted experimental data with more precisely obtained negative data may have a greater potential to influence a scientific field and therefore to be of greater overall value to society. For this to happen, it may be useful for journal editors to add an active policy for publishing negative data to their criteria for acceptance, to overcome the natural reluctance of researchers to submit data that supports the null hypothesis concept, namely, the experiment suggests that the hypothesis driving the work was wrong! As with submissions of positive study data, negative data need to challenge what is already published. We are certainly not advocating publishing negative study outcomes that could be predicted. We are suggesting criteria for publishing negative study data include that the studies were well conducted, include a significant amount of data, and have the potential to challenge or actively change existing paradigms (Table 2).

Table 2.

When to consider publishing negative data.

Is the result unexpected? Is it a sizable amount of data?

Is the study well performed and has it been replicated? Have type II errors (false negatives) been considered?

Does the data challenge existing paradigms?

Have the reasons for failing to support the existing paradigm been well explored?

Does it have the potential to save resources and animals in future studies?

Toxicologic Pathology Needs to Publish Negative Data

Toxicologic pathologists, and toxicologists, create large databases in support of safety assessment. In most cases, the data are part of a long developmental process, and most firms are reluctant to share data for perceived competitive reasons. Sistare et al. (2011) demonstrated a means of sharing a pharmaceutical database with protection of the propriety data, but as discussed earlier this has the potential to limit its usefulness.

Another major joint effort by pharmaceutical firms was the sharing of positive and negative data evaluating reference compounds in transgenic mouse models which eventually led to the adoption of the H2ras mouse model for carcinogenicity studies saving time, resources, and animals in product safety assessment (Robinson and MacDonald 2001; Jacobson-Kram, Sistare, and Jacobs 2004). The NTP/National Cancer Institute toxicology studies also clearly demonstrate the value of publishing both positive and negative data for the toxicology community at large. Over the years, NTP provided historical background information on the variable tumor incidence seen in the F344/N rat and the B6C3F1 mouse. The NTP results have subsequently been used retrospectively in a number of key publications to construct hypotheses for modes of action and structure–activity relationships between chemical series (Haseman and Lockhart 1993; Haseman et al. 1997; Haseman et al. 1984), a clear demonstration of utilizing negative and positive data in science.

Not publishing the negative NTP studies, alongside those showing carcinogenicity, would have seriously inhibited toxicological progress over the last 30 plus years. Thus, the authors of this article urge that consideration be given to publishing negative data when the results challenge existing paradigms, save others from repeating studies that have already been done, and continue to help reduce the number of animals and resources used in research. The approach of the Journal of Cerebral Blood Flow and Metabolism in publishing negative data in a single page, with the complete study online for further review, demonstrates an efficient means of sharing negative data. The editorial board of Toxicologic Pathology may want to consider whether creating a forum for publishing the results of negative safety assessment studies would serve an equally useful purpose.