Minimum Information about a Spinal Cord Injury Experiment: A Proposed Reporting Standard for Spinal Cord Injury Experiments

Abstract

The lack of reproducibility in many areas of experimental science has a number of causes, including a lack of transparency and precision in the description of experimental approaches. This has far-reaching consequences, including wasted resources and slowing of progress. Additionally, the large number of laboratories around the world publishing articles on a given topic make it difficult, if not impossible, for individual researchers to read all of the relevant literature. Consequently, centralized databases are needed to facilitate the generation of new hypotheses for testing. One strategy to improve transparency in experimental description, and to allow the development of frameworks for computer-readable knowledge repositories, is the adoption of uniform reporting standards, such as common data elements (data elements used in multiple clinical studies) and minimum information standards. This article describes a minimum information standard for spinal cord injury (SCI) experiments, its major elements, and the approaches used to develop it. Transparent reporting standards for experiments using animal models of human SCI aim to reduce inherent bias and increase experimental value.

Introduction

The lack of reproducibility in science has been recognized for decades. ^{1 –3} Recently, this issue has been highlighted by scientists in pharmaceutical companies who reported that the majority of published basic biomedical science experiments identifying potential therapeutic targets could not be replicated. ^4,5 Similarly, a project funded by the National Institute of Neurological Disorders and Stroke (NINDS) resulted in a high failure rate in replicating published pre-clinical results for treatment of spinal cord injury (SCI). ⁶ Strikingly, studies using RhoA/Rock inhibitors or stem cells to treat SCI have more favorable outcomes if the articles do not report whether investigators are blinded during behavioral testing. ^7,8 Challenges in interpreting primary endpoints of morphological and functional neurological regeneration make careful documentation of methods used in SCI experiments essential. ^{9 –11} In addition, concerns have been raised about a number of issues in neuroscience publications, including inappropriate statistics, ^12,13 low power (calculated from 49 meta-analyses of 739 primary studies to be 21% ¹⁴ ), and a call to decrease p values from the commonly used 0.05 to 0.005–0.001. ¹⁵ Changes in standard practices are needed to improve reproducibility and thus the translation of basic SCI research to the clinic. ¹⁶

Leaders in the neuroscience community have recommended specific changes in the way basic science studies are conducted and reported. ^6,17,18 In the larger scientific community, there has been extensive discussion of strategies to improve reproducibility. For example, litter-to-litter variability can have a large impact on animal behavioral studies. ¹⁹ Remarkably, between 2008 and 2013, there were at least 20 peer-reviewed publications presenting guidelines for in vivo pre-clinical studies. ²⁰ Common recommendations included appropriate sample size, randomization of animals to groups, the use of positive and negative controls, and blinding of scientists to treatments during outcome assessments.

In addition, the general lack of data availability is a major hindrance to interpretation and reanalysis of published experiments. Critical metadata regarding how experiments are done are often missing from published methods, and the raw data underlying graphs and figures are rarely included in publications. Many journals stipulate that primary data associated with a publication, such as microarray data, must be posted in public databases. However, an analysis of publications in the 50 journals with the highest impact factors in 2009 revealed poor compliance (only 47 of 500 articles) with full deposition of primary data. ²¹

Scientists and clinicians conducting clinical trials have had to comply with standardized design and reporting guidelines for many years. For example, a common data element (CDE) system is in place for stroke and SCI trials. ^22,23 A group of editors for some 400 journals have already set standards for publication of clinical trials through the CONSORT (Consolidated Reporting of Trials) guidelines, ^24,25 and, similarly, journal editors are calling for improvements in the conduct and reporting of pre-clinical research. ²⁶ However, the CDE concept is being applied slowly in the pre-clinical arena. A parallel worldwide effort is being launched under the umbrella of the Minimum Information about a Biomedical and Biological Investigation (MIBBI) project. ²⁷ Working groups associated with various organizations and ad-hoc groups have developed reporting standards. Perhaps the most widely used is the Minimum Information About a Microarray Experiment (MIAME). ²⁸ Similar approaches are being encouraged by the ARRIVE guideline (Animals in Research: Reporting in Vivo Experiments) ²⁹ and the CAMARADES initiative (http://www.camarades.info/) for experimental models of multiple sclerosis, traumatic brain injury, and stroke. ^{30 –32} Finally, reporting standards can help reviewers evaluate and critique grants and manuscripts. It is worth noting that although reporting standards encourage the use of best practices by focusing attention on critical concepts in a particular experimental domain, they are not to be confused with experimental practice standards. Pre-clinical research requires innovation that might be hampered if only existing models or assessment methods were allowed.

The adoption of a reporting standard not only improves transparency of research and encourages the use of best practices, but it also has the additional benefit of facilitating the aggregation and interrogation of large data sets in a given domain by annotating data and metadata using standardized terminology. ³³ The number of publications each year already exceeds the capacity of individuals to find, read, and absorb them, even in relatively small research areas, such as SCI. It is also difficult for researchers to compare variables across studies because of the lack of consistency in reporting experimental design parameters. Consequently, to ensure that valuable data are not lost or needlessly duplicated, these need to be collected into user-friendly knowledge bases. Moreover, as Minimum Information about a Spinal Cord Injury experiment (MIASCI) aims to reduce bias and thereby enhance the predictive value of experimental SCI modeling, the number of animals needed to verify or falsify a scientific hypothesis could be reduced. Such an effort is in line with the “3R” (Replacement, Refinement, and Reduction) initiative to increase animal welfare ³⁴ and reduce waste in biomedical research. ^35,36

To begin to address the problems outlined above, an international group of scientists studying SCI have worked collaboratively to develop a draft standard, termed MIASCI.

Methods and Results

In October 2012, a 3-day workshop, entitled “Growth Cones and Axon Regeneration: Entering The Age of Informatics” was held in New Orleans. The 35 participants (listed in Appendix B) spent substantial time in small working groups developing lists to describe important information relating to the performance of SCI experiments (i.e., the “metadata” concerning these experiments). After the meeting, the documentation from these discussions was collated and circulated to the meeting's advisory group for further review. A draft MIASCI checklist was assembled based on the guidelines recommended by the MIBBI project. ²⁷

The draft MIASCI was presented at four international meetings to obtain feedback from the SCI community. ^{37 –40} Finally, an online poll was conducted to obtain specific feedback concerning the importance of individual items in the checklist. Seventy-six SCI experts (of 125 invited to participate) from the global SCI research community participated in this poll. Thirty-four percent of the participants had more than 10 years of experience in the SCI field and 24% had 4–10 years of experience. Finally, individuals in the SCI community were invited to become part of the MIASCI Consortium (Appendix A). They reviewed the MIASCI checklist and the manuscript, made suggestions, and agreed to be co-author.

On December 11, 2013, a draft MIASCI was listed at the BioSharing portal (http://biosharing.org/bsg-000541), the current host of MIBBI checklists, and the MIASCI checklist (version 0.8) was posted at SourceForge (https://sourceforge.net/projects/miasci), a resource for open-source software development and distribution. The 1.0 version was posted at SourceForge on April 14, 2014, and is provided in the supplementary material. Please visit the MIASCI SourceForge site for the current version.

MIASCI data elements

Typical minimum information standards strive to minimize the number of required data elements to increase the likelihood that investigators will provide essential information. The average minimum information standard has approximately 50 required data elements. Because the design and analysis of SCI experiments varies widely, the draft MIASCI has approximately 250 required data elements. However, most SCI studies only cover a narrow range of these elements, and for a given study, the number of applicable data elements from the MIASCI is likely to be 100 or less.

The draft MIASCI has 11 major sections: investigator, organism, surgery, perturbagen, cell transplantation, biomaterials, histology, immunohistochemisty, imaging, behavior, and data analysis and statistics.

The section on organism covers important information, such as animal source, strain, and congenic status. Correct identification of strain and details about congenic status is increasingly recognized as essential information. Different mouse strains show different response to injuries and recovery patterns after SCI and optic nerve crush. ^{41 –43} The same rat strain from two different regional suppliers of the same commercial vendor can yield different outcomes in SCI experiments. ⁴⁴ Thus, the organism section also covers information about housing that has proven critical to the outcome of SCI experiments, such as animal batching, environmental enrichment, and light/dark cycle. ¹⁰ Animal batching (randomized, randomized block, and so on) can be complicated, ⁴⁵ especially when working with transgenic mice that might be difficult to produce in large numbers. Nonetheless, reporting how this is managed is important. Details about control groups (littermates, congenic status, and so on) should also be documented.

The surgery and behavior sections include whether the surgeon(s) are blinded to treatment group (concealed allocation), the type of injury and device used to inflict it, including whether compression was maintained after impact and for how long, as well as data elements regarding drugs (e.g., anesthetics, analgesics, and antibiotics) given to animals and hydration. Based on the recommendations of Landis and colleagues, ¹⁷ those sections also cover animal batching methods, power calculation methods, and estimation of effect size and details about whether observers were blinded to the experimental intervention. Small sample size in biological studies is a general problem, ¹⁴ but can be a special problem in SCI research, because animal experiments are especially time-consuming and expensive. Therefore, it is critical for scientists to consider and report this aspect of their work. In the behavior section, special attention is paid to collecting important metadata about the widely used BBB test, ⁴⁶ BMS test, ⁴¹ and various gait analysis methods.

A major focus of SCI research is in evaluating the ability of agents to improve recovery after injury. The drug discovery field has adopted the term “perturbagen” to refer to small molecules, peptides, antibodies, oligonucleotides, and so on, that alter a biological process by interfering with one or more molecular targets. Common perturbagens used in the SCI field are U.S. Food and Drug Administration–approved drugs, chemical compounds, naturally occurring bioactive agents, siRNAs, shRNAs, and cDNAs. The route (oral, intravenous, subcutaneous, minipump, and so on) and other aspects (time since injury and dose or doses) of administration of the perturbagen needs to be documented. The Minimum Information About a Cellular Assay (MIACA) project (http://miaca.sourceforge.net) proposed a standard that included data elements about perturbagens, such as vendor, catalog number, stock concentration, storage, and solvent. MIACA also covers experiments using viruses (type and titer). In accord with best practices in MIBBI and ontology development, we have chosen to reuse relevant sections of MIACA to describe perturbagen use in MIASCI.

The cell transplantation section requests information about cell source, isolation and purification methods, batch, and passage inspired by good laboratory practice/good manufacturing practice standards. Additional information about growth factors and differentiation methods is included, as is information about any immunosuppression methods or use of supporting growth factors administered at the time of cellular delivery. In addition, the time since injury, site, route, and volume of introduction are also required as well as the numbers of cells and numbers of times the cells are delivered and over what duration.

Biomaterials are also covered, because they are commonly used in SCI experiments. Domain experts recommended that details about composition, mechanical properties, manufacturer, and manufacturing methods be collected along with details about release kinetics, as well as mode and rate of metabolism, if the material is used to deliver a compound or biologic and whether any safety data are available regarding biodistribution after introduction.

Histology, immunohistochemistry, and imaging are three highly interrelated areas often used in evaluating the results of SCI experiments. The histology section of MIASCI deals with fixation, tissue processing, and axonal tracing methods. The immunohistochemistry section is devoted to collecting information about antibodies used in the research project. The Journal of Comparative Neurology has set a very high bar for describing antibodies and verifying the specificity of antibody labeling. ⁴⁷ Most other journals do not have such high standards, so better documentation of antibodies and their specificity remains a major concern ⁴⁸ (source and catalog number, at least, are essential, whereas batch information is very desirable). The Neuroscience Information Framework (NIF) has an antibody registry (www.antibodyregistry.org), which provides unique identifiers for antibodies, including those available from approximately 200 vendors. MIASCI will use these identifiers and will link to the NIF search tool to facilitate data entry into MIASCI. Basic information about the imaging platform is requested, especially information about image acquisition and analysis software and settings, as well as the specific counting and measurement methods used. ¹⁷ Software developers and computational biologists have discussed the critical importance of documenting software algorithms and settings. ^{49 –51} This includes archiving exact versions of programs used and version control of all custom scripts. When this logic is applied to image analysis, it requires documentation of software versions and settings, optical filters, and software filters, such as thresholds, used during image acquisition and analysis. We hope that the MIASCI will encourage the use of these imaging best practices.

The data analysis and statistics section is relatively brief, but covers concepts related to blinding of investigators, the kinds of technical and biological replicates used, prospective analysis plans and the statistical tests performed, normalization schemes, and positive and negative controls. These data elements are deemed critical by the SCI domain experts and are common recommendations in standards for life science research. ²⁰

An inevitable limitation is that a MIASCI statement relies on good faith and therefore cannot detect or prevent intentionally false statements. However, the utility of MIASCI could be tested experimentally, for example, by a prospective study comparing relevant publications before and after inauguration of the MIASCI standard. MIASCI acceptance and integration could be measured by analyzing whether experimental characterization according to the MIASCI statement is increasing. MIASCI efficacy could be investigated by determining whether the precision of the primary outcome measure effect size increased after inauguration of the MIASCI standard.

Summary

The MIASCI draft standard has been proposed to capture SCI experimental details, including the investigator, organism, behavior, surgery, perturbagens, histology, immunohistochemistry, imaging, biomaterials, cell transplantation, and data analysis. MIASCI provides a basis for establishing good laboratory practice in the modeling of SCI. This first version of MIASCI will require modifications and further development as new methodologies are applied to the study and treatment of SCI. In some cases, simply requiring the use of other standards, such as MIAME or minimum information about an RNA-Seq experiment, will be appropriate. In other cases, however, expanding or developing a new section may be needed. At present, the biomaterials section is minimal and there is no coverage of some molecular techniques, such as in situ hybridization or real-time polymerase chain reaction. Because scientists, reviewers, and editors use the MIASCI when reporting SCI studies, this should not only improve reporting, but also promote the adoption of best practices, such as randomization of animals to treatment groups, appropriate use of power analysis, and the blinding of scientists to treatment conditions. To facilitate the adoption of MIASCI, an overall framework, including an ontology and simple-to-use annotation tools, is being built and will be freely available for data annotation and submission. An important benefit of MIASCI adoption is that it will greatly facilitate the population of domain-specific databases, such as NIF, Regenbase, ³⁷ Adam Ferguson's SCI Database, ^52,53 Barbara Grimpe's SCI text mining initiative, ⁵⁴ the University of Dusseldorf Center for Neuronal Regeneration's SCI Database, ⁵⁵ and the CAMARADES project on meta-analysis of animal studies. ^30,32,36 Thus, establishment of transparent reporting standards in pre-clinical SCI research should facilitate experimental accuracy and lab-to-lab reproducibility, helping to speed up the development of novel therapeutic approaches to SCI.