Abstract
Omitting quality characteristics in animal stroke studies leads to an overestimation of the efficacy of candidate stroke drugs. Nevertheless, the methodological quality of preclinical stroke studies is often limited. As publishing of research results in high-impact journals is an important motivation for scientists, we analyzed whether study quality predicts high-impact publishing. Animal stroke studies of neuroprotective drugs that were recently investigated in clinical phase II/III trials were included in the analysis. Data on the study quality and other important study characteristics were extracted. Regression analyses were performed to estimate the effect of the study characteristics on the journal's impact factor. We identified 117 studies that investigated 12 different drugs. Study quality was not associated with the impact factor before (β=−0.2,
Introduction
Numerous candidate neuroprotectants showed beneficial effects in animal experimental stroke studies, whereas none of these agents was tested successfully in phase III clinical trials (O'Collins et al, 2006). As a result of this dilemma, the STAIR (Stroke Therapy Academic Industry Roundtable) was founded to develop recommendations regarding quality characteristics of preclinical stroke studies (STAIR, 1999; Fisher et al, 2009). It was shown that neglecting some of these quality issues, such as a random allocation of treatment or a blinded outcome assessment, leads to an overestimation of a drug's efficacy (Dirnagl, 2006). Despite the well-known importance of the study design, relevant study aspects were frequently disregarded in published animal stroke studies (van der Worp et al, 2005). Some potential reasons for neglecting quality criteria are the costs of extensive animal experiments and the difficulties in properly blinding a study when only one or two researches are involved. However, it was shown that meeting quality recommendations is feasible (Philip et al, 2009), which raises the question about the motivation for stroke scientists to include or not include quality aspects in their experiments. An important part and a major goal of research is publication. Despite some controversies, the journal's impact factor is widely used to judge the scientific quality of published research (Adam, 2002; Scully and Lodge, 2005). Publishing in journals with a high-impact factor is not only a question of prestige but also a requirement for subsequent funding (Adam, 2002). We hypothesized a lack of incentive to conduct animal experimental stroke studies that meet quality recommendations, because these are not relevant for publishing in high-impact journals. Therefore, we investigated the quality characteristics of animal experimental stroke studies and also whether these are associated with the impact factor. To include those preclinical studies that investigated the most promising drugs, we analyzed studies of neuroprotectants recently being tested in phase II/III clinical trials. In addition, we aimed to identify other relevant study characteristics that predict the publishing of animal experimental stroke studies with respect to a journal's impact factor. These study characteristics include the number of investigated pathophysiological mechanisms underlying a treatment, first investigation of a treatment in a stroke model, and combination of a drug with thrombolysis or another treatment.
Materials and methods
Retrieving the Literature
Animal experimental stroke studies of drugs recently investigated in clinical phase II/III trials were included in this analysis. For identification of these drugs, we systematically searched the database http://www.Clinicaltrials.gov (January 2000 to July 2009). The search strategy used the terms ‘acute stroke’ (conditions), and ‘Phase II’ or ‘Phase III’ (phase). Animal experimental stroke studies of agents that were identified in this manner were then searched using Pubmed (1974 to July 2009), Embase, and Biosis (2000 to July 2009). This strategy included the words ‘ischemia’ or ‘stroke’ or ‘infarct’ and ‘drug name’ or ‘abbreviated drug name.’ Only articles in English were included.
Selection of Studies, Data Extraction, and Identification of the Impact Factor
Studies selected for inclusion in this analysis must contain drug treatment initiation within 24 hours after the onset of ischemia in animal models of focal cerebral ischemia. Only articles in which outcome was measured as infarct size or in which behavioral deficits were assessed were included. Studies that purely investigated the mode of action of a drug in cerebral ischemia were excluded. Studies using neonatal models were excluded, because this model does not reflect human stroke, which usually occurs in the elderly.
The impact factor of the journals was extracted from the Science Citation Index (Institute for Scientific Information, 2009). Impact factors are published annually. They are calculated by dividing the number of citations in a given year to the source items published in that journal within the previous 2 years (Garfield, 2006). Therefore, the impact factor is primarily a measure of the journal rather than an individual article published within this journal. When published in 2009, the impact factor of 2008 was considered. Studies published in journals without an impact factor at the year of publication were not included in the analysis.
Assessment of Study Quality
The methodological quality of the included studies was evaluated according to a previously published quality scale (Minnerup et al, 2008, 2009). The items of the quality score were derived from the STAIR recommendations. The realization of 11 aspects of each study was evaluated: (1) dose–response relationship, (2) randomization of the experiment, (3) optimal time window of the treatment, (4) monitoring of physiologic parameters (such as temperature, glucose level, or blood pressure), (5) blinded outcome assessment, (6) assessment of at least two outcomes (infarct size and one functional outcome), (7) outcome assessment in the acute phase (1 to 7 days), (8) outcome assessment in the chronic phase (beyond 7 days), (9) animals with comorbidity (aged, diabetic, or hypertensive), (10) compliance with animal welfare regulations, and (11) statement of potential conflict of interests.
Evaluation of Other Study Characteristics
To evaluate the complexity with which the mode of action of a neuroprotective stroke drug was explored, we counted the number of the investigated pathophysiological mechanisms and applied techniques of each study. The following most frequently evaluated items were considered: (1) neurogenesis, (2) angiogenesis, (3) dendritogenesis and/or white matter reorganization, (4) brain local inflammation, (5) neurodegeneration and/or number of damaged neurons and other cells, (6) blood–brain barrier permeability, (7) apoptosis, (8) mitochondrial function, (9) brain growth factor synthesis, (10) hemorrhagic transformation, (11) use of genetically modified animals, (12) additional
Statistical Analysis
Regression analyses were performed using the general linear model command. To test for an association of study characteristics with the impact factor, we first performed 10 unadjusted regression analyses, followed by 1 adjusted regression model containing all the study characteristics. To explore the impact of the single quality items, we further performed one regression analysis containing the single items instead of the calculated quality score. All probability values are two tailed, and we considered
Results
Study Inclusion and Study Characteristics
We identified 12 different candidate neuroprotectants recently investigated in clinical phase II/III trials (Figure 1): Albumin, Activated Protein C, Arundic acid (ONO-2506), Citicholine, Edaravone (MCI-186), Erythropoietin (Epo), Granulocyte-Colony Stimulating Factor, Magnesium (Mg), Minocycline, NXY-059, Traxoprodil (CP-101,606), and Zonampanel (YM872). Of these, seven studies are still recruiting patients, three studies are completed, and two studies are terminated. Studies of DP-b99, Enecadin, and Piclozotan (SUN N4057) were not included in the analysis, as the results of animal experimental studies have not been published in peer-reviewed journals. Overall, 117 animal experimental stroke studies fulfilled the inclusion criteria (Supplementary Information). The mean impact factor was 4.6 (range, 0.7 to 30.6). A total of 55 studies (47.0%) were sponsored by industry. The middle cerebral artery occlusion model was used in 97 studies, an embolic stroke model in 17, and a photothrombotic model or a devascularization model in 3 studies. The majority of experiments were performed with rats (84 studies). In 23 studies, mice were used and only 1 study used gerbils. In two studies, both rats and mice were used. Rabbits were used in four studies. Only three studies used gyrencephalic animals, i.e., cats in one study and marmosets in two studies. In 15 studies, the neuroprotective drug was combined with thrombolysis and in 33 studies with an additional therapy other than thrombolysis. Only four studies additionally investigated the agent in an additional animal disease model, such as traumatic brain injury or multiple sclerosis (experimental allergic encephalomyelitis).
Flow chart for selection of studies. Overall, 102 studies were identified searching the database http://www.Clinicaltrials.gov. The terms ‘acute stroke’ (conditions), ‘Phase II,’ or ‘Phase III’ (phase) were used. In all, 44 studies were excluded as nonneuroprotective drugs, such as thrombolytic agents were used. Another 22 studies were excluded for the use of devices and other nonpharmaceutical therapies (such as electrical simulation and gait trainer). In 10 studies, more than one drug was investigated in stroke. Six studies investigated therapies in conditions other than stroke. Of three drugs, the results of animal experimental stroke studies were not published in peer-reviewed journals. Overall, 12 candidate stroke drugs were investigated in 117 animal studies of focal cerebral ischemia.
Impact of Study Quality on the Impact Factor
The median of quality checklist items was 6 (range 0 to 9). The quality score was not associated with the impact factor (Figure 2) before (β = −0.2,
Association between quality score and impact factor. The middle line is the median, and the outer bounds are the 25th and the 75th percentiles. There was no association between the quality score and the impact factor before (β = −0.2,
Regression coefficients (β) and s.e. showing the relationship between study characteristics and the impact factor
The regression coefficient (β) indicates the change in impact factor for 1-Unit change of the corresponding independent variable. For the classified study characteristics, substance, stroke model, and species, no β value is given as each single item of these categories has its own β-value.
Impact of Other Study Characteristics on the Impact Factor
The results of the regression analysis are shown in Table 1. The explained variance of our regression model was 52.3%. The number of investigated mechanisms and applied techniques was significantly associated with the impact factor (β = 1.5,
Discussion
Study Characteristics and Impact Factor
In this analysis of 117 animal experimental stroke studies of candidate neuroprotectants recently being investigated in phase II/III trials, half of the studies fulfilled only ≤6 of the 11 evaluated quality items. Randomized treatment allocation was reported in 50% and blinded outcome assessment was mentioned in 58%. A smaller previous analysis of animal stroke studies found that randomization was performed in 42% and blinding was reported in 40% of the included studies (van der Worp et al, 2005). These differences might be explained by the different search periods and the different drugs that were included in the analyses. It was recently shown that reasonable varieties regarding the fulfilment of quality characteristics in preclinical studies of different drugs exist (Sena et al, 2007; Philip et al, 2009). In our analysis, the study quality was not associated with the journal's impact factor. Moreover, none of the subitems of the quality score was significantly positively associated with the impact factor. Studies in which the dose–response relationship was investigated even had a significantly lower impact factor. However, the associations between the subitems and the impact factor must be interpreted with caution, as some quality characteristics had a low prevalence among the included studies and therefore this analysis might be underpowered. However, overall, our results suggest that there is no demand for including quality characteristics when publishing animal experimental stroke research in high-impact journals.
The number of investigated pathophysiological mechanisms and applied laboratory techniques was significantly associated with the impact factor. Each mechanism or technique that was reported within a study on average increased the impact factor by 1.4 (
Methodological Considerations
A potential weakness of our analysis might be a bias caused by unpublished studies. In a recent analysis, Sena et al (2010) showed that publication bias is highly prevalent in animal experimental stroke studies. The bias of publishing only positive results was pointed out by two meta-analyses of preclinical studies, which investigated NXY-059 in stroke models (Macleod et al, 2008; Bath et al, 2009). One of these analyses also contains unpublished studies, half of which showed neutral results. In particular, animal studies with negative results are often unpublished. As ignoring quality items promotes an overestimation of the efficacy of a treatment (Dirnagl, 2006), following quality recommendations might favor negative results and thereby prevent publication. In this case, unpublished negative studies might even have a higher quality compared with published studies. This would enforce our finding in a way that fulfilling quality characteristics is not only irrelevant for publishing in high-impact journals but also for publishing at all.
A limitation of our findings might be the fact that the STAIR-derived quality standards are not validated so far, as no neuroprotective drug was successfully translated from animal studies into a clinical trial. Therefore, one might criticize that meeting quality characteristics will certainly not predict the replication of animal experimental results in clinical studies. However, the importance of quality standards in preclinical studies regarding an appropriate estimate of a drugs efficacy, which is a requirement before investigation in a clinical trial, is underlined by recent publications (Macleod et al, 2008; Crossley et al, 2008).
Besides the investigation of the impact of the quality score on the impact factor, we aimed to determine other study characteristics that predict high-impact publishing. Therefore, we included a number of study features in the analysis, but we cannot rule out the possibility that some relevant study characteristics were omitted. It is particularly difficult to measure factors, such as the linguistic quality of an article or the reputation of the authors. However, the explained variance of the regression model used in our study was 52.3%, indicating that several relevant study characteristics were included in our model.
Future Implications and Conclusions
It is widely accepted that study characteristics bias the results of clinical trials (Chalmers et al, 1983; Balk et al, 2002; Gluud, 2006). Major medical journals with high-impact factors predominantly publish studies that meet quality items such as randomization and blinding, therefore contributing to the improvement in study quality (Lee et al, 2002). Nevertheless, quality varies among clinical studies and experimental design errors are common (Avram et al, 1985, 2002; Bain and Myles, 2005). For animal experimental stroke studies, there is growing evidence that study characteristics also bias estimates of a drug's efficacy (Macleod et al, 2005, 2008; van der Worp et al, 2007). However, as shown by our analysis, neither study quality as measured by a scale derived from the STAIR criteria nor relevant quality characteristics, such as blinding or randomization, are associated with publishing in high-impact journals. Therefore, we propose that these study attributes should be indispensably considered in animal stroke research and have to be reported in publications like the conflict of interest must be reported in the most relevant journals that publish preclinical stroke studies. We thereby concur with a recently published consensus statement demanding that only those studies should be considered for publication the ‘Methods’ section of which includes a description of how quality standards were addressed (Macleod et al, 2009).
Footnotes
W-R Schäbitz is an inventor on a patent application regarding the neuroprotective effects of G-CSF. EB Ringelstein is the principal investigator of a clinical study of G-CSF in acute stroke.
