Sage Journals: Discover world-class research

Abstract

Despite promising preclinical data, few novel stroke therapies have shown efficacy in man. Efforts to improve standards in conduct and reporting of preclinical research are ongoing. In clinical trials, inconsistency in outcome measures led to regulatory agencies and funders mandating use of a core set of functional outcomes. Our aim was to describe functional outcome measures in preclinical stroke and vascular cognitive impairment (VCI) studies. From 14 high impact journals (January 2005–December 2015 inclusive), 91,956 papers were screened with 1302 full texts analyzed for stroke (ischemic and hemorrhagic) and 56 for VCI studies. In total, 636 (49%) stroke and 37 (66%) VCI papers reported functional outcome measures. There were 74 different functional assessments reported in stroke and 20 in VCI studies. Neurological deficit scores (74%) and Morris water maze (60%) were most commonly used in stroke and VCI, respectively. However, inconsistencies in methods used to assess and score recovery were noted. Neurological and behavioural functional outcome measures are increasingly used in preclinical stroke or VCI studies; however, there is substantial variation in methods. A strict standardized outcome set may not be suitable for translational work, but greater consistency in choice, application and reporting of outcomes may improve the science.

Keywords

Rodent middle cerebral artery occlusion hemorrhagic stroke functional outcome measure vascular cognitive impairment

Introduction

The traditional translational pathway, where compounds are first trialed in animals and then humans, has provided limited success in stroke and vascular cognitive impairment (VCI). Many compounds show promise in early phase trials but fail to deliver benefit when tested in patients with cerebrovascular disease.¹ Following a number of neutral studies of putative neuroprotectants,² the stroke research community, pharmaceutical industry and regulatory agencies suggested methods to improve the translational pathway. Resulting guidance documents, from STAIR (Stroke Therapy Academic Industry Roundtable)^3,4 and others,^5,6 were designed to raise standards in both preclinical and clinical stroke research. To prove the efficacy of any new treatment, a robust measure of effect is required. In a disabling condition such as stroke, measures of function are important metrics of assessment. Functional measures are now recommended as the primary outcome in acute stroke interventional trials and certain regulatory agencies and funders mandate their collection in stroke studies. Similar efforts towards a consensus approach for clinical VCI research have recently been proposed.⁷

Systematic reviews of assessments applied in contemporary clinical trials have described substantial heterogeneity in choice and application of functional outcome measures in both stroke and VCI studies.^8,9 Even in more niche areas, such as assessment of post-stroke cognitive and mood disorders, there are almost as many outcome measures employed as there are studies.¹⁰ This variation in assessment is inefficient as it precludes meaningful comparative analyses of studies and complicates any attempt to pool data across studies. In recognition of this, consensus statements on the preferred functional outcomes for use in stroke and dementia clinical studies have been created.^11,12

There are numerous functional outcome assessments available for preclinical ischemic stroke and VCI studies, especially rodent models,¹³ and with choice comes the potential for inconsistency in assessment. Classically in preclinical stroke research, one measures sensorimotor impairment. Motor problems are common following stroke in man but additional impairments are also observed and an exclusive motor focus may be overly reductionist. In VCI models, assessments often rely on memory impairment, although novel assessments that capture other cognitive domains more aligned with the clinical VCI phenotype have been described.¹⁴ Arguments for the importance of using functional outcome measures in humans may also be true for animal studies. As in human cerebrovascular research, there are many potential functional assessment paradigms for different animal models.

The aim of this study was to characterize preclinical stroke (ischemic and hemorrhagic) and VCI trials from the last 11 years, looking at 14 highly cited, exemplar journals from the fields of cerebrovascular science. Our objectives were to describe the frequency of use, methodology applied for functional (sensorimotor, behavioural and neurocognitive) outcome measures as well as temporal trends. To put these results in context, we also described preference of species, sex and disease models employed.

Methods

Study design

Best practice guidance in systematic review (preferred reporting items for systematic review and meta-analysis (PRISMA)) was followed, where appropriate. As the aim was to describe the use of outcomes, no assessments of the quality of the included trials’ design, methods or conclusions were made. A pre-specified protocol (researchregistry1509) was followed. Our approach was based on a previous review of functional assessments in clinical stroke trials.^8,10

Paper screening

Following initial scoping of the literature, the search was limited to 14 journals, representing a broad field of translational cerebrovascular research. Choice of titles was made by the author team. Based on impact factor, reputation within the field and frequency of publication of relevant stroke and/or VCI studies, the senior authors suggested a list of potential titles that could represent the following themes: clinical stroke (three titles chosen in this area); pre-clinical stroke; experimental neurology; neuroscience; neurorehabilitation and vascular science. Through discussion, a consensus was reached as to the top two titles for each category. The primary criterion for selection was visibility within the international research community (Supplementary Table I). They were: Brain (Oxford University Press); Circulation (American Heart Association, AHA); Experimental Neurology (Elsevier); Hypertension (AHA), International Journal of Stroke (World Stroke Organization, Sage Publishers); Journal of Cerebral Blood Flow and Metabolism (International Society of Cerebral Blood Flow and Metabolism, Sage Publishers); Journal of Neuropathology and Experimental Neurology (American Association of Neuropathologists, Oxford University Press); Journal of Neuroscience (Society for Neuroscience); Journal of Stroke and Cerebrovascular Disease (National Stroke Association, Elsevier); Nature Neuroscience (Nature Publishing Group); Neurobiology of Disease (Elsevier); Neurorehabilitation and Neural Repair (Sage Publishers); Stroke (AHA); Translational Stroke Research (Springer).

Inclusion and exclusion criteria

The search was limited to original papers published January 2005–December 2015 inclusive to capture papers before and after quality guidelines such as STAIR.^3,4 Four independent researchers (T.M.H., C.O., Z.P., L.F.) hand searched the chosen publications and screened titles/abstracts. Inclusion criteria were: original research, published in one of the chosen journals, use of animal stroke or VCI model. To ensure no potential manuscripts were missed, all journal content was reviewed, including letters, editorials and short reports. Additional methodology described in online or paper supplements was assessed, where available. As the focus was around the content of published papers, where aspects of methodology were unclear, authors of the papers were not contacted. When looking for the original citation describing a model that was not described in the index publication, if more than three further citations were cross-checked and lacked original scaling, these methods were considered as not found. Where more than one paper described the same dataset, only the primary publication was included, unless other publications reported differing outcome measures.

The aim of the study was to collate stroke and VCI animal trials with functional outcome measures, where ‘functional outcome’ was defined as a quantified measure across any of the WHO-ICF domains of impairment, activity (disability) or participation (handicap). A functional measure was accepted, where it was used as either the primary or secondary end-point of the study. For the purposes of this review, ‘trial’ was defined as any research describing the effects of an active intervention. WHO criteria for stroke was used to define the scope and within the stroke rubric, models of both ischemic and hemorrhagic stroke were included. VCI studies were defined as studies, where vascular interventions were used to emulate one (or more) of the neuropathological changes associated with VCI and where the purpose of the model was to create dementia or a cognitive impairment phenotype.

Data extraction

From eligible papers, relevant data were extracted onto a standardized, piloted, proforma spreadsheet. Items of interest were: journal title, year of publication, animal model (sex, species ± genus), stroke model, VCI model, functional outcome measure(s) (primary and secondary). The name of the functional outcome assessment used was taken directly from the text of the paper. If the assessment was not named or described in the text, accompanying citations were referred to. Where an assessment was described as “modified”, content was compared to the primary scale. If fundamental aspects of measurement differed, the measure was included as a distinct outcome. A test was counted as a single score if it was a battery or composite or various assessments that were combined as a single result (e.g. neuroscore).

Functional assessments were categorized as either neurological scales (closest to impairment measures using WHO-ICF terminology) and functional tasks (closest to activity measures using WHO-ICF). Temporal trends in the use of functional outcomes were described.

Statistical analysis

Workflow was described with a PRISMA style flow diagram. Frequency was described as functional outcomes with the use of basic statistics, non-parametric or proportional, as required. Each category was described as number of papers as well as proportion. Results were reported as the top 10 measures. Interquartile range (IQR) was calculated to show data dispersion. Numbers of papers were compared with an unpaired t-test comparing articles grouped into two groups: published 2005–2010 and papers published 2011–2015, inclusive with 2010 chosen as a time-point when core outcomes and guidance in reporting of outcomes become the standard in clinical stroke research. All analyses were performed using Prism for Windows (4.00, Graphpad) software and p < 0.05 was considered significant.

Results

Frequency of functional measures and temporal trends

Of 91,956 papers screened, 23,802 had a stroke focus and 317 VCI. Primary screening revealed 2165 and 229, respectively, as potentially relevant and of these, 863 (40%) and 88 (38%), respectively, did not meet the inclusion criteria. For stroke, full text review was conducted on 1302 papers with 636 (49%) used for final analysis. The majority of exclusions (666, 51%) were due to no reporting of a functional measure. For VCI, 56 papers were assessed in full, 19 (34%) exclusions reported no functional outcome measures and 37 (66%) papers were included in the final analysis (Figure 1).

Figure 1.

Strategy implemented in the focused literature search. Papers were selected on the basis of set inclusion/exclusion criteria and after further comprehensive review, papers that had no reported functional outcome measures as their primary or secondary endpoint were excluded from the final analysis. This yielded 636 papers for stroke and 37 for vascular cognitive impairment (VCI).

The number of stroke (Figure 2(a)) and VCI- (Figure 2(b)) related papers reporting functional outcome measures increased over the 11 years studied. A significant increase in stroke papers describing functional outcome assessments was observed after 2010, from a median of 32.5 (range: 30–38; IQR: 7) papers annually before 2010 to a median of 82 (range:75–101; IQR:19.5) papers annually since 2011 (t-test; p < 0.0001).

Figure 2.

Number of papers published each year with functional assessments. The number of preclinical stroke (a) or VCI (b) papers reporting functional assessments over the 11 years of the review period. The number of papers per year in stroke and VCI, respectively were: 2005: 29,1; 2006: 30, 1; 2007: 32, 2; 2008: 37, 1; 2009: 38, 5; 2010: 33, 2; 2011: 97, 3; 2012: 75, 3; 2013: 101, 5; 2014: 82, 2; 2015: 82, 12. A significant increase in stroke papers describing functional outcome assessments was observed after 2010 (unpaired t-test; ***p < 0.0001 comparing articles from 2000 to 2010 vs. articles 2011–2015, inclusive). Data generated from 14 specified peer-reviewed journals (Supplementary Table I).

Animal species, sex and model preferences

Rodent models were most commonly used in both stroke and VCI studies (Figure 3(a)). In stroke papers, there was variation in the animal models employed, with 622 (98%) of the papers using rodent models (339 rat, 282 mice, 1 gerbil) and 18 (2%) using non-rodent animal models (6 rabbit, 3 monkey, 2 pig, 2 baboon, 3 macaque, 1 marmoset, 1 dog). All papers within the VCI scope used rodents, with 28 (76%) of papers carried out in mice and 9 (24%) in rats.

Figure 3.

The prevalence of species and sex of animals used in stroke and VCI papers. (a) The percentage of papers using each species in stroke and VCI studies. (b) The percentage of papers using each sex across stroke and VCI studies. Percentage was calculated as portion of total amount of papers included in the final analysis (n = 636 and n = 37, respectively).

There was a sex bias in animal models employed, with the majority using males, irrespective of species (Figure 3(b)). For stroke papers, 535 (84%) used males, 22 (3%) females, 28 (4%) both male and female and 51 (8%) sex unspecified. Males were used in 23 (62%) VCI studies, females in 1 (3%), both male and female in 6 (16%) and 7 papers (19%) did not specify sex. In both stroke and VCI studies, when females were used, it was most commonly in conjunction with male animals. Only 22 stroke and 1 VCI paper(s) used solely female animals.

There were 10 distinct disease models used in stroke and 11 in VCI studies (Supplementary Table II). Ischemic stroke was more commonly studied with 559 (88%) papers, while hemorrhagic stroke was investigated in 84 (13%) of papers. Transient middle cerebral artery occlusion (tMCAO) was the most common stroke model, used in 342 (54%) papers with permanent MCAO in 109 (17%) papers. Other models of ischemic stroke included thrombotic stroke, endothelin-1 induced stroke, hypoxia–ischemia and global ischemia. Hemorrhagic stroke papers used procedures directed to create an intracerebral, subarachnoid or intraventricular bleed (Supplementary Table II).

In VCI research, permanent global hypoperfusion was the most common model, utilized in 11 (30%) papers. Alzheimer’s disease (AD) models, such as amyloid precursor protein (APP) mutations, were the second most common model, with AD pathology being used on its own or in conjunction with other co-morbidities such as hypertension, dyslipidemia and stroke. Other models included post-stroke (tMCAO) dementia, microinfarcts, hypertension or dyslipidemia-induced vascular dementia and transforming growth factor-induced cerebral fibrosis (Supplementary Table II).

Variation in functional outcome measures used in stroke studies

There were 74 different functional outcome measures recorded across stroke papers (Supplementary Table III). Of these, 42 described functional tasks, where a specific behavior was monitored and recorded, and 32 described various neurological deficit scores (NDS). Examining only the top 10 outcome measures used in stroke studies, NDS were the most frequently named functional measure (Figure 4(a)), used in 471 papers with several variations used (Supplementary Table IV). The Bederson et al.¹⁵ NDS scale was the most commonly cited in 175 papers. Modified versions of original scales were considered as separate measures where scale content, scoring or application differed from the primary description. Ten different variations of scoring were found amongst those citing Bederson¹⁵, ranging from 0–3 to 0–20 (Supplementary Table V). In 29 (6%) papers, NDS scales were not referenced or the method was left unnamed. Of the specific functional tests, Rotarod was the most commonly used in 94 (10%) papers (Figure 4(a)). Rotarod, grid walk/foot fault, adhesive label removal, cylinder, and string/wire/swing tests were the five most frequently used sensorimotor assessments and together were present in 327 (34%) of studies.

Figure 4.

Top 10 functional assessments used in preclinical stroke or VCI papers. Top 10 functional assessments in (a) stroke outcome measures (n = 967) and (b) VCI outcome measures (n = 65) included in the final analysis. Percentage was calculated as a portion of the number of functional assessments made.

Considering functional outcome measures used according to the most commonly used stroke models, in papers using intraluminal filament to induce tMCAO or pMCAO, the NDS (81% or 46% respectively) and rotarod (17% or 21% respectively) were most commonly used (Table 1). For the endothelin-1 tMCAO model, the most commonly reported outcome measure was the skilled-reaching/staircase test and cylinder tests (both 19%) (Figure 5(a)). In thrombotic stroke models, the preferred outcome measure reported was the NDS (32%) and adhesive removal test (15%) (Figure 5(b)).

Table 1.

Prevalence of outcome measures used in papers using the intraluminal filament tMCAO and pMCAO stroke models.

Functional assessment	tMCAO, n(%)	pMCAO, n(%)
Neurological deficit score	246 (81)	48 (46)
Rotarod	53 (17)	22 (21)
String/wire/grip/swing test	31 (10)	8 (8)
Adhesive removal test	26 (9)	16 (15)
Beam/rope test	22 (7)	1 (1)
Grid walk/foot fault	22 (7)	13 (13)
Cylinder test	19 (6)	9 (9)
Locomotor activity	16 (5)	9 (9)
Corner turn test	14 (5)	8 (8)
Limb placing test	13 (4)	12 (12)
Morris water maze	12 (4)	11 (11)
Tail suspension/body swing	8 (3)	6 (6)
Side-walking/circling/rotation	5 (2)	2 (2)
Postural reflex	4 (1)	3 (3)
Gait analysis	4 (1)	2 (2)
T or Y or radial arm maze	3 (1)	1 (1)
Barnes maze	3 (1)	1 (1)
Elevated plus or O maze	2 (1)	NA
Skilled-reaching/staircase task	2 (1)	8 (8)
Ladder test	2 (1)	6 (6)
Novel object recognition	2 (1)	NA
Whisker/tactile	2 (1)	1 (1)
Passive avoidance	1 (0.3)	2 (2)
Fear conditioning	1 (0.3)	1 (1)
Parallel bar crossing	1 (0.3)	1 (1)
Inclined plane test	1 (0.3)	NA
Right forelimb resting motor threshold	1 (0.3)	NA
Drinking efficiency	1 (0.3)	NA
Social novel odour recognition task	1 (0.3)	NA
Hindlimb retraction	1 (0.3)	NA
Rotameter task	1 (0.3)	NA
Forced swim test	NA	1 (1)
Stress test (max speed)	NA	1 (1)
Step test	NA	1 (1)

Note: The preferred functional outcome measure used in papers using the intraluminal filament to induce tMCAO or pMCAO was determined. Percentage was calculated as a portion of all papers included (n = 305 and n = 104, tMCAO and pMCAO, respectively).

Figure 5.

Prevalence of outcome measures used in papers according to stroke model. The preferred functional outcome measure used in papers using (a) endothelin-1 or (b) the thrombotic model to induce experimental stroke was determined. Percentage was calculated as a portion of all outcome measures used (a: n = 48; b: n = 101).

The median number of outcome measures used per trial was 1 (range 1–10; IQR: 1). Only 57 (9%) stroke papers reported functional outcome measures as their primary endpoint. The prevalence of reporting more than one outcome measure demonstrated the majority (344 papers, 54%) used one (Figure 6(a)); this dropped to 159 papers, 25% for 2 measures and continued downwards with increasing number of measures (Figure 6(a)). The top two most commonly employed measures, NDS and Rotarod, were shared across rodent models. In the 20 non-rodent studies, NDS were used as the only functional measure in all but one study. Within the nine primate studies that applied NDS, there were five different original scales cited. The six rabbit studies all applied a different NDS. The two piglet stroke studies each employed a different version of the same NDS.

Figure 6.

Prevalence of the number of different outcome measures reported in preclinical stroke or VCI papers. The number of preclinical stroke papers using one or more functional outcome measures was determined for (a) stroke and (b) VCI papers. These represented for increasing number of outcome measures employed: (a) 1 = 344 papers (54%), 2 = 159 (25%), 3 = 81 (13%), 4 = 34 (5%), 5 = 7 (1%), 6 = 6 (1%), 7 = 1 (0.2%), 8 = 0 (0%), 9 = 1 (0.2%), 10 = 4 (1%) and (b) 1 = 20 papers (54%), 2 = 10 (27%), 3 = 4 (13 = 11%), 4 = 1 (3%), 5 = 0 (0%), 6 = 0 (0%), 7 = 0 (0%), 8 = 1 (3%), 9 = 0 (0%), 10 = 0 (0) 11 = 1 (3%). Percentage was calculated as a portion of all papers included.

Variation in functional outcome measures used in VCI studies

There were 20 different types of assessments used (Supplementary Table VI), which were predominantly tasks involving memory (8 assessments) or motor coordination (8 assessments). Only 7 (19%) VCI studies reported functional outcome measures as their primary endpoint. Examining, the top 10 most commonly used outcome measures in VCI papers (Figure 4(b)), demonstrated the Morris water maze (MWM)¹⁶ was most commonly used, irrespective of species and was used in 22 papers (with one completing 2 trials, hence 23 trials). Other commonly applied tasks were novel object recognition, T or Y radial maze, locomotor activity and NDS, found in 27 studies (Figure 4(b)). The median number of functional assessments used in VCI papers was 1 (range 1–11, IQR: 1). The prevalence of reporting more than one outcome measure demonstrated the majority (20 papers, 54%) used one (Figure 6(b)); this dropped to 10 papers, 27% for 2 measures and continued downwards with increasing number of measures (Figure 6(b)).

Within papers using the MWM, there was notable heterogeneity within the methodology of the test (Table 2). Some studies used a submerged platform (14 trials; 61%), whereas others a visible platform (4 trials; 17%) or a combination (5 trials; 22%). There were also differences in external cues used to assist in finding the platform (56% used cues versus 44% no cues) and whether these cues were visual or olfactory/auditory. There was a preference towards including a probe trial, where the platform is removed and the swimming latency is recorded to test retention memory, but this was inconsistent across all studies (78% and 22%, respectively). There was further variability in the number of acquisition days (between 0 and 10 days, IQR: 2.5) allowed before endpoint assessment; the number of trials performed per day (1–8 trials per day, IQR: 2); the timing of the probe trial (range 3–10 days, IQR: 1.25), and whether there were further trials after the probe trial (39% studies applying probe trials had post-probe trials, 61% did not).

Table 2.

Variation in the characteristics of Morris water maze (MWM).

Variation used	Papers using MWM, n (%)
Platform type (n = 23)
Submerged platform	14 (61)
Both types of platform	5 (22)
Visible platform	4 (17)
Cue type (n = 23)
Visual cues	12 (52)
No cues	10 (44)
Olfactory/auditory/spatial	1 (4)
Presence of probe trial (n = 23)
Probe trial	18 (78)
No probe trial	5 (22)

Note: Differences in characteristics of MWM used in the papers shown as differences in set-up, presence and type of cues as well as with the presence of a probe trial, where platform was removed from the pool. Percentages were calculated as total amount of papers using MWM (n = 23).

Discussion

The traditional paradigm for assessing efficacy in preclinical stroke and VCI trials has been a combination of neurological assessments with measurements of infarct volume. Use of functional assessments as primary outcome or to complement other outcomes is increasingly recommended in both clinical and preclinical research. In the present review, papers from 14 journals from the past 11 years were investigated to describe the functional outcome measures used in preclinical trials of stroke and VCI. Although functional assessments are mandated in clinical stroke trials, functional assessments were the primary outcome in only a minority of outcomes used in preclinical models. The landscape may be changing, as there was an increase in papers reporting functional outcome measures over the time horizon of the study. Where functional outcomes were used, there was substantial heterogeneity in the assessment and application of the assessment. There was greater consistency in the sex and species of animal model, although a predominant focus on male animals is not in keeping with best practice.

The observed increase in stroke papers reporting functional measures from 2010 onwards could be partly attributed to the publication of the revised STAIR guidelines in 2009.⁴ These guidelines emphasize not only the importance of using functional outcome measures, but also problems inherent in using multiple outcome measures. Indeed, a recent editorial by Zerna et al.¹⁷ highlighted the choice of endpoint remains challenging in the preclinical stroke field. While not as many papers were identified for VCI preclinical studies, those reporting outcome measures increased temporally across the 11-year period, which may reflect a shift in research focus towards dementia.¹⁸

The predominant (98%) or exclusive use of rodent models in stroke and VCI studies, respectively, is consistent with other animal disease models. Their use is favored for several reasons – maintenance costs are low, use is preferred ethically, availability of transgenic strains and the vascular anatomy is similar to human (reviewed in Macrae¹⁹ and Howells et al.²⁰). Considerations for stroke/VCI studies include that rodents have less white matter than humans since they have lissencephalic brains. STAIR⁴ recommend that once efficacy is established in a rodent model that larger animal models are studied prior to clinical translation. Another key consideration in the translation pipeline is the inclusion of disease-relevant comorbidities and risk factors, such as hypertension and aging for stroke/VCI studies, highlighted in STAIR and other recent guidelines.^4,21 Presence of existing comorbidities across the studies described herein was not included in the inclusion/exclusion criteria as based on previous studies, consideration of such co-morbidities occurred in a minority (3%) of reports of therapeutic interventions in stroke.²²

Even though women are more likely affected by post stroke disability,²³ males are more often used in preclinical research and within those studies included in this review, 84% of stroke and 62% of VCI studies used male animals. The focus on males most likely limits cohort sizes and variability through removal of the effect of the estrous cycle in females. However, the importance of considering sex as a biological variable has been discussed extensively.^4,24,25 This followed the publication of the National Institute of Health guidance highlighting that sex differences should be factored into experimental design.²⁶ Sex has significant effects on many biological processes (reviewed in Chauhan et al.²⁷) and certain interventions, such an inhibition or knockout of neuronal nitric oxide synthase (nNOS) or poly-ADP ribose polymerase-1 (PARP-1), have shown protection in male animals subjected to experimental stroke and exacerbation of injury in females.²⁸ Within the studies included here, when females were used, this was most commonly in conjunction with male animals. The recent publication of the ‘Sex and Gender Equity in Research’ guidelines²⁹ combined with grant review considerations may lead to a shift in the inclusion of both sexes in experimental stroke and VCI studies in coming years.

Eleven models of both stroke and VCI were described. Most stroke studies used an ischemic model with tMCAO being the most widely employed. For VCI, the permanent global hypoperfusion model was the most commonly used. The choice of experimental stroke model used will be tailored to the therapeutic intervention being trialed and considerations such as whether reperfusion should occur or the size and location of the lesion required. Indeed, differences in the choice of preferred outcome measure were evident comparing across stroke models. The relative merits and limitations of each model are reviewed elsewhere.^19,20,30,31 Similarly, for VCI studies, the biological question being addressed will determine the most appropriate model to use. Models that produce lesions in distinct locations require outcome assessments that are relevant to the affected brain region. For example, the use of the skilled reach/staircase test or cylinder test when the endothelin-1 stroke model was used reflects the cortical lesion produced. However, an exclusive focus on impairment measures specific to the lesion is a reductionist approach and potentially fails to capture the biological variability in response; the remote consequences of targeted lesions and the limited correlation between neuroanatomy and function. Global assessments of functional outcome should be relevant to any stroke lesion and could form part of a core set of outcomes complemented by other specific tests.

There were 74 different functional measures used in stroke preclinical studies. The most frequently used was NDS, although within this group there was marked heterogeneity in NDS used and the application of the test. These scales (or scores) generally reflect overall condition, assessing reflexes, simple motor function and balance (reviewed in Balkaya et al.³²). They are often preferred as they can be performed soon after experimental stroke and generally do not require specialized equipment. However, NDS are not a distinct behavioral measure per se, rather NDS are normally composed of various components that quantify the global stroke-related impairment over time with inherent limitations due to subjectivity of scores. The most commonly used NDS, Bederson,¹⁵ gives a total score of 0–3 in the original scale. This limited range of possible scores lacks sensitivity for assessing change with subsequent effects on sample size requirements and has led to a range of modified Bederson scales with a greater number of components.

A further challenge with the NDS was distinguishing between the methodologies used. From the NDS or battery tests described, most were named by referencing the original authors who first developed the tools, or by referencing authors who had previously used the tools. The referencing was further complicated by publication of subsequent and different modified versions of the originals, each with their own authors further modifying the scale. Comprehensive descriptions of the methodology of each assessment were infrequent and the original method was often left unclear. There were also multiple papers where methods were not clarified with appropriate citations or descriptions. Without clear methodology, interpreting between NDS becomes complex. Poor reporting makes it difficult to replicate experiments in independent cohorts and ultimately limits the validity of the research and potential to use the data for comparative or pooled analyses such as systematic reviews and meta-analyses.³³ Within the clinical arena, there have been numerous efforts leading to a consensus in preferred outcome measures in clinical trials³⁴ and our findings suggest that this could be applied in preclinical research.

In VCI studies, 20 different functional assessments were recorded. While common standards exist for identifying and describing cognitive impairment clinically, these recommendations do not extend to preclinical VCI study reporting.³⁵ Nevertheless, assessing cognitive impairment is crucial to validate the disease model and to further assess the effects of interventions. The MWM was the most commonly used outcome measure across all studies in mice or rats. However, there was clearly marked inconsistencies across the methodology used in these studies which would, again, make pooled analyses difficult. Indeed, this is consistent with a recent systematic review of AD mouse models where substantial variation in methodology in the use of the MWM was found with 57 studies using the probe phase using 59 different approaches.³⁶ The substantial variations in the characteristics of the MWM set-up alter which aspects of memory retention and learning are being assessed.³⁷ In stroke studies, use of a pre-determined performance criterion within assessment of the MWM has been recommended in order to avoid potential misinterpretation of data.³⁸ Furthermore, it has been shown that rats outperform mice when using the MWM as an assessment measure.³⁹ Many of the assessments used in VCI models were first proposed for animal models of other dementia pathologies, e.g. AD. Assessments that predominantly describe memory may miss important impairments in other domains commonly seen in VCI, for example the complex constructs of executive function. Recently, models used in VCI preclinical trials have been under review,⁴⁰ but for more effective clinical translation, the appropriateness of functional outcome measures for each model should also be evaluated.

There were strengths and limitations to the search conducted. The large and increasing stroke literature precluded a comprehensive review of outcomes across all published preclinical stroke and VCI trials. The analysis was limited to journals with a large readership across the disciplines of clinical and translational neurovascular research over an 11-year period. Consequently, the sampling frame is potentially biased. Important studies describing VCI could be published in a variety of journals. Our focus on cerebrovascular disease and non-inclusion of dementia specific titles may have affected the yield of VCI papers. However, the intention was to describe the outcome measures used in journals with the greatest scientific impact, rather than across the complete stroke and VCI literature. Our focus was predominantly on measures of sensorimotor or cognitive outcome. These may not be the only “functional” outcomes of relevance. Recent priority setting exercises have suggested that emotional and mood symptoms are the issues of greatest importance to stroke survivors.⁴¹ Although not comprehensive, the search strategy was systematic. A systematic approach was necessary for describing the assessment methods employed. As numerous papers only referred to the assessment method used, the stated functional outcome tool was cross-referenced, sometimes across several papers, to obtain the original description of the assessment method used. The intention was purely to describe current outcome assessment methods and no attempt was made to compare the strengths and weaknesses of different instruments or to assess the methodological approaches described in the trials. There have been recent efforts to evaluate the validity of various assessments in certain models of stroke,^42,43 and there would be value in expanding this evaluation to include the many stroke and VCI preclinical models described in the literature.

The need to determine neurological deficit and function as an outcome measure within stroke and VCI studies is imperative for potential clinical translation. The marked heterogeneity described is perhaps unsurprising as there are no agreed pre-set guidelines for functional assessment measures preclinically. Consistency across studies will facilitate easier between study comparisons and pooling of preclinical data. In clinical trials, work by groups such as EQUATOR (Enhancing the Quality and Transparency of Health Research) has raised standards in reporting of research methods. The work of collectives such as the CAMARADES group is trying to replicate this success in preclinical models. It is encouraging to see emerging guidance around preferred assessments, albeit this is limited to a specific aspect of cerebrovascular research. For example, the Stroke Recovery and Rehabilitation Roundtable have recently produced guidance for behavioural outcome measures.^44,45 Considerable efforts and advances have been made in improving design, conduct and analysis in stroke research,^21,45,46 with improvements in terms of rigor and reduced bias reflecting the changes implemented.⁴⁷ Indeed, in comparison to other areas, preclinical stroke research in particular, is performing well against targets to enhance reproducibility and promote translation.⁴⁸

In this context of increasing standardization of methods in clinical research, one possible interpretation of our results is that the preclinical stroke research community should move towards a core set of preferred functional outcomes measures. While few would argue against improving rigor and transparent reporting, restricting the choice of outcome assessments available may not suit translational and discovery science. The nature of preclinical research is often more exploratory compared to the confirmatory nature of clinical trials and indeed, over-standardization of laboratory studies may have the opposite intended effect and result in poor reproducibility.⁴⁹ For hypothesis generating pre-clinical studies, it is arguable whether functional assessment adds value to other surrogate outcomes such as neuroimaging. Perhaps a more suitable suggestion would be that for larger scale preclinical studies, researchers use at least one functional outcome measure with standardized approaches to assessment and scoring in addition to any assessments specific to the scientific question of interest.

In conclusion, we have demonstrated that functional outcome measures are increasingly being used in preclinical cerebrovascular research but when they are employed, substantial heterogeneity in the measures chosen and their application exists. This inconsistency in conduct and reporting limits the potential for comparative or meta-analyses. The clinical research community have developed preferred outcomes but strict control of the tests available to researchers may not always be suitable in preclinical work. There are other avenues available, including standardized operating procedures, standardized scoring criteria, guidance on reporting outcome assessments and many others. The preclinical research community must now work together to improve consistency and transparency and we would welcome any initiatives that look to develop these resources.

Supplemental Material

Supplemental material for Variability of functional outcome measures used in animal models of stroke and vascular cognitive impairment – a review of contemporary studies

Supplemental material for Variability of functional outcome measures used in animal models of stroke and vascular cognitive impairment – a review of contemporary studies by Tuuli M Hietamies, Caroline Ostrowski, Zhong Pei, Luyang Feng, Christopher McCabe, Lorraine M Work and Terence J Quinn in Journal of Cerebral Blood Flow & Metabolism

Footnotes

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: TMH is supported by a Cunningham Trust PhD studentship (CU15007.0001). TJQ is supported by a Chief Scientist Office/Stroke Association Clinical Senior Lectureship (TSA LECT 2015/05).

Acknowledgements

We thank Professor I. Mhairi Macrae for critical appraisal of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Authors’ contributions

TMH, CO, ZP, LF completed the data collation and primary analysis; TMH, CO, CM, LMW & TQ wrote the manuscript.

Supplementary material

Supplementary material for this paper can be found at the journal website:

References

van der Worp

Howells

Sena

et al.

Can animal models of disease reliably inform human studies?

PLoS Med 2010; 7: e1000245.

O'Collins

Macleod

Donnan

et al.

1,026 Experimental treatments in acute stroke. Ann Neurol 2006; 59: 467–477.

STAIR. Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 1999; 30: 2752–2758.

Fisher

Feuerstein

Howells

et al.

Update of the stroke therapy academic industry roundtable preclinical recommendations. Stroke 2009; 40: 2244–2250.

Macleod

Fisher

O'Collins

et al.

Good laboratory practice: preventing introduction of bias at the bench. Stroke 2009; 40: e50–52.

Howells

Sena

Macleod

. Bringing rigour to translational medicine. Nat Rev Neurol 2014; 10: 37–43.

Bordet

Ihl

Korczyn

et al.

Towards the concept of disease-modifier in post-stroke or vascular cognitive impairment: a consensus report. BMC Med 2017; 15: 107.

Quinn

Dawson

Walters

et al.

Functional outcome measures in contemporary stroke trials. Int J Stroke 2009; 4: 200–205.

Harrison

Noel-Storr

Demeyere

et al.

Outcomes measures in a decade of dementia and mild cognitive impairment trials. Alzheimers Res Ther 2016; 8: 48.

10.

Lees

Fearon

Harrison

et al.

Cognitive and mood assessment in stroke research. Stroke 2012; 43: 1678.

11.

Ritchie

Ropacki

Albala

et al.

Recommended cognitive outcomes in preclinical Alzheimer's disease: consensus statement from the European Prevention of Alzheimer's Dementia project. Alzheimers Dement 2017; 13: 186–195.

12.

Webster

Groskreutz

Grinbergs-Saull

et al.

Core outcome measures for interventions to prevent or slow the progress of dementia for people living with mild to moderate dementia: systematic review and consensus recommendations. PLoS One 2017; 12: e0179521.

13.

Schaar

Brenneman

Savitz

. Functional assessments in the rodent stroke model. Exp Transl Stroke Med 2010; 2: 13.

14.

Romberg

Bussey

Saksida

. Paying more attention to attention: towards more comprehensive cognitive translation using mouse models of Alzheimer's disease. Brain Res Bull 2013; 92: 49–55.

15.

Bederson

Pitts

Tsuji

et al.

Rat middle cerebral artery occlusion: evaluation of the model and development of a neurologic examination. Stroke 1986; 17: 472–476.

16.

Morris

. Developments of a water-maze procedure for studying spatial learning in the rat. J Neurosci Meth 1984; 11: 47–60.

17.

Zerna

Hill

Boltze

. Towards improved translational stroke research. Stroke 2017; 48: 2341.

18.

Luengo-Fernandez

Leal

et al.

UK research spend in 2008 and 2012: comparing stroke, cancer, coronary heart disease and dementia. BMJ Open 2015; 5: e006648. DOI: 10.1136/bmjopen-2014-006648.

19.

Macrae

. Preclinical stroke research – advantages and disadvantages of the most common rodent models of focal ischaemia. Br J Pharmacol 2011; 164: 1062–1078.

20.

Howells

Porritt

Rewell

SSJ

et al.

Different strokes for different folks: the rich diversity of animal models of focal cerebral ischemia. J Cereb Blood Flow Metab 2010; 30: 1412–1431.

21.

Bosetti

Koenig

Ayata

et al.

Translational stroke research. Stroke 2017; 48: 2632.

22.

O'Collins

Macleod

Donnan

et al.

Evaluation of combination therapy in animal models of cerebral ischemia. J Cereb Blood Flow Metab 2012; 32: 585–597.

23.

Di Carlo

Lamassa

Baldereschi

et al.

Sex differences in the clinical presentation, resource use, and 3-month outcome of acute stroke in Europe. Stroke 2003; 34: 1114.

24.

Clayton

. Studying both sexes: a guiding principle for biomedicine. FASEB J 2016; 30: 519–524.

25.

Clayton

Collins

. NIH to balance sex in cell and animal studies. Nature 2014; 509: 282–283.

26.

NIH, https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-102.html (accessed 24 August 2015).

27.

Chauhan

Moser

McCullough

. Sex differences in ischaemic stroke: potential cellular mechanisms. Clin Sci 2017; 131: 533.

28.

McCullough

Zeng

Blizzard

et al.

Ischemic nitric oxide and poly (ADP-ribose) polymerase-1 in cerebral ischemia: male toxicity, female protection. J Cereb Blood Flow Metab 2005; 25: 502–512.

29.

Heidari

Babor

De Castro

et al.

Sex and Gender Equity in Research: rationale for the SAGER guidelines and recommended use. Res Integrity Peer Rev 2016; 1: 2.

30.

Hossmann

. The two pathophysiologies of focal brain ischemia: implications for translational stroke research. J Cereb Blood Flow Metab 2012; 32: 1310–1316.

31.

Sutherland

Neuhaus

Couch

et al.

The transient intraluminal filament middle cerebral artery occlusion model as a model of endovascular thrombectomy in stroke. J Cereb Blood Flow Metab 2016; 36: 363–369.

32.

Balkaya

Trueman

Boltze

et al.

Behavioral outcome measures to improve experimental stroke research. Behav Brain Res 2018; 352: 161–171.

33.

Sena

Currie

McCann

et al.

Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab 2014; 34: 737–742.

34.

Lees

Bath

PMW

Schellinger

et al.

Contemporary outcome measures in acute stroke research. Stroke 2012; 43: 1163.

35.

Voelkl

Vogt

Sena

et al.

Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol 2018; 16: e2003693.

36.

Berge

Salman

RA-S

van der Worp

et al.

Increasing value and reducing waste in stroke research. Lancet Neurol 2017; 16: 399–408.

37.

Minnerup

Zentsch

Schmidt

et al.

Methodological quality of experimental stroke studies published in stroke journal: time trends and effect of the basic science checklist. Stroke 2016; 47: 267.

38.

Hachinski

Iadecola

Petersen

et al.

National Institute of Neurological Disorders and stroke-Canadian stroke network vascular cognitive impairment harmonization standards. Stroke 2006; 37: 2220.

39.

Egan

Vesterinen

Beglopoulos

et al.

From a mouse: systematic analysis reveals limitations of experiments testing interventions in Alzheimer's disease mouse models. Evidence Based Preclin Med 2016; 3: 12–23.

40.

Vorhees

Williams

. Morris water maze: procedures for assessing spatial and related forms of learning and memory. Nat Protocols 2006; 1: 848.

41.

Bingham

Martin

Macrae

et al.

Watermaze performance after middle cerebral artery occlusion in the rat: the role of sensorimotor versus memory impairments. J Cereb Blood Flow Metab 2012; 32: 989–999.

42.

Whishaw

Tomie

J-A

. Of mice and mazes: similarities between mice and rats on dry land but not water mazes. Physiol Behav 1996; 60: 1191–1197.

43.

Hainsworth

Allan

Boltze

et al.

Translational models for vascular cognitive impairment: a review including larger species. BMC Med 2017; 15: 16.

44.

Pollock

St George

Fenton

et al.

Top ten research priorities relating to life after stroke. Lancet Neurol 2012; 11: 209.

45.

Trueman

Diaz

Farr

et al.

Systematic and detailed analysis of behavioural tests in the rat middle cerebral artery occlusion model of stroke: tests for long-term assessment. J Cereb Blood Flow Metab 2017; 37: 1349–1361.

46.

Hernández-Jiménez

Peña-Martínez

Godino

MdC

et al.

Test repositioning for functional assessment of neurological outcome after experimental stroke in mice. PLoS One 2017; 12: e0176770.

47.

Bernhardt

Borschmann

Boyd

et al.

Moving rehabilitation research forward: developing consensus statements for rehabilitation and recovery research. Int J Stroke 2016; 11: 454–458.

48.

Corbett

Carmichael

Murphy

et al.

Enhancing the alignment of the preclinical and clinical stroke recovery research pipeline: consensus-based core recommendations from the Stroke Recovery and Rehabilitation Roundtable translational working group. Int J Stroke 2017; 12: 462–471.

49.

Ramirez

Motazedian

Jung

et al.

Methodological rigor in preclinical cardiovascular studies: targets to enhance reproducibility and promote research translation. Circ Res 2017; 120: 1916.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB