The role of systematic reviews in identifying the limitations of preclinical animal research,2000–2022: part 1

Introduction

A thousand years ago, the Persian physician Ibn Sina issued a strong warning about the pitfalls of using animals to develop medicines for humans:

Experiments should be carried out on the human body. If the experiment is carried out on the bodies of [other animals] it is possible that it might fail for two reasons: the medicine might be hot compared to the human body and be cold compared to the lion’s body or the horse’s body … The second reason is that the quality of the medicine might mean that it would affect the human body differently from the animal body … These are the rules that must be observed in finding out the potency of medicines through experimentation. Take note! ¹

While Ibn Sina’s advice to focus on humans was for the most part heeded for hundreds of years, the twentieth century saw an astounding increase in the number of animals used in research intended to have application to humans. Yet this has not borne fruit, and Ibn Sina’s advice remains relevant today, as Nasser et al. ² point out. A millennium after Ibn Sina wrote The Canon of Medicine, Dutch neurologists Janneke Horn and Martien Limburg published a systematic review of clinical trials of the neuroprotectant drug nimodipine.^3,4 It had been thought that nimodipine would reduce some of the adverse consequences of stroke, but the systematic review did not find this. The neurologists were bewildered; the clinical trials had only proceeded because studies in animals had indicated that the drug would be beneficial. Intrigued, they set out to retrieve all the relevant animal studies and examine them more closely. To their surprise, when these studies were reviewed systematically, the apparently beneficial effect in animals ‘disappeared’. ⁵ Their systematic review was one of the first to be conducted in the field of animal research and clearly demonstrated the value of synthesising studies which, if considered individually, might be misleading. Ibn Sina had warned about the problems of extrapolating findings from one species to another, but systematic reviews were pointing to further problems – limitations in the way research was being conducted and interpreted.

Two decades earlier, in what has been described as a rallying call for evidence synthesis,^6,7 the Scottish doctor, Archie Cochrane, had written: ‘It is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspeciality, adapted periodically, of all relevant randomised controlled trials’. ⁸ Collaborative reviews of certain clinical fields had been taking place sporadically (e.g. Chalmers et al. ⁹ ) but it was not until 1989 that the term ‘systematic review’ was first used. Not long after, in 1993, the international Cochrane Collaboration was established to generate robust evidence by systematically reviewing clinical trial data, so that informed decisions could be made about healthcare. ⁷ By the early 2000s, systematic reviews were a familiar entity within clinical research and considered vital for addressing uncertainties about the effects of treatments. ¹⁰ They provide a rigorous methodology for evaluating and synthesising a body of research so that, based on all the available relevant evidence, conclusions may be reached about an intervention’s effects.

UK

In 2002, clinical epidemiologists Peter Sandercock and Ian Roberts published a short commentary in The Lancet which made the case for conducting systematic reviews of animal studies on a more routine basis. ¹¹ The authors observed that about 1 in every 1000 MEDLINE records about human research was tagged as a meta-analysis compared with only 1 in every 10,000 records about animal research. Referring to the nimodipine research, they noted that, had the animal evidence been reviewed earlier, the 6400 or so patients who participated in the 22 clinical trials might have been spared the potential risk and inconvenience of taking part, never mind the costs of conducting the trials.

In 2002, Pandora Pound was working in the School of Social and Community Medicine (now Population Health Sciences) at the University of Bristol. With a background in the sociology of medicine, Pound was interested in the contribution of animal research to human medicine and had been discussing this with Susan Green, director of the patient safety charity SABRE, ¹² wondering whether systematic reviews of animal studies might throw light on the issue. Pound and epidemiologist Shah Ebrahim, who was head of the School at the time, decided to investigate, and invited clinical epidemiologists Ian Roberts, Peter Sandercock and Michael Bracken to collaborate. The result was a 2004 paper published in the BMJ entitled ‘Where is the evidence that animal research benefits humans?’ ¹³ The authors argued that little data were available to support the use of animals in preclinical research and suggested that systematic reviews of animal studies might generate the evidence necessary to answer the question they posed.

Since animal research was particularly controversial at that time, Ebrahim, out of courtesy, gave advance notice of the paper’s publication to the Dean of Medicine and the Director of Research at the University of Bristol. Because the School was in receipt of Medical Research Council (MRC) funding, Ebrahim also discussed it with Colin Blakemore, then Chief Executive of the MRC.All of those consulted strongly advised against publishing the paper, declaring that it would provide ammunition for the anti-vivisection movement and that it constituted an attack on essential biomedical research. The authors were also advised that they were not in a position to understand animal experiments. Disturbingly, Ebrahim was told that publishing the paper would not enhance his career prospects. In the early 2000s, tensions were running high. Many laboratories, including Huntingdon Life Sciences in Cambridgeshire (now Harlan Laboratories), were the focus of regular anti-vivisection protests and a small number of activists had been imprisoned for criminal activity. But a Select Committee inquiry into the use of animals in scientific research (commissioned by the House of Lords) concluded that animal experimentation was ‘a valuable research method which has proved itself over time’. ¹⁴ Within this febrile and polarised atmosphere, few academics questioned animal research for fear of being branded anti-vivisectionist (the press having successfully associated this with criminality). All of this helped solidify the long-held view of preclinical animal research as a ‘sacred cow’ – a practice beyond criticism or challenge.

Unsurprisingly then, the paper, which was reported in the national media, met with some hostility. Mark Henderson, then science correspondent at The Times newspaper (now Director of Corporate Affairs at the Wellcome Trust) attempted to publicly discredit the authors by referring to them as ‘the anti-vivisection lobby, or at least its law-abiding element’ in an article for The Times, ¹⁵ while the lobbying group ‘Coalition for Medical Progress’ (now Understanding Animal Research) attacked the paper and attempted to refute the claim that animal research lacked supporting evidence by referring to four disparate areas ‘where the advances are clear: polio, kidney dialysis, stomach ulcers and cystic fibrosis’, ¹⁶ In the ‘rapid responses’ to the paper published on the BMJ website, it was described by one critic as ‘spectacularly ill-judged’ and ‘scientifically invalid’ while another respondent stated that it ‘should never have been published in a peer-reviewed journal’. ¹⁷ Nevertheless, many of the responses indicated that the paper – and the debate that it was provoking – was welcome.

In what seems unlikely to have been a coincidence, on the same day the paper was published the UK’s Royal Society published a ‘guide’, the opening lines of which claimed, ‘Humans have benefited immensely from scientific research involving animals, with virtually every medical achievement in the past century reliant on the use of animals in some way’. ¹⁸ Blakemore publicly backed the Royal Society’s position, asserting ‘Animal research has contributed to virtually every area of medicine’. ¹⁹ Ironically, then, the paper’s publication was provoking exactly the sort of unsupported claims and assertions about the benefits of animal research that it highlighted as problematic. A few years later, Robert Matthews dissected claims such as those made in the Royal Society’s ‘guide’ and found that they were strikingly similar to an anonymous, one-page declaration by the US Public Health Service published in 1994, itself lacking any references or supporting evidence. After examining the statistical basis for such claims, Matthews argued that they should either be formally validated or replaced with statements capable of validation. ²⁰ Nevertheless, similar claims continue to be made.

It may have been that the title of the 2004 paper by Pound et al. ¹³ was inflammatory, or perhaps it was simply because, for the first time in a highly respected medical journal, a group of scientists was raising doubts about the evidence underlying the practice of animal research. Nevertheless, despite its vilification within much of the animal research community, the paper kick started a number of preclinical systematic reviews. Another positive outcome was that, as a result of Ebrahim’s discussions with Blakemore, the MRC provided a small pot of money for a comparison of systematic reviews of animal and human studies. A team led by clinical neurologist Malcolm Macleod successfully bid for this funding and went on in 2004 to form the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES) at Edinburgh University.

CAMARADES

The CAMARADES collaboration, ²¹ based in Edinburgh, conducts preclinical systematic reviews and provides support for other researchers conducting systematic reviews and meta-analyses of animal studies. The collaboration was first established to address ‘translational failures’ in preclinical stroke research, but soon broadened its scope to include other diseases. CAMARADES currently has five national co-ordinating centres and many more global participating centres. It aims to identify potential sources of bias in animal research, generate recommendations for improving the design and reporting of animal studies, and develop meta-analysis methodology so that it better applies to animal studies. As well as acting as a repository for completed reviews, it provides guidance and tools such as Systematic Review Facility (SyRF), ²² a free online systematic review platform.

In 2007, Malcolm Macleod’s team published the findings of their comparison of human and animal systematic reviews in the BMJ. ²³ The team identified six interventions for which there was unambiguous systematic review evidence of a treatment effect for humans: corticosteroids for brain injury; antenatal corticosteroids for neonatal respiratory distress; bisphosphonates for osteoporosis; antifibrinolytics for haemorrhage; thrombolytics for stroke; and a neuroprotective agent (tirilazad) for stroke. They then searched for and systematically reviewed all published and unpublished controlled animal studies for the same six interventions. They assessed the methodological quality of the animal studies (based on measures taken to prevent bias) as ‘poor’ across all six interventions. Comparing the results from the systematic reviews of animal studies with the systematic reviews of clinical studies, the authors found that two interventions (bisphosphonates for osteoporosis, thrombolytics for stroke) were concordant, i.e. the findings from the animal studies agreed with the findings from the human studies, one intervention was partially concordant (antenatal corticosteroids for preterm delivery) and three were discordant (corticosteroids for brain injury, tirilazad for stroke and antifibrinolytics for haemorrhage). They concluded that discordance was likely to be due to bias within the animal studies, or to failure of the animal models to adequately mimic human disease and treatment. This important study is one of only a few that have used systematic review methodology to understand how and whether animal studies translate to humans.

Netherlands

Meanwhile, in 2005, veterinarian Merel Ritskes-Hoitinga took up the post of Managing Director of the central animal facility at Radboud University Medical Center, Nijmegen, a post combined with a Professorship in Laboratory Animal Science. In 2006, she founded the 3Rs (Replacement, Reduction, Refinement) Research Centre, which aimed to help researchers implement the 3Rs with a view to improving animal welfare and laboratory animal science. Over time, however, she became disillusioned about the ability of the 3Rs to improve either. ²⁴ In 2008, she happened to be in Portugal for a conference where she attended a presentation by Malcolm Macleod. By the time Macleod uttered his closing words, Ritskes-Hoitinga had decided that preclinical systematic reviews would provide a more promising way of improving the quality of preclinical science than the 3Rs. She then met with Iain Chalmers, one of the founders of the Cochrane Collaboration, as well as Susan Green from SABRE, who was concerned about the ways in which poorly conducted animal research eventually impacted patients. ²⁵ Ritskes-Hoitinga went on to found the Systematic Review Centre for Laboratory Animal Experimentation, better known as SYRCLE ²⁶ in 2012 at Radboud University in the Netherlands.

In 2011, she invited members of the Dutch Parliament to visit her at the central animal facility in Nijmegen to discuss the need for preclinical systematic reviews. The visit made a strong impression on the members and perhaps also influenced the Partij voor de Dieren (Party for the Animals), which at the time had almost 2% of the votes and two of the 150 House of Representatives’ seats. ²⁷ A motion was passed in the Dutch Parliament the following year asking the government to make systematic reviews the norm in preclinical research as they had become in the clinical field. A second motion was passed in the Dutch Parliament in 2014, seeking to make systematic reviews a compulsory part of laboratory animal scientists’ training.

SYRCLE

SYRCLE launched with an international symposium in 2012 at Radboud University, Nijmegen. The symposium explored why scientific standards appeared to be lower for preclinical animal research despite its intended application to human health. Speakers focused on the challenges involved in adapting the methodology of systematic review to the preclinical field, so that animal research could be held to the same high standards as clinical research. The symposium, the first of its kind, had Macleod as the keynote speaker, with other speakers including Ian Roberts, Michael Bracken, Marlies Leenaars, Carlijn Hooijmans and Ritskes-Hoitinga. Esther Ouwehand, politician for the Dutch Party for Animals, sent a video message. ²⁸ Chalmers concluded the conference, ²⁹ noting that with more than 100 participants, the gathering was bigger than the first Cochrane meetings. He also reminded participants of the need to honour Janneke Horn, as one of the first people to conduct a preclinical systematic review. (Five years later she was awarded the FEDERA award ³⁰ for her pioneering preclinical systematic review.)

The Evidence-Based Toxicology Collaboration, ³¹ which advocates systematic reviews in toxicology, was founded in the United States at around the same time, and a close relationship developed with SYRCLE. The National Toxicology Program in the USA ³² actively promotes methods development and the harmonisation of systematic review approaches, and in collaboration with SYRCLE, developed a database of systematic reviews of animal studies. ³³ SYRCLE also joined Evidence Synthesis International, ³⁴ which was formed later to bring together organisations that were generating or using evidence syntheses. CAMARADES also collaborated with these groups, so relationships were formed that spanned the globe.

Realising that laboratory animal scientists would need training and support to conduct and facilitate systematic reviews, the SYRCLE group developed tools such as search filters for researchers to use when searching for animal studies,^35–37 as well as guidance on performing literature searches. ³⁸ They also provided guidance on evaluating the risk of bias in studies included in systematic reviews, ³⁹ performing meta-analysis on animal studies, ⁴⁰ grading the evidence, ⁴¹ and preparing, registering and publishing systematic review protocols. ⁴² As a result of the latter, the publication of preclinical systematic review protocols gained momentum, meaning that in addition to human studies, Prospero ⁴³ began registering and maintaining a database of protocols of preclinical systematic reviews relevant to human health. Additionally, the SYRCLE group published a checklist of items to include when reporting animal studies (both to improve reporting and to make subsequent systematic reviews easier), ⁴⁴ while a 2013 paper in PLOS Medicine highlighted the advantages of preclinical systematic reviews and the progress that had been made in the field. ⁴⁵ This paper emphasised the importance of the routine funding and conduct of preclinical systematic reviews to maximise transparency and avoid waste and unnecessary duplication. It also noted that cooperation between multiple stakeholders, including patients, was essential for further progress. A later paper made the case that preclinical systematic reviews can lead directly to achieving the 3Rs, as well as better science. ⁴⁶

Following some negative feedback from academia in response to the motions passed on systematic reviews, the Dutch Parliament commissioned some research to find out why. The research revealed that many scientists were unfamiliar with the methodology or value of systematic reviews. ⁴⁷ As a result, the Dutch Health Funder, ZonMw, created a ‘Synthesis of Evidence’ funding stream within the ‘More Knowledge with Fewer Animals’ programme, defining preclinical systematic reviews as animal-free innovations. It funded SYRCLE to offer one-*day workshops for researchers wanting to learn how to conduct systematic reviews and paid two months’ salary to those who went on to conduct a review. From 2012, when the funding started, until 2020, when the programme was evaluated, around 400 researchers had participated in 21 one-day workshops, learning the theory and practice of systematic reviews. A total of 88 participants began a systematic review with training from SYRCLE, and by the end of 2020, 38 had published their reviews. The programme evaluation concluded that the training raised awareness about the value of systematic reviews for research and researchers, as well as the need to bring greater scientific rigour to the conduct of animal studies, leading to an improvement in animal research quality. ⁴⁸

In 2016, following a request from the Dutch Ministry, SYRCLE began offering e-learning courses in how to conduct preclinical systematic reviews. By 2020 this e-learning (login code: syrcle) ⁴⁹ was being used by more than 4000 participants from 65 countries globally. An international ambassador network was also established to stimulate the adoption of preclinical systematic review methodology. Currently, 40 research ‘ambassadors’ in 16 countries have committed to encouraging its use locally.

Although there was initially some resistance to systematic reviews among preclinical researchers, Ritskes-Hoitinga calculated that there was a 35% reduction in the number of laboratory animals used at Radboud University since it began employing the methodology, while animal use in the Netherlands as a whole reduced over the same period by 15%. (In the UK, however, laboratory animal numbers rose to a peak of 4.4 million in 2015 before declining again, with 2.88 million animals being used in 2020.) In 2017, in recognition of the contribution of systematic reviews to higher scientific standards in the preclinical field, SYRCLE won the Cochrane REWARD Second Prize ⁵⁰ for reducing waste and increasing value in research.

Ritskes-Hoitinga had long wanted the preclinical field to join the Cochrane Collaboration and in 2010 she organised a workshop at a meeting of the Federation of European Laboratory Animal Science Associations in Helsinki to discuss the possibility of joining. A further meeting was held at the 8th World Congress on Alternatives and Animal Use in Life Sciences in 2011; this resulted in the Montreal Declaration on the Synthesis of Evidence to Advance the 3Rs Principles in Science, ⁵¹ as well as a paper ⁵² on the many advantages of preclinical systematic reviews, but no progress was made towards joining the Cochrane Collaboration. In 2013, SYRCLE was advised to apply to become a Cochrane Animal Study Methods Group in collaboration with CAMARADES. Consequently, an application was prepared, and the two groups met in London, paving the way for membership. Ritskes-Hoitinga and colleagues also made the case for an animal study methods group in an editorial published in the Cochrane Library in 2014. ⁵³ By this time, however, the Cochrane Collaboration had started to reorganise, making the application process uncertain and, as yet, the focus of the Cochrane Collaboration remains solely on clinical evidence.

Part 2, the concluding part of this series, will consider some of the evidence revealed by preclinical systematic reviews, and will be published in a forthcoming issue of the journal.