The Relevance of In Silico,In Vitro and Non-human Primate Based Approaches to Clinical Research on Major Depressive Disorder

Abstract

Major depressive disorder (MDD) is the most severe form of depression and the leading cause of disability worldwide. When considering research approaches aimed at understanding MDD, it is important that their effectiveness is evaluated. Here, we assessed the effectiveness of original studies on MDD by rating their contributions to subsequent medical papers on the subject, and we compared the respective contribution of findings from non-human primate (NHP) studies and from human-based in vitro or in silico research approaches. For each publication, we conducted a quantitative citation analysis and a systematic qualitative analysis of the citations. In the majority of cases, human-based research approaches (both in silico and in vitro) received more citations in subsequent human research papers than did NHP studies. In addition, the human-based approaches were considered to be more relevant to the hypotheses and/or to the methods featured in the citing papers. The results of this study suggest that studies based on in silico and in vitro approaches are taken into account by medical researchers more often than are NHP-based approaches. In addition, these human-based approaches are usually cheaper and less ethically contentious than NHP studies. Therefore, we suggest that the traditional animal-based approach for testing medical hypotheses should be revised, and more opportunities created for further developing human-relevant innovative techniques.

Introduction

According to the World Health Organization, depression is the leading cause of morbidity worldwide. It affects more than 300 million people of all ages and is a major contributor to the overall global burden of disease. ¹ People who suffer from depression are more prone to an early death either by suicide or through the development of other conditions such as cancer, heart disease or stroke. ^2,3 In addition, these patients are also more prone to a number of other disorders (e.g. osteoporosis) ⁴ that, although not life-threatening, do significantly impact not only quality of life but also public health and national economies.

Accordingly, major investment has been dedicated to research aiming to improve the understanding of all eight forms of depression. ⁵ Major depressive disorder (MDD) is the most severe type and the third leading cause of long-term disability. ⁶ Besides, the few studies that have comprehensively investigated the impact of MDD in Europe (from 2004 to 2010) have shown that MDD was the costliest brain disorder in Europe, accounting for at least 1% of the total European economy. ^7,8 In the United States, the economic burden of MDD alone was US$210.5 billion in 2010. ⁹

Clinical research is expensive, time-consuming and potentially ethically contentious. For instance, every patient who enrols in a clinical trial is subject to an increased level of risk with respect to deviations from their regular clinical care, particularly with regard to the occurrence of unexpected effects from exposure to a new treatment. Non-clinical (i.e. preclinical) research, often involving non-human animals and human-based in vitro and in silico approaches, is sometimes valuable in the early steps of biomedical research to simplify and accelerate drug and treatment discovery. However, to optimise the outcomes of this non-clinical research, it is crucial to evaluate the research approaches that might have the most potential for patient treatment results.

Animal-based research has been accepted as the ‘gold standard’ approach for preclinical biomedical research and testing since the second half of the 20th century. ¹⁰ Within this approach, non-human primate (NHP) research has been considered particularly relevant, due to the similarity between humans and NHPs. However, this similarity has led to NHPs being afforded various degrees of legal protection in different regions of the world. For example, Europe, ¹¹ the United States ¹² and New Zealand ¹³ have imposed considerable restrictions to the use of NHPs for scientific purposes. These restrictions are due to the understanding that subjecting NHPs to laboratory confinement alone, even before considering the use of any invasive or intrusive procedures, has resulted in psychosomatic injury, mutilation and physiological traits that have been compared to those exhibited by people with post-traumatic stress disorder. ^{13 –20} Moreover, NHPs are expensive to acquire ²¹ and are the most expensive animals to maintain. ²²

The legislation on animal use for experimental purposes of several countries (e.g. Directive 2010/63/EU) requires a cost–benefit assessment to be carried out prior to conducting a procedure on a non-human animal. For each project, the likely harm to the animal should be balanced against the potential benefits, and the project should only go ahead if the expected benefits outweigh the harms inflicted to the animals involved.

Considering all of the above, it is assumed that when research is conducted on NHPs, due to the ethical and economic concerns surrounding this practice, this research should provide highly relevant data that lead to concrete improvements in patient outcomes. While some authors assert that animal research approaches, and those involving NHPs in particular, are crucial for biomedical progress, ²³ an increasing number of evidence-based papers show that the contribution of animal-based research to the advancement of human healthcare has been poor, ²⁴ including in the case of MDD. ²⁵ However, it is yet to be established whether this poor contribution is due to the intrinsic limitations of all non-clinical research, or whether human-based (in vitro and in silico) non-clinical research approaches are more effective in helping biomedical progress, at least when seeking to understand complex disorders of a multifactorial origin, such as MDD.

In vitro and in silico methods that directly rely on human-based knowledge and/or material are thought to potentially allow for faster development of medical treatments. ^26,27 Usually, they are also more cost-effective than animal-based methods. However, despite yielding data of sufficient value to further disease understanding in humans, and providing the means to test new therapies, such non-animal methods are still judged against the standard biomedical research paradigm. Indeed, they are seen as incomplete on their own and considered to be preliminary steps prior to (often contradictory) animal testing. ^28,29

To shed light on this debate, the current study examines and compares the contribution of results from NHP studies, as well as from in silico-based and in vitro-based approaches, to clinical studies on MDD. This allows us to: (a) evaluate whether the low transferability of knowledge to clinical research is a common trait of all non-clinical research approaches; and (b) evaluate the specific relevance of NHP studies and human-based in silico and in vitro approaches to human clinical studies.

Considering the dominance of NHP studies within the current preclinical research paradigm, we expect the findings from these studies to have a higher contribution to subsequent clinical research than findings from in silico-based and in vitro-based studies. A similar or lower contribution from NHP studies would suggest that clinical research is becoming less reliant on this more costly and ethically questionable type of research, thus suggesting that the time for a paradigm shift has come.

Methods

The design of this study was based on a previously developed method consisting of a quantitative citation analysis and a systematic qualitative analysis of citations. ³⁰

Quantitative citation analysis

Bibliographic search

The citation analysis was performed between September 2016 and June 2017. The PubMed bibliographic database was searched for papers that described studies employing either NHPs, or in vitro or in silico research approaches, to investigate MDD. The following Medical Subject Heading (MeSH) search terms were used: ‘Depressive Disorder, Major’ AND MeSH terms: ‘primate’ OR ‘ape’ OR ‘macaque’ OR ‘macaca’ OR ‘rhesus’ OR ‘chimpanzee’ OR ‘bonobo’ OR ‘gorilla’ OR ‘gorila’ OR ‘Pan’ OR ‘orangutang’ OR ‘orang-utan’ OR ‘Orang utan’ OR ‘orangutan’ OR ‘ourang-outang’ OR ‘Pongo’ OR ‘gibbon’ OR ‘Hylobates’ OR ‘Colobus’ OR ‘Baboon’ OR ‘Papio’ OR ‘Mandrillus’ OR ‘Mandrill’ OR ‘Cebus’ OR ‘Cebuella’ OR ‘Brachyteles’ OR ‘Loris’ OR ‘Nycticebus’ OR ‘lemur’ OR ‘Callithrix’ OR ‘in silico’ OR ‘computer model’ OR ‘mathematical model’ OR ‘computer simulation’ OR ‘in vitro’ OR ‘cell culture’ OR ‘culture technique’ OR ‘cell line’ OR ‘organ culture’ OR ‘tissue culture’.

MeSH terms are a comprehensive list of key terms related to each human disorder, designed to identify all relevant studies in a given area. ³¹ Thus, searching for ‘Depressive Disorder, Major’ retrieves other nomenclatures for the same disorder (e.g. Melancholia). There were no exclusive MeSH terms for NHPs, so the search retrieved additional papers with non-human animals that were excluded by manual sorting. All in vitro-based and in silico-based papers that used animal data (e.g. rat cell line data) were also excluded.

Papers from scientific journals, books, research reports and conference proceedings written in English, Portuguese or Italian were included (being within the authors’ linguistic fluencies). PubMed filters were used, in order to exclude review papers (‘review’, ‘systematic review’, ‘meta-analysis’, ‘bibliography’), as well as editorials and other types of non-research papers (‘biography’, ‘auto-biography’, ‘comment’, ‘opinion paper’, ‘interview’), since the aim of the study was to evaluate the impact of original data. The search was restricted to publications prior to 31 December 2011, to allow adequate time for subsequent citation of papers. ³² Nineteen NHP study-based papers, 29 in silico-based papers and 38 in vitro-based papers describing data from original MDD research were retrieved (see Appendix 1).

Citation data

A citation analysis on the retrieved papers was performed by using the cited reference search facility within the Web of Science bibliographic database. For each retrieved paper, the subsequent papers that cited it were identified, and three types of citation data were recorded:

• – the total number of times that the retrieved paper was cited;

• – the total number of times that the retrieved paper was cited per research category; and

• – the total number of times that the retrieved paper was cited per research subject, that is, on MDD or other subjects, as detailed below.

Each citing paper was ascribed to one or more of the following eight research categories: ‘invasive animal research’; ‘human research’; ‘review’; ‘opinions’ (including editorials, comments or replies to comments); ‘in vitro’; ‘in silico’; ‘non-invasive animal research’ (e.g. observational studies with wild animals); and ‘other human studies’ (e.g. on social perceptions). The term ‘human research’ referred to any human-based research that might involve, among other things, the analysis of biological samples, epidemiological and behavioural studies, medical case studies and clinical studies. A citing paper could be allocated to more than one category, if it described different research approaches. Whenever the category of the citing paper could not be defined (due to language barriers or absence of an abstract), the paper was labelled as ‘not available’ and removed from further analysis.

Among the categories ‘human research’, ‘in silico’, ‘in vitro’ and ‘invasive animal research’, it was also recorded whether the citing paper focused on MDD or on other subjects.

Statistical analysis

To test for differences between the numbers of citations across research approaches, three generalised linear models (GLMs), each with a Poisson response and a log link function, were implemented. Each model tested one of the following response variables: (a) the total number of citations; (b) the total number of citations by papers in the category ‘human research’; and (c) the total number of citations by papers in the category of ‘human research’ that focused specifically on MDD. In each model, the only explanatory variable was the type of research approach, of which there were three: NHP studies, in silico-based approaches and in vitro-based approaches. The GLM’s goodness of fit was evaluated by visual inspection of the diagnostic plots. Additionally, a Gaussian GLM was used to evaluate whether the proportions of citations by human research papers, and by human research papers specifically on MDD, were different across the three approaches. The analyses were performed in R 3.6.1, ³³ by using the function glm. The results were considered significant when p < 0.05.

Systematic qualitative analysis of citations

Citing papers featuring human research specifically on MDD were systematically analysed by two independent raters, to qualitatively evaluate the contribution of knowledge from NHP studies, or from in vitro-based or in silico-based research approaches, to the respective human clinical study. Each study was rated according to the following classes, which were defined prospectively, as in Carvalho et al. ³⁰ :

• – Redundant: when the cited study was only mentioned among other studies as an example. In the case where multiple studies were used as examples of one or more points, the raters were instructed to rate the study as redundant only if there were older or human studies stating exactly the same points.

• – Minor relevance: when the cited study was cited in either the Discussion or the Introduction, to provide information not directly related to the hypothesis explored in the human study.

• – Relevant to the hypothesis: when the cited study was cited in the Introduction, to provide information relevant to the hypothesis explored in the human study.

• – Relevant to the methods: when the human study used the same methodology as that described in the cited paper, with the exception of species differences in the case of NHP study methods.

A paper considered to be ‘relevant’ could be both relevant for the hypothesis and the methods. The other options in the classes are mutually exclusive. In all cases, disagreement between the raters was resolved via detailed discussions until a consensus was reached.

Whenever it was not possible to assess the contribution of a cited paper to a human study due to unavailability of the full publication on the human study, the human research paper was labelled as ‘not available’ and removed from further analysis.

A statistical test was used for comparing proportions (Pearson’s χ ² test implemented via R’s prop.test function), in order to assess differences between the three cited approaches (i.e. NHP studies, and in vitro-based and in silico-based approaches). Since, even for the pair with the largest difference, the null hypothesis of equal proportions could not be rejected under the usual significance levels, corrections for multiple comparisons were not attempted.

Results

Citation analysis

NHP study-based results

Nineteen publications featuring NHP studies in the field of MDD research were retrieved, which were subsequently cited 841 times in total. Of these 19 papers, five featured both human and NHP data.

The subsequent citing papers belonged to the following categories: invasive animal research (312); reviews (245); human research (152); in vitro research (81); in silico research (14); non-invasive animal research (6); and opinions, including editorials, comments or replies to comments (4). Eighty-five citing papers were not categorised due to being unavailable or written in a language other than English, Portuguese or Italian.

Of the 312 citations by animal research papers, 63 were specifically focused on MDD; of the 152 citations by human research papers, 71 were specifically focused on MDD.

In silico-based approach results

Twenty-nine publications describing the use of in silico-based approaches in the context of MDD research were retrieved, which were subsequently cited 806 times in total. Of these 29 papers, seven featured both patient data and computer simulations.

The subsequent citing papers belonged to the following categories: human research (317); in silico research (193); reviews (193); invasive animal research (44); in vitro research (17); and opinions (17). Fifty-eight citing papers were not categorised due to being unavailable or written in a language other than English, Portuguese or Italian.

Of the 317 citations by human research papers, 94 specifically focused on MDD; of the 193 citations by in silico-based research papers, 36 specifically focused on MDD.

In vitro-based approach results

Thirty-eight publications describing the use of in vitro-based approaches in the context of MDD research were retrieved, which were subsequently cited 2,574 times in total. All of the in vitro-based papers used samples of human biological material, mostly being obtained from MDD patients (in 34 out of the 38 studies).

The subsequent citing papers belonged to the following categories: in vitro research (1,239), resorting to the use of human biological material (789), laboratory animal biological material (373) or biological material from both sources (12); human research (978), of which 189 studies solely used human participants without concurrent use of in vitro-based research approaches; reviews (844); invasive animal research (464), of which 79 studies solely used live animals without concurrent use of in vitro-based research approaches; opinions (27); and in silico research (16). One hundred and fifty-four citing papers were not categorised due to being unavailable or written in a language other than English, Portuguese or Italian.

Of the 978 citations by human research papers, 482 specifically focused on MDD; of the 1,239 citations by in vitro research papers, 487 specifically focused on MDD.

Comparison of citations of papers based on NHP studies, in vitro approaches and in silico approaches

An inspection of the diagnostic plots showed no reason for concern with regard to the GLM fit. Among the papers using an in vitro-based approach, one was frequently cited (711 citations). We performed the analysis both with and without this potential outlier and found no significant differences between the two scenarios.

The GLM estimated the average number of citations per paper for each of the three approaches (Figure 1 (a)). Each NHP paper was cited 3.73 times (standard error (SE): 0.03). Papers based on in silico approaches were cited less frequently than this (3.29 times; i.e. −0.44, SE: 0.05), and papers based on in vitro approaches were cited more frequently (4.23 times; i.e. +0.5, SE: 0.04). Both differences were statistically significant (p < 0.0001).

With regard to the average number of subsequent citations by human research papers (Figure 1 (b)), each NHP paper was cited 2.03 times (SE: 0.08). In comparison, papers based on in vitro and in silico approaches were more frequently cited (+1.09, SE: 0.09 and +0.33, SE: 0.10, respectively). These differences were statistically significant (p < 0.001).

When looking at the average numbers of citations by human research papers specifically focused on MDD (Figure 1 (c)), each NHP paper was cited 1.27 times (SE: 0.12), which was not statistically different from the number of citations of papers based on in silico approaches (−0.12, SE: 0.16). In these MDD-specific publications, papers based on in vitro approaches received, on average, more citations (+1.3, SE: 0.13) than papers based on NHP studies, and the difference was statistically significant (p < 0.001).

Figure 1.

The number of citations received by the retrieved papers, according to research approach. A bibliographic search was carried out to retrieve papers on MDD, which were categorised as based on NHP studies or in silico or in vitro approaches, according to the research method described. A citation analysis was then performed to identify papers that subsequently cited these retrieved papers. The graphs show: (a) the total number of times that the retrieved papers were cited, according to their research approach; (b) the number of times that the retrieved papers were cited by papers on human research, according to their research approach; and (c) the number of times that the retrieved papers were cited by papers on human research specifically focused on MDD, according to their research approach. For visualisation purposes, the largest observation in the ‘In vitro’ category was excluded from the data used to generate the graphs. MDD: major depressive disorder; NHP: non-human primate.

The estimated proportion of citations of NHP papers by human research papers was 0.13 (SE: 0.05). This proportion was significantly higher for papers based on in silico approaches (+0.20, SE: 0.07, p = 0.004) and also for papers based on in vitro approaches (+0.30, SE: 0.07, p < 0.0001).

The estimated proportion of citations of NHP papers by human research papers specifically focused on MDD was 0.06 (SE: 0.03), which was not significantly different from the proportion of citations of papers based on in silico approaches (+0.06, SE: 0.04, p = 0.1389). The proportion of citations in these MDD-specific publications, of papers based on in vitro approaches (+0.14, SE: 0.04), was significantly different from that of the NHP papers (p = 0.001).

Systematic qualitative analysis of citations

Of the 71 human research papers specifically focused on MDD that cited NHP papers, 50 (70%) were fully available for further analysis, along with 401 of the 482 (83%) human research papers on MDD that cited in vitro-based papers, and 58 of the 94 (62%) human research papers on MDD that cited in silico-based papers. It was judged that eight of 50 (16%), 15 of 58 (25%) and 100 of 401 (25%) of citations of papers based on NHP studies, in silico and in vitro approaches, respectively, were relevant to the hypothesis and/or the methods in the citing human research paper on MDD (see Table 1).

Table 1.

The relevance of cited NHP study-based, in silico-based or in vitro-based papers to subsequent (i.e. citing) human research papers focused on MDD.^a

Citations which are:	Papers based on NHP studies	Papers based on in silico approaches	Papers based on in vitro approaches	Total
Redundant or of minor relevance	42 (84%)	43 (75%)	301 (75%)	386
Relevant to the hypothesis or to the methods	8 (16%)	15 (25%)	100 (25%)	123
Total	50	58	401	509

MDD: major depressive disorder; NHP: non-human primate.

^a The relevance or redundancy of the cited paper to the hypothesis or methods of the citing MDD paper was evaluated by two independent raters.

Bold: total value.

The statistical test used to compare the proportions did not reveal any significant differences between the proportions of relevant citations between NHP–in vitro, NHP–in silico and in vitro–in silico (p = 0.31, 0.20 and 1, respectively).

Discussion

We quantitatively and qualitatively analysed the contribution of NHP, in vitro and in silico-based research approaches to the contemporary understanding of MDD. Of the three approaches analysed, NHP studies seemed to be the approach that was least likely to contribute to furthering progress in this field of human medical research. Of the three, the human-based in vitro approach seemed to influence human research to the greatest extent, judging by the number of citations. However, all three approaches seemed to be equally relevant in informing the hypothesis and/or methods of subsequent human research studies.

Overall, our results suggest that these less funded non-animal research approaches ³⁴ are more or equally effective than heavily invested animal-based research in reaching their final goal — which is to inform clinical research to improve human healthcare. Our quantitative results showed that in silico-based and in vitro-based approaches contributed more than NHP study-based approaches to human medical research, as the proportion of cited papers featuring the former two approaches was higher than the proportion of cited papers featuring the latter. NHP study-based papers were mainly cited by other papers on animal experimentation, which suggests that they are mainly contributing to subsequent animal research rather than to advances in human healthcare. In vitro studies seemed to be the most effective approach, since this approach received significantly more citations in total, and by human research papers either specifically focused on MDD or on other general medical areas.

Of the five analysed NHP study-based papers that were relevant to the citing human research papers on MDD in terms of their hypothesis, method or both, one featured both NHP and human research data. This paper was cited twice, and both citing papers referred to the human research data rather than to the NHP data. Another one of these five NHP papers was considered relevant to the methods and was cited once. The citing paper described both human and rhesus monkey data, and the citation was relevant to the methods used with the rhesus monkeys. After excluding these cases, only three out of the 19 NHP studies were relevant to the hypothesis and/or methods of the subsequent human research studies on MDD.

The results of our citation analysis also suggest that the widely accepted approach to testing medical hypotheses — which relies on in vitro-based and in silico-based research as a preliminary step prior to animal testing — is not actually working as intended, since clinical papers tend to cite in silico-based and in vitro-based papers directly too. However, citations of in silico-based and in vitro-based papers in subsequent publications on human clinical studies of MDD constituted a low percentage (50% or less) of the total citations received in all three analysed categories. This may be explained by the complexity of MDD, which shares certain genetic factors, phenotypic traits and possible neurologic pathways with a number of other disorders. Hence, a human study on anorexia might cite a non-clinical study on MDD focused on weight loss, since weight change is one of the symptoms of MDD.

As to the qualitative results, the judged relevance of the initially retrieved papers to the publications subsequently citing them was low for all three analysed research approaches. Even though a higher percentage of cited in silico-based and in vitro-based papers were relevant to the hypothesis and/or methods used by the citing clinical studies, the differences between the three approaches, in the extent of their judged relevance, were deemed insignificant. However, the size of the observed effect — where the proportion of citations of NHP-based papers was much lower than that of in silico-based or in vitro-based papers — suggests that, while not statistically significant (due to lack of statistical power), there might be a relevant practical difference.

Several important developments in in vitro technologies (e.g. organs-on-chips ³⁵ ) and in in silico technologies (e.g. advanced artificial intelligence based on sophisticated machine learning tools ³⁶ ) have been published since 2011. Such studies have been excluded from our analysis, in order to ensure that sufficient time is given to allow for subsequent citation of the resulting papers. However, it is reasonable to expect that these cutting-edge technologies are currently being widely used to generate and test new hypotheses in human medicine. ²⁷ Similarly, induced pluripotent stem cells, even though they have been worked on and developed for more than a decade, ³⁷ have only recently been recommended for MDD research. ³⁸ In light of the above, it would be interesting to repeat the current study a decade from now to investigate whether this has led to an increase in the number of subsequent citations of in vitro-based and in silico-based papers on MDD, in both MDD-focused and general human research publications.

We recognise that our study has certain limitations. Due to resource constraints, we were unable to use a greater number of search engines (e.g. CAB Abstracts). This would have increased the likelihood of retrieving all in silico-based, in vitro-based and NHP study-based papers on MDD, which would have increased our sample size and thus made it more comprehensive. Similarly, we were unable to examine the reference lists of many of the retrieved papers, in order to locate additional relevant papers. This inevitably means that some relevant publications might not have been identified. Because the sample size was small, our results should be interpreted with this caveat.

Finally, we are aware of the difficulty in objectively determining the relevance of a cited paper to the publication citing it. We used two different raters, in order to attempt to decrease any error in subjective assessment. Occasionally, the raters differed in their initial assessment, indicating that, even when the same criteria are used for assessment, differences can sometimes arise. However, our experience suggests that these differences would relate to only a small proportion of the papers assessed. Despite the limitations in the citation search and in the systematic qualitative analysis of the citation value, we consider that the method we followed is useful when evaluating the effectiveness of different research approaches. We hope that similar studies adopt this methodology, in order to investigate other medical disorders.

Our results suggest that the contribution of NHP studies to the current understanding of MDD is poor, and that other approaches with potentially superior relevance to humans should be used. Our results also shed light on the controversy around the efficacy of NHP-based research for investigating human disorders. This controversy is long-standing, with some authors claiming that their use is crucial for medical advancement, ²³ while others assert the opposite. ^39,40 However, ongoing scientific advances in non-animal methods for the acquisition of knowledge and the development of new treatments may provide future alternative solutions to help avoid the dilemmas and concerns surrounding NHP use.

Conclusions

To our knowledge, this is the first study to compare the effectiveness of original studies involving the use of NHP, in vitro and in silico research approaches to inform the medical research community within the MDD field. Our results suggest that, in this field of medical research, human-based in vitro and in silico research approaches are more promising than NHP studies, in generating new hypotheses and methods for subsequent clinical research.

Given the scientific advances in human-based research methods, we suggest that our methodology could be used in the future to analyse the impact of more recent technologies in informing human medical research. Such analysis could examine if and how the standard paradigm for testing medical hypotheses is still being followed, from applied research, through animal use in preclinical testing, and on to clinical research and development. It could also provide further insight into how the ‘gold standard’ that considers in vitro-based and in silico-based research approaches as merely preliminary steps prior to animal testing could be challenged and revised. Given the scientific and ethical solutions that innovative human-based approaches are providing, with relatively little investment when compared to the investments in animal-based research, a reallocation of resources is clearly warranted in favour of researching and developing the use of such approaches as part of human medical research.