Abstract
Before starting a new animal experiment, thorough analysis of previously performed experiments is essential from a scientific as well as from an ethical point of view. The method that is most suitable to carry out such a thorough analysis of the literature is a systematic review (SR). An essential first step in an SR is to search and find all potentially relevant studies. It is important to include all available evidence in an SR to minimize bias and reduce hampered interpretation of experimental outcomes. Despite the recent development of search filters to find animal studies in PubMed and EMBASE, searching for all available animal studies remains a challenge. Available guidelines from the clinical field cannot be copied directly to the situation within animal research, and although there are plenty of books and courses on searching the literature, there is no compact guide available to search and find relevant animal studies. Therefore, in order to facilitate a structured, thorough and transparent search for animal studies (in both preclinical and fundamental science), an easy-to-use, step-by-step guide was prepared and optimized using feedback from scientists in the field of animal experimentation. The step-by-step guide will assist scientists in performing a comprehensive literature search and, consequently, improve the scientific quality of the resulting review and prevent unnecessary animal use in the future.
Prior to setting up a new animal experiment, thorough analysis of previously performed experiments is paramount from both the scientific and ethical viewpoints. Through such analysis, the maximum amount of available information can be derived from previous work on the research topic and, as a result, unnecessary duplication of experiments can be prevented. Moreover, new insights that may arise from aggregating earlier work can be used to improve the design of future animal experiments. The method that is most suitable to carry out a thorough analysis of the literature is a systematic review (SR). 1 An SR implies that a review has been prepared using a systematic approach to minimize biases and random errors, and that the components of the approach are documented in a ‘Materials and methods’ section. 2
An essential first step in an SR is to retrieve all potentially relevant studies. 1 This requires adequate formulation of the research question, followed by a comprehensive search strategy. A comprehensive search strategy strongly underlies the quality of any literature review. 3 Not finding all relevant studies concerning a specific topic or omitting potentially relevant studies from a literature review may result in bias and, consequently, false or imprecise conclusions. When the omitted experiments are a random sample of all conducted experiments, only the precision of the outcome measure is decreased. However, when a particular selection of, for example, studies with non-significant or ‘negative’ results is not found and therefore not included, the SR will be biased towards a positive result. This so-called reporting bias may increase potential hazards involved in translating experimental outcomes to possible clinical benefits. 4–6 A systematic search may reduce the effect of reporting bias, but does not completely solve the problem. Some reporting bias and more specifically publication bias will remain and should be further investigated when analysing the results. A central register of performed and planned experiments as well as publishing the negative results of a research project might further diminish the problem of publication bias. 5
In contrast to the clinical field, where writing SRs and thus executing comprehensive search strategies are common practice, this is not the case within preclinical animal research.
7–8
This is surprising, since animal studies are often used as the foundation for clinical trials. One of the reasons may be that this approach of formally summarizing the results of previously performed experiments is not yet well known to many biomedical researchers designing animal experiments. Moreover, available guidelines from the clinical field cannot be copied directly to the situation within animal research. In addition, searching for and finding all the available literature on animal studies is not an easy task
9
, even though attempts have been made to make the search process easier. Hooijmans
To facilitate a structured, thorough and transparent search to identify all relevant animal studies (in both preclinical and fundamental science) concerning a specific research question, we developed an easy-to-use, step-by-step search guide which was optimized through the feedback from scientists in the field of animal experimentation. This step-by-step guide will assist scientists in performing a comprehensive literature search and, consequently, improve the scientific quality of the resulting review and prevent unnecessary animal use in the future.
Effective methodology and approach: the step-by-step guide
To prepare this step-by-step guide, we made use of the literature,
12–16
the
The basic steps on how to design and carry out comprehensive search strategies to identify potentially relevant animal studies on a specific research question are presented in Table 1 and explained more fully in the following sections.
Basic steps on how to design and carry out a comprehensive search strategy to identify potentially relevant animal studies on a specific research topic
Basic steps on how to design and carry out a comprehensive search strategy to identify potentially relevant animal studies on a specific research topic
1. Formulate research question
A focused, well-formulated research question is a prerequisite for designing an optimal search strategy.
Since the main components of the research question determine the structure and the sequence of the searches, it is important to carefully identify these components. Within animal research, a specific research question generally contains the following components: (1) intervention/exposure; (2) disease of interest/health problem; (3) animal/animal species/population studied; and (4) outcome measures. A well-formulated research question could then be: ‘What is the effect of [
To identify as many relevant papers as possible, searching in more than one database is recommended. An overview of health databases can be found at
Transform research question into search strategy
Once the research question is defined and the databases are selected, the search strategy can be prepared. Designing a search strategy for a specific database is a complex task. For building a comprehensive search strategy, one needs to be familiar with the specific search terms about the research topic and with the search engine's specific possibilities of searching. Literature searches can be difficult without a basic knowledge of the way information is organized and indexed. Basic and advanced courses in the use of major electronic bibliographic databases are often available at the institutional library. Taking such a course is strongly recommended. It is also advisable to involve an information specialist from the library for technical assistance when designing comprehensive search strategies. In Table 2, an overview of the detailed steps and an example of how to transform a research question into a search strategy in PubMed are given. These steps are more fully described and explained in Section 3.1. Section 3.2 describes how to transform a research question into a search strategy for other databases.
Detailed steps to transform research question into search strategy in PubMed
*Often scientists do not include SC4 (outcome measures) in the search strategy, because many abstracts do not contain a description of these outcome measures.
By including SC4 in the search strategy, there is a potential risk of missing relevant studies
Detailed steps to transform research question into search strategy in PubMed
*Often scientists do not include SC4 (outcome measures) in the search strategy, because many abstracts do not contain a description of these outcome measures.
By including SC4 in the search strategy, there is a potential risk of missing relevant studies
Split research question into critical search components
As mentioned in step 1 of Table 1, the research question often contains four components. These components can be translated into the following search components (SCs): (SC1) intervention/exposure; (SC2) disease of interest/health problem; (SC3) animal/animal species/population studied; and (SC4) outcome measures. Because, in many papers, outcome measures are only described in the main article and are rarely indexed, including SC4 (outcome measures) in a search strategy might increase the risk of missing relevant studies. SC4 (outcome measures) should therefore not always be included in the search strategy.
Identify relevant search terms for each SC
For each of the critical SCs, a separate search string needs to be developed. Each component-specific search string includes an extensive collection of appropriate search terms. The use of appropriate terminology is the cornerstone of an effective search. Using inappropriate terms may result in missing relevant studies or in too many irrelevant studies and a high number needed to screen/read. Search terms can be divided into standardized subject terms and free-text terms.
PubMed provides the MeSH database to explore relevant terminology and synonyms and to see the relationships between the terms. This can help to find the appropriate terminology.
For identification of MeSH terms and documentation of the search strategy, it can be practical to use a word processor, such as MS Word. The potentially relevant MeSH terms, including synonyms, e.g. entry terms, can be copied to the Word file. By studying the entry terms in the Word file, new relevant MeSH terms may be identified. The finally selected MeSH terms used in the search are then documented accordingly.
MeSH terms should be checked for appropriateness. The appropriateness of a single MeSH term can be evaluated by performing a PubMed search (add [MeSH] to search term in search bar) and assessing how the retrieved results cover the subject. When, on evaluation of the search results, it turns out that too many irrelevant studies are identified, or that the search resulted in too few hits, reformulation of the search terms is necessary.
To be complete, all possible relevant free-text terms need to be identified. It is practical to start with the MeSH terms and entry terms mentioned in the MeSH database of PubMed. Subsequently synonyms and related terms can be identified. Relevant articles (often reviews), Scopus, Google, etc. can be used for this purpose. Here, common text words may be identified as well as the index terms that the indexers have assigned to the articles. Free-text words should include all synonyms in singular and plural forms, UK and US English spelling, trade names, relevant abbreviations and trademarks. For example, when searching for ‘docosahexaenoic acid’ a frequently used abbreviation is DHA (Table 2, step B). Common spelling errors should also be covered; for example, ginko biloba for ginkgo biloba.
To economize the search string, truncation [adding * to a word stem] is often used in combination with free-text words. Truncation allows finding all the terms that begin with a given text string. For example, if you search statistic*, PubMed will simultaneously retrieve words such as statistic, statistics, statistical, etc. However, unwisely truncated terms may dramatically increase the number of unwanted results. The advice is to handle truncation with care and always check the search details for relevance.
To economize the search strategy, you can add [tiab] after a search term in PubMed and consequently your search will be restricted to the title and abstract of article citations and other less useful parts of the citation (such as author names and addresses) will not be searched. When [tw] is added after a search term, PubMed will search for that specific word in all subject-related fields in the article citation. This may result in irrelevant papers, for example, where the author's name is also the name of an intervention.
Free-text words should also be checked for appropriateness. The appropriateness of free-text terms can be evaluated by performing a PubMed search (add [tiab] or [tw] to the free-text terms in the search bar) and assess how the retrieved results cover the subject, how many hits there are and the context in the search history.
For identification of free-text terms and documentation of the search strategy, it can again be practical to use a word processor, such as MS Word. The finally selected free-text terms used in the search are documented as well.
Since the aim of the comprehensive search strategy is to retrieve as many relevant studies as possible, high sensitivity is necessary. Sensitivity, in this context, refers to a search that identifies as many research articles as possible that are potentially relevant. This may result in low precision and, consequently, in many hits. This is common to a comprehensive search strategy.
Repeat steps B1–B3 for SC2
After identifying relevant search terms for SC1, steps B1–B3 are repeated for SC2.
Combine search results for SC1 and SC2
After retrieving search results for SC1 and search results for SC2, they have to be combined into one search string in order to retrieve potentially relevant papers. Papers containing both SC1 and SC2 search results are then identified. The Boolean operator ‘AND’ can be used to combine search results for SC1 AND SC2 from the search history.
Evaluate search results
When developing a comprehensive search strategy, there needs to be a balance between comprehensiveness and precision. To be comprehensive, you need high sensitivity. As a result, the precision may be low and the number of hits may be high. This is not necessarily a problem since there is a third and maybe a fourth SC to combine the search result with to obtain more specific results and less hits.
Repeat steps B1–B3 for the remaining SCs (i.e. SC3 and occasionally SC4)
To prepare a search string for SC3 (animal/animal species/population studied) steps B1–B3 can be repeated. To identify all animal studies in PubMed, Hooijmans
As mentioned before, not all components of the research question lend themselves well to searching. For example, if SC4 is an outcome measure that is often not well described in the title or abstract of an article and is not well indexed with MeSH terms, then this component should not be included in the search strategy to prevent missing relevant studies.
Combine search results of all the separate SCs
In a final search, all separately developed search strings for each SC need to be combined to retrieve only the potentially relevant studies (Figure 1). This can be done by combining all the separate search results for each SC from the search history in PubMed using the Boolean operator ‘AND’. This results in final search string: SC1 AND SC2 AND SC3 (AND SC4). The final search string can be saved in a Word file and in PubMed for future use, to update the search results and to be able to report the exact search string in the review published. PubMed also has the functionality to automatically apply a specifically saved search string to most recently published or indexed articles. The reference details of the articles identified in this way will be sent to the researcher by email.
Combining components in the search strategy (adapted from Higgins and Green 3 )
Similar to step E, the final search results of search string SC1 AND SC2 AND SC3 (AND SC4) have to be evaluated for relevance. In case the search result SC1 AND SC2 AND SC3 (AND SC4) retrieves too many hits, the terminology in the search string can be adjusted and narrower terms can be selected. In order to be as complete as possible, it is quite usual to have 3000–5000 hits. However, one should not be set back by 8000–12,000 hits at this stage. In Section 5, we describe how to deal with these numbers of hits. When a search string is too narrow, low numbers of hits will be found (e.g. less than 300). Broader terms should be selected in order to be as complete as possible in a search. Developing search strategies is an iterative process in which the terms used are modified, based on what has already been retrieved. This means that based on the number of hits and their relevance, the search terms can be adjusted to being broader or narrower.
To restrict the number of hits in the search strategy, the use of limits in PubMed may seem attractive. However, limits should be used with caution to prevent missing potentially relevant studies and consequently introducing potential bias. From clinical literature reviews, for example, it is known that to prevent selection bias, there should be no restriction on the language of the identified studies, because ‘negative results’ are often published in non-English journals. 13 There should be a rationale for applying limits to a search strategy and it should be indicated in the text of the review.
Transfer search results into a reference manager program
The citations of the search result SC1 AND SC2 AND SC3 (AND SC4) are transferred and saved from PubMed into a reference manager program. Examples of such programs are: Endnote, Reference Manager, Procite and RefWorks. The saved citations in the reference program should at least include the reference details and the abstract.
Transform the research question into a search strategy for other databases
Besides through PubMed, other databases should be used to identify as many relevant papers as possible. The search strategy for PubMed described in Section 3.1 can be (slightly) adjusted and used to design a search strategy for other databases and search engines. One should be aware that different databases use different levels of keyword systems and different levels when indexing material, if any. Again it is advisable to involve an information specialist from the library when designing comprehensive search strategies, especially when you are not experienced in searching in specific databases.
EMBASE
A major biomedical database that also includes standardized subject terms is EMBASE. In EMBASE, the standardized subject terms are called EMTREE terms. A similar approach to find standardized subject terms and free-text words as used in Table 2 can be applied to prepare a search strategy in EMBASE. When searching on free-text terms in the search string of EMBASE in the Ovid interface ‘.ti,ab.’ is added to each search term. To find all animal studies in EMBASE, an animal search filter was developed by De Vries
Other databases
Besides MEDLINE and EMBASE, other databases could be considered. As stated before, for each database, the search-engine-specific search terminology needs to be found and applied. Not all databases use standardized subject terms. If a database does not use standardized subject terms, careful selection of free-text terms is very important. In many cases, a well-developed PubMed or EMBASE search strategy can be ‘translated’ for such databases by copying the search string per SC and stripping the field indicators, such as [MeSH], [tiab] and [tw]. Examples of databases not using standardized subject terms are Web of Science and Scopus.
Collect search results of all databases and remove duplicate citations
After developing a search strategy for each database and collecting the citations from the separate databases, the search results need to be combined into one file in the reference program. Citations that appear more than once can then be removed. Duplicate citations often appear because databases can encompass the same journals. After removing the duplicate citations, a list of potentially relevant papers is created.
Identify potentially relevant papers
The list of the potentially relevant papers needs to be screened for relevance of the studies with regard to the research question. The first screening can be done very quickly using only the title and abstract of the studies. To prevent bias in the selection process, it is very useful to have two observers screening the papers independently for relevance. The criteria used for the first screening are based on the SCs. For example, when the citation clearly does not describe an animal study, is not a primary study or is not studying the disease of interest, these citations can be removed since these are not relevant to the research question. Only clearly irrelevant citations should be removed (i.e. one should be over inclusive to prevent incorrect removal at this stage).
The citations resulting from the first screening undergo a second screening when writing an SR. The second screening is based on predefined in- and exclusion criteria and is also performed by two independent reviewers. Throughout the entire screening process for potentially relevant papers the reasons for removal of citations should be documented and reported to facilitate transparency and to make it possible for an interested reader to independently examine accuracy of study removal (Figure 2).
A fictive example of reporting on search results and reasons for exclusion of studies
Documenting and reporting the search process
The search processes should be documented in enough detail to ensure correct reporting in the (systematic) review and for the reproducibility of the searches in all databases used. Detailed documenting of the search process is also important to keep track of identified studies and to be able to update the search at a later state. Reporting criteria for the search process in SRs in human research are available. 15 In short, to be systematic, explicit and transparent, the scientist should always report: (1) all databases and other sources searched; (2) the dates of the last search for each database and the period searched; (3) full search strategies (including all search terms) for each database; and (4) any language or publication status restrictions used. An example of reporting on search results and reasons of exclusion of studies is given in Figure 2.
Concluding remarks
A comprehensive search strategy is a key element to ensure the quality of a (systematic) review and the validity of its findings. 3 Searching for pertinent literature on animal studies can be very difficult. 9 To facilitate a structured, thorough and transparent search process to identify potentially relevant animal studies concerning a specific research question (in both preclinical and fundamental science), we developed an easy-to-use, step-by-step search guide. The practical guide presented in this paper is meant to assist and stimulate scientists in performing a comprehensive literature search and adequately document and report on the search process. Transparency on the search process and adequate reporting makes it possible for others reading the review to judge the thoroughness of the search, and thereby the potential of bias in the review. This practical guide cannot be used to identify unpublished studies. Leaving out unpublished studies may result in an overestimation of the effect size because unpublished studies often contain negative data. 5 It should be noted that this step-by-step guide should be used with caution, because this guide is not exhaustive. It is highly recommended to always involve an information specialist for assistance when designing comprehensive search strategies.
Footnotes
Acknowledgements
This study is financially supported by a research grant from ZonMw programme ‘Dierproeven begrensd’ in The Netherlands (ZonMw is the Netherlands Organisation for Health Research and Development) (Grant no. 40-40100-96-8002) and the Executive Board of Radboud University Nijmegen. We thank the participants in our test and evaluation panel for their critical testing of and helpful comments on the step-by-step guide. We also thank Elmie Peters from the Medical Library of the Radboud University Nijmegen Medical Centre for critically reading the manuscript.