Sage Journals: Discover world-class research

Abstract

Over the past few decades, there has been major development in methods for evidence synthesis, which can lead to confusion as to which approaches to use and why. Several strategies can be used in systematic review approaches to reduce potential biases and errors. These strategies can be considered on a spectrum ranging from least to most likely to minimize biases and errors in the review process. Building on the existing literature of synthesis methods and biases, a five-level spectrum of systematicity in reviews is proposed in this paper. For each of the main steps of the review process (i.e. search, selection, data extraction, appraisal, and synthesis), potential biases are presented. Then, strategies are suggested and ordered based on their influence on potential biases and errors in the review process. The levels of systematicity suggested can help to distinguish the reviews based on their rigour. This paper can contribute to improving understanding of the variety of strategies that can be used at the different steps of a review process. This can be particularly useful for students and novice researchers seeking to understand the potential sources of bias and to choose suitable strategies for their review.

Keywords

systematic reviews risk of bias errors systematicity evidence synthesis

Background

Planning and conducting a systematic review can be challenging, especially for students and novice researchers. Indeed, the science of evidence synthesis has greatly evolved over the past few decades to take into account different perspectives and purposes in order to better inform decision-making (Aromataris, 2020; Gough & Thomas, 2016; Hong & Pluye, 2018). There currently exist a variety of types of reviews and synthesis methods (Moher et al., 2015). For example, Sutton et al. (2019) identified 15 review typologies and found a total of 48 different review types that can be categorized into seven families. This wide variety of types and methods can lead to confusion about which approaches to use and why. Their choice arises from the research question posed and the intended use of the review’s findings (Gough et al., 2019). Beside from typologies, there are available resources to help researchers decide which systematic review approach is right for them (e.g. Amog et al., 2022).

Within the different review types, there are steps that are commonly executed (e.g. searching, screening, data extraction, assessing, synthesizing, interpretating, and reporting) (Booth et al., 2021; Gough et al., 2017). Yet, the steps can be carried out in different ways (e.g. with one or two reviewers) for any review type, and these reflect recent consensus development and guidance on ‘rapid approaches’ to evidence synthesis (Garritty et al., 2024). The variation in the ways that each review step can be carried out reflects its systematicity. Systematicity has been defined as ‘a disposition towards organized, methodic, and orderly inquiry that uses various methods and processes to search, screen, assess, analyse, and interpret relevant information with a view to achieving a set of specific research goals’ (Paré et al., 2016, p. 596). Systematic review approaches include elements of systematicity that can reduce biases and errors and enable reliable inferences (Bird, 2019; Booth et al., 2016; Moher et al., 2015). These elements can be considered on a spectrum ranging from least to most likely to minimize biases and errors in the review process, which can refer to the ‘degree of systematization’ (Schick-Makaroff et al., 2016) or ‘level of systematicity’ (Booth et al., 2012; Paré et al., 2016). Understanding the potential risk of biases and errors for each variation at each stage is important to find the balance between available resources, quality of execution, and appropriateness of methods (Gough, 2021).

This paper builds on the existing literature of synthesis methods and biases (Buhn et al., 2017; Garritty et al., 2021; Shea et al., 2017; Tricco et al., 2017; Whiting et al., 2016) and the concept of systematicity (Bird, 2019; Booth et al., 2016; Paré et al., 2016) to better understand the risk of error for each variation. It proposes a spectrum of levels of systematicity in systematic review approaches. In the following sections, biases are discussed, and strategies are ordered based on their influence on biases and errors for each step of the review process. This can contribute to improving understanding of the variety of ways that can be carried out at the different steps of a review process. This can be particularly useful for students and novice researchers seeking to understand the potential sources of bias and choose the suitable strategies for their review.

Levels of Systematicity at Each Step of the Review Process

In general, biases and errors in systematic review approaches pertain to either the studies included in the review or the way the review is carried out (Gough & Thomas, 2016). Biases and errors can be found at different steps of the review process (i.e. search the literature, selection of relevant papers, data extraction, appraisal, and synthesis). In the following, some potential biases and errors (Table 1) at each step are presented and different levels of systematicity are suggested (Table 2).

Table 1.

Main Types of Bias.

Bias	Definition
Identification bias	This bias occurs when relevant studies on the topic of interest are not identified from the literature search.
Dissemination bias (or reporting bias)	This bias occurs when the nature, direction or strength of findings of a study influence its publication. Several forms can be found such as publication bias, selective outcome reporting bias, time-lag bias, language bias, multiple publication bias, citation bias, and media-attention bias.
Selection bias	This bias occurs when the included studies are not representative of the full range of evidence available.
Interpretation bias	This bias occurs when reviewers (mis)interpret results in a way that reflects their preconceptions, beliefs, or expectations, rather than being based on the data.

Table 2

Levels of Systematicity in Reviews

Review steps	(higher risk of bias)				(lower risk of bias)
Search	No source of information, no search (e.g. references known to the author)	One or several databases searched with general terms	One or several databases searched with a comprehensive search strategy	One or several databases searched with a comprehensive search strategy + some other sources of information	Several databases searched with a comprehensive search strategy + several other sources of information
Selection	No or unclear selection criteria	Emerging selection criteria	Predefined selection criteria	Predefined and pilot-tested selection criteria (with more restriction)	Predefined and pilot-tested selection criteria (with less restriction)
Selection	Single reviewer screens titles/abstracts and full-text papers	Single reviewer with verification by a 2nd reviewer of all excluded titles/abstracts and full-text papers, or Single reviewer with verification by 2nd reviewer of a random number of screened titles/abstracts and full-text papers	Single reviewer with verification by 2nd reviewer of all screened titles/abstracts and full-text papers	Dual reviewers independently screen a random number of titles/abstracts and full-text papers	Dual reviewers independently screen all titles/abstracts and full-text papers
Data extraction	No data extraction form	Unstructured form	Structured form	Structured and pilot-tested form (limited data)	Structured and pilot-tested form (complete data)
Data extraction	Single reviewer extracts data of included studies	Single reviewer with verification by a 2nd reviewer of data extracted of a random number of included studies	Single reviewer with verification by a 2nd reviewer of data extracted of all included studies	Dual reviewers independently extract data from a random number of included studies	Dual reviewers independently extract data from all included studies
Appraisal	No appraisal	Limitations reported in included studies	Informal quality appraisal	Formal quality appraisal using appropriate tools (not validated)	Formal quality appraisal using appropriate tools (validated)
Appraisal	Single reviewer appraises included studies	Single reviewer with verification by a 2nd reviewer of the appraisal of a random number of included studies	Single reviewer with verification by a 2nd reviewer of the appraisal of all included studies	Dual reviewers independently appraise a random number of included studies	Dual reviewers independently appraise all included studies
Synthesis	No synthesis	Single reviewer performs synthesis	Several reviewers involved in the synthesis	Several reviewers involved in the synthesis including methodological or content experts	Several reviewers involved in the synthesis including methodological and content experts

Step 1: Searching the literature

Two main biases can be found during the step of searching relevant studies. First, identification bias occurs when relevant studies on the topic of interest are not identified from the literature search. This bias is particularly important in some types of reviews where the missed relevant papers can greatly influence the findings such as systematic reviews that are used to make potentially life-threatening decisions. Second, reviews are influenced by dissemination bias (or reporting bias). This bias occurs when the nature, direction, or strength of findings of a study influence its publication (Song et al., 2013). It can include different forms of biases related with the dissemination process. For example, it has been shown that studies with positive results are more likely to be published (publication bias), report only outcomes that were significant (selective outcome reporting bias), be published more rapidly (time-lag bias), be published in English (language bias), be published more than once (multiple publication bias), be more frequently cited (citation bias), and be more likely to be covered by the media (media-attention bias) (Gough et al., 2017; Song et al., 2010). There is thus a risk of misleading conclusions. For example, it was found that the effect of psychological interventions for depression was overestimated when relying solely on published studies (Driessen et al., 2015).

Various strategies representing different levels of systematicity can be used to reduce identification bias that can range from no search strategy (thus, more susceptible to bias) to a search in several bibliographic databases and other sources of information (including the grey literature) (Table 2). First, it is necessary to develop a comprehensive search strategy with the help of an information specialist that can maximize recall and reduce search errors (McGowan & Sampson, 2005). In systematic reviews, the search strategy should be peer reviewed to identify search errors and improve the selection of search terms (McGowan et al., 2016).

Second, the use of multiple electronic bibliographic databases can allow to reduce the risk of identification bias. In systematic reviews, it is usually recommended to search a minimum of two databases (Garritty et al., 2021). Also, the choice of databases needs to be justified since there can be considerable overlap between some databases, and additional databases might not significantly minimize identification bias (Hirt et al., 2021). For example, Aagaard et al. (2016) found that searching in 10 additional databases (AMED, CINAHL, HealthSTAR, MANTIS, OT-Seeker, PEDro, PsycINFO, SCOPUS, SportDISCUS, and Web of Science) only increased the median recall by 2% when compared to searching in only three databases (MEDLINE, EMBASE, and CENTRAL). Also, Halladay et al. (2015) found that 84% of all papers included in 50 randomly sampled Cochrane reviews were indexed in PubMed. They concluded that the impact of using multiple databases beyond PubMed is modest for reviews on therapeutic interventions.

Third, using solely electronic bibliographic databases might not be enough to minimize identification bias. It is recommended to use several other sources of information (Aagaard et al., 2016). In addition to bibliographic databases, other search sources can be used such as follows: 1) search engines (e.g. Google, Google scholar, and Microsoft Academic), 2) specific websites (e.g. clinical trial registries and organizational websites), 3) databases for grey literature (e.g. ProQuest Dissertation and Thesis Global), 4) hand searching in specialized journals and books, 5) contact experts on the topic being reviewed to obtain additional references, and 6) backward and forward citation tracking (Newman & Gough, 2020). Using a variety of sources can help to increase the chance of identifying relevant articles. For example, Greenhalgh and Peacock (2005) found that only 25% of the 495 papers included in their review were identified from electronic database search. The other articles were found through tracking references of references (44%), personal knowledge (17%), citation tracking (7%), personal contacts (6%), and hand searching of key journals (5%).

Fourth, to address dissemination bias and since about half of completed research is published (Song et al., 2013), it is suggested to search for unpublished studies (Adams et al., 2016). Unpublished literature can be found with other sources such as databases of grey literature, relevant websites on the topic of the review, internet search engines, and trials registries (Adams et al., 2016; Song et al., 2013).

Step 2: Selecting relevant documents

One type of selection bias is reviewers’ voluntarily inclusion or exclusion of studies to support their position (Booth et al., 2016). This is similar to the concept of cherry-picking arising from confirmation bias, that is, choosing evidence confirming a position and rejecting evidence that contradicts that position (Mizrahi, 2015). A second type is random selection error that can occur due to ambiguous selection criteria or reviewers’ prior knowledge and understanding of the topic of interest (McDonagh et al., 2013). To minimize selection bias, strategies are based on the selection criteria and the number of reviewers involved in at this step (Table 2).

Regarding the selection criteria, it is recommended to define clear and unambiguous predetermined selection criteria so that reviewers have a common understanding of what publications need to be included and excluded. This ensures the selection process is transparent and consistent across publications (Newman & Gough, 2020). Also, it is suggested to pilot test the criteria on a small sample of titles and abstracts with all the members of the screening team. This strategy will help to calibrate the criteria before applying them to all the titles and abstracts (Garritty et al., 2021). Moreover, reviewers should be cautious about having criteria that are too restrictive criteria such as excluding papers based on reporting of outcomes, languages other than English, and place of publication (Hartling et al., 2017; McDonagh et al., 2013).

Regarding the number of reviewers involved in the selection step, it has been suggested that selection bias can be reduced with dual review. Several studies have compared single and double screening. For example, Gartlehner et al. (2020) found that single abstract screening missed 13% of relevant studies while dual screening missed only 3%. Similarly, a study comparing single to double independent screening of titles and abstracts found that the median proportion of missing studies in single screening was 5% (with a range from 0 to 57.8%) (Waffenschmidt et al., 2019). Moreover, this study found that the results were highly influenced by the level of experience of reviewers; the impact of missed studies on the findings of meta-analysis changed substantially from the screening conducted by reviewers with less experience while the impact was negligible for screening from experienced reviewers (Waffenschmidt et al., 2019). Moreover, different strategies can be used for the screening of titles and abstracts, and for full-text selection. For example, Stoll et al. (2019) compared complete (two independent reviewers for titles/abstracts and for full-text papers) and limited dual review (one reviewer for titles/abstracts and two independent reviewers for full-text papers). They found that the complete dual review increased the number of relevant studies by identifying more mistakenly excluded papers (0.4% compared with 0.2% in the limited dual review).

Step 3: Extracting data

Biases and errors during this step can be found when incomplete and wrong data are extracted. This can be due to misinterpretation of the information provided in the studies, omission of important data to extract, and errors when extracting the number of patients, means, standard deviations, and effect sizes (Gøtzsche et al., 2007; Li et al., 2019; Mathes et al., 2017). The frequency of data extraction errors is variable in systematic reviews. For example, a methodological review of six studies on data frequency errors found prevalence rates ranging from 8 to 70%, depending of outcomes and reviews (Mathes et al., 2017). Yet, these studies report that data extraction errors have low to moderate impact on the findings and conclusions from a review (Buscemi et al., 2006; Mathes et al., 2017).

Using a structured form can help to ensure accuracy and consistency in the process by providing information on what data to extract and how to code them (Table 2). It is recommended to pilot test the data extraction form before using it on all included studies (Li et al., 2019). This pilot testing is done by comparing the data independently extracted by several reviewers on a small number of studies. This can help identify unclear instructions as well as ambiguous, missing, or superfluous data (Li et al., 2019).

The number of reviewers can also influence the data extraction error rate (Table 2). It was shown that dual data extraction results in fewer errors than single data extraction or verification by a second reviewer (Mathes et al., 2017). When dual data extraction is not possible, verification by a second reviewer of data extracted of a percentage or all studies could help identify errors compared to single data extraction. It is usually advised to use more rigorous data extraction strategies for information that involves subjective interpretation as well as information essential to the interpretation of results such as outcome data used in the synthesis (Li et al., 2019).

Step 4: Appraising the quality of included documents

The appraisal of studies can be open to interpretation bias and is influenced by several factors (Deeks et al., 2003). For example, studies have shown that reviewers’ judgement of the quality of a study can be influenced by some characteristics of studies such as the author’s name and affiliation, the journal where the study was published, and the study results, which can lead to inconsistent assessments within and between reviewers (Morissette et al., 2011). Also, reviewers’ experience with quality appraisal and their methodological expertise are other factors that can influence their interpretation of the quality of a study (Dixon-Woods et al., 2007). Moreover, conflicts of interest can influence the appraisal. For example, a study found that systematic reviews with overlapping authors (i.e. authors of an overview that were also authors of some included systematic reviews) were rated of higher quality (Pieper et al., 2018).

To minimize interpretation bias, it has been recommended to conduct a formal appraisal using structured tools to make the process more transparent and explicit (Deeks et al., 2003; Whiting et al., 2017). A large number of risk of bias and critical appraisal tools have been developed over the past decades, which make it challenging to choose the most appropriate tools to use (Munthe-Kaas et al., 2019; Quigley et al., 2019; Whiting et al., 2017). Whenever possible, it is suggested to use a valid appraisal tool specific to the design of the studies included in the review (Garritty et al., 2021). Some online resources are available to help identify and choose appropriate validity assessment tools (e.g. https://www.latitudes-network.org/) (Whiting et al., 2024) or critical appraisal tools (e.g. https://www.catevaluation.ca/index.php/en/) (Hong et al., 2022).

In addition, involving two reviewers working independently is usually recommended (Whiting et al., 2016). When dual appraisal is not possible, having a second reviewer check the accuracy of the assessment performed by another reviewer for all or a sample of studies can be more rigorous than single review (Garritty et al., 2021).

Other strategies have been suggested such as performing quality appraisal under blinded conditions (e.g. removing authors’ and journal names). However, there is no consensus on this strategy since inconsistent results were observed between studies comparing blinded and unblinded assessment. For example, two studies found lower quality scores for blinded assessment, one found higher quality scores for blinded assessment, and three studies did not find any difference between blinded and unblinded assessment (Morissette et al., 2011).

Step 5: Synthesizing data

As seen in the previous steps, interpretation bias can also occur during the synthesis such as the risk of over-interpretation of study data and misinterpretation of findings (Booth et al., 2016). Although several publications have highlighted that misinterpretation of data is problematic and can influence the conclusion of a review (Bown & Sutton, 2010), to our knowledge, none has compared strategies to minimize this bias at this step.

To limit interpretation bias, a strategy is to involve several reviewers with the necessary methodological and content expertise to address the review questions during the synthesis step (Table 2). For example, when a meta-analysis is performed, it is usually advised to consult a statistician who can help to understand the data, make sure appropriate methods are used, deal with missing data, investigate heterogeneity, and interpret the findings of the synthesis (Deeks et al., 2019). Also, the findings from the synthesis can be checked by content experts as well as other parties such as practitioners and patients (Thomas et al., 2017).

Discussion and Conclusion

This paper focused on the concept of systematicity in review approaches and suggests different levels. It provides a range of strategies ordered based on their influence of reducing biases and errors (from low to high risk of bias) (Table 2). Different levels can be used at each step of a review. For example, Figures 1(a) and 1(b) illustrate the two opposite ends where all the levels are either the lowest (Figure 1(a)) or highest (Figure 1(b)). Within a review, researchers could also decide to put more rigour emphasis on some steps such as on the search, selection, and extraction (e.g. Figure 1(c)), or on the appraisal and synthesis (e.g. Figure 1(d)).

Figure 1.

Illustrations of Different Levels of Systematicity Used at Each Step of a Review.

During the planning stage of a review, different methodological decisions need to be made based on the aim of the review, needs of stakeholders, time, and resources available (e.g. cost, size, and expertise of the review team) (Garritty et al., 2021; Mathes et al., 2017; O’Hearn et al., 2021). While there is no clear guidance on the choice of level, some authors have proposed avenues to explore. For example, Rowe (2014) suggested that the level of systematicity should be higher for theoretical explanations and testing compared to theoretical development. Also, higher levels of systematicity may be needed in systematic reviews making health recommendations, especially when the recommendations have life-threatening consequences. Conversely, in reviews aiming at theoretical understanding a phenomenon, providing an overview of what has been done, or make recommendations for future research, it may not always be necessary to adopt the highest levels of systematicity. More empirical methodological research to support these ideas are warranted.

There is currently a variety of types and typologies of reviews that differ based on several dimensions such as the goals, topics, data types, coverage, and methods (Belaid & Ridde, 2020; Cooper, 1988; Grant & Booth, 2009; Kastner et al., 2012; Littell, 2018; MacEntee, 2019; Munn et al., 2018; Paré et al., 2015; Schryen et al., 2015; Sutton et al., 2019). This diversity shows that there is no one-size-fits-all approach to review the literature. The levels of systematicity suggested in this paper can be applied to different types of reviews since they focus on the strategies used within each step. They could help to distinguish the reviews based on their rigour.

Another implication of the levels of systematicity is for the appraisal of reviews. Several tools have been developed to appraise the quality of systematic review approaches. However, the available tools were mainly developed to assess the quality of a specific type of systematic review approaches such as realist and meta-narrative reviews (Wong et al., 2014), mixed methods systematic reviews (Jimenez et al., 2018), systematic reviews of randomized and non-randomized studies of healthcare interventions (Shea et al., 2017), and systematic reviews of interventions, diagnosis, prognosis, and aetiology (Whiting et al., 2016). More recently, a critical appraisal tool to assess the quality of different types of reviews for health promotion and prevention was developed (Heise et al., 2022). Several strategies listed in Table 2 can be found in the available critical appraisal tools for systematic review approaches.

This paper used existing literature, especially from rapid reviews (Garritty et al., 2021, 2024; Klerings et al., 2023; Nussbaumer-Streit et al., 2023; Tricco et al., 2017), to plot the review strategies on a continuum of levels of systematicity and provided an explanation based on their potential to reduce biases and errors. It is hoped that it will help people new to systematic reviews better understand the strategies and help them choose appropriate methods. Future research could refine and validate these levels. More comparative studies are needed to understand how the different levels can influence the trustworthiness of a review, what criteria to use to identify a level, whether different combinations of variations cause more or less biases, and how the levels should be adapted, especially for reviews using other synthesis logic such as qualitative evidence synthesis. In particular, there is also a need to show that ‘systematic’ review strategies can reduce bias (Greenhalgh et al., 2018), and to better understand the impact of biases on the findings of a review and on decision-making. Finally, the levels of systematicity will need to be adapted as research on systematic reviews approaches and tools evolve, such as the use of automation technologies to assist the review process (O’Connor et al., 2020).

Statements and Declarations

Footnotes

Acknowledgements

The authors would like to thank Dr. Paula Bush for providing constructive suggestions on a previous version of the manuscript.

Conflicting interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: QNH is supported by a Junior 1 Research Scholar Award from the Fonds de recherche du Québec - Santé (FRQS).

ORCID iDs

Quan Nha Hong

Ginny Brunton

References

Aagaard

Lund

Juhl

(2016). Optimizing literature search in systematic reviews–are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Medical Research Methodology, 16(1), 1–11. https://doi.org/10.1186/s12874-016-0264-6

Adams

Hillier-Brown

F. C.

Moore

H. J.

Lake

A. A.

Araujo-Soares

White

Summerbell

(2016). Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: Critical reflections on three case studies. Systematic Reviews, 5(1), 164. https://doi.org/10.1186/s13643-016-0337-y

Amog

Courvoisier

Mak

Booth

Godfrey

Hwee

Straus

S. E.

Tricco

A. C.

(2022). The web-based “Right Review” tool asks reviewers simple questions to suggest methods from 41 knowledge synthesis methods. Journal of Clinical Epidemiology, 147(1), 42–51. https://doi.org/10.1016/j.jclinepi.2022.03.004

Aromataris

(2020). Furthering the science of evidence synthesis with a mix of methods. JBI Evidence Synthesis, 18(10), 2106–2107. https://doi.org/10.11124/JBIES-20-00369

Belaid

Ridde

(2020). Une cartographie de quelques méthodes de revues systématiques. Centre Population et Développement. https://www.ceped.org/IMG/pdf/wp44.pdf

Bird

(2019). Systematicity, knowledge, and bias. How systematicity made clinical medicine a science. Synthese, 196(3), 863–879. https://doi.org/10.1007/s11229-017-1342-y

Booth

Papaioannou

Sutton

(2012). Systematic approaches to a successful literature review. Sage Publications.

Booth

Sutton

Clowes

Martyn-St James

(2021). Systematic approaches to a successful literature review (3rd ed.). Sage Publications.

Booth

Sutton

Papaioannou

(2016). Systematic approaches to a successful literature review (2nd ed.). Sage Publications.

10.

Bown

Sutton

(2010). Quality control in systematic reviews and meta-analyses. European Journal of Vascular and Endovascular Surgery, 40(5), 669-677. https://doi.org/10.1016/j.ejvs.2010.07.011. https://www.ejves.com/article/S1078-5884(10)00438-7/pdf

11.

Buhn

Mathes

Prengel

Wegewitz

Ostermann

Robens

Pieper

(2017). The risk of bias in systematic reviews tool showed fair reliability and good construct validity. Journal of Clinical Epidemiology, 91, 121–128. https://doi.org/10.1016/j.jclinepi.2017.06.019

12.

Buscemi

Hartling

Vandermeer

Tjosvold

Klassen

T. P.

(2006). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59(7), 697–703. https://doi.org/10.1016/j.jclinepi.2005.11.010

13.

Cooper

H. M.

(1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society, 1(1), 104–126. https://doi.org/10.1007/bf03177550

14.

Deeks

J. J.

Dinnes

D’Amico

Sowden

A. J.

Sakarovitch

Song

Petticrew

Altman

D. G.

International Stroke Trial Collaborative Group, & European Carotid Surgery Trial Collaborative Group . (2003). Evaluating non-randomised intervention studies. Health Technology Assessment, 7(27), 1–173. https://doi.org/10.3310/hta7270

15.

Deeks

J. J.

Higgins

J. P. T.

Altman

D. G.

Cochrane S, & Tatistical Methods Group . (2019). Analysing data and undertaking meta‐analyses. In Higgins

J. P. T.

Thomas

Chandler

Cumpston

Page

M. J.

Welch

V. A.

(Eds.), Cochrane handbook for systematic reviews of interventions (pp. 241–284). John Wiley & Sons.

16.

Dixon-Woods

Sutton

Shaw

Miller

Smith

Young

Bonas

Booth

Jones

(2007). Appraising qualitative research for inclusion in systematic reviews: A quantitative and qualitative comparison of three methods. Journal of Health Services Research and Policy, 12(1), 42–47. https://doi.org/10.1258/135581907779497486

17.

Driessen

Hollon

S. D.

Bockting

C. L.

Cuijpers

Turner

E. H.

(2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS One, 10(9), Article e0137864. https://doi.org/10.1371/journal.pone.0137864

18.

Garritty

Gartlehner

Nussbaumer-Streit

King

V. J.

Hamel

Kamel

Affengruber

Stevens

(2021). Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. Journal of Clinical Epidemiology, 130, 13–22. https://doi.org/10.1016/j.jclinepi.2020.10.007

19.

Garritty

Hamel

Trivella

Gartlehner

Nussbaumer-Streit

Devane

Kamel

Griebler

King

V. J.

Cochrane Rapid Reviews Methods Group . (2024). Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness. British Medical Journal, 384, Article e076335. https://doi.org/10.1136/bmj-2023-076335

20.

Gartlehner

Affengruber

Titscher

Noel-Storr

Dooley

Ballarini

König

(2020). Single-reviewer abstract screening missed 13 percent of relevant studies: A crowd-based, randomized controlled trial. Journal of Clinical Epidemiology, 121, 20–28. https://doi.org/10.1016/j.jclinepi.2020.01.005

21.

Gøtzsche

P. C.

Hróbjartsson

Marić

Tendal

(2007). Data extraction errors in meta-analyses that use standardized mean differences. Journal of the American Medical Association, 298(4), 430–437. https://doi.org/10.1001/jama.298.4.430

22.

Gough

(2021). Appraising evidence claims. Review of Research in Education, 45(1), 1–26. https://doi.org/10.3102/0091732x20985072

23.

Gough

Oliver

Thomas

(2017). An introduction to systematic reviews (2nd ed.). Sage Publications.

24.

Gough

Thomas

(2016). Systematic reviews of research in education: Aims, myths and multiple methods. The Review of Education, 4(1), 84-102. https://doi.org/10.1002/rev3.3068. https://onlinelibrary.wiley.com/doi/pdf/10.1002/rev3.3068

25.

Gough

Thomas

Oliver

(2019). Clarifying differences between reviews within evidence ecosystems. Systematic Reviews, 8(1), 170. https://doi.org/10.1186/s13643-019-1089-2

26.

Grant

M. J.

Booth

(2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

27.

Greenhalgh

Peacock

(2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. British Medical Journal, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68

28.

Greenhalgh

Thorne

Malterud

(2018). Time to challenge the spurious hierarchy of systematic over narrative reviews? European Journal of Clinical Investigation, 48(6), Article e12931. https://doi.org/10.1111/eci.12931

29.

Halladay

C. W.

Trikalinos

T. A.

Schmid

I. T.

Schmid

C. H.

Dahabreh

I. J.

(2015). Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. Journal of Clinical Epidemiology, 68(9), 1076–1084. https://doi.org/10.1016/j.jclinepi.2014.12.017

30.

Hartling

Featherstone

Nuspl

Shave

Dryden

D. M.

Vandermeer

(2017). Grey literature in systematic reviews: A cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology, 17(64), 1–11. https://doi.org/10.1186/s12874-017-0347-z

31.

Heise

T. L.

Seidler

Girbig

Freiberg

Alayli

Fischer

Haß

Zeeb

(2022). CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Medical Research Methodology, 22(1), 1–10. https://doi.org/10.1186/s12874-022-01821-4

32.

Hirt

Bergmann

Karrer

(2021). Overlaps of multiple database retrieval and citation tracking in dementia care research: A methodological study. Journal of the Medical Library Association, 109(2), 275–285. https://doi.org/10.5195/jmla.2021.1129

33.

Hong

Q. N.

Bouix-Picasso

Ruchon

(2022). Creation of an online inventory for choosing critical appraisal tools. Education for Information, 38(2), 205–210. https://doi.org/10.3233/Efi-211567

34.

Hong

Q. N.

Pluye

(2018). Systematic reviews: A brief historical overview. Education for Information, 34(4), 261–276. https://doi.org/10.3233/Efi-180219

35.

Jimenez

Waddington

Goel

Prost

Pullin

White

Lahiri

Narain

Bhatia

(2018). Mixing and matching: Using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400–421. https://doi.org/10.1080/19439342.2018.1534875

36.

Kastner

Tricco

A. C.

Soobiah

Lillie

Perrier

Horsley

Welch

Cogo

Antony

Straus

S. E.

(2012). What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Medical Research Methodology, 12(1), 114. https://doi.org/10.1186/1471-2288-12-114

37.

Klerings

Robalino

Booth

Escobar-Liquitay

C. M.

Sommer

Gartlehner

Devane

Waffenschmidt

Cochrane Rapid Reviews Methods Group . (2023). Rapid reviews methods series: Guidance on literature search. BMJ Evidence-Based Medicine, 28(6), 412–417. https://doi.org/10.1136/bmjebm-2022-112079

38.

Higgins

J. P.

Deeks

J. J.

(2019). Collecting data. In Higgins

J. P. T.

Chandler

Cumpston

Page

M. J.

Welch

V. A.

(Eds.), Cochrane handbook for systematic reviews of interventions (2nd ed., pp. 109–141). John Wiley & Sons. https://training.cochrane.org/handbook/current/chapter-05

39.

Littell

(2018). Conceptual and practical classification of research reviews and other evidence synthesis. The Campbell Collaboration. https://doi.org/10.4073/cmdp.2018.1

40.

MacEntee

M. I.

(2019). A typology of systematic reviews for synthesising evidence on health care. Gerodontology, 36(4), 303–312. https://doi.org/10.1111/ger.12439

41.

Mathes

Klaßen

Pieper

(2017). Frequency of data extraction errors and methods to increase data extraction quality: A methodological review. BMC Medical Research Methodology, 17(1), 152. https://doi.org/10.1186/s12874-017-0431-4

42.

McDonagh

Peterson

Raina

Chang

Shekelle

(2013). Avoiding bias in selecting studies. Agency for Healthcare Research and Quality. https://www.ncbi.nlm.nih.gov/books/NBK126701/

43.

McGowan

Sampson

(2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association, 93(1), 74–80.

44.

McGowan

Sampson

Salzwedel

D. M.

Cogo

Foerster

Lefebvre

(2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40–46. https://doi.org/10.1016/j.jclinepi.2016.01.021

45.

Mizrahi

(2015). Historical inductions: New cherries, same old cherry-picking. International Studies in the Philosophy of Science, 29(2), 129–148. https://doi.org/10.1080/02698595.2015.1119413

46.

Moher

Stewart

Shekelle

(2015). All in the family: Systematic reviews, rapid reviews, scoping reviews, realist reviews, and more. Systematic Reviews, 4(1), 183. https://doi.org/10.1186/s13643-015-0163-7

47.

Morissette

Tricco

A. C.

Horsley

Chen

M. H.

Moher

(2011). Blinded versus unblinded assessments of risk of bias in studies included in a systematic review. Cochrane Database of Systematic Reviews, 2011(9), Article MR000025. https://doi.org/10.1002/14651858.MR000025.pub2

48.

Munn

Stern

Aromataris

Lockwood

Jordan

(2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research Methodology, 18(5), 1–9. https://doi.org/10.1186/s12874-017-0468-4

49.

Munthe-Kaas

H. M.

Glenton

Booth

Noyes

Lewin

(2019). Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: First stage in the development of the CAMELOT tool. BMC Medical Research Methodology, 19(1), 113. https://doi.org/10.1186/s12874-019-0728-6

50.

Newman

Gough

(2020). Systematic reviews in educational research: Methodology, perspectives and application. In Zawacki-Richter

Kerres

Bedenlier

Bond

Buntins

(Eds.), Systematic reviews in educational research (pp. 3–22). Springer.

51.

Nussbaumer-Streit

Sommer

Hamel

Devane

Noel-Storr

Puljak

Trivella

Gartlehner

Cochrane Rapid Reviews Methods Group . (2023). Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment. BMJ Evidence-Based Medicine, 28(6), 418–423. https://doi.org/10.1136/bmjebm-2022-112185

52.

O’Connor

A. M.

Glasziou

Taylor

Thomas

Spijker

Wolfe

M. S.

(2020). A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: A summary of significant discussions at the fourth meeting of the International Collaboration for automation of systematic reviews (ICASR). Systematic Reviews, 9(1), 1–6. https://doi.org/10.1186/s13643-020-01351-4

53.

O’Hearn

MacDonald

Tsampalieros

Kadota

Sandarage

Jayawarden

S. K.

Datko

Reynolds

J. M.

Bui

Sultan

Sampson

Pratt

Barrowman

Nama

Page

McNally

J. D.

(2021). Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: A cross-sectional study. BMC Medical Research Methodology, 21(1), 142. https://doi.org/10.1186/s12874-021-01335-5

54.

Paré

Tate

Johnstone

Kitsiou

(2016). Contextualizing the twin concepts of systematicity and transparency in information systems literature reviews. European Journal of Information Systems, 25(6), 493–508. https://doi.org/10.1057/s41303-016-0020-3

55.

Paré

Trudel

M.-C.

Jaana

Kitsiou

(2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52(2), 183–199. https://doi.org/10.1016/j.im.2014.08.008

56.

Pieper

Waltering

Holstiege

Büchter

R. B.

(2018). Quality ratings of reviews in overviews: A comparison of reviews with and without dual (co-)authorship. Systematic Reviews, 7(1), 63. https://doi.org/10.1186/s13643-018-0722-9

57.

Quigley

J. M.

Thompson

J. C.

Halfpenny

N. J.

Scott

D. A.

(2019). Critical appraisal of nonrandomized studies—a review of recommended and commonly used tools. Journal of Evaluation in Clinical Practice, 25(1), 44–52. https://doi.org/10.1111/jep.12889

58.

Rowe

(2014). What literature review is not: Diversity, boundaries and recommendations. European Journal of Information Systems, 23(3), 241–255. https://doi.org/10.1057/ejis.2014.7

59.

Schick-Makaroff

MacDonald

Plummer

Burgess

Neander

(2016). What synthesis methodology should I use? A review and analysis of approaches to research synthesis. AIMS Public Health, 3(1), 172–215. https://doi.org/10.3934/publichealth.2016.1.172

60.

Schryen

Wagner

Benlian

(2015). Theory of knowledge for literature reviews: An epistemological model, taxonomy and empirical analysis of IS literature. In Thirty Sixth International Conference on Information Systems. ICIS.

61.

Shea

B. J.

Reeves

B. C.

Wells

Thuku

Hamel

Moran

Moher

Tugwell

Welch

Kristjansson

Henry

D. A.

(2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008. https://doi.org/10.1136/bmj.j4008

62.

Song

Hooper

Loke

(2013). Publication bias: What is it? How do we measure it? How do we avoid it? Open Access Journal of Clinical Trials, 2013(5), 71–81. https://doi.org/10.2147/OAJCT.S34419

63.

Song

Parekh

Hooper

Loke

Y. K.

Ryder

Sutton

A. J.

Hing

Kwok

C. S.

Pang

Harvey

(2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14(8), 1–220. https://doi.org/10.3310/hta14080

64.

Stoll

C. R. T.

Izadi

Fowler

Green

Suls

Colditz

G. A.

(2019). The value of a second reviewer for study selection in systematic reviews. Research Synthesis Methods, 10(4), 539–545. https://doi.org/10.1002/jrsm.1369

65.

Sutton

Clowes

Preston

Booth

(2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information and Libraries Journal, 36(3), 202–222. https://doi.org/10.1111/hir.12276

66.

Thomas

O'Mara-Eves

Harden

Newman

(2017). Synthesis methods for combining and configuring textual or mixed methods data. In Gough

Oliver

Thomas

(Eds.), Introduction to systematic reviews. Sage Publications.

67.

Tricco

A. C.

Langlois

E. V.

Straus

S. E.

(2017). Rapid reviews to strengthen health policy and systems: A practical guide. World Health Organization. https://apps.who.int/iris/bitstream/10665/258698/1/9789241512763-eng.pdf

68.

Waffenschmidt

Knelangen

Sieben

Bühn

Pieper

(2019). Single screening versus conventional double screening for study selection in systematic reviews: A methodological systematic review. BMC Medical Research Methodology, 19(1), 132. https://doi.org/10.1186/s12874-019-0782-0

69.

Whiting

Savovic

Higgins

J. P.

Caldwell

D. M.

Reeves

B. C.

Shea

Davies

Kleijnen

Churchill

group

(2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69(1), 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005

70.

Whiting

Wolff

Mallett

Simera

Savović

(2017). A proposed framework for developing quality assessment tools. Systematic Reviews, 6(204), 1–9. https://doi.org/10.1186/s13643-017-0604-6

71.

Whiting

Wolff

Savović

Devine

Mallett

(2024). Introducing the LATITUDES network: A library of assessment tools and training to improve transparency, utility and dissemination in evidence synthesis. Journal of Clinical Epidemiology, 174(21), Article 111486. https://doi.org/10.1016/j.jclinepi.2024.111486

72.

Wong

Greenhalgh

Westhorp

Pawson

(2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist and Meta-narrative Evidence Syntheses - Evolving Standards) project. Health Services and Delivery Research, 2(30), 1–252. https://doi.org/10.3310/hsdr02300

Helping Trainees Understand the Strategies to Minimize Errors and Biases in Systematic Review Approaches

Abstract

Keywords

Background

Levels of Systematicity at Each Step of the Review Process

Step 1: Searching the literature

Step 2: Selecting relevant documents

Step 3: Extracting data

Step 4: Appraising the quality of included documents

Step 5: Synthesizing data

Discussion and Conclusion

Statements and Declarations

Footnotes

Acknowledgements

Conflicting interest

Funding

ORCID iDs

References