Qualitative Evidence Syntheses Within Cochrane Effective Practice and Organisation of Care: Developing a Template and Guidance

Abstract

A growing number of researchers are preparing systematic reviews of qualitative evidence, often referred to as ‘qualitative evidence syntheses’. Cochrane published its first qualitative evidence synthesis in 2013 and published 27 such syntheses and protocols by August 2020. Most of these syntheses have explored how people experience or value different health conditions, treatments and outcomes. Several have been used by guideline producers and others to identify the topics that matter to people, consider the acceptability and feasibility of different healthcare options and identify implementation considerations, thereby complementing systematic reviews of intervention effectiveness.Guidance on how to conduct and report qualitative evidence syntheses exists. However, methods are evolving, and we still have more to learn about how to translate and integrate existing methodological guidance into practice. Cochrane’s Effective Practice and Organisation of Care (EPOC) (www.epoc.org) has been involved in many of Cochrane’s qualitative evidence syntheses through the provision of editorial guidance and support and through co-authorship. In this article, we describe the development of a template and guidance for EPOC’s qualitative evidence syntheses and reflect on this process.

Keywords

qualitative meta-analysis/synthesis methods in quanlitative inquiry meta-synthesis qualitative evaluation secondary data analysis

Background

Synthesising qualitative research evidence is becoming a well-established research method (Noyes et al., 2021). A growing number of healthcare researchers are preparing systematic reviews of qualitative evidence, often referred to as ‘qualitative evidence syntheses’. Many of these reviews are designed and used to support policy and programmatic decisions, often alongside systematic reviews of intervention effectiveness.

Typically, healthcare researchers prepare qualitative evidence syntheses to explore how people experience or value different healthcare conditions, treatments and outcomes. Decision-makers, guideline producers and others can then use this evidence to identify the topics that matter to people, consider the acceptability and feasibility of different healthcare options for different people and under which conditions and identify implementation considerations (Downe et al., 2019a; Lewin et al., 2019; Glenton et al., 2019). For instance, Cochrane’s first qualitative evidence synthesis from 2013 (Glenton et al., 2013) explored people’s views and experiences of lay health worker programmes. This review was used to provide information about the acceptability and feasibility of these programmes for a World Health Organization guideline on shifting tasks from one type of health worker to another (WHO-World Health Organization recommendations, 2012). Qualitative evidence syntheses can therefore complement other types of evidence by providing important insights into people’s values, views and experiences that may otherwise not be included in decision-making processes(Downe et al., 2019; Lewin et al., 2019; Lewin & Glenton, 2018).

As more qualitative evidence syntheses are conducted, guidance on conducting and reporting these types of reviews has also become more sophisticated. Recent guidance includes the ENTREQ reporting guidance for qualitative evidence syntheses (Tong et al., 2012), the EMERGE reporting guidance for meta-ethnographies (France et al., 2019) and a series of papers published by the Cochrane’s Qualitative and Implementation Methods Group (Noyes et al., 2018a, 2018b) that provides methodological guidance on conducting qualitative and mixed-method evidence syntheses. However, methods are still evolving, and we still have more to learn about how to translate and integrate existing guidance into practice.

Cochrane Editorial Teams: An Ideal Environment for Cumulative Learning and Continuous Quality Improvement

Cochrane’s editorial teams provide an ideal environment for developing, testing and operationalising systematic review methodology. Cochrane editorial teams differ from those in traditional journals in that they actively and collaboratively support authors throughout the entire process, from inception to publication of each review. Cochrane also expects authors to follow specific quality criteria regarding review conduct and reporting. Cochrane editorial teams facilitate this by sharing formal guidance with review authors as well as providing several rounds of tailored editorial feedback to ensure that these quality criteria are met.

Most Cochrane groups focus on systematic reviews of intervention effectiveness, primarily based on randomised trials. However, Cochrane is preparing an increasing number of qualitative evidence syntheses. The Cochrane Library currently includes 30 qualitative evidence syntheses and synthesis protocols as well as several mixed-methods reviews. Several more are in the pipeline. Cochrane’s Effective Practice and Organisation of Care (EPOC)’s editorial team (www.epoc.org) has been involved in many of these, either through the provision of editorial guidance and support or through co-authorship.

The relative novelty of qualitative evidence synthesis methods has meant that EPOC’s support to and collaboration with review authors have been particularly close. Most of these review teams have regular contact with the editorial team, often including 2–3-day face-to-face meetings. Through this process, review authors have been able to draw directly from, and build on, the experiences and lessons of other review authors; adopt what works; and avoid what does not work. The nature of Cochrane’s editorial process, combined with a particularly close collaboration with review teams, has given EPOC an excellent opportunity for cumulative learning and continuous quality improvement.

The qualitative evidence syntheses that EPOC have been involved in represent a case series that has allowed us to learn more about how to prepare qualitative evidence syntheses in practice.

To support new review authors and editors, EPOC decided to consolidate these lessons in a qualitative evidence synthesis template for protocols and full reviews (Glenton et al., 2020a). This resource will also likely be useful to review authors carrying out qualitative evidence syntheses outside of Cochrane. In the following, we describe the development of the template and reflect on this process.

Developing the Template

One of the main aims of the EPOC template was to improve the quality and transparency of qualitative evidence synthesis reporting. We therefore broadly followed the stages described by the Equator Network for the development of reporting guidelines (Equator Network, 2018). However, we also identified the need for more advice about review conduct, particularly for certain review components. This aspect of the template was also incorporated into the same development process. Both aspects are described in more detail below.

Stage 1: Identifying the Need for a Qualitative Evidence Synthesis Template

EPOC published Cochrane’s first qualitative evidence synthesis in 2013 (Glenton et al., 2013). This has since been followed by a number of other EPOC qualitative evidence syntheses. Prior to the development of our template, review authors and the editorial team relied on existing Cochrane and non-Cochrane methodological guidance, as described above. In addition, our reviews received feedback from external peer reviewers and, in most cases, a member of Cochrane’s Qualitative and Implementation Methods Group.

This pre-existing guidance and advice was helpful but not always sufficient. Firstly, editors and review authors needed to ensure that our reviews met the strict and specific reporting requirements for all Cochrane reviews. However, these reporting requirements, originally designed for reviews of effectiveness, did not transfer adequately to the process of documenting qualitative method and methodologies and reporting findings. Secondly, we lacked guidance for certain elements of a qualitative evidence synthesis, including the use of sampling approaches, dealing with studies published in multiple languages, addressing review author reflexivity and conflict of interest issues, developing implications for practice and for future research and preparing abstracts and plain language summaries. Where guidance existed, we sometimes struggled to interpret or apply this guidance in practice, often because the methods were difficult to operationalise (Tricco et al., 2016). Moreover, existing guidance was often geared towards specific qualitative synthesis approaches, such as meta-ethnography (France et al., 2019).

Initially, the editorial team addressed these challenges by working with individual review teams to develop solutions as well as by asking review teams to share their experiences, including their published and unpublished materials, with other review teams who were at earlier stages in the review process. However, we quickly realised that these approaches were not efficient, and in 2018, the editorial team decided to create more structured guidance.

Stage 2: Getting Ready to Develop the Template

To prepare for the development of the template, we began by establishing a core group who would lead this work. This group was made up of four of EPOC’s six qualitative evidence synthesis editors (MAB, SD, CG, and SL). All four have backgrounds in primary qualitative research as well as experience in preparing qualitative evidence syntheses. Three of them (MAB, CG, and SL) are also co-conveners of the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) project group and regularly train review authors in the use of GRADE-CERQual for assessing confidence in findings from qualitative evidence syntheses (Lewin et al., 2018a). In addition, one of EPOC’s managing editors (EP) joined the core group. While she did not have a background in qualitative research, she had an in-depth knowledge of the editorial processes for these reviews.

In addition, we consulted with a wider group throughout the development process. This included EPOC’s information specialist (MJ), two other EPOC qualitative evidence synthesis editors (KD, SMB) and the lead authors of the qualitative evidence syntheses (CG, SL, SD, SMB, SA, HA, SC, KD, CH, AKS, HM, WO, ES, LV, AX, and MAB). These authors had mixed backgrounds, but each review team included at least one person with primary qualitative research experience and at least one person with systematic review experience. We also shared drafts of the template with other relevant stakeholders within and outside of Cochrane, including the Cochrane Implementation and Qualitative Methods Group and commissioners of Cochrane qualitative evidence syntheses such as the World Health Organization.

Finally, we identified relevant Cochrane and non-Cochrane guidance for the reporting and conduct of systematic reviews in general and qualitative evidence syntheses specifically. This included guidance that we wanted to draw upon as well as guidelines that we would be expected to comply with, including resources from ENTREQ (Tong et al., 2012), EMERGE (France et al., 2019), Cochrane’s Qualitative and Implementation Methods Group (Noyes et al., 2018a, 2018b); EPOC, the Cochrane Handbook and PRISMA (Page et al., 2021).

Stage 3: Developing the Template

Pre-Meeting Activities: Extracting and Assessing the Contents of Existing Syntheses

To prepare for our developmental work, we extracted data from our current qualitative evidence syntheses. At this stage, EPOC was managing ten qualitative evidence syntheses at different stages of the editorial process (Glenton et al., 2013; Munabi-Babigumira et al., 2017; Karimi-Shahanjarini et al., 2019; Odendaal et al., 2020; Xyrichis et al., 2021; Cooper et al., 2019a; Cooper et al., 2019b; Moloi et al., 2020; Bohren et al., 2019; Downe et al., 2019b), and members of the editorial team were also co-authoring two qualitative evidence syntheses managed by the Cochrane Consumers and Communication Group (Ames et al., 2017; Ames et al., 2019a). One EPOC editor (CG) extracted the content from these 12 published and unpublished protocols and reviews and organised this content according to the different sections of a qualitative evidence synthesis (i.e. title, abstract, methods section, results section, etc.).

Face-to-Face Consensus Meeting

The core team then had one face-to-face meeting and several teleconferences to assess the contents of the reviews, section by section; consider feedback from review authors, review commissioners and other stakeholders; as well as comments from peer reviewers to individual reviews; and to discuss what we considered to be acceptable solutions for each section. EPOC’s information specialist (MJ) also provided input throughout this process, especially with regard to the guidance on search methods.

When attempting to reach consensus, we aimed to ensure that the template followed the theoretical and philosophical principles of qualitative research (Denzin & YSe, 2011). We also aimed to ensure that the template was in line with existing guidance provided by EPOC, the Cochrane Qualitative and Implementation Methods Group, the Cochrane Handbook and other appropriate sources.

Writing Up and Piloting the Template

The core team, with help from EPOC’s information specialist, drafted a template, as well as complementary guidance documents for sections of the review that we thought required more detailed information.

Two of the twelve syntheses on which we based our template had been published when we started this work, while ten were still ongoing. The EPOC editors and managing editor sent the first draft of the template to EPOC review authors that were currently working on their qualitative evidence syntheses and asked them to send feedback. The EPOC editors and managing editor also tested these early versions on protocols and reviews as part of the editorial process and kept notes of areas that required further work. The template was also sent to other relevant stakeholders within Cochrane for feedback.

One EPOC editor (CG) then used this feedback to update the template. Version 1 of the template was then agreed upon by the core group, as well as by the managing editor and information specialist.

Stage 4: Updating the Template

Version 1 of EPOC’s qualitative evidence synthesis template was based on our experiences with twelve qualitative evidence syntheses: ten from Cochrane EPOC and two from Cochrane Consumers and Communication and co-authored by EPOC editors. In 2019, we developed a Version 2 of the template in response to experiences with two mixed-methods reviews and a rapid qualitative evidence synthesis published by Cochrane EPOC (Vasudevan et al., 2018; Agarwal et al., 2020; Houghton et al., 2020). In addition, the editorial team kept a running list of issues that we had identified when using the template, including where clarifications or additional information were needed. The editorial team continues to gather this information to inform the next version of the template.

The Current Template

An overview of EPOC’s qualitative evidence synthesis template can be seen in Figure 1. The latest version of the full template is freely available on EPOC’s website (https://bit.ly/3vgPcAg). The template includes the following:

Suggested subheadings for each section of the review.

An explanation of the type of content each section should include.

Where appropriate, proposals for standardised text.

Links or references to additional information (for instance, other Cochrane resources).

Appendices with examples of how each section could be written.

Figure 1.

Overview of EPOC’s qualitative evidence synthesis template structure.

The explanations and examples we have included have focused on topics that have received less attention in other sources of guidance (Figure 1). These include the following topics:

Preparing abstracts and plain language summaries for qualitative evidence syntheses.

Including and translating studies in multiple languages.

Approaches for study sampling.

Considering review author reflexivity.

Developing ‘Implications for practice’ sections.

Developing ‘Implications for research’ sections.

Preparing declarations of interest statements.

We discuss each of these topics further in a forthcoming study (Glenton et al., 2020).

In addition, EPOC has developed separate but complementary guidance on how to prepare a plain language summary for an EPOC qualitative evidence synthesis (https://bit.ly/36oT6dT) and how to sample studies (https://bit.ly/3gEX2gJ).

Overarching Issues when Developing the Template

A key topic of discussion as we developed the template concerned the amount of guidance it was appropriate to give. We were cautious of being overly prescriptive and wanted to ensure that we would continue to learn from new review teams bringing methodological insights and using different synthesis approaches. At the same time, we wanted to develop a review structure and content that would encourage good practice and that would make the work of review authors, editors and end users easier.

We decided to offer suggestions of standardised text in situations where we wanted to encourage specific methodological practices or reporting standards. This included parts of the methods section, specifically descriptions of how review authors had searched for, selected, translated and sampled studies; how they assessed any methodological limitations of studies; and how they assessed confidence in the review findings. In other sections where we wanted to encourage flexibility, we avoided standardised text; for instance, in the sub-section on data analysis and synthesis. We also included appendices with examples, particularly where we believed that our reviews offered practical suggestions that might not be available in other sources of guidance; for instance, in the sections on reflexivity and implications for practice.

Topics for Future Development

In addition to these topics, our template covers elements of a qualitative evidence synthesis that have been discussed elsewhere, including in guidance provided by the Cochrane Qualitative and Implementation Methods Group (Noyes et al., 2018a). While these topics have received more attention, they are also likely to benefit from further worked examples, including examples using a broader range of qualitative evidence synthesis methods, and prompts for review teams to consider how to operationalise methodological guidance. For instance, we need more knowledge about how best to assess the methodological limitations of qualitative studies in the context of a qualitative evidence synthesis (Munthe-Kaas et al., 2019; Munthe-Kaas et al., 2018; Noyes et al., 2018b); the extent to which ‘dissemination bias’ operates within qualitative research and how this may influence the type of studies we search for (Harris et al., 2018; Toews et al., 2016; Toews et al., 2017); how best to integrate findings from qualitative evidence syntheses with findings from related effectiveness reviews (Harden et al., 2018); and how to apply GRADE-CERQual in the context of more interpretive findings (Lewin et al., 2018b).

We expect the template to continue to improve as we gain more experience. Issues that we hope to work on in future version of the template include the following: processes for translating studies written in multiple languages; approaches to summarising review findings in abstracts and plain language summaries; and processes for applying and reporting reflexivity in the context of qualitative evidence syntheses. In reviews that have used sampling strategies and have findings that we assessed as being of low or very low confidence, we also plan to explore how review authors can utilise studies that were eligible but not originally sampled to address this (Ames et al., 2019b).

Usefulness of the Template for Future Authors of Qualitative Evidence Syntheses

One of our key motivations when developing the template was to offer new review authors the opportunity to learn from other review authors’ experiences and to ensure that editors do not have to repeatedly point to the same gaps or flaws. EPOC now routinely offers the template to new review authors and has also made it freely available to other authors of qualitative evidence syntheses within and outside Cochrane.

By drawing on a series of worked examples of qualitative evidence syntheses, we have been able to address gaps in current guidance and translate existing guidance into practical advice. The template offers review authors a review structure, describes content, provides examples and suggests standardised text. This type of practical support to review authors has previously proved very helpful to both editors and authors in other situations. For instance, Cochrane Consumers and Communication has developed a template and practical guidance for review authors of complex communication reviews (Ryan & Hill, 2019). According to the Group’s editors, the review template has ‘greatly improved the transparency of reporting’ in their reviews, and the use of standardised wording for authors to adapt has led to ‘major improvements in the coherence and comprehensiveness of the description of methods as a whole’ (Ryan & Hill, 2019). The editorial team is now spending less time querying and correcting review authors’ descriptions of methods and results, and now have more time to focus on conceptual coherence and clarity, and on ‘supporting authors in interpreting the meaning of findings and in developing consistent messages about these’ (Ryan & Hill, 2019). EPOC editors have had similar experiences. The template has also provided welcome guidance to review authors embarking on their first qualitative evidence synthesis and has provided more experienced review authors with information about Cochrane’s specific expectations. Most recently, the template helped support authors of a rapid qualitative evidence synthesis prepared as part of Cochrane’s response to the COVID-19 pandemic by providing standardised text that could be adapted rapidly (Houghton et al., 2020). The success of the template lies partly in striking a balance between instruction and flexibility, so that qualitative evidence synthesis authors can be guided, but not constricted in the development of their reviews. We believe that this resource will be helpful for all researchers developing and reporting qualitative evidence syntheses.

Conclusion

The qualitative evidence syntheses that Cochrane EPOC has been involved in represent a case series that has allowed us to learn more about how to undertake a qualitative evidence synthesis in practice. This work has helped to create a bridge between the existing conceptual guidance on conducting qualitative evidence syntheses and the practical steps in implementing this. A good understanding of the principles underlying qualitative research is necessary to conduct a qualitative evidence synthesis well. This template can support people who have this background but who are novice review authors or review authors new to Cochrane. We have consolidated the lessons we have learnt in a template designed for future authors of EPOC qualitative evidence syntheses. We also expect the template and guidance to be useful to review teams conducting qualitative evidence syntheses in other contexts and believe that it can improve the conduct and reporting of these reviews, including their transparency and consistency.

Footnotes

Acknowledgments

Work on the template was partly made possible through financial support from The Norwegian Agency for Development Cooperation (Norad), the WHO and the South African Medical Research Council.

Declaration of Conflicting Interests

The authors declared that there is no conflict of interest with respect to the research, authorship, and/or publication of this article

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Work on the template was partly made possible through financial support from The Norwegian Agency for Development Cooperation (Norad), the WHO and the South African Medical Research Council.

ORCID iDs

Claire Glenton

Simon Lewin

Soo Downe

Catherine Houghton

Willem Odendaal

References

Agarwal

Glenton

Henschke

Tamrat

Bergman

Fønhus

M. S.

Mehl

G. L.

Lewin

(2011). Tracking health commodity inventory and notifying stock levels via mobile devices: A mixed methods systematic review. The Cochrane database of systematic reviews, 10(10), Cd012907.

Ames

H. M.

Glenton

Lewin

(2017). Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: A synthesis of qualitative evidence. The Cochrane database of systematic reviews, 2(2), Cd011787.

Ames

Glenton

Lewin

(2019b). Purposive sampling in a qualitative evidence synthesis: A worked example from a synthesis on parental perceptions of vaccination communication. BMC Medical Research Methodology, 19(1), 26.

Ames

H. M

Glenton

Lewin

Tamrat

Akama

Leon

(2019a). Clients’ perceptions and experiences of targeted digital communication accessible via mobile devices for reproductive, maternal, newborn, child, and adolescent health: a qualitative evidence synthesis. The Cochrane database of systematic reviews, 10(10), Cd013447.

Bohren

M. A.

Berger

B. O.

Munthe-Kaas

Tunçalp

Ö.

(2019). Perceptions and experiences of labour companionship: A qualitative evidence synthesis. The Cochrane database of systematic reviews, 3(3), Cd012449.

Cooper

Schmidt

Ryan

Leon

Mavundza

Burnett

Tanywe

A. C.

Wiysonge

C. S.

(2019a). Factors that influence acceptance of human papillomavirus (HPV) vaccination for adolescents: A qualitative evidence synthesis. Cochrane Database of Systematic Reviews. Advance online publication. https://doi.org/10.1002/14651858.CD013430.

Cooper

Schmidt

B. M.

Sambala

E. Z.

Swartz

Colvin

C. J.

Leon

Betsch

Wiysonge

C. S.

(2019b). Factors that influence parents’ and informal caregivers’ acceptance of routine childhood vaccination: A qualitative evidence synthesis. Cochrane Database of Systematic Reviews. Advance online publication. https://doi.org/10.1002/14651858.CD013265.

Denzin

N. K.

YSe

(2011). The SAGE handbook of qualitative research (4th ed.). Thousand, Oaks : SAGE.

Downe

Finlayson

K. W.

Lawrie

T. A.

Lewin

S. A.

Glenton

Rosenbaum

Barreix

Tunçalp

Ö.

(2019a). Qualitative evidence synthesis (QES) for guidelines: Paper 1—Using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Research Policy and Systems, 17(1), 76.

10.

Downe

Finlayson

Tunçalp

Ö.

Gülmezoglu

A. M.

(2019b). Provision and uptake of routine antenatal services: a qualitative evidence synthesis. The Cochrane database of systematic reviews, 6(6), Cd012392.

11.

Equator Network (2018). How to develop a reporting guideline. https://www.equator-network.org/toolkits/developing-a-reporting-guideline/.

12.

France

E. F.

Cunningham

Ring

Uny

Duncan

E. A. S.

Jepson

R. G.

Maxwell

Roberts

R. J.

Turley

R. L.

Booth

Noyes

(2019). Improving reporting of meta-ethnography: The eMERGe reporting guidance. BMC Medical Research Methodology, 19(1), 25.

13.

Glenton

Bohren

M. A.

Downe

Paulsen

E. J.

Lewin

(2020a). EPOC qualitative evidence synthesis: Protocol and review template. Version 1.1 on behalf of Effective Practice and Organisation of Care (EPOC). In: EPOC Resources for review authors. Oslo: Norwegian Institute of Public Health.

14.

Glenton

Colvin

C. J.

Carlsen

Swartz

Lewin

Noyes

Rashidian

(2013). Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: Qualitative evidence synthesis. The Cochrane database of systematic reviews, 10(10), CD010414.

15.

Glenton

Lewin

Downe

Paulsen

Munabi-Babigumira

Johansen

Agarwal

Ames

Cooper

Daniels

(2020). Preparing Cochrane qualitative evidence syntheses, differences from reviews of effectiveness, and implications for guidance in a Cochrane context. Manuscript submitted for publication.

16.

Glenton

Lewin

Lawrie

T. A.

Barreix

Downe

Finlayson

K. W.

Tamrat

Rosenbaum

Tunçalp

Ö.

(2019). Qualitative evidence synthesis (QES) for guidelines: Paper 3—Using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Research Policy and Systems, 17(1), 74.

17.

Harden

Thomas

Cargo

Harris

Pantoja

Flemming

Booth

Garside

Hannes

Noyes

(2018). Cochrane Qualitative and Implementation Methods Group guidance series-paper 5: Methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. Journal of Clinical Epidemiology, 97, 70-78.

18.

Harris

J. L.

Booth

Cargo

Hannes

Harden

Flemming

Garside

Pantoja

Thomas

Noyes

(2018). Cochrane Qualitative and Implementation Methods Group guidance series-paper 2: Methods for question formulation, searching, and protocol development for qualitative evidence synthesis. Journal of Clinical Epidemiology, 97, 39-48.

19.

Houghton

Delaney

Smalle

Glenton

Booth

Chan

X. H. S.

Devane

Biesty

L. M.

(2020). Barriers and facilitators to healthcare workers’ adherence with infection prevention and control (IPC) guidelines for respiratory infectious diseases: A rapid qualitative evidence synthesis. Cochrane Database of Systematic Reviews, 2020, CD013582. DOI: 10.1002/14651858.CD013582

20.

Karimi-Shahanjarini

Shakibazadeh

Rashidian

Hajimiri

Glenton

Noyes

Lewin

Laurant

Colvin

C. J.

(2019). Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: A qualitative evidence synthesis. The Cochrane database of systematic reviews, 4(4), Cd010412.

21.

Lewin

Bohren

Rashidian

Munthe-Kaas

Glenton

Colvin

C. J.

Garside

Noyes

Booth

Tunçalp

Ö.

Wainwright

Flottorp

Tucker

J. D.

Carlsen

(2018b). Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: How to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Implementation Science, 13(Suppl 1), 10.

22.

Lewin

Booth

Glenton

Munthe-Kaas

Rashidian

Wainwright

Bohren

M. A.

Tunçalp

Ö.

Colvin

C. J.

Garside

Carlsen

Langlois

E. V.

Noyes

(2018a). Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Science, 13(Suppl 1), 2.

23.

Lewin

Glenton

(2018). Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization. International Journal for Equity in Health, 17(1), 126.

24.

Lewin

Glenton

Lawrie

T. A.

Downe

Finlayson

K. W.

Rosenbaum

Barreix

Tunçalp

Ö.

(2019). Qualitative evidence synthesis (QES) for guidelines: Paper 2—Using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Research Policy and Systems, 17(1), 75.

25.

Moloi

Daniels

Cooper

Odendaal

W. A.

Naledi

Goliath

Leon

(2020). Healthcare workers’ perceptions and experience of primary healthcare integration: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews, (5). DOI:10.1002/14651858.CD013603.

26.

Munabi-Babigumira

Glenton

Lewin

Fretheim

Nabudere

(2017). Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. The Cochrane database of systematic reviews, 11(11), Cd011558.

27.

Munthe-Kaas

Bohren

M. A.

Glenton

Lewin

Noyes

Tunçalp

Ö.

Booth

Garside

Colvin

C. J.

Wainwright

Rashidian

Flottorp

Carlsen

(2018). Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 3: How to assess methodological limitations. Implementation Science, 13(Suppl 1), 9.

28.

Munthe-Kaas

H. M.

Glenton

Booth

Noyes

Lewin

(2019). Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: First stage in the development of the CAMELOT tool. BMC Medical Research Methodology, 19(1), 113.

29.

Noyes

Booth

Cargo

Flemming

Garside

Hannes

Harden

Harris

Lewin

Pantoja

Thomas

(2018a). Cochrane Qualitative and Implementation Methods Group guidance series-paper 1: Introduction. Journal of Clinical Epidemiology, 97, 35-38.

30.

Noyes

Booth

Flemming

Garside

Harden

Lewin

Pantoja

Hannes

Cargo

Thomas

(2018b). Cochrane Qualitative and Implementation Methods Group guidance series-paper 3: Methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. Journal of clinical epidemiology, 97, 49-58.

31.

Noyes

J. B. A.

Cargo

Flemming

Harden

Harris

Garside

Hannes

Pantoja

Thomas

(2021). Chapter 21: Qualitative evidence. In Higgins

J. P. T.

Chandler

Cumpston

Page

M. J.

Welch

V. A.

(Eds.). Cochrane Handbook for Systematic Reviews of Interventions Volume Version 6.2. Cochrane. Available from www.training.cochrane.org/handbook.

32.

Odendaal

W. A.

Anstey Watkins

Leon

Goudge

Griffiths

Tomlinson

Daniels

(2021). Health workers’ perceptions nd experiences of using mHealth technologies to deliver primary healthcare services: A qualitative evidence synthesis. The Cochrane database of systematic reviews, 3(3), Cd011942.

33.

Page

M. J.

Moher

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

McKenzie

J. E.

(2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. Bmj: British Medical Journal, 372, n160.

34.

Ryan

Hill

(2019). Supporting implementation of Cochrane methods in complex communication reviews: Resources developed and lessons learned for editorial practice and policy. Health Research Policy and Systems, 17(1), 32.

35.

Toews

Booth

Berg

R. C.

Lewin

Glenton

Munthe-Kaas

H. M.

Noyes

Schroter

Meerpohl

J. J.

(2017). Further exploration of dissemination bias in qualitative research required to facilitate assessment within qualitative evidence syntheses. Journal of Clinical Epidemiology, 88, 133-139.

36.

Toews

Glenton

Lewin

Berg

R. C.

Noyes

Booth

Marusic

Malicki

Munthe-Kaas

H. M.

Meerpohl

J. J.

(2016). Extent, awareness and perception of dissemination bias in qualitative research: An explorative survey. PLoS One, 11(8), e0159290.

37.

Tong

Flemming

McInnes

Oliver

Craig

(2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Medical Research Methodology, 12, 181.

38.

Tricco

A. C.

Soobiah

Antony

Cogo

MacDonald

Lillie

Tran

D'Souza

Hui

Perrier

Welch

Horsley

Straus

S. E.

Kastner

(2016). A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method. Journal of Clinical Epidemiology, 73, 19-28.

39.

Vasudevan

Henschke

Glenton

Lewin

Maayan

Eyers

Fønhus

M. S.

Tamrat

Mehl

G. L.

(2018). Birth and death notification via mobile devices. Cochrane Database of Systematic Reviews, (7). DOI:10.1002/14651858.CD012909.pub2.

40.

WHO-World Health Organization recommendations (2012). Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting. Geneva: World Health Organization.

41.

Xyrichis

Iliopoulou

Mackintosh

N. J.

Bench

Terblanche

Philippou

Sandall

(2021). Healthcare stakeholders’ perceptions and experiences of factors affecting the implementation of critical care telemedicine (CCT): Qualitative evidence synthesis. The Cochrane database of systematic reviews, 2(2), Cd012876.