Qualitative Evidence Syntheses Within Cochrane Effective Practice and Organisation of Care: Developing a Template and Guidance

Abstract

A growing number of researchers are preparing systematic reviews of qualitative evidence, often referred to as ‘qualitative evidence syntheses’. Cochrane published its first qualitative evidence synthesis in 2013 and published 27 such syntheses and protocols by August 2020. Most of these syntheses have explored how people experience or value different health conditions, treatments and outcomes. Several have been used by guideline producers and others to identify the topics that matter to people, consider the acceptability and feasibility of different healthcare options and identify implementation considerations, thereby complementing systematic reviews of intervention effectiveness.Guidance on how to conduct and report qualitative evidence syntheses exists. However, methods are evolving, and we still have more to learn about how to translate and integrate existing methodological guidance into practice. Cochrane’s Effective Practice and Organisation of Care (EPOC) (www.epoc.org) has been involved in many of Cochrane’s qualitative evidence syntheses through the provision of editorial guidance and support and through co-authorship. In this article, we describe the development of a template and guidance for EPOC’s qualitative evidence syntheses and reflect on this process.

Background

Synthesising qualitative research evidence is becoming a well-established research method (Noyes et al., 2021). A growing number of healthcare researchers are preparing systematic reviews of qualitative evidence, often referred to as ‘qualitative evidence syntheses’. Many of these reviews are designed and used to support policy and programmatic decisions, often alongside systematic reviews of intervention effectiveness.

Typically, healthcare researchers prepare qualitative evidence syntheses to explore how people experience or value different healthcare conditions, treatments and outcomes. Decision-makers, guideline producers and others can then use this evidence to identify the topics that matter to people, consider the acceptability and feasibility of different healthcare options for different people and under which conditions and identify implementation considerations (Downe et al., 2019a; Lewin et al., 2019; Glenton et al., 2019). For instance, Cochrane’s first qualitative evidence synthesis from 2013 (Glenton et al., 2013) explored people’s views and experiences of lay health worker programmes. This review was used to provide information about the acceptability and feasibility of these programmes for a World Health Organization guideline on shifting tasks from one type of health worker to another (WHO-World Health Organization recommendations, 2012). Qualitative evidence syntheses can therefore complement other types of evidence by providing important insights into people’s values, views and experiences that may otherwise not be included in decision-making processes(Downe et al., 2019; Lewin et al., 2019; Lewin & Glenton, 2018).

As more qualitative evidence syntheses are conducted, guidance on conducting and reporting these types of reviews has also become more sophisticated. Recent guidance includes the ENTREQ reporting guidance for qualitative evidence syntheses (Tong et al., 2012), the EMERGE reporting guidance for meta-ethnographies (France et al., 2019) and a series of papers published by the Cochrane’s Qualitative and Implementation Methods Group (Noyes et al., 2018a, 2018b) that provides methodological guidance on conducting qualitative and mixed-method evidence syntheses. However, methods are still evolving, and we still have more to learn about how to translate and integrate existing guidance into practice.

Cochrane Editorial Teams: An Ideal Environment for Cumulative Learning and Continuous Quality Improvement

Cochrane’s editorial teams provide an ideal environment for developing, testing and operationalising systematic review methodology. Cochrane editorial teams differ from those in traditional journals in that they actively and collaboratively support authors throughout the entire process, from inception to publication of each review. Cochrane also expects authors to follow specific quality criteria regarding review conduct and reporting. Cochrane editorial teams facilitate this by sharing formal guidance with review authors as well as providing several rounds of tailored editorial feedback to ensure that these quality criteria are met.

Most Cochrane groups focus on systematic reviews of intervention effectiveness, primarily based on randomised trials. However, Cochrane is preparing an increasing number of qualitative evidence syntheses. The Cochrane Library currently includes 30 qualitative evidence syntheses and synthesis protocols as well as several mixed-methods reviews. Several more are in the pipeline. Cochrane’s Effective Practice and Organisation of Care (EPOC)’s editorial team (www.epoc.org) has been involved in many of these, either through the provision of editorial guidance and support or through co-authorship.

The relative novelty of qualitative evidence synthesis methods has meant that EPOC’s support to and collaboration with review authors have been particularly close. Most of these review teams have regular contact with the editorial team, often including 2–3-day face-to-face meetings. Through this process, review authors have been able to draw directly from, and build on, the experiences and lessons of other review authors; adopt what works; and avoid what does not work. The nature of Cochrane’s editorial process, combined with a particularly close collaboration with review teams, has given EPOC an excellent opportunity for cumulative learning and continuous quality improvement.

The qualitative evidence syntheses that EPOC have been involved in represent a case series that has allowed us to learn more about how to prepare qualitative evidence syntheses in practice.

To support new review authors and editors, EPOC decided to consolidate these lessons in a qualitative evidence synthesis template for protocols and full reviews (Glenton et al., 2020a). This resource will also likely be useful to review authors carrying out qualitative evidence syntheses outside of Cochrane. In the following, we describe the development of the template and reflect on this process.

Developing the Template

One of the main aims of the EPOC template was to improve the quality and transparency of qualitative evidence synthesis reporting. We therefore broadly followed the stages described by the Equator Network for the development of reporting guidelines (Equator Network, 2018). However, we also identified the need for more advice about review conduct, particularly for certain review components. This aspect of the template was also incorporated into the same development process. Both aspects are described in more detail below.

Stage 1: Identifying the Need for a Qualitative Evidence Synthesis Template

EPOC published Cochrane’s first qualitative evidence synthesis in 2013 (Glenton et al., 2013). This has since been followed by a number of other EPOC qualitative evidence syntheses. Prior to the development of our template, review authors and the editorial team relied on existing Cochrane and non-Cochrane methodological guidance, as described above. In addition, our reviews received feedback from external peer reviewers and, in most cases, a member of Cochrane’s Qualitative and Implementation Methods Group.

This pre-existing guidance and advice was helpful but not always sufficient. Firstly, editors and review authors needed to ensure that our reviews met the strict and specific reporting requirements for all Cochrane reviews. However, these reporting requirements, originally designed for reviews of effectiveness, did not transfer adequately to the process of documenting qualitative method and methodologies and reporting findings. Secondly, we lacked guidance for certain elements of a qualitative evidence synthesis, including the use of sampling approaches, dealing with studies published in multiple languages, addressing review author reflexivity and conflict of interest issues, developing implications for practice and for future research and preparing abstracts and plain language summaries. Where guidance existed, we sometimes struggled to interpret or apply this guidance in practice, often because the methods were difficult to operationalise (Tricco et al., 2016). Moreover, existing guidance was often geared towards specific qualitative synthesis approaches, such as meta-ethnography (France et al., 2019).

Initially, the editorial team addressed these challenges by working with individual review teams to develop solutions as well as by asking review teams to share their experiences, including their published and unpublished materials, with other review teams who were at earlier stages in the review process. However, we quickly realised that these approaches were not efficient, and in 2018, the editorial team decided to create more structured guidance.

Stage 2: Getting Ready to Develop the Template

To prepare for the development of the template, we began by establishing a core group who would lead this work. This group was made up of four of EPOC’s six qualitative evidence synthesis editors (MAB, SD, CG, and SL). All four have backgrounds in primary qualitative research as well as experience in preparing qualitative evidence syntheses. Three of them (MAB, CG, and SL) are also co-conveners of the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) project group and regularly train review authors in the use of GRADE-CERQual for assessing confidence in findings from qualitative evidence syntheses (Lewin et al., 2018a). In addition, one of EPOC’s managing editors (EP) joined the core group. While she did not have a background in qualitative research, she had an in-depth knowledge of the editorial processes for these reviews.

In addition, we consulted with a wider group throughout the development process. This included EPOC’s information specialist (MJ), two other EPOC qualitative evidence synthesis editors (KD, SMB) and the lead authors of the qualitative evidence syntheses (CG, SL, SD, SMB, SA, HA, SC, KD, CH, AKS, HM, WO, ES, LV, AX, and MAB). These authors had mixed backgrounds, but each review team included at least one person with primary qualitative research experience and at least one person with systematic review experience. We also shared drafts of the template with other relevant stakeholders within and outside of Cochrane, including the Cochrane Implementation and Qualitative Methods Group and commissioners of Cochrane qualitative evidence syntheses such as the World Health Organization.

Finally, we identified relevant Cochrane and non-Cochrane guidance for the reporting and conduct of systematic reviews in general and qualitative evidence syntheses specifically. This included guidance that we wanted to draw upon as well as guidelines that we would be expected to comply with, including resources from ENTREQ (Tong et al., 2012), EMERGE (France et al., 2019), Cochrane’s Qualitative and Implementation Methods Group (Noyes et al., 2018a, 2018b); EPOC, the Cochrane Handbook and PRISMA (Page et al., 2021).

Stage 3: Developing the Template

Pre-Meeting Activities: Extracting and Assessing the Contents of Existing Syntheses

To prepare for our developmental work, we extracted data from our current qualitative evidence syntheses. At this stage, EPOC was managing ten qualitative evidence syntheses at different stages of the editorial process (Glenton et al., 2013; Munabi-Babigumira et al., 2017; Karimi-Shahanjarini et al., 2019; Odendaal et al., 2020; Xyrichis et al., 2021; Cooper et al., 2019a; Cooper et al., 2019b; Moloi et al., 2020; Bohren et al., 2019; Downe et al., 2019b), and members of the editorial team were also co-authoring two qualitative evidence syntheses managed by the Cochrane Consumers and Communication Group (Ames et al., 2017; Ames et al., 2019a). One EPOC editor (CG) extracted the content from these 12 published and unpublished protocols and reviews and organised this content according to the different sections of a qualitative evidence synthesis (i.e. title, abstract, methods section, results section, etc.).

Face-to-Face Consensus Meeting

The core team then had one face-to-face meeting and several teleconferences to assess the contents of the reviews, section by section; consider feedback from review authors, review commissioners and other stakeholders; as well as comments from peer reviewers to individual reviews; and to discuss what we considered to be acceptable solutions for each section. EPOC’s information specialist (MJ) also provided input throughout this process, especially with regard to the guidance on search methods.

When attempting to reach consensus, we aimed to ensure that the template followed the theoretical and philosophical principles of qualitative research (Denzin & YSe, 2011). We also aimed to ensure that the template was in line with existing guidance provided by EPOC, the Cochrane Qualitative and Implementation Methods Group, the Cochrane Handbook and other appropriate sources.

Writing Up and Piloting the Template

The core team, with help from EPOC’s information specialist, drafted a template, as well as complementary guidance documents for sections of the review that we thought required more detailed information.

Two of the twelve syntheses on which we based our template had been published when we started this work, while ten were still ongoing. The EPOC editors and managing editor sent the first draft of the template to EPOC review authors that were currently working on their qualitative evidence syntheses and asked them to send feedback. The EPOC editors and managing editor also tested these early versions on protocols and reviews as part of the editorial process and kept notes of areas that required further work. The template was also sent to other relevant stakeholders within Cochrane for feedback.

One EPOC editor (CG) then used this feedback to update the template. Version 1 of the template was then agreed upon by the core group, as well as by the managing editor and information specialist.

Stage 4: Updating the Template

Version 1 of EPOC’s qualitative evidence synthesis template was based on our experiences with twelve qualitative evidence syntheses: ten from Cochrane EPOC and two from Cochrane Consumers and Communication and co-authored by EPOC editors. In 2019, we developed a Version 2 of the template in response to experiences with two mixed-methods reviews and a rapid qualitative evidence synthesis published by Cochrane EPOC (Vasudevan et al., 2018; Agarwal et al., 2020; Houghton et al., 2020). In addition, the editorial team kept a running list of issues that we had identified when using the template, including where clarifications or additional information were needed. The editorial team continues to gather this information to inform the next version of the template.

The Current Template

An overview of EPOC’s qualitative evidence synthesis template can be seen in Figure 1. The latest version of the full template is freely available on EPOC’s website (https://bit.ly/3vgPcAg). The template includes the following:

Suggested subheadings for each section of the review.

An explanation of the type of content each section should include.

Where appropriate, proposals for standardised text.

Links or references to additional information (for instance, other Cochrane resources).

Appendices with examples of how each section could be written.

Figure 1.

Overview of EPOC’s qualitative evidence synthesis template structure.

The explanations and examples we have included have focused on topics that have received less attention in other sources of guidance (Figure 1). These include the following topics:

Preparing abstracts and plain language summaries for qualitative evidence syntheses.

Including and translating studies in multiple languages.

Approaches for study sampling.

Considering review author reflexivity.

Developing ‘Implications for practice’ sections.

Developing ‘Implications for research’ sections.

Preparing declarations of interest statements.

We discuss each of these topics further in a forthcoming study (Glenton et al., 2020).

In addition, EPOC has developed separate but complementary guidance on how to prepare a plain language summary for an EPOC qualitative evidence synthesis (https://bit.ly/36oT6dT) and how to sample studies (https://bit.ly/3gEX2gJ).

Overarching Issues when Developing the Template

A key topic of discussion as we developed the template concerned the amount of guidance it was appropriate to give. We were cautious of being overly prescriptive and wanted to ensure that we would continue to learn from new review teams bringing methodological insights and using different synthesis approaches. At the same time, we wanted to develop a review structure and content that would encourage good practice and that would make the work of review authors, editors and end users easier.

We decided to offer suggestions of standardised text in situations where we wanted to encourage specific methodological practices or reporting standards. This included parts of the methods section, specifically descriptions of how review authors had searched for, selected, translated and sampled studies; how they assessed any methodological limitations of studies; and how they assessed confidence in the review findings. In other sections where we wanted to encourage flexibility, we avoided standardised text; for instance, in the sub-section on data analysis and synthesis. We also included appendices with examples, particularly where we believed that our reviews offered practical suggestions that might not be available in other sources of guidance; for instance, in the sections on reflexivity and implications for practice.

Topics for Future Development

In addition to these topics, our template covers elements of a qualitative evidence synthesis that have been discussed elsewhere, including in guidance provided by the Cochrane Qualitative and Implementation Methods Group (Noyes et al., 2018a). While these topics have received more attention, they are also likely to benefit from further worked examples, including examples using a broader range of qualitative evidence synthesis methods, and prompts for review teams to consider how to operationalise methodological guidance. For instance, we need more knowledge about how best to assess the methodological limitations of qualitative studies in the context of a qualitative evidence synthesis (Munthe-Kaas et al., 2019; Munthe-Kaas et al., 2018; Noyes et al., 2018b); the extent to which ‘dissemination bias’ operates within qualitative research and how this may influence the type of studies we search for (Harris et al., 2018; Toews et al., 2016; Toews et al., 2017); how best to integrate findings from qualitative evidence syntheses with findings from related effectiveness reviews (Harden et al., 2018); and how to apply GRADE-CERQual in the context of more interpretive findings (Lewin et al., 2018b).

We expect the template to continue to improve as we gain more experience. Issues that we hope to work on in future version of the template include the following: processes for translating studies written in multiple languages; approaches to summarising review findings in abstracts and plain language summaries; and processes for applying and reporting reflexivity in the context of qualitative evidence syntheses. In reviews that have used sampling strategies and have findings that we assessed as being of low or very low confidence, we also plan to explore how review authors can utilise studies that were eligible but not originally sampled to address this (Ames et al., 2019b).

Usefulness of the Template for Future Authors of Qualitative Evidence Syntheses

One of our key motivations when developing the template was to offer new review authors the opportunity to learn from other review authors’ experiences and to ensure that editors do not have to repeatedly point to the same gaps or flaws. EPOC now routinely offers the template to new review authors and has also made it freely available to other authors of qualitative evidence syntheses within and outside Cochrane.

By drawing on a series of worked examples of qualitative evidence syntheses, we have been able to address gaps in current guidance and translate existing guidance into practical advice. The template offers review authors a review structure, describes content, provides examples and suggests standardised text. This type of practical support to review authors has previously proved very helpful to both editors and authors in other situations. For instance, Cochrane Consumers and Communication has developed a template and practical guidance for review authors of complex communication reviews (Ryan & Hill, 2019). According to the Group’s editors, the review template has ‘greatly improved the transparency of reporting’ in their reviews, and the use of standardised wording for authors to adapt has led to ‘major improvements in the coherence and comprehensiveness of the description of methods as a whole’ (Ryan & Hill, 2019). The editorial team is now spending less time querying and correcting review authors’ descriptions of methods and results, and now have more time to focus on conceptual coherence and clarity, and on ‘supporting authors in interpreting the meaning of findings and in developing consistent messages about these’ (Ryan & Hill, 2019). EPOC editors have had similar experiences. The template has also provided welcome guidance to review authors embarking on their first qualitative evidence synthesis and has provided more experienced review authors with information about Cochrane’s specific expectations. Most recently, the template helped support authors of a rapid qualitative evidence synthesis prepared as part of Cochrane’s response to the COVID-19 pandemic by providing standardised text that could be adapted rapidly (Houghton et al., 2020). The success of the template lies partly in striking a balance between instruction and flexibility, so that qualitative evidence synthesis authors can be guided, but not constricted in the development of their reviews. We believe that this resource will be helpful for all researchers developing and reporting qualitative evidence syntheses.

Conclusion

The qualitative evidence syntheses that Cochrane EPOC has been involved in represent a case series that has allowed us to learn more about how to undertake a qualitative evidence synthesis in practice. This work has helped to create a bridge between the existing conceptual guidance on conducting qualitative evidence syntheses and the practical steps in implementing this. A good understanding of the principles underlying qualitative research is necessary to conduct a qualitative evidence synthesis well. This template can support people who have this background but who are novice review authors or review authors new to Cochrane. We have consolidated the lessons we have learnt in a template designed for future authors of EPOC qualitative evidence syntheses. We also expect the template and guidance to be useful to review teams conducting qualitative evidence syntheses in other contexts and believe that it can improve the conduct and reporting of these reviews, including their transparency and consistency.