Can a Meta-ethnography Be Updated by Different Reviewers? Reflections From a Recent Update

Abstract

Over the last decade, there has been a proliferation of published meta-ethnographies. Yet, strategies and techniques for updating have not received the same attention, rendering answers to important methodological questions still elusive. One such question has to do with who can perform an update. Although it is not uncommon for quantitative systematic reviews and statistical meta-analyses to be updated by different reviewers, qualitative synthesists might find themselves caught between a rock and a hard place. On the one hand, as meta-ethnography constitutes an interpretation three times removed from the lived experience of the participants in the original studies, it could be argued that an update by different reviewers might add an extra layer of interpretation. By comparison, updating by the same reviewers could give rise to concerns about the robustness of updated findings, as an implicit drive for making new data fit the original work might be difficult to control for. We recently reported the findings of our attempt to update an earlier meta-ethnography of primary care antibiotic prescribing, conducted by a different team of reviewers. In this article, we wish to contribute to the emerging debate on the necessity of promoting a culture of updating in qualitative evidence synthesis, by discussing some of the practical and methodological issues we considered at each stage of the process and offering lessons learnt from our experience.

Introduction

Updating – typically defined as ‘a process aiming to identify new evidence to incorporate into a previously completed systematic review’ (Moher & Tsertsvadze, 2006) or, more recently, as ‘a new edition of a published systematic review with changes that can include new data, new methods, or new analyses to the previous edition’ (Garner et al., 2016) – has become common practice for quantitative systematic reviews and statistical meta-analyses. Since its inception in 1995, the Cochrane Database of Systematic Reviews established as a principle that all reviews registered should be kept up to date, due to the recognition that research findings have expiration dates and need to be regularly revisited. Several organisations, including Cochrane, have recommended updating every 2 years, while there is also evidence to suggest that a considerable number of clinically relevant reviews may become obsolete within 1 year of publication or even less (Shojania et al., 2007). Clearly, the question of whether updating is needed has been adequately addressed in the field of quantitative research, and emerging issues currently include the frequency with which reviews should be updated and the development of surveillance systems to identify potentially out-of-date reviews (Ahmadzai et al., 2013; Bashir et al., 2018; Garner et al., 2016; Garritty et al., 2010; Soll, 2008; Tugwell et al., 2011).

Methods for systematically reviewing and synthesising published qualitative health research emerged in the mid-1990s, in response to an increasing demand from policy makers, guideline developers, and health care professionals for review evidence that goes beyond ‘what works’, to provide an understanding of the experiences, perceptions and beliefs of those directly involved, as well as contextual factors that may affect implementation (Flemming & Noyes, 2021). In 2013, Cochrane published its first qualitative synthesis, the results of which proved critical in the formulation of the World Health Organization’s OptimizeMNH guidance recommendations (Gulmezoglu et al., 2013). Despite concerns that the process of synthesising multiple qualitative studies might diminish the depth, complexity and richness of individual works (Thorne, 2016), the results of published qualitative syntheses are now increasingly being used to guide the development of clinical practice guidelines (Wang et al., 2020), pointing towards the usefulness of such endeavours when rigorously conducted. There are now over 30 different approaches falling under the umbrella term of ‘qualitative evidence synthesis’ (e.g., thematic synthesis, framework synthesis, meta-study, meta-summary, meta-ethnography and critical interpretive synthesis), and although these methods have evolved considerably over the last decade, some have been subject to more development and testing than others (Flemming & Noyes, 2021). Meta-ethnography, for instance, initially developed by sociologists in education Noblit and Hare (1988) and often viewed as the qualitative equivalent to meta-analysis, currently constitutes one of the most frequently used and best developed methods for synthesising published qualitative health research (France et al., 2019). A key element of the approach is that, rather than simply aggregating data, it uses comparative understanding to generate a theory which could have greater explanatory power than the individual studies.

Paradoxically enough, however, similar methodological interest in the updating of qualitative syntheses has not yet taken off, raising questions about whether existing findings are up to date. Although examples of updated meta-ethnographies can be traced back to the late 2000s, published updates are still rare (Daker-White et al., 2014; Lang et al., 2013; McCann et al., 2013; McInnes & Chambers, 2008; Noyes & Popay, 2007; Rodriguez-Prat et al., 2017), whereas the first discussion of why, when and how to perform an update came only in 2016. In a methodological paper, France et al. (2016) argued that ‘since beliefs, experiences, health care contexts and social phenomena change over time, the continued relevance of the findings from meta-ethnographies cannot be assumed’. Building on the scarce available evidence, as well as their own experiences and reflections on the topic, they went on to propose three possible methods of updating, using the analogy of rebuilding a house: (a) ‘Extending and renovating the original house’ (i.e., adding to and revising the original meta-ethnography to incorporate new articles); (b) ‘Building a new house next door to the original and comparing the two houses’ (i.e., doing a new, stand-alone synthesis of the new articles and, subsequently, comparing new findings to the earlier ones) and (c) ‘Knocking down the house and rebuilding it’ (i.e., starting the analysis again from scratch and creating a single overarching synthesis that incorporates both old and new articles).

The number of published meta-ethnographies is increasing rapidly. A search on Web of Science, using the term ‘meta-ethnograph*’ as topic, yielded 890 references, out of which 125 were published in 2020, as opposed to 26 in 2010. This not only renders the development and/or further refinement of updating techniques indispensable, but also brings to the forefront questions about who is most appropriate to plan and conduct an update. Although it is not uncommon for quantitative systematic reviews and meta-analyses to be updated by different reviewers, qualitative synthesists might find themselves caught between a rock and a hard place. On the one hand, as meta-ethnography constitutes an interpretation three times removed from the lived experience of the participants in the original studies, given that meta-ethnographers seek to give their own interpretation of original authors’ interpretations of research participants’ accounts (Sandelowski & Barrosso, 2007), it could be argued that an update by reviewers who were not involved in the original work might add an extra layer of interpretation. By comparison, updating by the same reviewers could give rise to concerns about the robustness of updated findings, as an implicit drive for making new data fit the original work might be difficult to control for. McCann et al. (2013), for instance, justified their decision to conduct a new, stand-alone meta-ethnography and subsequently compare its findings with the ones generated by their previous synthesis, on the grounds of addressing reasonable concerns about ‘forcing a fit’.

In a recent paper (Germeni et al., 2018), we reported the findings of our attempt to update an earlier meta-ethnography of primary care antibiotic prescribing, and argued that in the same way that updated meta-analyses can inform about whether health care interventions continue to be safe and effective, updated qualitative syntheses can provide evidence on whether these remain relevant to individuals’ changing needs, values and experiences. Here, we wish to expand on the methodological discussion around the updating of qualitative syntheses, by providing a reflexive account of the process we followed. As this was the first time that an update of an interpretative synthesis was carried out by a different team of reviewers, we discuss major challenges and options faced at each stage of the process and offer lessons learnt from our experience.

Practical and Methodological Considerations

Our process of updating comprised six key stages: (1) Identifying an out-of-date qualitative synthesis; (2) Determining the role of the original team; (3) Deciding on the aim of the updated review; (4) Updating the original search strategy and selection criteria; (5) Critically appraising and synthesising the studies and (6) Reflecting on the value of the update (Figure 1). Practical and methodological issues we considered at each stage are discussed in detail below.

Figure 1.

Key stages of updating a systematic review and synthesis of qualitative evidence.

Stage 1: Identifying an Out-of-Date Qualitative Synthesis

Our team consisted of a psychologist, three medical sociologists, a general practitioner (GP) and an information specialist. Most of us had considerable experience in conducting systematic reviews and syntheses of qualitative evidence, and one of the members was a co-convenor for the Cochrane Qualitative and Implementation Methods group. Starting from different perspectives, we were all broadly interested in the topic of antimicrobial resistance. Most importantly, we considered that there have been several recent developments (e.g., the joint Declaration on Combating Antimicrobial Resistance, signed in 2016 at the World Economic Forum in Davos), which may have impacted research in the field. Therefore, we set out to search for any published qualitative reviews on the topic and were able to locate three: The first (Teixeira Rodrigues et al., 2013) was a systematic review of published qualitative literature on physicians' perceptions of factors influencing antibiotic prescribing, but the authors had made no attempt to synthesise identified findings. The second (Fleming et al., 2015) was a systematic review and thematic synthesis of factors influencing antibiotic prescribing in long-term care facilities; however, it was published in 2015, just a year before we started our own work. The third (Tonkin-Crine et al., 2011), which was the one that we eventually chose to update, ticked several boxes: (a) it focused on GPs’ attitudes and experiences of antibiotic prescribing for acute respiratory tract infections (ARTIs), including their views of interventions aimed at more prudent prescribing; (b) the authors had systematically reviewed relevant literature and had synthesised available qualitative findings using the technique of meta-ethnography and (c) the paper was published in 2011, meaning that the volume of studies published within the five-year period 2011–2016 was likely to be substantial enough to warrant an update.

Stage 2: Determining the Role of the Original Team

Recognising that the process of synthesising qualitative research is essentially interpretative, a crucial question that we had to confront early in the process was: Can different reviewers perform an update? And, if so, how do we ensure that differences in conclusions reached are in fact due to different findings, and not due to differences in interpretations between the original and the new team? In their seminal paper on inter-rater reliability in qualitative research, Armstrong et al. (1997) showed that different researchers do not necessarily share divergent interpretations of the same raw material; rather, there seems to be a concordance, at least at a level of situating themes within a wider framework. We concluded that updating by different reviewers – though involving somehow members of the original team – might not just represent a compromise solution, but could also prove an efficient way to move forward. Issues that need to be considered in this case focus primarily on the level of the original reviewers’ involvement, mainly associated with the stage of the process at which they will be consulted and the degree of input they will be asked to provide (Figure 2). It could be, for example, that members of the initial team collaborate with the new team throughout all stages of the updating process. This approach was recently adopted in an update exploring patients’ experiences of the wish to hasten death; two authors of the original meta-ethnography joined the new team, and this triangulation of researchers was claimed to have ‘injected a fresh perspective’ (Rodriguez-Prat et al., 2017). We wanted to explore whether a one-off consultation would be useful, and only contacted the lead author of the original work when our synthesis was well advanced. We found this approach worthwhile, as it allowed us to quickly gather information on the accurate essence and nuances of the original work, while exploring the perspectives of the two teams.

Figure 2.

The 3 Cs of involvement: Ways for involving original reviewers in the updating of meta-ethnographies.

Stage 3: Deciding on the Aim of the Updated Review

We started describing our process of updating, by reporting our disciplinary backgrounds. Equally important, however, seems to be a consideration of our cultural backgrounds. Four of our team members were of British origin (same as the original team); yet, two came from Southern European countries (Greece and Spain) and were well aware that over-the-counter sale of antibiotics is still a common phenomenon in many parts of the world and, as such, cannot be overlooked in a review seeking to address the global dimensions of the problem. Furthermore, the fact that we came from three different European countries meant that we had all experienced very diverse primary care systems and were conscious that, although GPs are typically the main primary care actors, various other disciplines are involved in the delivery of primary care and their role may vary widely across countries. We, therefore, decided to expand the original review question to encompass, not only GPs, but all primary care professionals who may prescribe or dispense antibiotics for ARTIs (e.g., nurses and pharmacists). Our updated review question was thus formulated as follows: What are primary care professionals’ attitudes and experiences of antibiotic prescribing/dispensing for ARTIs and what are their views of interventions aimed at more prudent prescribing/dispensing?

Stage 4: Updating the Original Search Strategy and Selection Criteria

In practice, expanding the original review question meant two things: (a) that we would need to re-design and re-run the original search strategy and selection criteria; and (b) that we would have to start the analysis again from scratch, thus creating a single overarching synthesis that would incorporate both old and new articles. In France et al.’s (2016) words, this would be equivalent to ‘knocking down the house and rebuilding it’, an approach to updating that has not been previously used, but was proposed by the authors as potentially useful in cases where a different team of reviewers conducts the update. We therefore went on to revise the original search strategy to reflect the aims of the updated review. Specifically, the original database search used only a subject heading for ‘general practitioner’ for the population of interest. For our update search, we expanded the list of terms to include nurses and pharmacists, who can also prescribe or dispense antibiotics in some contexts. Furthermore, while the original review used ‘prescri*’ as a search term for prescribing, we also added ‘overprescribing’, ‘dispensing’, ‘responsible use’, ‘over the counter’, ‘non prescription’ and ‘self medicat*’, to identify literature related to antibiotic dispensing. We searched the same five electronic databases as the original team (with the addition of ASSIA to capture studies with a social science leaning), applying no date restrictions. As we wanted to capture more of a global context, we also applied no language restrictions, although authors of other meta-ethnographies have advocated for the inclusion of only English language publications, to prevent cultural and linguistic bias in translations (Tong et al., 2008). Our search yielded 507 unique citations, out of which 53 met our inclusion criteria. Of those 53 studies, more than half (28) were published after the publication of the original meta-ethnography, justifying our decision to conduct an update. All of the original review studies were identified and included in the update.

Stage 5: Critically Appraising and Synthesising the Studies

Only a small number of published meta-ethnographies have included more than 40 studies (Toye et al., 2014), and the challenges of critically appraising and synthesising a larger number of papers can be easily understood. The authors of the original meta-ethnography applied the Critical Appraisal Skills Programme (CASP) checklist to assess the quality of the 12 studies selected for inclusion. Yet, we felt that applying the CASP on 53 studies would be a resource-intensive exercise, which might provide us with important information about their reporting quality, but would not help us evaluate them in terms of their interpretive merit. Instead, drawing on the categorisation proposed by Dixon-Woods et al. (2007), we classified studies based on two criteria: (a) their relevance to the review objectives; and (b) their analytic/explanatory power. We had already excluded irrelevant and fatally flawed ¹ studies during full-text screening, so the primary goal of this exercise was to enable in-depth familiarisation with available material rather than make exclusion decisions. In an attempt to make the work more manageable, however, we did consider synthesising only ‘key’ papers, that is, papers classified as likely to make an important contribution due to their high relevance and high analytic/explanatory power. Yet, we soon abandoned this plan, as immersion in the primary studies made us realise that even less conceptually rich papers could provide important contextual information. As a result, we kept all papers in the synthesis, but went on to group them according to their thematic focus (usual care vs. intervention studies) and perform two separate line-of-argument syntheses. The first synthesis resulted in the development of a conceptual model of antibiotic prescribing and dispensing, showing how primary care professionals may choose to present themselves differently in the context of ARTI consultations (the expert self, the benevolent self, and the practical self), depending on the range of intrapersonal, interpersonal and contextual situations in which they find themselves. The second synthesis led to the development of a typology of ARTI intervention acceptance, depicting four possible ways in which primary care professionals may experience quality improvement interventions (compromise, ‘supportive aids’, source of distress and unnecessary). Thus, using France et al.’s (2016) analogy of house building, our process of updating did not exactly result in ‘knocking down the house and rebuilding it’, but rather in ‘knocking down the house and building two new houses’. In our view, this indicates the wide range of possibilities that might exist when updating a qualitative synthesis.

Stage 6: Reflecting on the Value of the Update

The final stage of our process involved reflecting on what the updating achieved. For this purpose, we created a table juxtaposing the key characteristics of the original and updated meta-ethnographies (Table 1). In quantitative systematic reviews and meta-analyses, it is often the case that the updating does not substantively alter the original findings; nonetheless, increasing the number of included studies can improve precision of effect estimates, demonstrate wider applicability of the effect, or allow additional comparisons and subgroup analyses to be performed (Cumpston & Chandler, 2021). Likewise, in our case, including 41 additional studies enabled us to perform two separate syntheses, one offering valuable insights into the factors that may lead to inappropriate prescribing in usual care, and the other exploring primary care professionals’ experiences of interventions aimed at more prudent prescribing. Although the results of our usual care synthesis were similar to the findings of the original meta-ethnography, inclusion of more studies added depth to concepts already identified and led to a more nuanced understanding of the phenomenon. Table 2 shows an example of this. Moreover, inclusion of recent evidence on primary care professionals’ experiences of interventions allowed us to draw important conclusions about the possible ways that these may be employed. In contrast to the original meta-ethnography, which identified five key aspects that interventions should incorporate to maximise acceptability, we found that one-size-fits-all approaches are doomed to result in variable uptake, as different professionals experience the same elements in a very different way. We concluded that acceptability of interventions is likely to increase if these are context-sensitive and take into account primary care professionals’ varying roles and changing priorities.

Table 1.

Overview of original and updated meta-ethnographies.

	Original meta-ethnography	Updated meta-ethnography
Aim	To examine GPs’ attitudes and experiences of antibiotic prescribing and interventions aimed at more prudent prescribing for ARTIs	To examine PCPs’ attitudes and experiences of antibiotic prescribing or dispensing and interventions aimed at more prudent prescribing/dispensing for ARTIs
Search strategy	5 databases (MEDLINE, EMBASE, PsycINFO, CINAHL, Web of Science) were searched from inception to May 31, 2011	6 databases (MEDLINE, EMBASE, PsycINFO, CINAHL, Web of Science, ASSIA) were searched from inception to June 29, 2016
No of included papers	12	53
Quality assessment	The CASP checklist was used to assess quality of included studies	Studies were evaluated on the basis of their analytic/explanatory power and relevance to the review objectives
Synthesis approach	Meta-ethnography	Meta-ethnography
Synthetic products	An explanatory model presenting how GPs perceive antibiotic prescribing and interventions aimed at more prudent prescribing	A conceptual model of antibiotic prescribing and dispensing in primary care
		A typology of ARTI intervention acceptance
Implications for practice	To maximize acceptability, interventions aimed at more prudent prescribing for ARTIs should incorporate five aspects: (1) allow GPs to reflect on their own prescribing; (2) help decrease uncertainty about appropriate ARTI management; (3) educate GPs about appropriate prescribing; (4) facilitate more patient-centred care and (5) be beneficial to implement in practice	One-size-fits-all approaches are doomed to result in variable uptake, as different professionals experience the same elements in a very different way
		Acceptability of interventions is likely to increase if these are context-sensitive and take into account PCPs’ varying roles and changing priorities

Note. GPs’ = general practitioners’; PCPs’ = primary care professionals’; ARTIs = acute respiratory tract infections; CASP = critical appraisal skills programme.

Table 2.

Example of synthesised concepts from original and updated meta-ethnographies.

Original meta-ethnography
Final synthesised concept	Translated concepts	Summary definition of translated concepts
Perceptions of potential conflict with patients	GPs’ view of patient expectations	GPs’ perception of patient expectations and experiences of patient demand for antibiotics.
	GPs’ view of patient satisfaction	GPs’ opinion of the importance of patient satisfaction and the perception of whether patients will go elsewhere for treatment.
Perceptions of how to provide patient-centred care	GPs’ view of how they can help their patients	GPs’ perception of what they personally can do to help their patients and what types of help they can offer (e.g., emotional, practical).
	GPs’ view of providing explanations to patients	GPs’ perceptions of how easy or difficult it is to explain prescribing decisions to patients.
	Decision making may be influenced by patients’ concern	GPs’ prescribing decisions may be influenced by a patient’s fear or concerns about their illness.
	Decision making may be influenced by patients’ non-clinical context	GPs’ prescribing decisions may be influenced by the patient’s work, social life or socio-economic background in addition to their illness.
Updated meta-ethnography
Final synthesised concept	Translated concepts	Summary definition of translated concepts
The benevolent self	Dissatisfaction in not meeting patient expectations	Many primary care professionals feel that once a patient makes the effort to come into the clinic, it is unsatisfying not to be able to offer a solution. Concerns of being perceived as ‘having done nothing’ for the patients or not being ‘proper doctors’ if they do not prescribe antibiotics are common.
	Desire to avoid conflict and maintain a good relationship with patients	Building and maintaining a good relationship with their patients is viewed as a priority for health care professionals working in primary care and several admit that they would not jeopardise this ‘for the sake of a prescription for penicillin V’.
	Beneficence/non-maleficence	Primary care professionals justify their prescribing decisions on the basis of a desire to do their best for the patients. Although some report prioritizing potential resistance problems and longer-term issues, the majority feels that their priority should be ‘the patient in front of them’ and his/her immediate needs. The desire to ‘help’ the patient is not restricted to treating a patient that is ill, but involves a broader consideration of the circumstances in an individual’s life, such as the environment in which he/she lives, his/her socio-economic status or vulnerability on the job market, as well as plans for leisure activities.

Note. GPs’ = general practitioners’.

Conclusions

The importance of regularly updating quantitative systematic reviews and statistical meta-analyses is now well established. As science evolves with the accumulation of new evidence, health care interventions previously considered to be safe and effective may in the future prove to be harmful or ineffective, and vice-versa. Likewise, ignoring changes in individuals’ preferences, needs and values, or changes in health care contexts, could undermine the applicability of qualitative syntheses. In this article, we have sought to contribute to the emerging debate on the necessity of promoting a culture of updating in qualitative evidence synthesis, by discussing some of the practical and methodological issues we considered when updating an earlier meta-ethnography, conducted by a different team of reviewers. As updating by the same reviewers might not always be possible for a variety of reasons, such as resource constraints, dispersal of teams or lack of motivation, we have demonstrated that different reviewers can successfully perform an update of an interpretative synthesis, provided that they involve somehow members of the original team. We have also provided a framework for involvement, in the hope that it will serve as a useful guide for research teams embarking on similar endeavours.

In conducting this update, we encountered several challenges that we have sought to document here. Most importantly, this was a labour-intensive process that took almost a year to complete, with a researcher (the lead author) working full-time on the project. If the original team had performed the update, it is very likely that this would have been completed in a shorter timeframe. Indeed, although theory development is often cited as one of meta-ethnography’s key strengths, this also means that the outcome of a meta-ethnography is much more complex than other forms of evidence synthesis and, as such, additional findings cannot simply be ‘added’ to earlier ones in the same way as new data may be fed into an existing statistical meta-analysis. We had to spend a considerable amount of time to get to grips with the particular ‘whats’ and ‘hows’ of the original work and, although contacting the initial team proved very helpful, ways for minimising the time needed to complete an update (e.g., using machine learning approaches to aid with the screening process) should be more carefully examined in the future.

Metrics for citations, and article accesses and downloads, are often used as proxy or indicators for the currency and relevance of a review (Garner et al., 2016). We believe that the same could apply for determining the value of an update. Our updated meta-ethnography was published 3 years ago and has received so far more than 25 citations, whereas it has been accessed around 2000 times. Soon after its publication, it was also chosen to be summarised as a Signal by the National Institute for Health Research (NIHR) Dissemination Centre, due to its relevance to decision makers. As in qualitative research the updating debate is now starting to unfold, evidence like this might be useful to convince the sceptics.