An exploration of analytical tools to conduct a realist synthesis and demonstrate programme theory development: An example from a realist review

Abstract

Introduction

A realist literature review involves iterative processes, with searches, appraisal and synthesis occurring simultaneously. Whilst this lends itself to theory development, the synthesis process often lacks transparency. This has led to unanticipated challenges for novice realist researchers, particularly PhD candidates.

Methods

The aim of this paper is to contribute to the realist methodological knowledge base by outlining the analytical tools of coding, consolidating and conceptual mapping used within a realist review. Specifically, how these techniques aid the synthesis process and demonstrate the development of valid and evidence informed programme theories.

Results

A worked example is provided to illustrate how; (i) coding techniques using realist logic can evidence a rigorous synthesis process; (ii) the use of consolidating techniques facilitates data management and aids theory development; and (iii) conceptual mapping demonstrates programme theory development.

Conclusions

Recommendations for novice realist researchers include defining and documenting the analytical tools for conducting a rigorous realist synthesis to provide transparency about how valid programme theories were developed. Plus, theory development checks can be built into the structure of the review process.

Keywords

Realist synthesis analysis techniques realist review

Introduction

A realist literature review offers an alternative to a traditional systematic review that is theory-driven and gives value to stakeholder knowledge.¹ The World Health Organisation and UK Medical Research Council acknowledge the utility of the approach within health research.^2,3 As such, the realist approach is becoming widely used within health research by both experienced and novice researchers, however Pawson et al.⁴ warns that realist research could prove challenging for novice researchers. Challenges reported by doctoral students,⁵ and examiners alike⁶ relate to demonstrating the synthesis process and programme theory development. There are only a handful of published examples of a realist synthesis process, none of which offered a “perfect fit” for the current review. As such, the analytical tools of coding, consolidation and conceptual mapping were utilised to conduct a realist synthesis within a realist literature review⁷. This paper presents the authors reflections on an alternative application of these analytical tools and provides details of how they can also demonstrate programme theory development. The approaches outlined within this paper were developed through trial and error for the needs of a specific realist review, therefore this does not offer a ‘how to’ guide on conducting a realist synthesis. However, it is hoped that novice realist researchers find the authors’ reflections useful for documenting programme theory development.

To understand the application of analytical tools within a realist synthesis, it is important to review the key features of realist research including (1) a focus on ontology, (2) the use of realist logic to generate causal explanations, and (3) retroductive reasoning. Firstly, realist research focuses on exploring ontological depth, whereby reality is understood as consisting of three interacting layers: the empirical (reality i.e. observable), the actual (reality i.e. generated through the activation of mechanisms) and the real (reality i.e. unobservable in which latent mechanisms exist).^8,9 For example, the application of critical realism within disability research argues that the experience of disability has a biological basis (empirical), but that ableist social structures cause impairments such as stigma increasing isolation (the actual), resulting in psychological distress at an individual level (the real).¹⁰ It is argued that the physiological experience of disability is inextricably linked to social and psychological experiences of illness, and so all layers of reality need to be determined when researching within health.¹⁰ As such, identifying interactions between different layers of reality enables the researcher to go beyond the descriptive and develop theory to explain how a programme operates within specific contexts.¹¹ A recent paper by Wiltshire & Ronkainen¹² proposes a critical realist thematic analysis procedure that develops themes based on ontological layers. Whilst this approach offers potential to demonstrate programme theory development, realist logic was not used to articulate the theory development process, which is an essential feature of the realist research approach established by Pawson & Tilley.¹

The second key feature of realist research is the use of realist logic, which underpins the methodological approach established by Pawson & Tilley.¹ Realist logic articulates causal explanations as a relationship between context, mechanism and outcome (CMO). A context refers to the conditions in which an intervention operates and is stratified across social structures from cultural norms to the social networks of an individual.¹³ A mechanism is the generative force that produces an outcome, and can be understood as a resource provided by an intervention, to which actors respond.¹⁴ Identifying mechanisms provides the explanatory insights and enables theory development.¹³ The outcome tends to refer to the phenomenon under review and may be intentional or not. Within a realist synthesis, identifying these elements and how they interact establishes CMO configurations. Therefore, CMO configurations reflect interacting layers of reality and provide explanations of how and why mechanisms are activated within a particular context, to produce the outcomes under review.^11,15,16 CMO configurations are combined to develop a programme theory – a theory to explain how an intervention works.^11,16

Unearthing CMOs within different layers of reality requires retroductive reasoning, the third key feature of the approach. Retroductive reasoning is a type of inference that goes beyond inductive and deductive thinking to ‘theorise and test hidden mechanisms,’¹³ p. 122). Retroduction involves abductive thinking whereby data or knowledge outside of a theoretical framework is incorporated into the synthesis.^8,13,17,18 Jagosh et al.¹³ notes that abductive thinking is a ‘hunch-driven’ approach that utilises creativity combined with common sense to theorise on contextual features and potential mechanisms that are not explicit. For example, Jagosh et al.¹³ reflected that once an outcome has been identified, researchers can draw on both the data, and their own knowledge on the topic, to make inferences about related context and mechanisms. Therefore, retroductive inference enables the identification of both visible and hidden mechanisms that are involved in generating outcomes, and the specific contexts in which they activate.^8,13,19 It is worth noting that the iterative nature of realist research requires inferences to be constantly tested against the data ensuring any CMOs abductively generated are ultimately evidence informed.¹¹

The use of realist methodology to examine complex social interventions has increased within health and social sciences over the past two decades.^13,15,20,21 A realist literature review is one such method and is a distinct type of systematic review that synthesises literature using realist logic.¹⁵ The aim of a realist review is to answer questions about ‘how, why, for whom and in what context does an intervention work’?’ which is achieved through developing, testing and refining programme theory.^11,15,16 As such, conducting a realist review is an iterative process, with searches, appraisal selection and synthesis processes overlapping and informing each stage of the review.^11,20 There are several key texts that provide guidance on strategies for conducting a realist review, in addition to publication standards developed through the Realist And Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) project.^{11,15,16,19,22,23} The synthesis process is described as finding and aligning evidence using realist logic to make sense of the different contributions of the data and refine theory.^15,19 However, there are a lack of published examples of the analytical tools used to conduct a realist synthesis that demonstrates the process of programme theory development.^13,17,21,23 The lack of guidance presents an unintended challenge for the novice researcher who face difficulties demonstrating programme theory development.^5,6 Such difficulties are compounded by the large volume of complex data that makes the initial stages of a synthesis ‘messy.’^1,22 One proposed solution is to document reflections to illustrate the researcher’s decision making processes.^17,24 This paper proposes an alternative solution through the application of the analytical tools of coding, consolidating and conceptual mapping. These analytical tools can be utilised for both conducting realist synthesis and, through their documentation, evidence programme theory development. Examples of this process will be drawn from a realist review conducted in 2021 by Peters et al.⁷ to demonstrate how; (i) coding techniques using realist logic can evidence a rigorous synthesis process; (ii) the use of consolidating techniques facilitates data management and aids theory development; and (iii) conceptual mapping shows programme theory development.

Review background

A realist review was conducted between January-September 2021 to answer the research question, how, why and in what context do community-based arts activities enable the identity change recovery process from serious mental illness?⁷. A brief overview of the review process is provided here, however full details of the review protocol are available via the University of Huddersfield²⁵. The current realist review was conducted using four search cycles: background; programme theories; empirical evidence; and middle-range theory searches²² resulting in 42 searches on nine databases, plus citation searches and literature suggestions. A total of 9034 hits were generated throughout the review (after removing duplicates where possible), resulting in a final 22 reports and articles contributing to the synthesis. Figure 1 outlines how the search, appraisal, and synthesis processes were iteratively conducted throughout the review. This diagram also highlights the process of theory development within the synthesis column, from (1) an initial programme theory; (2) to five programme theories; (3) of which three were selected to focus the review; (4) resulting in one overarching programme theory. A worked example of the analysis and synthesis is provided in the supplementary materials, extracts of which are included within this article.

Figure 1.

A task-and-time diagram outlining the realist review and synthesis process (adapted from,¹¹ pg 103).

Given the iterative way in which the searches and synthesis are conducted, it is important to continuously check developing CMOs and theories to improve the validity of the findings. Wiltshire and Ronkainen¹² argue that the rigour of a synthesis and the validity of findings are improved by checking for interpretive validity (how well the interpretation relates to the data) and empirical adequacy (how well an idea is supported by the data). Ronkainen and Wiltshire²⁶ also argue that ontological plausibility should be considered given that realist research does not aim to reflect observable reality. Ontological plausibility refers to causal explanations that encompass stratified layers of reality.²⁶ The use of realist logic within the analysis represents ontological depth through the CMO configurations,^11,15,16 providing a useful indicator of ontological plausibility.

Two strategies were undertaken throughout the current review to check the validity of CMO configurations and programme theory development. First, the author checked developing CMOs and theories against the data throughout the synthesis process to ensure inferences continued to be informed by the evidence.^1,11 As the synthesis progressed ‘check-in’ documents were developed to aid this process. These documents outlined the programme theory, a conceptual map of the CMO configurations that informed the programme theory, and all the data extracts that informed each of the CMO configurations. Bringing together the different stages of the synthesis within a ‘check-in’ document provided a useful tool to review the findings and ensure interpretations were consistent throughout the synthesis. The ‘check-in’ documents also allowed for ontological plausibility to be reviewed by presenting a programme theory against its constituent CMO configurations. An extract from a check-in document can be seen in phase 2 of the worked example provided in the supplementary material. ‘Check-in’ documents were used at two points in the review, first prior the expert panel consultation and second following the final two search cycles.

The second strategy used to check the validity of the research included consultations with key stakeholders. Coding and theory development was conducted by the current author and discussed regularly with colleagues involved in the review⁷, plus a panel of experts were recruited for the specific purpose of checking the salience of the review findings. Consultations with an expert panel of stakeholders is an important feature of a realist review.¹⁹ An expert panel of 11 individuals were recruited for the current review including activity coordinators, facilitators, and individuals with lived experience of SMI. Two consultations were held, the first at the mid-point of the review and the second at the end of the review. The aim of the consultations was to present the programme theories and discuss their relevance and salience against lived experience. Feedback from these consultations informed programme theory development, thus improving the validity of the findings (see phase 3 in the worked example).

Methods

This section outlines the methods of the analysis. Once a literature item was selected, three analytical tools were employed whereby (i) data extracts were coded using realist logic to develop CMO configurations; (ii) CMO configurations from across the literature sources were then brought together and consolidated, then (iii) mapped to develop programme theory. Recent papers have provided examples of coding and consolidating techniques within a realist synthesis,^{12,13,17,21,27} however no one approach was a “perfect fit” for the current review. As such a trial-and-error approach was taken, resulting in two coding techniques being used - coded annotations and conceptual codes; and two consolidation techniques - similarity and thematic groups. Previous research has included conceptual maps to help explain programme theory.^28–31 During the current review, conceptual maps were also utilised for programme theory development. The following sections present a description of each analytical tool, followed by details of their rigorous application to develop valid and evidence informed programme theories within the current review.

Coding

The first step in analysing a literature item was to code the data using realist logic. Several different approaches have been used to undertake this task such as categorising data extracts as either a context, mechanism, or outcome¹³; applying labels to literature extracts^16,23; applying themes to the data²¹; coding observable CMO configurations¹⁷; and dyadic coding which involves coding two CMO elements (i.e. CM, CO, MO).²⁷ Jagosh et al.¹³ also argue that if an outcome can be identified within a data extract, retroductive reasoning can be used to label contexts and mechanisms. It should be noted that as a theory driven approach, the synthesis is driven by an initial programme theory.^11,32,33 Despite this however, determining what constitutes as a CMO is a common issue faced within realist research.^13,34 An alternative approach to coding was taken within the current review using annotations. A literature item was annotated with interpretations relating to the initial theory, plus any new explanatory insights gleaned. CMO codes were then added to the annotation and adjusted as needed as the synthesis progressed. An example of coded annotations is outlined within the worked example, an extract of which is presented in Figure 2. Within figure two, boxes 1a and 1b provides two data extracts, and boxes 2a and 2b outline the annotations of the data. CMO labels have also been added to the annotation in brackets.

Figure 2.

Extract from sections 1 and 2 from the worked example (see supplementary materials for full details). The early stages of the synthesis is demonstrated and how coding techniques were applied from (1a & 1b) data extracts to (2a & 2b) annotations with CMO labels applied, which are then (3) consolidated based on similarity. This CMO was then (4) grouped thematically with thematically with similar CMO configurations to form a programme theory and, (5) conceptual codes were drawn out from the programme theory.

Annotating the data provides documentation of the researcher’s interpretations using realist logic. Retroductive reasoning can also be demonstrated using coded annotations. For example, the annotation underlined within box 2a was not explicit within the data but rather postulated from the narratives about context. Therefore, coded annotations offer an audit trail of interpretations and inferences. Figure 2 also highlights how CMO labels may change through the synthesis process. For example, the outcome of ‘making friends’ identified in box 2a was consolidated into the mechanism ‘connected to others’ in box 3 (highlighted in bold). Such changes are expected within a realist synthesis as new evidence informs and refines the findings.¹³ As noted above, such retroductive inferences were iteratively checked against the data and validated through peer and stakeholder consultations.

The second coding technique utilised was conceptual coding, which featured in the latter stages of the synthesis process. In particular, a technique was needed to identify potential commonalities across programme theories, which was an issue that was highlighted during an expert panel consultation. Conceptual coding was achieved by applying a summarising term or phrase to a key element of a CMO configuration or programme theory. An example can be seen in boxes 4 and 5 in Figure 2. Box 4 outlines a programme theory that has been developed from several CMO configurations (one of which is outlined in box 3) that were grouped together as thematically similar. Box 5 outlines the conceptual codes that were applied to each CMO element of the programme theory, for example the context of feeling judged was coded as ‘stigma.’ Once applied, conceptual codes were used to map programme theory thus aiding theory development (see section on conceptual mapping).

Consolidating

A common argument for the use of consolidation techniques is to help manage large amounts of complex data through data reduction. For example, Jackson & Kolla²⁷ reduced data using frequency and seeking the most common mechanisms that relate to a desired outcome. However, this approach was not used within the current review, as focusing on frequency may result in important CMOs being overlooked. Another technique to manage and reduce data is to consolidate based on similarity. For example, Pawson et al.¹⁵ states that an initial stage in defining the scope of a review is to group or categorise conceptually similar initial programme theories. This was not required for the current review as there was only one initial programme theory, however the strategy of grouping similar CMO configurations was utilised in different stages of the review. Furthermore, the process of consolidation can go beyond data management and presents another way to evidence the synthesis process.

In the first stages of the synthesis CMO configurations were compared as they were developed. Over 60 CMO configurations were developed within the current review, demonstrating the need for data management. When two or more configurations offered a similar explanation there were consolidated, bringing together all the associated data extracts and rewording the configuration as required. The consolidated CMO was then reviewed against the accumulated data extracts to ensure interpretive validity and empirical adequacy. To provide an audit trail of the consolidation process a record was created whereby CMO configurations were listed and notes were made about those that were merged. An extract from the consolidation record is provided in Table 1 which illustrates a list of eight CMO configurations that were consolidated into four configurations (highlighted in bold).

Table 1.

An extract from the CMO consolidation document, outlining notes on merging similar CMO configurations and thematic grouping.

CMO Configuration	Consolidation notes	Thematic group
34. The practitioner saw you as a student yet acknowledged any struggles you faced (C). This made you feel supported and valued (O)	Merge with 40 due to similar content on social connections.	N/A
35. You can use the process of creating art (I) to experience emotions in a way that is detached from yourself as it exists outside of your mind. This gives you a new perspective to safely reflect on these (M) and learn from them (O).	CMO 35 retained	Safe communication
36. If you face stigma due to mental illness (C), then going to an arts organisation (I) feels safe (O) because there is no focus on your illness, the space is welcoming and you are treated like a person not a patient (M).	CMO 36 retained	Feeling Safe
37. If you face stigma due to mental illness (C), but then people at the arts organisation (I) are positive and are not judgmental then you feel positive and welcomed (M). This makes you feel like you are accepted in this place (O).	Merged with 36 due to similar content on stigma	N/A
38. If you face stigma due to mental illness (C), but the arts organisation (I) looks nice then you feel welcome (M). This makes the place feel like a safe place to be (O).	Merged with 36 due to similar content on stigma	N/A
39. If your illness is a barrier to engage in past creative passions or interests (C), engaging in an arts group (I) can be a way to engage with this again, reigniting these desires (M). This provides a positive link with your past self.	CMO 39 retained	Past self
40. When you feel isolated (C) and join a new arts group (I) with people similar to you, (M) it makes you feel safe and able to make friends (O). CMO 40. Rewritten: CMO: If you feel isolated (C) then joining an arts for mental health group (I) makes you feel safe to be yourself (O) because the group members are similar to you and you are treated as a person, making you feel accepted and connected to the group (M).	CMO 40 Rewritten	Connect with others

After CMO configurations had been consolidated they were grouped by conceptual similarity, achieved by applying thematic labels to the configurations, exemplified in the final column of Table 1. Thematic grouping allows for demi-regularities to be drawn across CMO configurations, thus demonstrating programme theory development. Applying themes also demonstrates interpretive validity by providing insight into the researcher’s conceptualisation of the data. Once grouped together the CMO configurations were mapped for interactions and synthesised into a programme theory.

Conceptual mapping

A conceptual map is a tool to visually represent information such as mind maps and flow charts. They are particularly helpful when working with complex systems as a map can provide structure and demonstrate the relationship between ideas.³⁵ Visualising CMO through diagrams and mapping was established in the seminal realist texts by Pawson and Tilley¹ and Pawson,¹¹ as this technique lends itself to the CMO structure of realist logic. Detailed conceptual mapping techniques are being utilised more frequently within realist research as a useful tool to work with complexity. Many studies have utilised this technique to visualise the programme theory at a high level of abstraction, usually applying conceptual codes to represent full CMO configurations,^28,31 and demonstrating interactions between configurations.^29,30 Coles et al.³⁶ utilised conceptual mapping to enable stakeholder engagement and support programme theory development, although only context was mapped in this example.

In contrast, the current review used conceptual maps to develop programme theory by illustrating the interactions between CMO elements at the lowest level of abstraction. Mapping took place during the later stages of the synthesis, in particular following the expert panel consultation, which highlighted links between several programme theories. A technique was needed to explore these links and develop programme theories further. This was achieved by applying conceptual codes to each CMO element in the programme theory and then mapping the conceptual codes into programme theory. An example of this process is outlined in Figure 3, drawn from phase 3 of the worked example in the supplementary material.

Figure 3.

An example of the conceptual mapping process to demonstrate programme theory development. Conceptual codes are applied to two developing programme theories which are then mapped for interactions.

Through this process connections could be drawn across programme theories and theories redeveloped. The check-in documents were used to check for validity and ontological plausibility by reviewing the conceptual maps against data extracts.

As the review progressed new CMO configurations that offer insights, refined, or refuted elements of the programme theory were conceptually coded and the theory maps amended. Through this process the conceptual maps provided a useful tool to record programme theory development. An example of how new data was incorporated into the mapping process is outlined in phase 3 of the worked example provided in the supplementary material.

Documenting tools

Within the current review, several different software packages were trialled to meet the needs of the review, including NVivo and Microsoft Office programmes. Through a trial-and-error process, four approaches were used to document the synthesis and programme theory development, outlined in Table 2.

Table 2.

An outline of the software used for each analysis tool and the purpose of document created.

Analysis	Software used	Documentation purpose
Annotated coding	Nvivo	Demonstrate the researcher’s interpretations of the data using realist logic.
Consolidating	Microsoft Excel	Track the consolidation process as CMO configurations are merged by similarity. Demonstrate the researcher’s interpretations of the data through thematic grouping.
Conceptual coding and mapping	Microsoft Powerpoint	Demonstrate programme theory development
Validity checks	Microsoft Word	Check the validity and ontological plausibility of developing theories and CMO configurations against data extracts.

To annotate literature items during the initial stages of the review, literature items were imported into NVivo for coding and data extraction, which helped to manage the large amount of qualitative data. However, given the large number of CMO configurations being developed, it became difficult to track the consolidation process as NVivo does not offer a version control function. As an alternative, CMO configurations were extracted into an Excel document, with notes made on merging similar CMO configurations and applying themes (see Table 1). This document was largely used within the first stages of the review prior to the expert panel consultation. Once CMO configurations were developed, they were extracted to Microsoft Powerpoint to apply conceptual codes and map into programme theory, offering insight into programme theory development. Lastly, a key documenting tool was the creation of ‘check-in’ documents (outlined in review background) that brought together conceptual maps, CMO configurations and data extracts within a word document. These documents proved to be a valuable way to review the programme theory at different levels of abstraction, demonstrate theory development and check theory validity and ontological plausibility.

Discussion

Evidencing the iterative process of a realist review is challenging, particularly for novice researchers as there is a lack of publications on the practicalities of conducting a realist synthesis. Transparency of the theory development process is a key publication standard,²³ however, this does not relate to replicability of findings given that realist research aims to develop theory.¹⁵ The current article aims to contribute to the realist methodological knowledge base by reflecting on a worked example of a synthesis from a realist review. A limitation of the worked example provided within this article is that it only provides a snapshot of the overall review process. The details laid out in the supplementary materials do not reflect the documentation tools used to synthesise the data, however, the worked example aims to visualise the synthesis process. This article outlined how a realist synthesis was successfully conducted through the alternative application of the analytical tools coding, consolidating and conceptual mapping. Therefore, defining and documenting the analytical tools utilised to conduct a rigorous realist synthesis can provide transparency about how valid programme theories were developed.

The synthesis conducted within the current review was not without its challenges. In particular, the iterative nature of the realist review meant that coding, consolidating and conceptual mapping did not occur in a linear fashion, but rather overlapped and repeated throughout the review. The cyclic nature of a realist synthesis is unavoidable to ensure programme theory is evidence informed (i.e. embedded within the data) and not evidence based.¹ However, the iterative process was difficult to document, given the multiple documentation methods used within the current review. Documenting the synthesis process is a commonly cited issue within realist research.^17,24 For example, Gilmore et al.¹⁷ found that despite using Nvivo effectively during the initial stages of a synthesis, manual documentation was ultimately used as programme theories developed. This highlights the difficulties of using one platform for a realist synthesis. However, using several different documenting techniques within the current review meant the analysis became separated, which distanced the synthesis from the data. A solution to this was the development of ‘check-in’ documents, to regularly bring together data extracts, CMO configurations, conceptual maps, and programme theory. Drawing together the analysis into one document enabled developing CMO configurations and programme theories to be checked for interpretive validity, empirical adequacy, and ontological plausibility. ‘Check-in’ documents therefore provide a comprehensive record of programme theory development. Future reviews could employ this documentation strategy after each search cycle to provide a structured approach to documenting and validating programme theory development.

Each analytical tool also provided an insight into how programme theories were developed. Coded annotations reveal the researcher’s decision-making process by demonstrating how data has been interpreted using realist logic, as well as abductive and retroductive reasoning. Embedding initial interpretations in the literature also improves interpretive validity and empirical adequacy. The use of annotation allows for similarities to be drawn across potentially heterogeneous data extracts, which aids consolidation and the development of CMO configurations. The application of CMO labels still proved challenging and labels changed as the synthesis progressed. However, this appears to be a common feature of the realist synthesis process and has been suggested that it promotes retroductive thinking.^13,34 Applying conceptual codes to CMO elements during later stages of the review also aided the synthesis. Holton³⁷ argues that conceptual codes are an essential link between data and theory as they provide a condensed view of a phenomenon and allows patterns to be discerned. Using conceptual codes also provided anchor points within the theory from a detailed conceptual map to a higher level of theory abstraction. Therefore, conceptual codes help to navigate theory at different levels of abstraction (see phase 6 of the worked example). Both coding techniques also served two other important purposes within the current review. First, the codes helped to define search terms throughout the review, which enhanced the rigour of theory development through empirical adequacy. Conceptual terms also enabled relevant literature items to be found across different contexts, an important feature of the review process.¹⁵

Consolidation techniques served two purposes. First, consolidating CMO configurations by similarity facilitated data reduction. Second grouping CMO configurations by theme evidenced how they were synthesised into programme theories. Thematically grouping CMO configurations highlights demi-regularities that form the basis of a programme theory.^16,17 However, to evidence this process effective documentation is needed, which proved to be a challenge within the current review. Gilmore et al.¹⁷ effectively used NVivo to track changes through the use of memo’s and creating new codes. However, this approach was not practical for the current review given the large amount of messy data generated. Tracking the consolidation process within an Excel sheet offered an alternative solution, however this process separates the synthesis from the data. It is also not a practical solution for a larger research team, demonstrating a need for software that offers version control in order to record the consolidation process.

A novel approach was taken to utilising conceptual mapping to visualise programme theory at a low level of abstraction and evidence programme theory development. A potential limitation of mapping is that it loses some of its explanatory power through the use of conceptual codes. The structured nature of a conceptual map could also be viewed as a reductionist way of illustrating CMO configurations.³⁸ The programme theory could be interpreted as stages that need to be moved through to reach an outcome. This is not the case within the current review as the programme theory offers a number of potential mechanisms and outcomes. CMO configurations can be understood as both feedback loops and pathways, that explain the phenomenon under review. A solution may be to use causal loop diagrams as illustrated by Singh et al.³⁹ Causal loop diagrams emphasise the interactions between mechanisms and mediators linked to a particular outcome. Singh et al.³⁹ argues this mapping technique is a more effective way to demonstrate non-linear feedback relationships. However, it is important to ensure that the CMO elements are still represented within a causal feedback diagram, in line with realist logic that defines a realist synthesis. Documenting visual map developments poses the same issues as consolidation with mapping becoming separated from the data. It is possible to create visual maps within NVivo, however it is still a separate entity from the data and coding, highlighting the need for documenting software that can track and illustrate non-linear relationships. Within the current review ‘check-’in documents’ were used to collate analysis elements into one place to review the validity and ontological plausibility of developing theories. As such, these check-in documents present an effective solution to review separately documented analysis techniques, and they also meet the RAMESES publication standards to provide a worked example from the synthesis.²³ As such, a series of check-in documents could be used to demonstrate how a synthesis progresses over time as data is extracted throughout the review. Such checks could be built into the review process and structured against search cycles to create a systematic approach to evidencing programme theory development.

Conclusion

This article has demonstrated how the analytical tools of coding, consolidating and conceptual mapping can be used to both conduct a realist synthesis and demonstrate programme theory development. Further work is needed to address issues of documentation, which could help streamline the synthesis process. Key lessons for novice realist researchers include defining the analytical tools used to conduct a synthesis and highlight how these tools demonstrate theory development. If no one piece of software can be used, check-in documents could be an effective alternative to meet the requirements of the RAMESES publication standards. A structured approach could be taken to producing ‘check-in’ documents by creating one after each search cycle. Ultimately a realist synthesis requires a flexible approach and researchers should adapt their analysis to suit the needs of the review.

Supplemental Material

Supplemental Material - An exploration of analytical tools to conduct a realist synthesis and demonstrate programme theory development: An example from a realist review

Supplemental Material for An exploration of analytical tools to conduct a realist synthesis and demonstrate programme theory development: An example from a realist review by Louisa Anne Peters in Research Methods in Medicine & Health Sciences

Footnotes

Author contributions

LAP: Study conceptualization, data curation, formal analysis, methodology development, project administration, resources & software set-up, validation, visualization and writing.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Louisa Anne Peters

Supplemental Material

Supplemental material for this article is available online.

References

Pawson

Tilley

. Realistic evaluation. London: Sage, 1997.

Skivington

Matthews

Simpson

, et al. A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 2021; 374: n2061. DOI: 10.1136/bmj.n2061.

Wong

. Methods Commentary: Realist reviews in health policy and systems research. In Langlois

ÉV

Daniels

Akl

(Eds.) Evidence Synthesis for Health Policy and Systems: A Methods Guide. Geneva: World Health Organization, 2018b. https://www.ncbi.nlm.nih.gov/books/NBK569577/

Pawson

Greenhalgh

Harvey

, et al. Realist synthesis: an introduction. 2004.

Realist PhD Network . Demonstrating PT development. Jiscmail, 2022. https://www.jiscmail.ac.uk/cgi-bin/wa-jisc.exe?A1=ind2207&L=REALIST-PHD

Howe

. The realist PhD viva. (2021). https://mediaspace.nottingham.ac.uk/media/Nottingham+Realism+Group+-+The+Realist+PhD+Viva/1_p2fw9nq0

Peters

Louisa

Gomersall

Booth

Lucock

. Community arts, identity and recovery: a realist review of how community-based arts activities enables the identity change recovery process from serious mental illness. Journal of Community & Applied Social Psychology. 2023; 1–25. doi:10.1002/casp.2751, In this issue.

Jagosh

. Retroductive theorizing in Pawson and Tilley’s applied scientific realism. J Crit Realism 2020; 19(2): 121–130. DOI: 10.1080/14767430.2020.1723301.

Raduescu

Vessey

. Methodology in critical realist research: the mediating role of domain specific theory. In: 15th Americas Conference on Information Systems 2009, San Francisco, CA, 6–9 August, 2009.

10.

Bhaskar

Danermark

. Metatheory, interdisciplinarity and disability research: a critical realist perspective. Scand J Disabil Res 2006; 8(4): 278–297. DOI: 10.1080/15017410600914329.

11.

Pawson

Evidence-based policy: a realist perspective. London: Sage, 2006.

12.

Wiltshire

Ronkainen

A realist approach to thematic analysis: making sense of qualitative data through experiential, inferential and dispositional themes. J Crit Realism 2021; 20(2): 159–180. DOI: 10.1080/14767430.2021.1894909.

13.

Jagosh

Pluye

Wong

, et al. Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment. Res Synth Methods 2014; 5(2): 131–141. DOI: 10.1002/jrsm.1099.

14.

Dalkin

Greenhalgh

Jones

, et al. What’s in a mechanism? Development of a key concept in realist evaluation. Implement Sci 2015; 10(1): 49. DOI: 10.1186/s13012-015-0237-x.

15.

Pawson

Greenhalgh

Harvey

, et al. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005; 10(Suppl 1): 21–34. DOI: 10.1258/1355819054308530.

16.

Wong

Greenhalgh

Westhorp

, et al. RAMESES publication standards: Realist syntheses. BMC Med 2013; 14: 21.

17.

Gilmore

McAuliffe

Power

, et al. Data analysis and synthesis within a realist evaluation: toward more transparent methodological approaches. Int J Qual Methods 2019; 18, 1609406919859754. DOI: 10.1177/1609406919859754.

18.

Meyer

Lunnay

The application of abductive and retroductive inference for the design and analysis of theory-driven sociological research. Socio Res Online 2012; 18(1): 12.

19.

Wong

Data gathering in realist reviews: looking for needles in haystacks. In Emmel

Greenhalgh

Manzano

, et al. (Eds.). Doing realist Research. (1st ed.). London: Sage, 2018a.

20.

Booth

Briscoe

Wright

The “realist search”: a systematic scoping review of current practice and reporting. Res Synth Methods 2020; 11(1): 14–35. DOI: 10.1002/jrsm.1386.

21.

Rycroft-Malone

McCormack

Hutchinson

, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci 2012; 7(1): 33. DOI: 10.1186/1748-5908-7-33.

22.

Booth

Wright

Briscoe

. Scoping and searching to support realist approaches. In Emmel

Greenhalgh

Manzano

, et al. (Eds.). Doing realist Research. (1st ed.). Sage, 2018.

23.

Wong

Westhorp

Pawson

, et al. Realist synthesis: RAMESES Training Materials, 2013. https://ramesesproject.org/media/Realist_reviews_training_materials.pdf

24.

Dalkin

Forster

Hodgson

, et al. Using computer assisted qualitative data analysis software (CAQDAS; NVivo) to assist in the complex process of realist theory generation, refinement and testing. Int J Soc Res Methodol 2021; 24(1): 123–134. DOI: 10.1080/13645579.2020.1803528.

25.

Peters

Louisa

. How do community-based arts activities enable the identity change recovery process from serious mental illness: a protocol for a realist review. (Unpublished). University of Huddersfield Repository. 2022. https://eprints.hud.ac.uk/id/eprint/35766/.

26.

Ronkainen

Wiltshire

. Rethinking validity in qualitative sport and exercise psychology research: a realist perspective. Int J Sport Exerc Psychol 2021; 19(1): 13–28. DOI: 10.1080/1612197X.2019.1637363.

27.

Jackson

Kolla

. A new realistic evaluation analysis method: linked coding of context, mechanism, and outcome relationships. Am J Eval 2012; 33(3): 339–349. DOI: 10.1177/1098214012440030.

28.

Fick

Muhajarine

. First steps: creating an initial program theory for a realist evaluation of Healthy Start-Départ Santé intervention in childcare centres. Int J Soc Res Methodol 2019; 22(6): 545–556. DOI: 10.1080/13645579.2019.1595375.

29.

Papoutsi

Mattick

Pearson

, et al. Social and professional influences on antimicrobial prescribing for doctors-in-training: a realist review. J Antimicrob Chemother 2017; 72(9): 2418–2430. DOI: 10.1093/jac/dkx194.

30.

Smith

Lunn

Wong

, et al. Optimising GPs’ communication of advice to facilitate patients’ self-care and prompt follow-up when the diagnosis is uncertain: a realist review of ‘safety-netting’ in primary care. BMJ Qual Saf 2022; 31(7): 541–554. DOI: 10.1136/bmjqs-2021-014529.

31.

Stevens

Hughes

Hulme

, et al. Depenalization, diversion and decriminalization: a realist review and programme theory of alternatives to criminalization for simple drug possession. Eur J Criminol 2022; 19(1): 29–54. DOI: 10.1177/1477370819887514.

32.

Flynn

Schick-Makaroff

Levay

, et al. Developing an initial program theory to explain how patient-reported outcomes are used in health care settings: methodological process and lessons learned. Int J Qual Methods 2020; 19: 1609406920916299. DOI: 10.1177/1609406920916299.

33.

Marchal

Kegels

Van Belle

. Theory and realist methods. In Emmel

Greenhalgh

Manzano

, et al. (Eds.). Doing realist Research. (1st ed.). Sage, 2018.

34.

Shaw

Gray

Baker

, et al. Mechanisms, contexts and points of contention: operationalizing realist-informed research for complex health interventions. BMC Med Res Methodol 2018; 18(1): 178. DOI: 10.1186/s12874-018-0641-4.

35.

Novak

. Concept mapping: a strategy for organizing knowledge. In Forster

Wallace

(Eds.) Learning Science in the Schools (pp. 229–245). Routledge, 1995.

36.

Coles

Anderson

Maxwell

, et al. The influence of contextual factors on healthcare quality improvement initiatives: a realist review. Syst Rev 2020; 9(1): 94. DOI: 10.1186/s13643-020-01344-3.

37.

Holton

. The coding process and its challenges | grounded theory review. Grounded Theiry Review: An Internetional Journal 2010; 9(1). https://groundedtheoryreview.com/2010/04/02/the-coding-process-and-its-challenges/

38.

Mowles

. Complex, but not quite complex enough: the turn to the complexity sciences in evaluation scholarship. Evaluation 2014; 20(2): 160–175. DOI: 10.1177/1356389014527885.

39.

Singh

Kovacs

Cassidy

, et al. A realist review to assess for whom, under what conditions and how pay for performance programmes work in low- and middle-income countries. Soc Sci Med 2021; 270, 113624. DOI: 10.1016/j.socscimed.2020.113624.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.90 MB