Abstract
Background:
Realist studies represent an increasingly popular approach for exploring complex interventions’ successes and failures. The theory-driven approach seeks to explain “what works, how, why, in which contexts, for whom, and to what extent” using context–mechanism–outcome (CMO) configurations. When the approach was first developed, CMO configurations were the method for expressing causal explanations. Increasingly, realist studies have been conducted using different variations of the heuristic such as strategy–context–mechanism–outcome (SCMO) configurations or intervention–context–actor–mechanism–outcome (ICAMO) configurations. Researchers have highlighted a lack of methodological guidance regarding which additional explanatory factors can be included in configurations (e.g., strategies, interventions, actors). This article aims to clarify and further develop the concept of configurations by discussing how explanatory factors could be robustly added to the original CMO configuration as put forward by Pawson and Tilley.
Comparing the use of different types of configurations:
We draw on two of our own studies, one which formulated CMO configurations and one which formulated SCMO configurations, and on an evidence scan of realist studies. We explored the effects these different configurations had on studies’ findings and highlight why researchers chose CMOs or SCMOs. Finally, we provide recommendations regarding the use of configurations. These are as follows: Using additional explanatory factors is possible but consider the research scope to select the configuration appropriate for the study; Be transparent about the choice in configuration and include examples of configurations; Further studies about the use of additional explanatory factors are needed to better understand the effects on each step in the realist evaluation cycle; and New ways of disseminating realist findings are needed to balance transparency regarding the use of configurations.
Conclusions:
Adding explanatory factors is possible and can be insightful depending on the study’s scope and aims; however, any configuration type must adhere to the rule of generative causation.
Contributions to the Literature
Realist configurations have been applied in a variety of ways yet many researchers struggle to apply configurations in a way appropriate to their studies due to a lack of clear guidance or best-practice literature. By sharing our experiences of conducting realist studies, we hope to contribute to the debate of when and why additional explanatory factors can be added to the “original” context–mechanism–outcome (CMO) configuration. This article highlights important issues to consider when choosing a configuration type and provides recommendations for ensuring realist studies are transparent so others can critically examine the approach and thus evaluate studies’ results.
Background
Realist studies, namely realist evaluations and realist reviews, were first developed based on the idea that studies should not only indicate whether an intervention works or not but should highlight “what works, how, in which contexts, and for whom” (Pawson & Tilley, 1997). According to Pawson and Tilley (1997), realist studies start with, and are based on, program theories, which are initial hypotheses about how a program (component) may or may not work, in which contexts, leading to particular outcomes. Based on these initial program theories, a research design, for example, what data are needed and how it should be collected, is formed to enable the testing of the program theories.
After the data collection phase, data analysis is directed toward formulating and refining configurations that explain which (aspects of) interventions work, for whom, under what circumstances, and to what extent (Wong et al., 2016). These configurations are embedded within program theories and set out the causal links between the context (C) and mechanism (M) to explain how an outcome (O) was produced (Marchal et al., 2012; see Table 1 for the conceptualizations of the C, M, and O). When realist studies were originally developed, CMO configurations were outlined as a heuristic to aid researchers to think in terms consistent with realist causal links (Kastner et al., 2019; Pawson & Tilley, 1997; Wye et al., 2014).
Conceptualizations of Realist Concepts.
While there are many different schools of realism, this article specifically focuses on the realist approach first put forward by Pawson and Tilley (1997). Table 1 and Figure 1 highlight how the authors have conceptualized important realist terms and interpret generative causation within this school. Figures such as the one included in this article and others like it, for example, Dalkin et al.’s 2015 CMO framework, are meant as a heuristic for realist approaches and generative causation and are therefore not meant as a one-size-fits-all instrument for realist studies.
Generative explan (Pawson, 2008).
Variations in Realist Configurations
After the CMO configuration was introduced to understand causality, some authors have added explanatory factors to the CMO configuration, for example, the intervention–context–actor–mechanism–outcome (ICAMO) configuration. Some of these researchers explained they had expanded on the CMO configuration because they felt it helped them to think and analyze in a realist way and to unpack different aspects of the intervention(s) under investigation (Abejirinde et al., 2018).
Apart from the abovementioned reason, few published papers provided insight into the use and reasoning behind these additional explanatory factors. We therefore scanned the literature of the past 10 years. We searched the Embase database and Google Scholar using the terms “realist evaluation,” “realist synthesis,” “realist study,” and “realist review” and included primary studies that claimed to apply the realist approach. We found over 300 studies, which were self-proclaimed realist studies. About a third of the studies referred to the use of configurations and half of these had included examples of configurations (either in the main text or in tables, appendices, and visualizations). The vast majority of studies, which had mentioned the use of configurations and/or provided examples of configurations, had used CMO configurations to analyze the data. Several studies had included additional explanatory factors in their configurations; for example, strategy–context–mechanism–outcome (SCMO) configurations, context–intervention–mechanism–outcome (CIMO) configurations, and ICAMO configurations were used (see Table 2 for examples of these configurations). Overall, we found that many of the papers identified in the evidence scan had reported their configurations in such a way that it was difficult to decipher which factors within the configurations were functioning as context to activate which mechanism and thus cause which outcome. This lack of transparency and clarity made it difficult to understand researchers’ rationale for using the realist approach, why they choose to add additional explanatory factors, and what the causal processes were for outcomes within program theories (Pawson & Manzano-Santaella, 2012).
Variations in Configuration Types Presented in Different Realist Papers.
Abbreviation: HCW, health care workers.
aTaken from a summary table.
Comparing the Use of Different Configurations
Based on the authors’ experiences of formulating CMOs or SCMOs in our own separate studies (De Weger et al., 2020; Van Vooren et al., 2020) and the examples of the literature scan, we firstly hypothesized that
Concrete examples are needed to illustrate the potential benefits of additional factors to the original CMO configuration format because at present such examples are conspicuous by their absence. Thus, for this article, we revisited the CMO and SCMO configurations we developed in two of our own realist evaluations. We went back to the original interview transcripts and formulated CMO configurations for the original SCMO study and vice versa. Our goal was to explore and illustrate the benefits and/or drawbacks of formulating CMO and SCMO configurations into a different format—highlighting, where relevant, their differences and its effects on the results.
Two Illustrative Case Studies
When initially carrying out both studies, we purposively reflected on which type of configuration would best suit the studies’ research aims and scopes and the breadth of information available (see Table 3 for more detailed information on the case studies). Study A aimed to explore how community engagement (CE) is understood and being operationalized in the Dutch health care system (De Weger et al., 2020). It examined engaged citizens’ and professionals’ perceptions and experiences of CE approaches. The aim of this study was to unpack the relationships and dynamics between citizens and professionals by doing a deep-dive analysis of mechanisms (and the CMO causal processes) on a more granular level.
Illustrative Case Studies.
In Study B, the development of nine Dutch Population Health Management sites was monitored and analyzed using SCMO configurations (Van Vooren et al., 2020). We wanted to stay close to professionals’ needs and perceptions and thus provided practical insights for professionals to successfully develop toward population health management (PHM). The study therefore focused on the strategies that were implemented by the PHM sites. In order to highlight how strategies were implemented within, and impacted by, their contexts and how this triggered certain mechanisms to produce specific outcomes, strategies were added as an explanatory factor to the CMO heuristic. For this study, strategies were conceptualized as intended plans of action (Jagosh et al., 2015 ) “aimed at the reorganisation and integration of public health, health care, social care and community services including ‘partner’ sectors (e.g., housing, transport), to promote the Triple Aim and develop into a health and wellbeing system” (Van Vooren et al., 2020, p.38). Strategies in this study can be compared to the concept of interventions that are implemented in the context, which triggers mechanisms and causes a certain outcome (Lacouture et al., 2015).
In both studies, the same realist evaluation cycle was used (Marchal et al., 2012; Pawson & Tilley, 1997). Figure 2 highlights how the choice in configuration influenced each step in the studies’ cycles. This figure shows the importance of choosing the most appropriate configuration type as it influences how initial program theories are expressed, how data are collected and analyzed, and ultimately how program theories are refined. While this process is streamlined in Figure 2, we acknowledge the realist process is iterative.
Influence of configuration type on realist evaluation cycle.
Discussion and Comparison of Our Own Experiences of Using Different Types of Configurations
In Study A, we had originally expressed our realist causal explanations in the form of CMOs, in Study B as SCMOs. When we went back to reanalyze the original transcripts in Study A to develop SCMOs, we firstly found that the
Additionally, we found that the
Third, while we chose our type of configuration based on our studies’ aims and scope, we found that the choice of using additional explanatory factors within a realist configuration
In developing our original and revised configurations (i.e., CMOs and SCMOs), we have come to appreciate that the core purpose of any type of configuration is to provide realist causal explanations. While elaborations to CMOs can help address studies’ scope and aims, we noted that guidance is needed regarding the use of additional factors. Incorporating additional factors into the original CMO configuration could distract from and undermine any realist causal explanation provided, especially as there is currently limited information available on how to add explanatory factors in a methodologically sound manner. Furthermore, the use of additional explanatory factors raises ontological issues, which have rarely been discussed in published realist methodological texts. For example, within the realist philosophy of science, the ontological “status” of a strategy as an additional explanatory factor remains unclear. In other words, what and/or how does a strategy relate to CMO configurations? While such issues require more discussion and methodological development, we have found it conceptually useful to see strategies as processes that are deliberately employed to alter or manipulate that which is functioning as context within a CMO configuration. This means that strategies can be used to change the context in such a way that it activates the right mechanism to give us the desired outcome.
Recommendations Based on Comparing the Use of Different Configurations and the Evidence Scan
Based on our reexamination of the data from Studies A and B and the evidence scan, we recommend the following as guidance for those wishing to apply realist approaches in a consistent and coherent manner. 1. Using additional explanatory factors is possible but consider the research scope to select the configuration appropriate for the study
While the starting point for explaining the causation for outcomes should take the form of the CMO configuration, based on our reexamination and the evidence scan, adding explanatory factors to the CMO configuration can be useful, depending on research projects’ aims and scope. However, realist researchers should consider the (possible) effects of this choice. As highlighted in the reexamination, because of the exploratory nature and in-depth understanding needed in Study A, we found that explaining causation in the form of CMO configurations helped us to extract and analyze data on a granular and personal level and to generate new theories on community involvement, so no additional focus on, for example, strategies was needed. In comparison, in Study B, adding “strategies” as an explanatory factor to CMO configurations helped us to more explicitly explore how strategies were related (if at all) to causal processes, in line with the study’s aim of refining theories on successful PHM strategies. In addition, Mukumbang et al. (2018) found that adding the explanatory factors of “actor” and “intervention” to the CMO configuration helped them to analyze the effect of the same interventions on different actors—that is, for whom different interventions worked in different contexts. Adding explanatory factors may therefore be more appropriate for studies, which have a specific focus on additional factors like strategies or actors. These factors may also help to remind researchers to specify whom the causal explanation relates to and/or which intervention or strategy is related to a particular CMO configuration. What our experiences and the literature scan above show, is that there should be a clear rationale for choosing a configuration type. Future studies could further unpack which types of configurations are especially useful for which types of studies, for example, using different configurations for different levels of focus such as more granular-level data (Study A) or more operational-level data (Study B).
A potential problem with the addition of explanatory factors like “strategy” (i.e., “S”) to the original CMO configuration is the risk of confusion regarding the exact nature of the causal explanation. Regardless of the addition of factors into the original CMO configuration developed by Pawson and Tilley, it must be remembered that it is something that is functioning as context that “triggers” or activates a mechanism which in turn produces an outcome (Pawson, 2013). This is the way causation is explained within realist studies and the addition of any factors should not obfuscate this core explanatory form. In other words, regardless of additional explanatory factors, anyone reading realist studies’ findings should be able to understand that this outcome was caused by this mechanism, which was in turn “triggered” by this context. In our evidence scan, we found that many published papers that claimed to be realist studies provided lists of contextual factors, mechanisms, outcomes, and potentially other explanatory factors, without explicitly describing the causal link between the factors. Such analyses and unconfigured reporting are contrary to the quality and reporting standards for realist studies (Wong et al., 2014, 2017 ) and the methodological rigor of such work has been questioned (Pawson & Manzano-Santaella, 2012). Such unconfigured reporting causes confusion because it is unclear what the actual causal explanation is—that is, which factor (e.g., context, intervention, or strategy, or actor) activates mechanisms that cause the outcome. Ultimately, whether additional factors are used or not, a deep understanding of the CMO configuration and generative causation is required within realist studies. Additional factors can be used to highlight specific aspects within the generative causation (in order to address studies’ specific scopes and aims).
2. Be transparent about the choice in configuration and include examples of configurations
Building on from the RAMESES reporting standards I and II (Wong et al., 2014, 2017) and from authors such as Gilmore et al. (2019), we argue that realist studies should be written up transparently in order to provide clear insights into the methodological and analytical processes (including configurations). We further suggest that to ensure realist studies can be critically examined, researchers should clearly describe which configuration type they have used. As the CMO configuration could be seen as the original configuration type, researchers who choose to use a different configuration type should explain their alternative.
When we investigated realist papers through the abovementioned evidence scan, we found that of those papers that had included definitions of configurations’ explanatory factors, factors were defined and operationalized differently. Dalkin et al. (2015) and Marchal et al. (2012) had already highlighted such differences in the concept of “mechanism.” Additionally, we found that the terms “interventions,” “strategies,” and “program” (components) are interpreted and used differently in different configuration types. This may mean there is a risk of the terms being conflated. By clearly articulating which configuration type has been used and by providing conceptualizations of concepts used, realist papers can provide the transparency needed for others to judge the value of the methodological approaches used.
3. Further studies about the use of additional explanatory factors are needed to better understand the effects on each step in the realist evaluation cycle
As stated before, we expect that the choice in configuration type influences important steps within the evaluation cycle, including, for example, how initial program theories are expressed, how data are collected and analyzed, and how program theories are refined. By discussing our use of CMOs and SCMOs in the CE and PHM studies, we hope to open up the debate about the effects of additional explanatory factors in each step of the realist evaluation cycle (see Figure 2). Based on our own experiences, we have seen that choosing a configuration type influences how and what data are collected and analyzed. We suspect that differences in where information is captured regarding the causal processes within configurations may lead to a different focus within the analysis and may therefore help shape different program theories. We have, however, not examined this possible influence on the generation of program theories.
Further realist methodological studies are needed to advance thinking about the implications, and use of additional explanatory factors, and how this affects each step in the realist evaluation cycle, including data collection, analysis, and theory development. In this way, such studies could provide further guidance for selecting appropriate configurations.
4. New ways of disseminating realist findings are needed to balance transparency regarding the use of configurations
The realist approach can be used to provide professionals with insights into what works, how, in which conditions, and for whom, enabling them to tailor interventions to their specific contexts. However, based on our own experience, we know it is difficult to portray complex and rich realist findings, regardless of the configuration type used, in a scientifically transparent manner that also clearly and succinctly communicates the key points relevant to professionals. To ensure the realist approach remains useful, researchers should strive to develop new ways of clearly disseminating complex information in a way that is manageable for professionals. One way to do this is through visualizing configurations (e.g., Bertotti et al., 2017; Fick & Muhajarine, 2019; Gilmore et al., 2019; Pagatpatan & Ward, 2017). For example, Pawson and Tilley (1997), Jagosh et al. (2015) and Dalkin et al. (2015) have provided helpful visualizations in the form of equations, a ripple effect, and a process. Clear visualizations of configurations that explicitly show causation between the different explanatory factors could play a pivotal role in ensuring realist findings connect more with professionals.
Conclusion
Realist studies are inherently flexible approaches for making sense of complex phenomena, provided the studies seek to understand generative causation. However, this flexibility also means there is no one protocol or template for conducting realist studies, which may be why many realist researchers seek more methodological guidance. By drawing on our own experiences, an evidence scan, and a reanalysis of our findings, we provided recommendations on using additional explanatory factors. Adding explanatory factors is possible and can be insightful depending on the study’s scope and aims; however, we would argue that any configuration type must explain the causal link between context, mechanism, and outcome and any additional explanatory factors must adhere to that rule of generative causation.
Footnotes
Authors’ Note
E. De Weger and N. J. E. Van Vooren share equally attributed authorship. Views expressed in this article do not necessarily represent those of the funders or UKCRC. Funders had no role in the article's preparation or in the decision to publish it. The evidence scan's reference list is available upon request.
Author Contributions
E. De Weger and N. J. E. Van Vooren conceptualized the manuscript and wrote the first draft. S. Dalkin, G. Wong, B. Marchal, H. W. Drewes, and C. A. Baan contributed to the conceptualization, provided feedback on the manuscript’s drafts, and contributed to additional conceptualization and writing of the final product. All authors reviewed and approved the submitted version of the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: G. Wong’s salary is partly supported by the Evidence Synthesis Working Group of the National Institute for Health Research School for Primary Care Research (NIHR SPCR) [Project Number: 390]. S. Dalkin is a member of Fuse, the Centre for Translational Research in Public Health (
). Funding for Fuse from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, the National Institute for Health Research, under the auspices of the UKCRC, is gratefully acknowledged.