Sage Journals: Discover world-class research

Abstract

Medical Research Council guidelines recognise the need to optimise complex interventions prior to full trial through greater understanding of underlying theory and formative process evaluation, yet there are few examples. A realist approach to formative process evaluation makes a unique contribution through a focus on theory formalisation and abstraction. The success of an intervention is dependent on the extent to which it gels or jars with existing provision and can be successfully transferred to new contexts. Interventions with underlying programme theory about how they work, for whom, and under which circumstances will be better able to adapt to work with (rather than against) different services, individuals, and settings.

In this methodological article, we describe and illustrate how a realist approach to formative process evaluation develops contextualised intervention theory that can underpin more adaptable and scalable interventions. We discuss challenges and benefits of this approach.

Keywords

complex intervention development feasibility and piloting formative process evaluation programme theory development realist evaluation

Introduction

Coming up with solutions to complex problems in complex systems is challenging and requires in-depth understanding of the nature of interventions and their implementation contexts (Wong et al., 2016). Medical Research Council (MRC) guidance emphasises the use of evaluation to build theory, understand causal mechanisms and optimise delivery during feasibility and piloting, but does not include illustrative case studies of formative process evaluations (Craig et al., 2013; Moore et al., 2015). The small, but emergent, number of published formative process evaluations evaluate interventions that have been running for some time (Evans et al., 2015), although there are notable exceptions (Scorgie et al., 2017).

Complex interventions are more than the sum of their parts and should be better theorised to reflect this (Hawe et al., 2004). To build implementation and delivery knowledge it is helpful to understand how context (individual, social, cultural, organisational) interacts with intervention components and underpinning mechanisms to bring about desired outcomes. Yet under-theorising of how interventions depend on their social contexts is typical in public health evaluations (Hawe, 2015; Macintyre and Petticrew, 2000; Moore et al., 2015) and only limited mention of the role of context in shaping implementation and causal processes is made in the MRC complex intervention guidance (Fletcher et al., 2016).

Realist evaluation

Realist evaluation builds, tests and refines theory about how causal mechanisms, including human agency, and contexts (individual- and system-level), interact to produce outcomes (intended or unintended) (Hawkins, 2014; Pawson, 2013). Realist evaluation is a form of theory-driven evaluation that is grounded in a realist philosophy of science (Wong et al., 2016). Generative mechanisms are seen to be discernible within the social systems in which they operate (Bhaskar, 2013; Sayer, 2000). It is characterised by an iterative process of identification, appraisal, and synthesis of diverse forms of evidence in the form of programme theories (Pearson et al., 2015). Candidate theories about how an intervention works identified from a variety of sources (e.g. interviews, practitioner notes, service reports) are consolidated into overarching programme theory about how an intervention works, for whom and under which circumstances (Pawson, 2013). Mechanisms are conceived of as the way in which a programme’s resources or opportunities interact with the reasoning of individuals and produce behaviour change (Pawson et al., 1997), and these are held to be enabled or constrained by a wider configuration of factors (‘contexts’ (Pawson, 2006; Pawson et al., 2005). A realist evaluation aims to explain how the resources of an intervention interact with the reasoning of the individuals (mechanisms) to produce different patterns of outcomes in different contexts.

The MRC guidelines for process evaluations acknowledge the suitability of realist evaluation approaches for process evaluations and their increasing influence, particularly in public health (Moore et al., 2015). In consideration of realist and complexity theory principles, pragmatic (e.g. realist) process evaluation of pre-existing interventions at the beginning of the intervention development phase (MRC guidelines for complex intervention development and evaluation (Craig et al., 2013)) to build underlying intervention theory, would improve external and socio-ecological validity and create more sustainable implementation procedures (Fletcher et al., 2016). Realist formative process evaluation has the potential to support the development of interventions in the feasibility and piloting phase that are flexible and adaptable to context, and thus more likely to succeed when scaled up and/or spread to new locations for outcome evaluation.

In this article, we bring together in constructive dialogue different approaches to evaluation (realist, formative, process) and we use findings to illustrate in practice how this combined evaluation approach can be used to good effect. Realist formative process evaluation (‘realist FPE’) combines a more sociologically-oriented, theory-engaged, iterative and emergent realist tradition with a more applied, process-driven, decision-maker facing technology assessment tradition. Realist FPE addresses two critical objectives in complex intervention development and evaluation: the need for formative evaluation in the feasibility and piloting phase to build the theory behind the intervention prior to outcome evaluation (i.e. full trial) (Craig et al., 2013), and the need for greater attention to be paid to context during initial delivery and effective scale-up during later implementation (Moore et al., 2015). We demonstrate how this can be done in a complex project that crosses conventional fields (criminal justice system, health, and social care).

In this article we:

Briefly outline the ENGAGER research programme (Box 1)

Describe how a realist approach was applied in a formative process evaluation (‘realist FPE’)

Illustrate with findings how realist FPE critically develops underlying theory behind an intervention and builds implementation and delivery knowledge for full trial

Elucidate the distinct contribution of realist FPE to developing adaptable and scalable complex interventions for full trial

Discuss the challenges of applying realist FPE

The terms ‘mechanism’, ‘context’, and ‘outcome’ used throughout refer specifically to realist concepts (defined in Box 2).

Box 1.

The ENGAGER intervention.

Our examples are drawn from a realist FPE that was embedded in the pilot trial of the ‘ENGAGER’ intervention. Although specifics of the ENGAGER intervention are used here to illustrate methodological points, we do not describe this complex intervention in detail. We refer interested readers to full descriptions of this intervention and its development (see Lennox et al., 2017; Pearson et al., 2015) prior to the full trial (Kirkpatrick, 2018).
Offenders have high rates of mental health problems (Brooker et al., 2002; Grubin et al., 1997; Singleton et al., 1998), distrust, substance misuse, homelessness and relationship difficulties and high rates of comorbidity (Georgiadis et al., 2016), but poor continuity of service provision on release from prison (Byng et al., 2012; Williamson, 2006) and minimal access to mental health care (Forrester et al., 2013; Moore et al., 2015).
The ENGAGER practitioners work with male prisoners within 12 weeks of their release date, and 3-5 months’ post-release. An individualised plan is built around a shared understanding of the links between his thinking, feelings, and behaviour. The ENGAGER practitioner works with a participant’s personal strengths and abilities, professionals in other services, and friends and family, to mobilise available resources in prison and the community around a participant’s goals.
We used a realist review (Pearson et al., 2015), alongside case studies, focus groups and discussion with an expert stakeholder group, including peer researchers, to produce the initial programme theory for the ENGAGER intervention. We also incorporated extant behaviour change theory, selected according to context, which proposed both how practitioners should work with offenders and how implementation should be achieved in the form of an implementation delivery platform (i.e. practitioner manual, training, and supervision). The initial programme theory as used to inform initial intervention development is shown in Figure 1, where each of the arrows represents a group of context-mechanism-outcome configurations (e.g. Supplementary Figure S1) which we hypothesised would produce each intermediate outcome. Feedback loops operate between the intermediate outcomes at all levels.
This initial programme theory has three important aspects. Firstly, ENGAGER practitioners are the main intervention resource (their behaviour in these dyadic relationships), which interacts with the agency of the prison leavers and with other health and social care professionals to produce outcomes. Secondly, based on an understanding that individual offenders’ problems and contexts would be diverse, flexibility of practitioner responses to individual need was prioritised over replication of fixed intervention components. Thirdly, a concern with implementation – ensuring change happens – was built in from the start in the form of a comprehensive implementation platform.

Figure 1.

The ENGAGER initial programme theory.

Box 2.

Definition of Realist terms as applied in the ENGAGER realist FPE.

Context is defined as social/cultural/organisational/practitioner/individual characteristics required for a particular mechanism to produce a desired outcome; mechanism as the interaction of people’s reasoning with the resources that the ENGAGER practitioners offer to influence a change in behaviour or action; and outcomes as 1) a change in the thinking, emotions, or behaviour (e.g. engagement) of prison leavers, and 2) a change in the thinking, emotions, or behaviour (i.e. work-practices) of health and social care professionals working with prison leavers, that occur as a result of accessing resources provided by the ENGAGER practitioners.

Applying a realist approach in a formative process evaluation

Design

The realist FPE aims relate to the three phases of analysis and theory building (Table 1) required to refine the implementation platform and develop a core programme theory (Figure 2).

Table 1.

Realist formative process evaluation phases.

Phase	Action	Aim
Phase 1	Data collection and rapid feedback to ENGAGER practitioners during the pilot trial	Optimise delivery and opportunities for learning
Phase 2	Initial analysis of full data set to inform substantive changes to the key components and implementation platform after the pilot trial	Prioritise key intervention functions to inform delivery planning (e.g. team structure and manual revisions)
Phase 3	In-depth analysis to elaborate core programme theory – how the intervention works, for whom, when, and in what way	Inform detailed focus and priorities of the manual and training (implementation platform) for intervention delivery in the full trial outcome evaluation

Figure 2.

Design of realist FPE.

Data collection

Face-to-face, semi-structured realist interviews (in which candidate theories are discussed with interviewees; Manzano, 2016; Pawson, 1996) were digitally audio-recorded and transcribed verbatim. Realist interview schedules (Supplementary Table S2) were constructed using the context-mechanism-outcome (CMO) configurations in the initial programme theory (Supplementary Table S1). The developing programme theory informed iterative revision of interview schedules. Interviews were conducted with: 4 ENGAGER practitioners (2 in South West site (SW), 2 in North West site (NW); each at three time points: early, mid, and late in intervention delivery); 3 ENGAGER supervisors (2 NW, 1 SW) (each at three time points: early, mid, and late in intervention delivery); 14 prison leavers (6 NW – 2 in prison, 4 in community; 8 SW – 3 in prison, 6 in community); 5 criminal justice, health and/or social care professionals (2 NW, 3 SW; 2 prison services, 3 community services).

Prison leavers were recruited using a combination of opportunity and theoretically driven purposive sampling. Opportunity sampling was used when approaching those with lower levels of engagement (e.g. returned to prison). Purposive sampling was used to recruit those with experience of the salient issues in the emerging programme theory, e.g. few contacts in prison prior to release. We purposively sampled other health and social care professionals to include professionals proving challenging to involve in ENGAGER.

ENGAGER practitioners used a pro forma to record notes about what they had done in each session with prison leavers, or on a participant’s behalf, and any significant issues that had been discussed (all notes for 19 prison leavers’ contacts in NW, 18 in SW). We examined scanned copies of practitioner notes to describe what components, where, and over what time were delivered.

ENGAGER practitioners digitally audio-recorded 24 of their one-to-one sessions (12 NW, 12 SW; 19 prison, 5 community) with 17 prison leavers. Recordings of sessions in the community, as opposed to in prison, were particularly challenging to obtain as these sessions evolved to be more flexible and less ‘formal’ to meet the needs of the prison leavers (e.g. not necessarily at a table in a quiet private room). A sub-set of the recordings obtained were purposively sampled to represent different stages of the intervention delivery and sessions at particular stages of the intervention (close to prison release; close to final contact with their ENGAGER practitioner).

Ethics

The study received National Health Service (NHS) Ethical approval from the National Research Ethics Service (NRES) Committee East of England (13/EE/0249) and National Offender Management Service (NOMS) approval (2013-187). In addition, local NHS Research and Development approvals were obtained from the healthcare providers in each prison establishment.

Data analysis

This realist evaluation was a form of interpretive case study, thus rigour here embraces the principles of interpretive case study in general (Greenhalgh et al., 2009; Stake, 1995). Following these principles, care was taken to ensure immersion in the data and at the field site as far as practicable, and to define and justify the ‘case’. The case was defined as the implementation of the ENGAGER intervention in two prison and community locations, one in the NW and one in the SW of the UK. The ‘case’ provided rich contextualised evidence of how the people (ENGAGER practitioners, participants and other practitioners) interacted with one another and the resources and limitations in the two implementation contexts.

The intervention was discussed in depth regularly with practitioners delivering it, their supervisors and the research team across both sites. Time was spent at each field site to achieve immersion and recordings of delivery in practitioner sessions with participants were listened to. Reflexivity was encouraged in both researchers and research prison leavers. Theory was developed iteratively as emerging data were analysed and disconfirming cases and alternative explanations explicitly sought. Interpretations were explored with all stakeholders (including practitioners, policy-makers, and men with lived experience of leaving prison) in regular focus groups.

As expressed by Greenhalgh and colleagues (2009: 396–7):

the realist methodology cannot be expressed simply in technical or sequential terms (first do X, like this, then move on and do Y, like this). Rather, it uses all the following approaches judiciously and in combination: Organizing and collating primary data and producing preliminary thematic summaries of these …; Presenting, defending, and negotiating particular interpretations of actions and events both within the research team and also to the stakeholders themselves …; Testing these interpretations by explicitly seeking disconfirming or contradictory data …; Considering other interpretations that might account for the same findings.

During the pilot a researcher (SLB) monitored data and raised any potential problems with feasibility or fidelity of the intervention with the wider research team in regular meetings. Where appropriate, the research team provided rapid feedback to the practitioner team to redouble efforts toward fidelity or to make changes to the intervention.

During the pilot we also presented, defended, and negotiated interpretations of the evidence within the research team and in wider stakeholder groups meetings. Emergent issues from the analysis process were also explored in subsequent semi-structured interviews with the ENGAGER supervisors, practitioners, prison leavers and other health and social care professionals.

In the formal analysis after the pilot trial ended, the qualitative data were analysed using Framework analysis (Ritchie and Spenser, 1994) as we wanted to build on and adapt the deductively derived initial ENGAGER programme theory. Data were sifted, charted and sorted in accordance with this initial programme theory, which formed the structure of the framework in NVivo 10 software package for data analysis (QSR International Pty Ltd, 2012). The framework matrices (Supplementary Table S3) were organised as follows: columns were CMOs or partial CMOs of interest; rows were one type of data for one participant. For example, one participant might have three rows, one for an interview in prison, one for practitioners’ case notes, and one for an interview in the community after release. In the framework matrix, one box represented relevant evidence from one type of data for one participant (practitioner or prison leaver) in relation to one piece of programme theory. This structure enabled our thematic analysis of evidence across all sources related to the initial programme theory.

Each stage of the Framework analysis was conducted by one researcher (SLB or AS), and reviewed by a second to raise issues (CQ or SLB), which were resolved through discussion. During this process, gaps in evidence or conflicting evidence were marked to be explored more deeply in the process evaluation of the main trial of the ENGAGER intervention.

Prior to completion of the detailed Framework analysis, pragmatic considerations and time constraints required that the analysis was fed in to the decision-making process for the intervention delivery (see discussion for serendipitous benefits of working in this way). The CMO configurations at this stage were expressed as if-then statements, or narrative summaries, of the evidence for each component of the intervention. The results of this stage informed a stakeholder consensus group meeting and fed in to decisions about which components of the intervention should be prioritised, amended or discarded in the main trial.

In the final stage of mapping and interpretation, one researcher (SLB) consolidated core CMOs from across all of the potentially important components of the intervention (see Supplementary Material). Consolidation was done by merging by outcome, mechanism, or context, depending on which was deemed to be most useful (see Pearson et al.’s, 2015, supplementary file 4 for a detailed Powerpoint presentation and audio description of this consolidation process). Three researchers (CQ, RB, MP) reviewed the evidence and agreed the core mechanisms. Two researchers sense-checked the theory from the point of view of their immersion in the intervention delivery at the two sites (CL, TK). An experienced qualitative researcher not immersed in the data or delivery acted as a critical friend to facilitate reflexivity (CO). In this way components of the intervention identified as most important for delivering outcomes were prioritised as four key ‘intervention functions’ (i.e. the main ways in which intervention resource/s disrupted or changed the system). The core programme theory underpinning each of the key intervention functions and how they worked together was elaborated. The results of this process were used to provide further detail within the manual and inform training before and at the start of the delivery of the intervention in the subsequent Randomised Controlled Trial (RCT).

Illustration of how realist FPE critically develops underlying theory behind an intervention and builds implementation and delivery knowledge for full trial

We prioritised four main intervention functions from the initial programme theory as most important in delivering the desired outcomes of the ENGAGER intervention: 1) building trust and engagement, 2) doing practical and emotional work, 3) using the mentalisation-based approach, and 4) sharing participant’s personal goals and action plans with their health and social care professionals. For the former two prioritised intervention functions we first present illustrative findings and then the process by which the initial programme theory and related implementation platform were elaborated, emphasised, and prioritised to form a core ENGAGER programme theory and implementation platform for the full ENGAGER RCT. For each, we show in flowcharts:

1) initial programme theory;

2) rapid feedback of findings during the pilot trial and immediate formative changes to implementation;

3) formal Framework analysis findings and substantive changes to the implementation and delivery after the pilot trial; and

4) elaborated, emphasised and prioritised core programme theory.

The first example highlights substantive changes made to the core ENGAGER intervention programme theory, while the second example highlights changes to the implementation programme theory. Similar results for the remaining two intervention functions are provided in the supplementary material (Supplementary material: Illustrative findings from key intervention functions 3 & 4).

Building trust and engagement

Illustrative findings: Release day working

ENGAGER practitioner interviews during the pilot trial raised the concern that, for a range of practical reasons, including sudden early release, the length of time and number of contacts with ENGAGER participants in prison prior to release was far lower than intended, often as little as two weeks and 1–3 contacts. The practitioners, who were used to building trust and engagement in formal sessions, were concerned about not having sufficient time to build trust before release. However, our emerging findings highlighted that for the men receiving the intervention, anxiety about release made it difficult to do the substantive therapeutic work in sessions close to release from prison that our initial theory had anticipated would support engagement. What they most valued instead was the practical and emotional support that ENGAGER practitioners provided during their release day (‘release day working’). In this extract from an interview with ‘Dave’ soon after release, we can see the impact that being met at the gate had on his engagement:

[when you get out on your own] everything that you planned in [prison], oh, I’m gonna do this and I’m gonna do that, it just … it’s just gone. Gone out your head. And by the time you know it, you’re out of your face somewhere and you’ve missed all your appointments and then you [are] just recalled anyway. I don’t think I would have got to any of the appointments if … if I didn’t have [ENGAGER practitioner] there. I’d have jumped on the train, probably gone and got some beers, gone and got some Valium and then probably ended up in a ditch or something. [ENGAGER practitioner] just took me to every appointment … With the ENGAGER people … I can focus on getting all my stuff out the way so I’m not in trouble straight away already before I even start.

Similar themes came up in a number of interviews with men interviewed soon after release. It emerged that one of the most important ways in which work in prison built engagement was the setting up of release day working, which was critical for building engagement. Interviews with prison leavers emphasised that rather than pre-release meetings it was this release day working, including meeting at the gate, supporting transportation to appointments, attending appointments, and facilitating sobriety, that was most important for building their trust and engagement (for example, see Box 3, which shows one piece of programme theory developed from these interviews).

Box 3.

Illustrative if-then statements: Release day working.

IF a participant would usually go straight out from prison and use or drink and get stuck in same rut AND he is met at the prison gate by his practitioner (whom he knows from their prison meetings) and that practitioner goes with him to all of his release day appointments at services, advocates for him and supports communication with other services, THEN the participant has less time and less inclination to use substances and/or alcohol on his release day, is more likely to get home without using or drinking and to feel proud of himself for his release day abstinence AND THEN he will feel increased trust in his ENGAGER practitioner and will want to engage with them in the community.

The contribution of realist FPE: Trust and engagement

The realist FPE contributed to changes to implementation and delivery of trust and engagement in the following ways (Figure 3):

1) prioritised release day working as a key driver for trust and engagement, and

2) informed formative and substantive changes to programme theory and implementation platform, including details for tailoring delivery to people with particular contexts (e.g. anxiety about drug use on release day); and emphasising the importance of release day working in the manual, supervision and training for ENGAGER practitioners.

Figure 3.

Formative and substantive changes to the implementation platform and programme theory for ‘trust and engagement’.

Emotional support

Illustrative findings: Emotional and practical support

Our interviews highlighted early concerns of ENGAGER practitioners that they were unable to provide ‘emotional support’ and that they were spending a lot of time instead on practical support. However, we observed in interviews with both ENGAGER practitioners and prison leavers that practical support was in fact a critical way in which the ENGAGER practitioners were delivering emotional support. Men in the intervention described how the practitioner supporting them in getting their benefits, in finding accommodation, or getting to appointments with other professionals such as probation, showed that the ENGAGER practitioner cared about them and was willing to go the extra mile for them.

The feedback from practitioners and prison leavers in the pilot trial highlighted the importance of both 1) ‘practical support’ and 2) ‘working with crises’ in delivering emotional support. Practical support was prevalent in the pilot trial delivery of the intervention but was not theorised in terms of how it also contributed to emotional support. Prison leavers described how everyday practical support from the ENGAGER practitioner built their trust, self-esteem, and feeling of being cared for, which supported their engagement with their ENGAGER practitioner (Box 4). ENGAGER practitioners discussed how anxiety about imminent release made prison leavers less able to focus on longer-term or more abstract conversations, such as the links between thinking, feeling, and behaviour, but able to focus on short-term practical issues to do with their release and return to the community.

Some other services described the everyday ‘crises’ that this population present with as a barrier to getting to the ‘real work’. ENGAGER practitioner interviews showed how these crises provide an opportunity for rich, engaging, and person-centred emotional support. The MB approach of ‘micro-slicing’ in detail everyday life events that go wrong when in the community was used to identify and address recurring problems in the participant’s life. For example, supporting him to understand what happens just before he ‘sees red’ and acts aggressively.

Box 4.

Illustrative if-then statements: Emotional and practical support.

• IF an ENGAGER Practitioner uses practical support around a participant’s personal goals (i.e. in supporting actions in his Shared Action Plan), THEN the participant will feel that someone genuinely cares about him and wants to help him AND will increasingly trust his ENGAGER Practitioner.
• IF an ENGAGER Practitioner provides practical support and uses everyday crises to explore links between his feelings, behaviour, and actions with the participant, THEN the work shared between the ENGAGER Practitioner and the participant will provide emotional support in a way that the participant finds helpful, that focuses on his personal experiences and goals, and that he is able to engage with.

The contribution of realist FPE: Emotional support

The realist FPE contributed to changes to implementation and delivery of ENGAGER emotional support in the following ways (Figure 4): 1) elaborated and emphasised two forms of delivering emotional support in the core programme theory, and 2) made substantive changes to the implementation platform (manual and training), including detailed examples and description of how ENGAGER delivers emotional support, when to use these two forms of delivering emotional support (practical in prison and crisis-based in the community), for whom (e.g. tailor type of support to anxiety levels), and in what way.

Figure 4.

Formative changes and substantive changes to the implementation platform and programme theory for emotional support.

Inter-relationships between the four prioritised functions of the ENGAGER intervention

We found important and unanticipated inter-relationships between the intervention functions. The realist FPE supported us to capture and understand these interactions in the core programme theory, by illustrating the operation of the CMO configurations across the intervention functions, and thus to articulate them in the manual and training.

In practice, in delivering the intervention, ENGAGER practitioners merged the intervention functions in the initial programme theory to produce something greater than the parts, and it worked in inter-related, unexpected and important ways. In the initial programme theory: the mentalisation-based (MB) approach was intended to support ENGAGER practitioners to engage prison leavers by helping practitioners to manage and respond to prison leavers’ emotions during sessions; the shared understanding of the links between thinking, feeling and acting was to be developed in prison and used to guide future sessions; emotional support was delivered through one-to-one working in empathic and person-centred ways. These three functions had not been conceptualised in terms of how they facilitated and enhanced each other, as identified from the analysis.

ENGAGER practitioners in the act of delivering these intervention functions used the MB approach to deliver emotional support in discussing and caring about ‘crises’, and in turn, the understanding over time of these everyday crises formed the basis of the shared understanding and thus the shared action plan. Through linking these functions together using a realist approach, highlighting the nested CMO configurations involved in ENGAGER, the ‘what to do when, for whom and in what way’ details emerged across these intervention functions. For example, with the evolution of the shared understanding in response to prison leavers’ changing ability to focus on non-practical issues over time, the shared plan evolved from an early focus on short-term practical goals (like finding somewhere to live) toward more abstract longer-term goals (like building stronger relationships) as work with an individual progressed. Through the opportunities presented by returning to the community and re-engaging in problematic patterns of thinking, feeling, and behaving, the ENGAGER practitioners could use the MB toolkit to support prison leavers to build engagement, understanding, and progress toward goals, which in turn supports the engagement of other practitioners’ resources around the prison leavers’ goals.

A specifically realist approach to formative process evaluation focuses on building the underlying theory about how an intervention works, for whom, and under what circumstances. This approach meant we could articulate the prioritised functions of the ENGAGER intervention and critically how they interact to mobilise the resources of health and social care professionals around the goals of men nearing and after release from prison who have common mental health problems (Figure 5; prioritised functions 1-3 are in the top row; Prioritised function 4 is represented by ‘shared understanding’ and ‘shared action plan’)

Figure 5.

How ENGAGER mobilises resources around a prison leaver’s goals during and after the intervention.

The distinct contribution of realist FPE

This article brings together in constructive dialogue different approaches to evaluation (realist evaluation, formative evaluation, process evaluation) and shows how the resulting realist FPE contributed to elaborating theory and changes to practice to good effect. We demonstrate realist FPE in a complex project that crosses conventional fields (criminal justice system, health, and social care).

In the piloting and feasibility stage, intervention strategy is necessarily only partially understood. In formative evaluation original intentions should and can be adapted through a process of trial, error and reflection. A realist approach to formative process evaluation makes a distinct contribution over and above other forms of formative and action research through its special focus on theory formalisation and theory abstraction. It also addresses the issue that new interventions are always introduced into existing service arrangements, and that their success is dependent on the extent to which they gel or jar with existing provision, or can be adapted to enable them to work with existing services.

This article makes a unique contribution to an emerging body of scholarship. It brings together different evaluation traditions in a novel way to show how a realist FPE develops underlying programme theory and iteratively feeds this in to intervention development and implementation knowledge to prepare adaptable and scalable interventions for full trial. Realist FPE takes a realist approach to formative process evaluation, uniquely adding in-depth theory formalisation and abstraction to other formative, process, and action research approaches. Its focus on intervention function and moderating context in design and evaluation of complex interventions uniquely creates interventions that can be tailored to individuals (Hawe et al., 2004, 2009) and scaled for full trial. We demonstrate how realist FPE is particularly useful at the feasibility stage of intervention development in cases where there are no comparable pre-existing interventions to evaluate (see Fletcher et al., 2016; Moore et al., 2015).

This article makes a novel methodological contribution to the developing field of realist evaluation. The key methodological issues in realist research can be summarised as conceptual, such as ‘what is a mechanism?’ (Dalkin et al., 2015; Williams, 2018); paradigmatic, such as debates about the (in)commensurability of experimental and realist approaches (Bonell et al., 2012, 2016, 2018; Marchal et al., 2013; Van Belle et al., 2016); and practical, such as how a realist approach can be applied in low-resource settings (Marchal et al., 2018) or in economic evaluation (Anderson et al., 2018). Our article addresses the last type of these methodological challenges, namely the gap in knowledge about the utility of and how to apply a realist approach in early phase studies to inform intervention development. It also adds some empirical evidence to the paradigmatic debate about whether RCTs can be ‘realist’. Realist approaches used in intervention development in preparation for RCTs can produce interventions, such as ENGAGER, that consist of a transferable framework of core intervention functions. Critically, the underlying programme theory details how to adapt this ‘mid-range intervention’ to local context (local service provision, individual deliverer or participant contexts, and so on). In this way, the intervention itself can ‘hold’ some of the complexity and account for critical (intended) variation in implementation and delivery that the RCT would otherwise control out (see also Hawe et al., 2004). In such RCTs, fidelity to delivery of the mid-range intervention includes its adaptation to each local context, according to the underlying programme theory. In the ENGAGER full trial we have added a further level of ‘meta-supervision’ in order to deliver the intervention optimally by responding to changing circumstances (prison, organisational configuration, practitioners). The process evaluation alongside the main trial will explore this function and also recommend how to balance the need to continue adapting in any future spread without moving too far from key function.

We now reflect on our learning about this new methodological approach.

Realist FPEs unique contribution of theory formalisation and abstraction

Our specifically realist approach to formative process evaluation uniquely adds a focus on building the underlying theory about how an intervention works, for whom, and under what circumstances. This allows articulation of key intervention functions and critically how they interact. This contribution is critical in creating interventions that are 1) adaptable, and 2) scalable. It supports the prioritisation of intervention resources. It produces information at different levels of abstraction that enable both clear and detailed articulation of what practitioners need to do, for whom, in what circumstances, and, more abstractly, of the overarching ways in which the intervention achieves its outcomes. We discuss each of these benefits in turn.

Creating adaptable interventions: Prioritising and elaborating key intervention functions

Realist FPE leads to a clear articulation prior to full trial of what works about an intervention, whom it works for, under which circumstances and in what way. Our Realist FPE of the ENGAGER intervention clarified that it worked through four inter-related intervention functions that together led to the mobilisation of the resources of health and social care professionals around prison leavers’ goals. Realist FPE’s focus on encapsulating key intervention functions (rather than form) enables practitioners to tailor the resulting intervention (see Hawe et al., 2004, 2009). The clarification of core intervention functions and their contextual contingency produces clear articulation of how to tailor delivery to individual and local service provision contexts. Interventions that are more adaptable to individual/local context are more likely to work for more people and thus to be ‘effective’ when undergoing outcomes evaluation in a trial. Where function is clearly articulated in underlying programme theory, a trial can focus on fidelity to function rather than form (Hawe et al., 2004) and both intervention and trial can take in to account more of the critical contextual variation.

Creating scalable interventions: Implementation and delivery knowledge for a trial

Realist FPE provides rich implementation and delivery knowledge for a main trial: the analysis feeds in to both the form of the implementation platform (e.g. increasing the frequency and changing the style of supervision) and to its content (e.g. providing detailed information in the manual about what practitioners should do, for whom, when, and in what way). This knowledge produces critical information for full trial of how the intervention can be adapted and scaled for success in other implementation contexts. This increases the likelihood of a successful future trial by creating optimal implementation and delivery in new contexts and thus the likelihood of an intervention being delivered in a way that optimises the chances of it being effective. A clearly articulated underlying theory will also provide a clear theory for testing in the main trial process evaluation, which will inform any further roll-out of the intervention.

Prioritising intervention resources

Realist FPE informs decisions to prioritise (or drop) components of the intervention through elaboration of key intervention functions. By building understanding in an iterative way about how a new intervention works and whether specific components appear helpful or not and are context contingent or not, realist FPE provides timely information regarding how to focus limited resource in delivery. In the evaluation embedded in the ENGAGER pilot trial, it was clear early on that meeting men multiple times in the prison setting to build trust and the shared understanding was generally not feasible. At the same time, interviews with prison leavers revealed that release day working was considerably more important to them than prison contacts. Practitioners were advised to prioritise release day work (despite it taking a full day) over multiple contacts in prison. Focusing limited resource on what is more likely to work is critical for the success of complex interventions.

Levels of analysis, abstraction and outputs

In any qualitative analysis different levels of abstraction of the findings will be required for a variety of project outputs and stakeholders. Through a process of iterative programme theory development at increasing levels of abstraction, Realist FPE produces evidence at three levels of abstraction: 1) emergent findings at the level of the data (i.e. no formal analysis performed) inform rapid changes to the intervention during the pilot trial; 2) detailed findings relating to key intervention functions at the penultimate stage of analysis (consolidated if-then descriptions) add contextual nuance to the manual and supervision content to support tailored delivery; 3) final analysis findings in the form of ‘core programme theories’ that emphasise key overarching functions and their interaction and further elaborate at an abstract level how the intervention works, for whom, and under what circumstances.

Our understanding of the benefits of these three levels of abstraction of realist FPE findings to intervention development came to us somewhat serendipitously, during the responsive and sometimes intuitive decision-making required by evaluations embedded within a wider programme of research. We describe this serendipity in relation to each of the three levels of abstraction.

The rapid and iterative changes introduced to the ENGAGER intervention and delivery in response to emerging findings during the pilot were not planned but maximized the potential to create an optimum model prior to full trial. These changes share similarities with other quality improvement approaches such as the use of Plan-Do-Study-Act cycles in Quality Improvement (see Taylor et al., 2013) and action research (e.g. Waterman et al., 2001).

After the pilot trial, during the formal Framework analysis, time pressures required that the analysis feed in to full trial intervention design prior to the final stage of analysis. At this stage of analysis groups of if-then statements describe the data within each column of the framework (i.e. related to one CMO configuration in the initial programme theory). These if-then statements, although abstracted from the data, still contained fine-grained contextualised information that at the next level of abstraction (the final stage of Framework analysis) was necessarily lost. This detailed contextual information provided critical detail for practitioners to tailor their delivery to the context and the individual. This was invaluable for writing practical manual content in the form of ‘if this, then that, unless the other, in which case try this’.

The final stage of the Framework analysis involved emerging core programme theories from across all groups of if-then statements. This stage resulted in elaborated, prioritised and abstract theory that was suitable for articulating the more generalised intervention model and core programme theory for the full outcome evaluation.

The challenges of applying realist FPE

Managing the level of context

A challenge for using realist evaluation alongside a pilot trial to inform intervention development was managing the level of context. For pragmatic reasons we focused mostly on the individual-level contexts at play in the dyadic relationships that are the vehicle for change in the ENGAGER intervention. This focus supported the formative development of the ENGAGER practitioner manual, supervision, and training to support practitioners to deliver the intervention in a way that was responsive to prison leavers’ needs in light of their individual-level contexts. The pilot only operated in two settings and we did not focus on fit of intervention into the two contrasting contexts. This is reflected in the final ENGAGER theory presented, where contexts are mostly individual- rather than higher-level (e.g. organisational or cultural).

Dealing with complexity

In this article we describe and illustrate the process of iterative development of programme theory at increasingly higher levels of abstraction and how this process identifies and elucidates the key intervention functions (and the interaction between functions that creates emergent impact greater than the sum of the individual functions). The ENGAGER intervention is a complex intervention operating in a complex system, and pinning down core mechanisms and the contexts on which they are contingent to produce particular outcomes (CMOs) is challenging. There are many inter-related mechanisms operating at many levels across the different prison and community contexts, and it was sometimes difficult to maintain the analytical gaze on the intended mechanisms as described in the initial programme theory. Using the analytical tool of discrete CMOs within a Framework analysis approach puts to one side much (but certainly not all) of the complexity, inter-relatedness, and contextual richness, but this was necessary to end up with useable conclusions from the evaluation to inform intervention development for the full RCT.

The challenge of bounding scope

A classic challenge when using realist approaches is prioritising which aspects of the initial programme theory to evaluate and bounding the extent and/or depth of the evaluation. The initial programme theory was large and complex, with many inter-related parts. Choosing which were likely to be important and thus which to focus evaluation resource on proved challenging at first. Working in multi-disciplinary teams with depth and breadth of expertise and experience can result in a broad range of ideas of what is likely to ‘work’ in the intervention to be piloted. Prioritisation of theories about what works was achieved through the process of iterative consultation and input in to the developing theory from a wider group of topic experts, experienced policy-makers and implementers, and most critically, people with lived experience and the practitioners and supervisors delivering the intervention.

Conclusions and recommendations

Realist FPE informs early phases of intervention development and evaluation (MRC guidelines) to produce complex interventions that are both adaptable to individual contexts and scalable to new implementation contexts. Over and above other forms of formative evaluation and action research its fundamental value is in the rich contextualised underlying programme theory that it produces. It is this programme theory and the process of increasing its level of abstraction through the stages of the evaluation, that produces the different levels of abstraction of theory that can both inform detailed delivery information for practitioners – how to adapt the delivery of the intervention to different aspects of the individuals to whom they are delivering it and their contexts – as well as the more abstract programme theory that articulates the overarching mechanisms, how they interact with each other, and the contexts that moderate them. This is a rich understanding of an intervention in terms of the contexts that enable the intervention resource to interact with people’s reasoning to produce desired outcomes. This contextualized theory helps people involved in delivering and implementing an intervention to make informed decisions as to how to best adapt it to both individuals receiving it and local service provision.

We recommend that the MRC intervention development and evaluation guidelines describe realist FPE and its unique contribution to the development and evaluation of complex interventions for full trial that are more adaptable and scalable.

Supplemental Material

Supplementary_material_EVI – Supplemental material for Building programme theory to develop more adaptable and scalable complex interventions: Realist formative process evaluation prior to full trial

Supplemental material, Supplementary_material_EVI for Building programme theory to develop more adaptable and scalable complex interventions: Realist formative process evaluation prior to full trial by Sarah Louise Brand, Cath Quinn, Mark Pearson, Charlotte Lennox, Christabel Owens, Tim Kirkpatrick, Lynne Callaghan, Alex Stirzaker, Susan Michie, Mike Maguire, Jennifer Shaw and Richard Byng in Evaluation

Footnotes

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/ or publication of this article: ENGAGER is funded by the National Institute for Health Research (NIHR) under its Programme Grant for Applied Research Programme (Grant number: RP-PG-1210-12011). This research was also supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula at the Royal Devon and Exeter NHS Foundation Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health.

Sarah Louise Brand’s research focus is improving health and/or social service transitions for vulnerable populations. Sarah uses realist approaches to build implementation/delivery knowledge to tailor interventions to individuals/contexts.

Cath Quinn (Weyer Brown) is a mixed methods researcher, particularly within realistic evaluation frameworks. Cath’s research has focused on the mental health of socially marginalised groups, particularly offenders.

Mark Pearson’s research focuses on knowledge translation. He uses theory-driven methods, including realist approaches, to conduct research that informs the development and evaluation of complex interventions.

Charlotte Lennox is a chartered psychologist and lecturer in Forensic Psychology at the University of Manchester.

Christabel Owens is an associate professor of Public Mental Health at the University of Exeter, where she specialises in the use of qualitative research methods.

Tim Kirkpatrick is a research fellow at Plymouth University and Programme Manager for the Engager Study.

Lynne Callaghan works in community and health psychology, with an interest in addressing health inequalities in traditionally marginalised groups, in particular those within the criminal justice system.

Alex Stirzaker is a clinical psychologist within Swindon LIFT primary care psychological therapy service. Her research focuses on how psychological understanding helps develop interventions that support prison leaver mental health.

Susan Michie is a professor of Health Psychology and Director of the Centre for Behaviour Change, University College London. She applies health behaviour change theory to intervention development, evaluation and implementation.

Mike Maguire is a professor of Criminology, University of South Wales, with interest in prisoner rehabilitation/resettlement. He is a member of the Correctional Services Accreditation Panel.

Jennifer Shaw is a professor in Forensic Psychiatry at University of Manchester interested in translating research evidence in to practice to improve forensic mental health services.

Richard Byng is a general practitioner and mixed methods researcher with an interest in developing and evaluating complex interventions for those with complex needs and from disadvantaged groups.

References

Anderson

Hardwick

Pearson

et al . (2018) Using realist approaches to explain the cost-effectiveness of programmes. In: Emmel

Greenhalgh

Manzano

et al . (ed.) Doing Realist Research. London: SAGE.

Bhaskar

(2013) A Realist Theory of Science. London: Routledge.

Bonell

Fletcher

Morton

et al . (2012) Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine 75: 2299–306.

Bonell

Moore

Warren

et al . (2018) Are randomised controlled trials positivist? Reviewing the social science and philosophy literature to assess positivist tendencies of trials of social interventions in public health and health services. Trials 19: 238.

Bonell

Warren

Fletcher

et al . (2016) Realist trials and the testing of context-mechanism-outcome configurations: A response to Van Belle et al. Trials 17: 478.

Brooker

Repper

Beverley

et al . (2002) Mental Health Services and Prisoners: A Review. Sheffield: Mental Health Task Force.

Byng

Quinn

Sheaff

et al . (2012) COCOA: Care for Offenders Continuity of Access. Final report NIHR Service Delivery and Organisation programme. Available at: http://www.netscc.ac.uk/hsdr/files/project/SDO_FR_08-1713-210_V01.pdf

Craig

Dieppe

Macintyre

et al . (2013) Developing and evaluating complex interventions: The new Medical Research Council guidance. International Journal of Nursing Studies 50: 587–92.

Dalkin

Greenhalgh

Jones

et al . (2015) What’s in a mechanism? Development of a key concept in realist evaluation. Implementation Science 10: 49.

10.

Evans

Murphy

Scourfield

(2015) Implementation of a school-based social and emotional learning intervention: Understanding diffusion processes within complex systems. Prevention Science 16: 754–64.

11.

Fletcher

Jamal

Moore

et al . (2016) Realist complex intervention science: Applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation 22: 286–303.

12.

Forrester

Exworthy

Olumoroti

et al . (2013) Variations in prison mental health services in England and Wales. International Journal of Law and Psychiatry 36: 326–32.

13.

Georgiadis

Byng

Coomber

et al . (2016) The social, relational and mental health characteristics of justice-involved men in the south-west England. The Journal of Forensic Psychiatry & Psychology 27: 835–52.

14.

Greenhalgh

Humphrey

Hughes

et al . (2009) How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Quarterly 87: 391–416.

15.

Grubin

Birmingham

Mason

(1997) The Durham Remand Study. Report for HM Prison Service/Northern and Yorkshire Regional Health Authority.

16.

Hawe

(2015) Lessons from complex interventions to improve health. Annual Review of Public Health 36: 307–23.

17.

Hawe

Shiell

Riley

(2004) Complex interventions: How ‘out of control’ can a randomised controlled trial be? BMJ: British Medical Journal 328: 1561–3.

18.

Hawe

Shiell

Riley

(2009) Theorising interventions as events in systems. American Journal of Community Psychology 43: 267–76.

19.

Hawkins

(2014) The case for experimental design in realist evaluation. Learning Communities: International Journal of Learning in Social Contexts 14: 46–59.

20.

Kirkpatrick

(2018) Evaluation of a complex intervention (Engager) for prisoners with common mental health problems, near to and after release – study protocol for a randomised controlled trial. BMJ Open 8(2): e017931.

21.

Lennox

Kirkpatrick

Taylor

et al . (2017) Pilot randomised controlled trial of the ENGAGER collaborative care intervention for prisoners with common mental health problems, near to and after release. Pilot and Feasibility Studies 4: 15.

22.

Macintyre

Petticrew

(2000) Good intentions and received wisdom are not enough Journal of Epidemiology and Community Health 54(11): 802–3.

23.

Manzano

(2016) The craft of interviewing in realist evaluation. Evaluation 22: 342–60.

24.

Marchal

Kegels

Van Belle

(2018) Theory and realist methods. In: Emmel

Greenhalgh

Manzano

et al . (eds) Doing Realist Research. London: SAGE.

25.

Marchal

Westhorp

Wong

et al . (2013) Realist RCTs of complex interventions – An oxymoron. Social Science & Medicine 94: 124–8.

26.

Moore

Audrey

Barker

et al . (2015) Process evaluation of complex interventions: Medical Research Council guidance. BMJ 350: h1258.

27.

Pawson

(1996) Theorizing the interview. British Journal of Sociology 47(2): 295–314.

28.

Pawson

(2006) Evidence-based Policy: A Realist Perspective. London: SAGE.

29.

Pawson

(2013) The Science of Evaluation: A Realist Manifesto. London: SAGE.

30.

Pawson

Greenhalgh

Harvey

et al . (2005) Realist review-a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy 10: 21–34.

31.

Pawson

Tilley

(1997) Realistic Evaluation. London: SAGE.

32.

Pearson

Brand

Quinn

et al . (2015) Using realist review to inform intervention development: Methodological illustration and conceptual platform for collaborative care in offender mental health. Implementation Science 10: 134.

33.

QSR International Pty Ltd (2012) NVivo qualitative data analysis Software. 10 ed.

34.

Ritchie

Spenser

(1994) Qualitative data analysis for applied policy research. In: Bryman

Burgess

(eds) Analyzing Qualitative Data. London: Routledge.

35.

Sayer

(2000) Realism and Social Science. London: SAGE.

36.

Scorgie

Vearey

Oliff

et al . (2017) ‘Leaving no one behind’: Reflections on the design of community-based HIV prevention for migrants in Johannesburg’s inner-city hostels and informal settlements. BMC Public Health 17: 482.

37.

Singleton

Gatward

Meltzer

(1998) Psychiatric morbidity among prisoners in England and Wales. London: Stationery Office.

38.

Stake

(1995) The Art of Case Study Research. London: SAGE.

39.

Taylor

McNicholas

Nicolay

et al . (2013) Systematic review of the application of the plan–do–study–act method to improve quality in healthcare. BMJ Quality and Safety 23(4): 290–8.

40.

Van Belle

Wong

Westhorp

et al . (2016) Can ‘realist’ randomised controlled trials be genuinely realist? Trials 17: 313.

41.

Waterman

Tillen

Dickson

et al . (2001) Action research: A systematic review and guidance for assessment. Health Technology Assessment 5(23): 1–166.

42.

Williams

(2018) Making up mechanisms in realist research. In: Emmel

Greenhalgh

Manzano

et al . (ed.) Doing Realist Research. London: SAGE.

43.

Williamson

(2006) Improving the health and social outcomes of people recently released from prisons in the UK: A perspective from primary care. London: The Sainsbury Centre for Mental Health.

44.

Wong

Westhorp

Manzano

et al . (2016) RAMESES II reporting standards for realist evaluations. BMC Medicine 14: 96.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB