A new opportunity for enhancing trial efficiency: Can we investigate intervention implementation processes within trials using SWAT (study within a trial) methodology?

Abstract

Background

A study within a trial (SWAT) is a self-contained research study embedded within one or more host trials to evaluate or explore alternative ways of delivering or organising a particular trial process. There is limited evidence of SWATs evaluating trial processes other than recruitment and retention.

Purpose

Embedding a SWAT into a host trial provides a potential method of evaluating an aspect of intervention implementation, such as engagement or compliance with the intervention.

Research Design

This paper presents two case studies of SWATs which aim to test the use of video animations to improve intervention implementation, with particular focus on enhancing understanding, engagement and compliance. These are important aspects of intervention implementation as they are directly linked to intervention effectiveness and therefore, important to study.

Results

In this paper, we present the potential benefits of conducting SWATs of intervention implementation processes as well as discussing the methodological considerations for embedding a SWAT of this nature within a host trial. Benefits include the opportunity to test minor refinements to intervention implementation within trials through robust randomised SWATs, and the possibility of increasing trial efficiency by maximising the quality or quantity of intervention implementation. Methodological considerations surrounding the design and conduct of the SWAT as well as statistical and health economics considerations are discussed in this paper.

Conclusions

This paper presents a novel application of SWAT methodology in investigating intervention implementation processes within trial conduct.

Introduction

A study within a trial (SWAT) is a self-contained research study embedded within one or more host trials to evaluate or explore alternative ways of delivering or organising a particular trial process. ¹ SWATs contribute to the evidence base for improving trial efficiency, without affecting the scientific integrity of the host trial. Typically SWATs have evaluated small refinements to trial processes relating to recruitment and retention,^2,3 although SWATs investigating how best to refine other trial processes, such as data quality, monitoring risks or dissemination, are becoming more common.^4,5

Intervention implementation can be defined as ‘the structures, resources and process through which delivery of the intervention is achieved and the quantity and quality of what is delivered ⁶ ’. To date, SWATs are not widely used to evaluate competing trial processes relating to an aspect of intervention implementation. The few SWAT examples identified within the Northern Ireland SWAT repository ⁷ test the influence of different healthcare professionals on intervention delivery ⁸ ; reminders to intervention recipients to improve adherence ⁹ ; and the use of additional virtual follow-ups and automatic, rather than manual, intervention adjustments on intervention compliance. ¹⁰ These studies are crucially different to recruitment or retention SWATs in that they can only operate within the intervention arm(s) of the host trial. It is unclear, however, if these studies have systematically assessed whether embedding a SWAT within only the intervention arm of a randomised trial compromises the scientific integrity of the host trial or presents other methodological or conduct challenges.

In this paper, we propose that such SWATs are an informative and potentially important methodology for the understanding and refinement of implementation processes of complex interventions. We present two examples where we have designed studies to test the use of video animations to improve intervention implementation, particularly in relation to enhancing understanding, uptake, engagement and compliance. In this context, uptake is defined as the participant proceeding with the intervention following the initial visit from an intervention deliverer. Engagement is an important aspect of intervention implementation, which in this context, refers to a participant’s understanding and responsiveness to the intervention. ¹¹ An essential element of understanding whether an intervention is delivered with fidelity is compliance, which refers to the extent to which intervention content, the frequency, and duration of the intervention delivery are as intended. ¹¹ We consider how this novel application of SWAT methodology fits within the context of the updated MRC guidance (2021) on developing and evaluating complex interventions. ¹² We discuss benefits of conducting SWATs of intervention implementation processes and consider some methodological and conduct issues for the SWAT itself and for the host trial, highlighting where these may differ from recruitment or retention SWATs.

Case studies

To illustrate our proposal, we summarise the key features of two SWATs designed to evaluate an additional mode of information provision during intervention delivery (Figures 1 and 2). Both SWATs are embedded within definitive randomised trials and test a similar intervention, the use of video animation, but differ in key aspects of host trial design, intervention, setting and population; and in the SWAT rationale, randomisation and outcomes. The differences in design and conduct present different challenges, as discussed later in the paper (Discussion).

Figure 1.

SWAT 1 summary (awaiting upload to the SWAT repository).

Figure 2.

SWAT 2 summary. ¹⁴

Discussion

Fit with guidance for complex intervention research

The development and evaluation of complex interventions is iterative in nature and usually informed by a range of methods with different approaches taken. Recent updated guidance from the UK Medical Research Council (MRC) published in 2021 recommends that evidence on the impact of complex interventions in health care settings accumulates over four phases: intervention development or identification, feasibility, evaluation and implementation. ¹² Intervention refinement is highlighted as a core element to consider at each stage of the new framework

Within the earliest phase of intervention development there is no universally accepted methodological approach for identifying the necessary components of a complex intervention, including how it is best implemented. A recent systematic methods overview of approaches employed for intervention development identified seven categories (partnership-generated; target population-centred; evidence and theory-based; implementation-based; efficiency-based; stepped or phased development intervention-specific methods) or some combination of these categories. ¹⁵ As a result a mix of methods are often employed to explore key areas of uncertainty regarding intervention development and refinement, such as evidence synthesis, stakeholder workshops, qualitative methods (e.g. interviews, focus groups) and quantitative methods (e.g. observational studies, feasibility trials). Irrespective of the different methods employed, evidence is built up iteratively based on an evolving programme theory, with changes to the intervention supported by transparent reporting of the rationale for changes. Programme theory describes how an intervention is expected to lead to its effects and under what conditions. ¹⁶

Process evaluations conducted alongside feasibility and pilot trials often then test key areas of uncertainty in relation to intervention development and implementation, as well as trial procedures. ¹⁷ During this phase of research, mixed-method research can identify a number of different refinements with potential to improve the implementation of a complex intervention in real world settings. However, research methods employed during the feasibility stage are not designed to test the relative effectiveness of different refinements of a complex intervention, nor can it provide definitive evidence on the impact of potential refinements to an intervention which might enhance the underlying programme theory.

Crucially, the MRC guidance recommends that intervention development and refinement is rare in the evaluation phase of efficacy and effectiveness research, the implication of which is that interventions, (or at least the core components), do not change or evolve within the context of a definitive trial. ¹² We agree that core components of an intervention should not change during a definitive trial evaluation. Instead, we suggest that SWATs can be used to test ‘minor’ or ‘small’ refinements in intervention implementation processes in trial evaluations. We outline specific pre-conditions under which these SWATs would be appropriate:

(1) two modes of delivering a specific aspect of the intervention are indicated, based on evolving programme theory and evidence generated from the intervention feasibility phase and process evaluation;

(2) there is equipoise regarding which of the competing delivery modes is likely to prove most effective;

(3) the uncertainties regarding intervention implementation are localised to one aspect of the intervention programme theory (e.g. two options with potential to improve an outcome such as intervention uptake).

Coupled with these necessary pre-conditions, we propose that SWATs testing refinements in intervention implementation should only be undertaken when there is potential to generate new knowledge relating to how best implement interventions and that SWAT results are likely to be more widely generalisable to building our knowledge base around intervention design.

Benefits

Currently, few SWATs focus on intervention implementation processes within trials. These SWATs have the potential to test a minor refinement to the intervention delivery procedure, such as those designed to improve engagement or compliance with the intervention, at either intervention recipient or deliverer level or both.

We recognise that if trial conduct issues relating to how well an intervention was delivered were identified during a definitive trial, then responsive amendments to these intervention processes may have been made. For example, a trial team may have taken action to improve intervention engagement, compliance or fidelity, by providing extra information to intervention recipients, implementing additional reminders to intervention recipients, deliverers or both, delivering booster or top-up training or increasing the frequency and/or duration of supervision. These actions may or may not have been introduced systematically and equally across all those delivering (or receiving) the intervention. The value of such amendments may have been explored within a process evaluation, or with a post-hoc quantitative analysis, such as through the use of causal modelling. What we propose here is that minor amendments to the implementation of the trial intervention could be evaluated systematically through randomised SWATs, as these studies are designed to provide a robust, unbiased assessment of the effect of such changes.

Furthermore, we propose that SWATs of this type could increase trial efficiency by maximising the number of trial participants receiving (more of) the intervention as intended. This could help avoid diluting the potential to detect treatment effects, which could arise with reduced intervention implementation or with lower levels of intervention compliance. Trials, in which intervention delivery is better optimised, require fewer numbers of patients than those where treatment effect dilution is accounted for in power calculations.

Considerations for the design and conduct of the study within a trial

Study within a trial intervention choice

The choice of intervention implementation process under evaluation in the SWAT will present challenges. As described above, we are not proposing SWATs to test changes to the core components of an intervention. Researchers need to consider which aspects of the strategies to facilitate intervention implementation can be subject to minor amendments, without changing the trial intervention itself. Results from earlier intervention development, refinement, feasibility testing and process evaluation can be used to identify where there are uncertainties about the best way to deliver an aspect of the intervention. SWATs can be designed to test small refinements to that particular intervention delivery process.

Intervention implementation is a broad concept, encompassing both the quantity and quality of intervention delivery. Understanding differing elements of implementation, such as uptake, engagement or compliance, will also help to identify a suitable target for intervention within a SWAT. It is important to consider uptake and engagement when measuring implementation since these link directly with intervention effectiveness. Similarly, compliance is important in understanding the extent to which the intervention is delivered with fidelity.

Delivery of the study within a trial intervention

In recruitment or retention SWATs, delivery of the SWAT intervention is often fully within the control of the central trial team or the research team at site. For SWATs evaluating intervention implementation processes, delivery of the SWAT intervention is often by a third party, such as a clinical team or voluntary sector organisation, outside of the central trial/SWAT team and this brings additional challenges. It can be difficult to ensure the SWAT intervention is delivered at the correct time, in the correct manner or to the correct participants. When the SWAT intervention is dependent on technology, this can bring additional challenges, especially in multi-centre trials if different technology, equipment or software varies across sites.

Ensuring that there is no contamination within an intervention implementation SWAT can be challenging in individually randomised host trials, even if adopting cluster randomisation for the SWAT. Intervention deliverers may share resources or methods for delivering interventions and it becomes more complex for the trial team to identify and minimise all contamination threats. The importance and value of the SWAT can be highlighted during training to maximise adherence and prevent contamination, but ultimately the central trial team has limited control. Furthermore, duplication of activities for the trial team can arise when planning training for intervention deliverers. Separate training sessions may be required for intervention delivery with and without the SWAT intervention, and this will require clear communication with the intervention deliverers about the SWAT to avoid potential confusion.

Statistical considerations

Randomisation level: In common with recruitment or retention SWATS, intervention implementation SWATs require careful thinking about the most appropriate level at which to randomise. Randomisation within the SWAT does not automatically follow the randomisation choice for the host trial. Often when embedding SWATs in host trials randomising individual patients, the SWAT randomisation needs to be undertaken at the level of the intervention deliverer, as the implementation process is modified. This aims to maximise correct use of the alternative implementation processes and avoid the potential for contamination between SWAT arms. This is seen in our first example (Figure 1).

In contrast in cluster randomised host trials, when both the trial intervention and the alternative implementation process tested in the SWAT are delivered at the cluster level, cluster-level randomisation should also be used for the SWAT. This is illustrated with our second example, where SWAT randomisation follows immediately after a cluster is randomised to the intervention arm in the host trial, to ensure that the alternative intervention implementation process is included at the intervention training session for staff in that cluster.

SWAT outcomes: Recruitment or retention SWATs typically adopt simple outcomes, such as recruitment rate or follow-up rate, which are relevant to all trials. SWATs testing alternative intervention implementation processes are faced with a more difficult choice as to the most appropriate outcome measure. As intervention implementation is a broad concept, encompassing both the quantity and quality of intervention delivery, there is no single, common outcome appropriate to use within these SWATs. Instead, the SWAT outcome should be carefully aligned to the aspect of implementation that it is seeking to improve. The first example SWAT (Figure 1), adopts a simple outcome measure of participant uptake as measured by patient attendance at the first intervention session, as the SWAT evaluates an additional information provision method used only at initial contact between intervention deliverers and recipients. Other implementation processes may be designed to improve other aspects such as intervention compliance or intervention fidelity, in which case SWAT outcome should measure rates of compliance or fidelity, such as in the second SWAT example (Figure 2).

Sample size: The sample size for a SWAT is usually based on the available sample in the host trial, which can lead to an individual SWAT being underpowered. Potential lack of power, especially to detect the modest improvements in SWAT outcomes most likely to arise from minor changes to a trial process, is an issue across all SWATs. This problem is exacerbated, however, in SWATs investigating intervention implementation processes, as the SWAT would typically include participants or sites only in the intervention arm of the host trial.

Meta-analysis of SWATs is encouraged to increase the power of the analysis of SWATs ¹⁸ and to provide more precise estimates of the effect of changes in trial processes. For SWATs investigating intervention implementation processes, meta-analysis may prove challenging than that for recruitment or retention SWATs, as there may be high heterogeneity due to variations in the host trials, the delivery settings, the outcomes measured for the SWAT, or the type of implementation process evaluated in the SWAT.

Analysis: The timing of analysis of the SWAT will depend on the outcomes being considered. Trial teams may be keen to implement any findings showing improved trial processes or intervention delivery, so interim analysis for the sample of patients with data available before the end of the trial may be appropriate for outcomes that are not analysed as main outcomes in the host trial. If, however, SWAT outcomes are related to the primary and secondary outcomes of the host trial, the analysis of the SWAT may need to be delayed until the end of the host trial follow-up period, to preserve the scientific integrity of the host trial.

Other study within a trial conduct considerations

When embedding a SWAT within a host trial, the aim is to achieve the objectives of the SWAT without compromising the scientific integrity host trial. There can be logistical issues in incorporating a SWAT in a host trial(s) that need consideration, ideally in advance. Design issues for randomisation and outcomes have been discussed above, but there are also more practical considerations. Deciding how and when to conduct the randomisation for the SWAT in the least disruptive manner for the host trial is vital. Trial and data management processes in the host trial may require adaptation to accommodate the SWAT data collection. Planning this carefully in advance will help to avoid confusion and minimise burden for participants, staff and researchers. In common with recruitment and retention SWATs, it is likely that additional work will be required to embed the SWAT in a host trial and this may have financial as well as workload implications, so should be costed for in funding applications. Simple costs relating to additional printing, postage and phone calls may be required, but as can be seen from our example SWATs, it is likely additional costs, for instance relating to video animation design and production or training intervention deliverers, can significantly increase the overall cost of the SWAT.

Considerations for the host trial

Statistical analysis

Recruitment and retention SWATs do not affect the host trial’s intervention processes and analysis of the effectiveness or efficacy of the host trial intervention can be conducted without reference to the SWAT. When undertaking SWATs designed to test changes in intervention implementation in the host trial, analysis to derive treatment effect estimates must account for the possibility of an interaction effect between the intervention in the host trial and the SWAT intervention. The analysis approach should be chosen to reflect the key research question that the trial is looking to answer, for example, whether to assess the host trial intervention effect averaged over the two SWAT arms or whether the groups should be assessed separately. The analysis plan for the host trial should pre-specify how the SWAT will be accounted for in the host trial analysis. The impact of the SWAT intervention on intervention delivery should also be accounted for during secondary analyses, and when examining mediating effects, for instance when adopting causal inference approaches.

Health economics considerations

Health economic evaluation methods are often embedded within definitive trials to estimate the cost and cost-effectiveness of new technologies. ¹⁹ SWATs evaluating the effectiveness of alternative methods for delivering an intervention may therefore have implications for the resources costed. For example, the new intervention delivery method being evaluated may incur additional upfront costs to develop it (for example, designing and producing a video animation), as well as affecting the ongoing costs associated with intervention implementation (e.g. staff time taken to deliver intervention may be increased, or decreased depending on the change to delivery processes). The costs and resources used within the context of a SWAT should be documented within the health economic evaluation, and subsequent analysis adjusted accordingly. The analysis approach may mirror that taken for the statistical analysis. Additionally, the economic evaluation may require sensitivity analyses with varying assumptions relating to the extent to which the SWAT intervention is adopted within the trial intervention if it were to be implemented into clinical practice.

Considerations for process evaluations within the host trial

Process evaluations are valuable for further understanding the complexities of intervention implementation and are therefore complementary to conducting a SWAT. Whereas a SWAT is used to evaluate one specific change in the trial process, process evaluations take a broader view and can investigate in other ways, multiple changes in the trial. Process evaluations typically incorporate qualitative aspects such as interviews and observations where researchers can get a more in-depth understanding of the factors that influence engagement and compliance, and how much the participants understand the intervention. ⁶ When planning the SWAT researchers should consider developing a separate programme theory and logic model which includes the aspect of delivery under evaluation e.g. video animation. This will assist in understanding the mechanisms by which the additional intervention aspect may or may not have an effect. Researchers will also need to consider what will be examined as part of a process evaluation as this may influence the data collection methods. For example, questions specific to the SWAT may need to be added to some of the topic guides for semi-structured interviews.

Depending on the trial design, there can be challenges involved when conducting a SWAT alongside a process evaluation. If participants are individually randomised as part of the trial, researchers should be mindful of potential burden if participants are asked to take part in separate interviews, for the host trial process evaluation and the SWAT. To minimise the potential burden on participants, process evaluation researchers and SWAT researchers could ensure they do not contact the same participants for interviews. However, this does reduce the pool of potential participants available for each element which can be problematic when trying to purposively sample. Where possible, this could be avoided entirely by adding additional questions relevant to the SWAT to process evaluation interview topic guides, thus eliminating the need for separate interviews.

Conclusion

In this paper, we have proposed that it is possible to embed SWATs to investigate and refine a relatively unexplored area of trial conduct, namely intervention implementation processes. The benefits of this are that it provides a systematic and rigorous way of testing minor refinements which may enhance the intervention implementation process. If the refinements are shown to enhance intervention implementation, they could be used as evidence based strategies to facilitate implementation of complex interventions in future trials.

There are methodological considerations associated with the design and conduct of the SWAT pertaining to intervention choice and delivery as well as statistical considerations surrounding randomisation, outcomes and analysis. Considerations for the host trial relating to statistical analysis and health economics have also been discussed alongside unique considerations associated with conducting a process evaluation alongside a SWAT.

Study within a trials evaluating trial processes relating to intervention implementation could be particularly useful at the feasibility stage, ¹² when more refinement of an intervention is permitted. Caution must be applied within the context of definitive trials and the impact of such SWATs on the host trial should be carefully considered, especially when assessing the influence on treatment effect estimates within the host trial.

We suggest that these SWATs supplement, rather than replace, evidence from process evaluations embedded in trials, and together address key uncertainties about the best ways to implement interventions into practice. We would encourage other trialists to consider the value of SWATs to systematically explore minor amendments to implementation processes when evaluating complex interventions, but be mindful of methodological implications.