A systematic review of economic evaluations alongside studies within a trial (SWATs) for improving recruitment and retention in randomised controlled trials

Abstract

Aim

To review the cost-effectiveness of strategies to improve participant recruitment and retention in randomised controlled trials.

Methods

All included studies from the latest Cochrane recruitment and retention reviews were considered. To identify articles published since the Cochrane reviews, electronic databases were searched until March 2021. Hand searching of conference databases and journals was also undertaken. The inclusion criteria included Studies within a Trial (SWATs). The main outcome was the incremental cost-effectiveness ratio (ICER). Quality assessment of papers used the Cochrane risk of bias 1 tool. The CRD guidance was used to assess the quality of economic evaluation. Random-effect meta-analyses were undertaken. The GRADE certainty of evidence was applied for each strategy, and Trial Forge Guidance 2 was used for strategies included in meta-analyses to evaluate the uncertainty of the findings. Cost-effectiveness ranks summarise the cost-effectiveness of all strategies.

Results

We identified 6569 records and included 29 SWATs (earliest conducted in 1999 and latest in 2021) including more than 35,800 participants. There is no strategy we would recommend trial teams and researchers adopt with complete statistical certainty. Recruitment strategies which could be cost-effective include financial incentives, trial-branded pens, telephone reminders and pre-notification leaflets. Retention strategies which could be cost-effective include vouchers and trial-branded pens.

Conclusion

Future SWATs should replicate existing recruitment and retention strategies, rather than evaluate novel ones. We recommend that economic evaluations be carried out alongside all future SWATs, costs and benefits be recorded transparently, and the cost-effectiveness of existing recruitment or retention strategies be evaluated.

Keywords

Study within a trial embedded trial recruitment to RCTs retention in RCTs economic evaluation alongside SWATs

Introduction

Recruitment of participants into randomised controlled trials (RCTs) is usually poor.¹ Under-recruited, and hence under-powered, trials result in research waste.² Another main challenge with RCTs is attrition, which occurs when recruited participants fail to complete follow-up assessments. A systematic review of 151 trials associated with the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme has found the median retention rates to be 89%.³ Poor retention not only diminishes the power of the trial but also can introduce attrition bias, thus threatening the statistical analysis of RCTs.⁴

Studies Within A Trial (SWATs) are a study design for identifying strategies to improve recruitment and retention in RCTs.^5,6 SWATs’ primary objective is to improve trial methodology and efficiency.⁶ Two systematic reviews have appraised the evidence on the effectiveness of strategies for improving recruitment² and retention⁷ in RCTs. Both reviews have implied poor progress on identifying effective recruitment/retention strategies.

Moreover, no appraisal of the evidence on the cost-effectiveness of recruitment and retention strategies has been undertaken so far. Given the anticipated high direct and indirect costs of poor recruitment and retention rates,^8,9 economic evaluations of recruitment and retention strategies alongside SWATs are useful. More broadly, an “economic evaluation offers an organised consideration of the range of possible alternative courses of action and the evidence of the possible effects of each. This is more likely to lead to better decisions that improve overall social value”.¹⁰ There is an urgent need to strengthen the evidence arising from SWATs towards identifying strategies that could improve recruitment and retention in RCTs. There is a further need to develop a framework for the economic analysis of SWATs to enable research organisations to make more informed decisions.

This review accumulates and critically appraises the existing evidence on economic evaluations alongside SWATs for improving recruitment and retention in SWATs. The primary aim is to improve trial efficiency by increasing the evidence available for making trial process decisions. The secondary aim is to make recommendations for improvement of future economic evaluations alongside recruitment and retention SWATs.

Methods

A protocol is registered on PROSPERO (CRD record code: 42021236824), in line with the 2020 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines.¹¹

The studies eligible for inclusion were quasi- or fully randomised SWATs. The corresponding host trials of SWATs had to be quasi- or fully randomised, and within the context of healthcare or any field applicable to healthcare settings. Hypothetical studies (i.e. studies that ask potential patients whether they would participate in a trial that will not take place in reality) were excluded as these were assessed to have a high Cochrane risk of bias,² and hence were also excluded from the retention review.⁷

Any strategies designed to improve recruitment and/or retention of participants in RCTs were eligible for inclusion in the study. The target population was any potentially eligible trial participants. For SWATs associated with improving recruitment, the strategies were aimed at potential trial participants who could be recruited to a host trial. For SWATs associated with improving retention, the strategies were aimed at already randomised trial participants who were asked to provide follow-up data. In contrast to the Cochrane recruitment and retention reviews, strategies aimed at collaborators or research ethics committees were not considered.^2,7 There were no restrictions regarding comparators. There are several potential types of economic evaluation alongside SWATs, including cost-effectiveness analysis, cost-benefit analysis, cost-consequence analysis, and cost-utility analysis. Therefore, the primary outcome was reported in terms of the incremental cost-effectiveness ratio (i.e. the incremental cost per additional patient recruited or per additional participant retained), the (monetary) net benefit of a given strategy or the willingness to pay (WTP) for a given strategy. The secondary outcomes were any costs and health utilities (benefits) of recruitment/retention strategies. The measures of effect could be reported as incremental/unit/total costs, or incremental utilities/effects/benefits expressed in recruitment or retention rates. If the primary outcome in a study was unavailable but its secondary outcomes were reported appropriately, such a study would not be excluded on these grounds.

As this review focused on SWATs of recruitment or retention strategies that included economic evaluations in their analyses, the SWATs from the most recent Cochrane recruitment² and retention⁷ reviews were considered in the study selection process. Further potential SWATs were identified after the final dates of the study searches in these Cochrane reviews, that is, on and after 12 February 2015 until 3 March 2021 for recruitment strategies² and on and after 1 March 2020 until 3 March 2021 for retention strategies.⁷ Thus, we developed a search strategy for the identification of more recent SWATs on recruitment and/or retention strategies that involved economic evaluation. The search strategy is available in Supplemental Material 2.

We searched the following electronic databases:

• MEDLINE (OVID)

• Embase (OVID)

• CINAHL

• Cochrane Methodology Review Group Specialised Register (CMR) in the Cochrane Library

• Science Citation Index and Social Citation Index

• ERIC (EBSCO)

• PsycINFO (OVID)

• Scopus

Hand searching of conference abstracts associated with SWATs was also undertaken. Journals were also hand searched, including ClinicalTrials.gov, OpenTrials, EU Clinical Trials Register, Cochrane Central Register of Controlled Trials (CENTRAL) and the Online Resource for Recruitment research in Clinical triAls (ORRCA). The search dates were the same as those for the electronic databases.

The titles, abstracts, and full texts of identified records were independently screened by two authors (AG and AE). We independently extracted the data through a standardised data extraction form on Microsoft Word, which included information on both the host trial and the SWAT. The data extraction form provided information about the host trial name, design, location, clinical setting, population, intervention(s), and comparator(s) (Supplemental Material 1). With respect to SWATs, the data extraction form provided information about the design, the strategy (-ies), the comparator(s), study objective, time horizon, frequency and timing of strategy, measure(s) of benefit and costs, type of economic evaluation, numbers, and proportions of participants in the intervention and control groups, results of health economic outcomes in the intervention and control groups, and perspective adopted in the economic evaluation (Supplemental Material 1). No automation tools were applied in the data collection process. We did not need to contact any study investigators to obtain further data or to ask for clarification of published data.

In line with the Cochrane recruitment and retention reviews’ methodology, the Version 1 Cochrane risk of bias tool was used to appraise the quality of the included studies.¹² For included studies that were obtained from the recruitment and retention reviews,^2,7 the risk of bias presented in these reviews was assumed to be valid, and hence their quality appraisal was adopted from the corresponding reviews^2,7 to avoid duplication of effort. In addition, a descriptive quality assessment was independently undertaken to assess the quality of the included SWATs (including those from the Cochrane reviews) with respect to their economic evaluation. Such assessment followed explicitly the University of York’s Centre for Reviews and Dissemination (CRD) guidance on systematic reviews of economic evaluations by considering the following: methods of deriving the effectiveness data, cost analysis, valuation and measurement of health benefit, methods of synthesising the costs and effects, and, if applicable, analysis of uncertainty.¹³ To allow for broader inclusion of studies associated with economic evaluations alongside SWATs, studies with a high risk of bias, or studies with a low quality of economic evaluation, or studies that were not peer-reviewed, were still included if they met the inclusion criteria.

Random-effect meta-analyses, through the Cochran-Menthel-Haenszel weighting method, were undertaken for the primary outcome. Since all included SWATs were associated with cost-effectiveness analysis, this was carried out by initially obtaining the odds ratios (ORs) of the recruitment or retention rates for each strategy. Then, the ORs were converted to effect sizes, that is incremental recruitment or retention rates, by dividing the natural logarithm of the OR with 1.81.¹⁴ This conversion is assumed for continuous outcomes,¹⁴ such as the incremental recruitment or retention rate, and hence we applied such a conversion in our study.

(i.e. effect size = incremental recruitment rate/retention rate = $\frac{\ln (O R)}{1.81}$ ).

The Cochran-Menthel-Haenszel method was applied for weighting the incremental costs of each strategy from each included study, to obtain the aggregate figure for the incremental cost of each strategy. The final step was to calculate the incremental cost per patient recruited or participant retained for each strategy by dividing the incremental cost with the incremental recruitment or retention rate. 95% confidence intervals are presented for the primary outcome, the OR, and the incremental recruitment or retention rate. RevMan was the software used for meta-analysis.¹⁵ The figures for the primary outcome were adjusted to 2019 USD Purchasing Power Parity (PPP) rates. The use of PPP, defined by the International Monetary Fund (IMF) as “the rate at which the currency of one country would have to be converted into that of another country to buy the same amount of goods and services in each country”,¹⁶ can reflect more accurately any cost variations among countries. We anticipated the included studies to be potentially subject to between-group (study) heterogeneity; hence the I² statistic, ranging from 0% to 100%, was computed for all strategies, whose primary outcome was obtained through the inclusion of multiple SWATs. The greater a reported value of I², the greater the extent of between-study heterogeneity. More details about the meta-analyses of each recruitment and retention strategy can be found in Supplemental Material 3 and Supplemental Material 4 respectively.

Following meta-analysis, the GRADE approach was applied to the effect measure (i.e. the OR) and consequently the primary outcome (i.e. the ICER), to assess the certainty of the evidence for each recruitment and retention strategy; this tool explores the extents of risk of bias, imprecision, inconsistency, indirectness, and publication bias in the included studies. Such an assessment was undertaken by the two reviewers (AG and AE), and details about the GRADE assessment of each recruitment and retention strategy can be found in Supplemental Material 5. Furthermore, the Trial Forge Guidance 2 was explicitly used to qualitatively assess whether more SWATs should be conducted for recruitment and/or retention strategies included in the study’s meta-analyses.¹⁷ The assessment using Trial Forge Guidance 2 comprises of five criteria: risk of bias, imprecision, inconsistency, balance of benefit and disadvantage to participants, and balance of benefit and disadvantage to the host trial.¹⁷ Such an assessment was undertaken by the two reviewers (AG and AE), and details about the application of Trial Forge Guidance 2 to each recruitment and retention strategy comprising of at least two SWATs can be found in Supplemental Material 6.

Cost-effectiveness ranks of strategies for improving recruitment and retention in RCTs are presented. The rank was based on the mean ICER of each recruitment or retention strategy, whether the lower 95% odds ratio confidence intervals indicate the associated strategy is significantly effective, and the GRADE certainty of evidence.

Results

Searching of records

The full texts of 68 studies from the recruitment review² and 71 studies from the retention review⁷ were assessed for inclusion to our study. Following the searches, 8113 records were retrieved from the electronic databases overall. 28 additional studies were identified from manually searched registers. Nine studies were identified from hand searching reference lists of two studies that were retrieved from the electronic databases. After deduplication, 6569 records were screened, and 267 full texts were assessed for eligibility. 22 studies were included in this review. A PRISMA flow diagram is shown in Figure 1.

Figure 1.

PRISMA flow diagram for the systematic review.

Characteristics of the included studies

Three studies had more than one SWATs,^18–20 and hence 22 studies (29 SWATs) were included in this review. Nine studies (15 SWATs) assessed recruitment strategies, whereas 15 studies (16 SWATs) evaluated retention strategies. Two studies included SWATs with strategies targeting both recruitment and retention.^21,22 The characteristics of the included studies, that is, study (author, date, country), host trial design, participants, SWAT intervention(s) (and comparator(s)), and SWAT outcome(s), are presented in Table 1.

Table 1.

Characteristics of the included studies.

Study	Host trial design	Participants	SWAT interventions	SWAT outcome(s)
Jennings et al.¹⁸ Country: United Kingdom	Prospective randomised open blinded end point (PROBE) design	People aged 60 or over taking long‐term NSAIDS for arthritis	Intervention: Invitation letter with a fixed payment of £100. Comparator: An invitation letter with no fixed payment incentive	Increase in consented patients with incentive
Jennings et al.¹⁸ Country: United Kingdom	Prospective randomised open blinded end point (PROBE) design	People aged 60 or over with chronic hyperuricaemia in conditions where urate deposition has already occurred	Intervention: Invitation letter with a fixed payment of £100. Comparator: An invitation letter with no fixed payment incentive	Increase in consented patients with incentive
Jennings et al.¹⁸ Country: United Kingdom	Randomised controlled trial (RCT); open, parallel, double-blind	People aged 18–79 with newly diagnosed hypertension	Intervention: Invitation letter with a fixed payment of £100. Comparator: An invitation letter with no fixed payment incentive	Increase in consented patients with incentive
Jennings et al.¹⁸ Country: United Kingdom	Randomised controlled trial (RCT); open, parallel, double-blind	People aged 18–79 with uncontrolled blood pressure	Intervention: Invitation letter with a fixed payment of £100 Comparator: An invitation letter with no fixed payment incentive	Increase in consented patients with incentive
Jennings et al.¹⁸ Country: United Kingdom	Randomised controlled trial (RCT); open, parallel, double-blind	People aged 18–80 with at least one component of the metabolic syndrome	Intervention: Invitation letter with a fixed payment of £100 Comparator: An invitation letter with no fixed payment incentive	Increase in consented patients with incentive
Free et al.¹⁹ Country: United Kingdom	Randomised controlled trial (RCT); pilot, single-blind	People aged 16 and above who are smokers and willing to stop smoking in next month	Intervention: Research staff sending a text message regarding the newly available online registration facility Comparator: Research staff calling the participants’ mobile numbers to register them for the trial (no text message)	Consent to be randomised into the Txt2stop trial (i.e. host trial) within 2 weeks
Free et al.¹⁹ Country: United Kingdom	Randomised controlled trial (RCT); pilot, single-blind	People aged 16 and above who are smokers and willing to stop smoking in next month	Intervention: Letter containing study and consent information and a £5 note Comparator: Participants received the normal trial procedures	Consent to be randomised into the Txt2stop trial (i.e. host trial) within 2 weeks
Free et al.¹⁹ Country: United Kingdom	Randomised controlled trial (RCT); pilot, single-blind	People aged 16 and above who are smokers and willing to stop smoking in next month	Intervention: Four text messages over 1 week containing quotes from existing participants Comparator: No text messages	Consent to be randomised into the Txt2stop trial (i.e. host trial) within 2 weeks
Miller et al.²⁹ Country: United States	Two randomised controlled trials (RCTs). No further information about their designs	Participants aged 18–75 with DSM‐IV dysthymic disorder, double depression (major depression superimposed on antecedent dysthymia), or chronic major depression	Intervention: Phone screening by a senior investigator Comparator: Phone screening by a trained research assistant	Proportion of participants recruited to the two host trials
Bell et al.³² Country: United Kingdom	Randomised controlled trial (RCT); pragmatic, unblinded, two-arm, parallel	Females aged 70–85 who are not currently on prescription medication to prevent osteoporotic fractures before randomisation	Intervention: Trial-branded pen with the 60-month follow-up questionnaire Comparator: 60-month follow-up questionnaire alone	Questionnaire return rate
Cunningham-burley et al.³³ Country: United Kingdom	Randomised controlled trial (RCT); two-arm, 1:1 randomisation	NHS staff who are subject to a trust dress code	Intervention: Trial-branded pen with the 14-week follow-up questionnaire Comparator: 14-week follow-up questionnaire alone	Questionnaire return rate
Clark et al.³⁹ Country: United Kingdom	Randomised controlled trial (RCT); two-arm, 1:1 block randomisation	Smokers who are aged 35 or more, who are invited to undertake a series of case-finding tools, which comprise lung function tests and several symptom based case-finding questionnaires, for the potential identification of COPD.	Intervention: Electronic prompt (i.e. SMS or email) to return the study questionnaire Comparator: No prompt	Questionnaire return rate
Cochrane et al.³⁷ Country: United Kingdom	Cohort, pragmatic, two-arm, open randomised controlled trial (RCT)	People aged 65 or older who either have experienced a fall in the last year or feel worried about falling in the future	Intervention: Personalised text message, as a retention strategy Comparator: Generalised text message	Questionnaire return rate
Hardy et al.³⁴ Country: United Kingdom	Multicentre randomised controlled trial (RCT)	Adult women who are nulliparous, have a single cephalic presentation, greater than or equal to 37 weeks’ gestation, intend spontaneous vaginal birth, are in second stage of labour and with an effective mobile epidural in situ	Interventions 1) An incentive cover letter with a promise of a £10 gift voucher on the return of a completed questionnaire. The covering letter thanked participants for their time and effort. All reminder letters included a sentence about the incentive Comparator: No incentive mentioned in the cover letter. If the questionnaire was not returned, all reminder letters included the promise of a £10 gift voucher on the return of a completed questionnaire	Questionnaire return rate
Gates et al.³⁰ Country: United Kingdom	Cluster randomised controlled trial (RCT)	Participants who attended emergency departments (EDs) with an acute whiplash injury of whiplash-associated disorder grades I-III were eligible for step 1. People who attended EDs with whiplash injuries and had persistent symptoms 3 weeks after ED attendance were eligible for step 2	Intervention: £5 gift voucher, redeemable at a range of shops with their questionnaire, and a covering letter thanking participants for their time and effort Comparator: No gift voucher, and a standard covering letter	Questionnaire return rate
James et al.³⁶ Country: United Kingdom	Cohort, pragmatic, two-arm, open randomised controlled trial (RCT)	People aged 65 or older who either have experienced a fall in the last year or feel worried about falling in the future	Interventions 1) a branded pen and a standard cover letter 2) a branded pen and a social incentive cover letter 3) No pen and a social incentive cover letter Comparator: No pen and a standard cover letter	Questionnaire return rate
Kenyon et al.³⁵ Country: United Kingdom	Cohort, pragmatic, two-arm, open randomised controlled trial (RCT)	Children whose mothers joined the MRC ORACLE trial. Their mothers have had preterm, prelabour rupture of the fetal membranes (pROM). The parents of the survived children are the participants in the SWAT.	Intervention: Monetary incentive (£5 voucher redeemable at high street stores) together with the reminder questionnaire Comparator: No monetary incentive. The same reminder questionnaire was sent	Questionnaire return rate
Khadjesari et al.²⁰ Country: United Kingdom	Randomised controlled trial (RCT); 2-arm, randomisation stratified by age and gender	People who visited DownYourDrink while browsing the web, and who had an AUDIT-C score greater than 5	Intervention: Participants who did not complete the first follow-up questionnaire within 1 week received either a £5 Amazon.co.UK voucher, £5 donation to cancer research UK, or entry in a £250 prize draw in the second prompt for completion of questionnaires Comparator: No incentive. The participants received another prompt for completion of questionnaires	Questionnaire return rate
Khadjesari et al.²⁰ Country: United Kingdom	Randomised controlled trial (RCT); 2-arm, randomisation stratified by age and gender	People who visited DownYourDrink while browsing the web, and who had an AUDIT-C score greater than 5	Intervention: Participants received a £10 Amazon.co.UK voucher in the first prompt for completion of questionnaires. Comparator: No incentive. The participants received another prompt for completion of questionnaires	Questionnaire return rate
Marsh et al.³¹ Country: United Kingdom	Cluster randomised controlled trial (RCT)	Parents of children aged 3–12 months registered with 36 participating practices in Nottingham	Intervention 1) Postal administration with financial incentive (£2 voucher) once the completed diary had been received or postal group without financial incentive 2) Telephone administration with financial incentive (£2 voucher) once a completed diary had been received or telephone group without financial incentive Comparator: No postal or telephone administration; either with or without financial incentive	Diary return rate
Treweek et al.³⁸ Country: United Kingdom	Randomised controlled trial (RCT); four-centre, 1:1 parallel group	Women aged 50–70 who are overweight and attending routine breast screening in four Scottish breast screening service centres	Intervention: Pre-notification card Comparator: No pre-notification card	Proportion of participants attending the host trial primary outcome measurement visit (i.e. retention)
Whiteside et al.²¹ Country: United Kingdom	Cohort, pragmatic, two-arm, open randomised controlled trial (RCT)	People aged 65 or older who either have experienced a fall in the last year or feel worried about falling in the future	Intervention; branded pen with trial invitation pack Comparator: No pen, but with trial invitation pack	1) Randomisation rate 2) Proportion of participants who remained in the trial at 3 months post randomisation, i.e. retention
Jolly et al.²² Country: United Kingdom	Pragmatic, multicentre randomised controlled trial (RCT)	People aged 18 or older who are on the practice COPD register and have mild dyspnoea	Interventions The practices recruiting participants for the host trial accessed standard printed patient information materials, as well as a multimedia information resource, developed by patient and public involvement (PPI) contributors and researchers Comparator The practices accessed standard printed patient information materials, with no extra multimedia information resource	1) Recruitment rate 2) Retention rate (after 6 months) 3) Retention rate (after 12 months)
Bracken et al.²⁵ Country: Australia	Randomised controlled trial (RCT): Multicentre, double-blind	Men aged 50–74 years, obese or overweight, with prediabetes or newly diagnosed type 2 diabetes, and a low serum testosterone	Intervention: SMS reminder text which provided key enrolment information as well as including a peripheral cue based on the concept of social proof Comparator: Telephone reminder, with a reminder call script used by staff members	Attendance rate
Arundel et al.²⁶ Country: United Kingdom	Cohort randomised controlled trial (cRCT); two-arm, pragmatic, open, multicentre	Patients aged 65 years and over who have attended routine podiatry services within the past 6 months, have had one fall in the past 12 months; or one fall in the past 24 months requiring hospital attention; or report a fear of falling on their baseline questionnaire in the past 4 weeks	Intervention: A pre-notification leaflet, 2–3 weeks before the trial recruitment pack Comparator: Trial recruitment pack only	Randomisation rate
Rogers et al.²⁷ Country: United Kingdom	Randomised controlled trial (RCT): Prospective, open-label, blinded	People aged 60 or over, taking allopurinol for chronic gout, and with additional cardiovascular risk factors	Intervention: DVD presentation containing an audio-visual presentation explaining the background and operation of FAST, and a standard invitation pack Comparator: Standard invitation pack only	Randomisation rate
Hancocks et al.²⁸ Country: United Kingdom	Randomised controlled trial (RCT): Pragmatic, multicentred, parallel, two group	People aged 18 or over, who are smokers and smoke at least 10 cigarettes per day (for at least 1 year)	Interventions 1) Full invitation pack from a GP. 2) Single-page invitation from a GP. Comparator: Text message invitation	Recruitment rate
Cook et al.²³ 15 european countries	Randomised controlled trial (RCT): Pragmatic, multicountry, adaptive, two group, phase IV	People aged 1 or over, who have sudden onset of self-reported fever, with at least one respiratory symptom (cough, sore throat, running or congested nose) and one systemic symptom (headache, muscle ache, sweats or chills or tiredness) during a period of increased influenza activity	Intervention: Unconditional monetary incentive of £20 given to participants at recruitment, as an intervention to boost retention in the host trial Comparator: Conditional monetary incentive of £20 given to participants only once a questionnaire had been returned	Diary return rate
Dorling et al.²⁴ Countries: Ireland, United Kingdom	Randomised controlled trial (RCT): Parallel, multicentre, two group	Infants born at <32 weeks’ gestation or a weight of <1500 g, who were receiving <30 mL/kg/day of milk at trial enrolment	Intervention: The first paper follow-up letter to parents would include a promise of an incentive (£15 (€15 for Irish residents) gift voucher redeemable at high-street shops) after receipt of a completed form Comparator: The first paper letter to parents would enclose the incentive (£15 (€15 for Irish residents) gift voucher redeemable at high-street shops) before the receipt of a completed form	Questionnaire return rate

All SWATs of retention strategies were already included in the retention review.⁷ However, two of these SWATs had data that were publicly inaccessible at the time of the publication of the retention review.^23,24 These SWATs eventually became publicly accessible in journals. As there was an uncertainty regarding which data the reviewers from the retention review had accessed, the papers’ risk of bias was re-assessed.

For recruitment strategies, four included studies (four SWATs) were retrieved from the electronic databases,^22,25–27 one study (one SWAT) was retrieved from manual searching,²⁸ four studies (10 SWATs) were already included in the recruitment review,² and one study (one SWAT) was already included in the retention review.²⁰

Most SWATs had individually randomised designs; however, two studies (two SWATs) were quasi-randomised,^29,30 and two studies (two SWATs) were cluster randomised.^22,31

Primary outcomes were available in seven out of 22 studies (11 out of 29 SWATs) and reported in terms of the incremental cost per additional patient recruited/incremental cost per additional participant retained, respectively. Accordingly, the incremental cost-effectiveness ratio (ICER), for recruitment strategies, was defined as:

I C E R = I n c r e m e n t a l c o s t p e r a d d i t i o n a l p a t i e n t r e c r u i t e d = \frac{\begin{array}{l} C o s t o f r e c r u i t m e n t s t r a t e g y \\ - c o s t o f b a s e l i n e s t r a t e g y \end{array}}{\begin{array}{l} R e c r u i t m e n t r a t e o f r e c r u i t m e n t s t r a t e g y \\ - R e c r u i t m e n t r a t e o f b a s e l i n e s t r a t e g y \end{array}}

(1)

For retention strategies, the ICER was defined as:

I C E R = I n c r e m e n t a l c o s t p e r a d d i t i o n a l p a r t i c i p a n t r e ta i n e d = \frac{\begin{array}{l} C o s t o f r e t e n t i o n s t r a t e g y \\ - c o s t o f b a s e l i n e s t r a t e g y \end{array}}{\begin{array}{l} R e t e n t i o n r a t e o f r e t e n t i o n s t r a t e g y \\ - R e t e n t i o n r a t e o f b a s e l i n e s t r a t e g y \end{array}}

(2)

Therefore, cost-effectiveness analysis was the sole method of economic evaluation alongside the included SWATs. The primary outcome was manually computed by the reviewers in the remaining 15 studies (18 SWATs), using the incremental costs and the incremental recruitment and/or retention rates of a given strategy to obtain the ICER. The perspective adopted by all economic evaluations related to the trial teams, that is, the reported effects and costs of recruitment or retention strategies were direct and associated with the trial teams’ budget. In total, 35,864 participants from 29 SWATs were involved.

The Cochrane risk of bias was assessed to be high for four included studies, unclear for nine studies, and low for nine studies (Table 2). In terms of the quality of the economic evaluation, this was assessed to be low for seven studies, moderate for six studies, unclear for one study, and high for eight studies (Table 3). One included study affected the quality appraisal detrimentally, as a full text was unavailable.²⁶

Table 2.

Cochrane risk of bias in the included studies.

Table 3.

Quality of economic evaluation in the included studies.

Recruitment strategies

Financial incentives

The ICER of a financial incentive, against no financial incentive, was estimated from two studies (six SWATs).^18,19 With an odds ratio of 1.65 (95% CI: 0.86, 3.18) and an incremental cost of US$133.44, it costs US$476.57 (95% CI: from US$208.50 to N/A¹) to recruit an additional patient (see Table S1 and Figure S1 in Supplemental Material 3 for more details). All SWATs have a low Cochrane risk of bias^18,19, but moderate quality of economic evaluation^18,19. The I² statistic is 49%, signalling evidence of substantial between-study heterogeneity. There are three potential sources of between-group heterogeneity; (1) variations in healthcare settings across the host trials of the SWATs; (2) variations in the monetary incentives among SWATs and (3) variations in the populations across SWATs. The GRADE certainty of evidence is moderate for this recruitment strategy, due to inconsistency. In line with Trial Forge Guidance 2,¹⁷ the GRADE criterion is met, the cumulative meta-analysis criterion is met, the PICOT criterion² is partially met, the balance of benefit and disadvantage to participants criterion is not met, and the balance of benefit and disadvantage to host trial criterion is not met. We suggest further studies including different monetary incentives be conducted in the future so that a figure of additional patients recruited by a US$1 increase in monetary incentive be obtained.

Nudge interventions

The ICER of nudge interventions against usual recruitment procedures was estimated from three studies (three SWATs).^19,22,27 Nudge interventions related to recruitment included: quotes from existing participants over text messages, a multimedia information resource that was developed through patient and public involvement (PPI) contributors and researchers, and a DVD presentation containing an audio-visual presentation explaining the host trial. With an odds ratio of 1.13 (95% CI: 0.72, 1.77) and an incremental cost of US$22.00, it costs US$314.29 (95% CI: from US$68.75 to N/A) to recruit an additional patient (see Table S2 and Figure S2 in Supplemental Material 3 for more details). The risk of bias is high for one study²⁷, unclear for one study²², and low for one study.¹⁹ In addition, the quality of economic evaluation is moderate for two studies^19,27 and low for one study.²² The I² statistic is 74%, signalling evidence of substantial between-study heterogeneity. There are four potential sources of such heterogeneity; (1) variations in healthcare settings across the host trials of the SWATs; (2) variations in the “nudge interventions”; (3) variations in the populations across SWATs and (4) variations in the designs of the included SWATs. The GRADE certainty of evidence is very low for this recruitment strategy, due to risk of bias, inconsistency, and indirectness of the included studies. In line with Trial Forge Guidance 2,¹⁷ the GRADE criterion is met, the cumulative meta-analysis criterion is met, the PICOT criterion is partially met, the balance of benefit and disadvantage to participants criterion is met, and the balance of benefit and disadvantage to host trial criterion is partially met. Therefore, further replications of SWATs associated with nudge interventions are encouraged.

Screening of a trial by a senior investigator

The cost-effectiveness of this strategy was estimated according to a single SWAT.²⁹ Screening for the host trial undertaken by a senior investigator, versus screening for the host trial undertaken by a research assistant, is not cost-effective, since the odds ratio is 0.19 (95% CI: 0.11, 0.32), with an incremental cost of US$37.05 (see Table S3 in Supplemental Material 3 for more details). Given the low sample size of the included study,²⁹ and its high Cochrane risk of bias, the GRADE certainty of evidence is low due to imprecision and risk of bias. The included study has a moderate quality of economic evaluation.

Text messages versus telephone calls

The cost-effectiveness of this strategy was estimated according to a single SWAT.¹⁹ With an odds ratio of 3.47 (95% CI: 1.27, 9.48) and an incremental cost of US$22.00, it costs only US$4.41 (95% CI: from US$2.45 to US$23.38) to recruit an additional patient (see Table S3 in Supplemental Material 3 for more details). Given the sample size of the included study,¹⁹ and its low Cochrane risk of bias, the GRADE certainty of evidence is moderate due to imprecision. The included study has a moderate quality of economic evaluation.

Pre-notification leaflet

The cost-effectiveness of this strategy was estimated according to a single SWAT.²⁶ With an odds ratio of 1.17 (95% CI: 0.87, 1.57) and an incremental cost of US$2.25, it costs US$25.97 (95% CI: from US$9.00 to N/A) to recruit an additional patient (see Table S3 in Supplemental Material 3 for more details). Given the sample size of the included study²⁶, and its low Cochrane risk of bias, the GRADE certainty of evidence is moderate due to imprecision. The included study has a high quality of economic evaluation.

Telephone reminders versus text reminders

The cost-effectiveness of this strategy was estimated according to a single SWAT.²⁵ With an odds ratio of 1.37 (95% CI: 0.95, 1.98) and an incremental cost of US$3.98, it costs US$23.37 (95% CI: from US$10.47 to N/A) to recruit an additional patient (see Table S3 in Supplemental Material 3 for more details). Given the sample size of the included study²⁵, and its low Cochrane risk of bias, the GRADE certainty of evidence is moderate due to imprecision. The included study has a high quality of economic evaluation.

Invitation packs by GP

The cost-effectiveness of this strategy was estimated according to a single SWAT.²⁸ With an odds ratio of 7.75 (95% CI: 1.04, 57.97) and an incremental cost of US$1.13, it costs US$1.00 (95% CI: from US$0.50 to US$57.47) to recruit an additional patient (see Table S3 in Supplemental Material 3 for more details). However, since these figures were obtained from an abstract ²⁸, the GRADE certainty of evidence is very low due to risk of bias, imprecision, indirectness and publication bias. The included study has an unclear quality of economic evaluation.

Trial-branded pen

The cost-effectiveness of this strategy was estimated according to a single SWAT.²¹ With an odds ratio of 1.04 (95% CI: 0.65, 1.66) and an incremental cost of US$0.47, it costs US$21.41 (95% CI: from US$1.68 to N/A) to recruit an additional patient (see Table S3 in Supplemental Material 3 for more details). Given the sample size of the included study,²¹ and its low Cochrane risk of bias, the GRADE certainty of evidence is moderate due to imprecision. The included study has a high quality of economic evaluation.

In line with Trial Forge Guidance 2,¹⁷ we encourage all the recruitment strategies to be replicated in future SWATs, but the comparison of pre-notification leaflet against no leaflet.

Ranking recruitment strategies

The cost-effectiveness rank of the eight recruitment strategies is available in Table 4. Providing financial incentives might be an effective recruitment strategy, but its ICER is relatively high, at US$476.57, thus questioning its cost-effectiveness; more SWATs of financial incentives with moderate amounts (i.e. significantly less than £100) are needed to estimate the ICER. The following may be considered cost-effective strategies: providing a telephone reminder versus a SMS reminder, or a branded pen versus no pen, or a pre-notification leaflet versus no leaflet. However, whereas their corresponding ICERs are relatively low, their OR lower bounds signal they may not actually be effective recruitment strategies. Providing primary text message, versus primary call and no text message, might be a very cost-effective strategy; however, more SWATs of this strategy are needed since its GRADE certainty of evidence is moderate. It remains uncertain whether the remaining recruitment strategies are cost-effective since their GRADE certainty of evidence is either low or very low.

Table 4.

Cost-effectiveness rank of different recruitment strategies.

Cost-effectiveness rank of different recruitment strategies
Rank	Strategy	Number of SWATs	Sample size	GRADE certainty of evidence	ICER
1	Financial incentive versus no financial incentive	6	1506	Moderate	$476.57 ($ 208.50, N/A)
2	Branded pen with trial invitation pack versus no pen	1	1862	Moderate	$21.41 ($1.68, N/A)
3	Telephone reminder versus SMS reminder	1	709	Moderate	$23.37 ($10.47, N/A)
4	Pre-notification leaflet versus no leaflet	1	4314	Moderate	$25.97 ($9.00, N/A)
5	Primary text message versus primary call and no text message	1	937	Moderate	$4.41 ($2.45, $23.38)
6	Invitation pack from a surgeon versus text message	1	1267	Very low	$1.00 ($0.50, $57.47)
7	Nudge intervention versus usual recruitment	3	6054	Very low	$314.29 ($68.75, N/A)
8	Screening for the host trial undertaken by a senior investigator versus screening undertaken by a research assistant	1	347	Low	N/A (ineffective)

Overall, there is no complete certainty up to date on which recruitment strategies would be cost-effective for trial teams to use for recruiting eligible patients to their trials. Nevertheless, strategies such as financial incentives, trial-branded pens, telephone reminders and pre-notification leaflets could possibly provide recruitment benefits to future trials in a cost-effective manner. More evidence is needed to determine the cost-effectiveness of such strategies.

Retention strategies

Trial-branded pen

The ICER of providing a trial-branded pen versus no pen was estimated from three studies (three SWATs).^21,32,33 With an odds ratio of 1.14 (95% CI: 1.00, 1.30) and an incremental cost of US$0.52, it costs US$6.98 (95% CI: from US$3.63 to N/A) for an additional participant to be retained in a host trial (see Table S4 and Figure S3 in Supplemental Material 4 for more details). One included study has a low Cochrane risk of bias,²¹ whereas the remaining studies have an unclear risk of bias.^32,33 All studies have a high quality of economic evaluation. The I² statistic is negligible at 0%, signalling no evidence of substantial between-study heterogeneity. However, there are four potential sources of such heterogeneity; (1) variations in healthcare settings across the included SWATs; (2) variations in retention periods among SWATs; (3) variations in the populations across the SWATs and (4) variations in the SWATs’ designs. The GRADE certainty of evidence for this retention strategy is moderate, due to inconsistency. In line with Trial Forge Guidance 2,¹⁷ the GRADE criterion is met, the cumulated evidence criterion is not met, the PICOT criterion is partially met, the balance of benefit and disadvantage to participants criterion is met and the balance of benefit and disadvantage to host trial criterion is not met. Therefore, we argue further SWATs associated with trial-branded pens as a retention strategy to be undertaken.

Financial incentives

The ICER of providing financial incentives versus no incentives was estimated from three studies (three SWATs).^20,30,34 With an odds ratio of 1.33 (95% CI: 1.15, 1.53) and an incremental cost of US$8.20, it costs US$15.89 (95% CI: from US$10.65 to US$32.42) for an additional participant to be retained in a host trial (see Table S5 and Figure S4 in Supplemental Material 4 for more details). Once the quasi-randomised SWAT included in the three SWATs of financial incentives is removed, the result is not significantly affected: the OR slightly falls to 1.32 (95%: 1.01, 1.73), the incremental cost increases to US$11.08 and the ICER slightly increases to US$22.05 (95% CI: from US$11.17 to US$615.21). The Cochrane risk of bias is low in one study,³⁴ unclear in one study,²⁰ and high in one study.³⁰ Furthermore, two studies^20,34 have a low quality of economic evaluation, whereas one study³⁰ has a high quality of economic evaluation. The I² statistic is 37%, signalling low evidence of substantial between-study heterogeneity. There are two sources of such heterogeneity; (1) variations in retention periods among SWATs and (2) variations in the monetary incentives among SWATs. The GRADE certainty of evidence for this retention strategy is moderate due to risk of bias. In line with Trial Forge Guidance 2,¹⁷, the GRADE criterion is met, the cumulative meta-analysis criterion is not met, the PICOT criterion is partially met, the balance of benefit and disadvantage to participants criterion is partially met, and the balance of benefit and disadvantage to host trial criterion is not met. We encourage replications of further SWATs associated with financial incentives as a strategy for improving participant retention in RCTs. Three further studies (or three SWATs) were not included in this meta-analysis.^20,31,35 The results of these studies, and the reasons for which they were not included are available in Supplemental Material 7.

Nudge interventions

The ICER of a nudge intervention versus usual retention was estimated from three studies (three SWATs).^22,36,37 Nudge interventions related to retention included: a personalised text message instead of a generalised one, a multimedia information resource that was developed through patient and public involvement (PPI) contributors and researchers, and a social incentive cover letter instead of a standard one. With an odds ratio of 1.14 (95% CI: 0.94, 1.39) and an incremental cost of US$0.84, it costs US$11.55 (95% CI: from US$4.61 to N/A) for an additional participant to be retained in a host trial (see Table S6 and Figure S5 in Supplemental Material 4 for more details). The Cochrane risk of bias is low in two studies,^36,37 and unclear in one study.²² The quality of economic evaluation is high in one study,³⁶ moderate in one study,³⁷ and low in one study.²². The I² statistic is negligible at 0%, signalling low evidence of substantial between-study heterogeneity. However, there are still three sources of such heterogeneity; (1) variations in “nudge” interventions among SWATs; (2) variations in retention periods among SWATs and (3) variations in the SWATs’ designs. The GRADE certainty of evidence for this retention strategy is moderate due to inconsistency. In line with Trial Forge Guidance 2,¹⁷ the GRADE criterion is met, the cumulative meta-analysis criterion is met, the PICOT criterion is partially met, the balance of benefit and disadvantage to participants criterion is not met, and the balance of benefit and disadvantage to host trial criterion is partially met. We encourage replications of further SWATs associated with nudge interventions for improving participant retention in RCTs.

Unconditional monetary incentive versus conditional monetary incentive

The ICER of an unconditional monetary incentive, versus a conditional one, was estimated from two studies (two SWATs).^23,24 With an odds ratio of 0.90 (95% CI: 0.31, 2.64) and an incremental cost of US$18.61, such a strategy is not cost-effective, since its estimated odds ratio is less than 1 (see Table S7 and Figure S6 in Supplemental Material 4 for more details). The Cochrane risk of bias is unclear for both studies, whereas the quality of economic evaluation is high in one study²³ and moderate in the other study.²⁴ The I² statistic is 93%, demonstrating high evidence of substantial between-study heterogeneity. There are five potential sources of such heterogeneity; (1) variations in healthcare settings between the host trials of the included SWATs; (2) variations in the populations between SWATs; (3) differences in the interventions between SWATs; (4) variations in retention periods between SWATs and (5) variations in the SWATs’ designs. The GRADE certainty of evidence is low, due to risk of bias and inconsistency. In line with Trial Forge Guidance 2,¹⁷ the GRADE criterion is met, the cumulative meta-analysis criterion is met, the PICOT criterion is partially met, the balance of benefit and disadvantage to participants criterion is met, and the balance of benefit and disadvantage to host trial criterion is met. We highly encourage replications of further SWATs comparing unconditional with conditional monetary incentives for improving participant retention in RCTs.

Pre-notification card

The cost-effectiveness of this strategy was estimated according to a single SWAT.³⁸ With an odds ratio of 1.26 (95% CI: 0.99, 2.19) and an incremental cost of US$1.02, it costs US$4.86 (95% CI: from US$2.76 to US$N/A) to retain an additional participant in a host trial (see Table S8 in Supplemental Material 4 for more details). Given the low sample size of the included study³⁸ and its low Cochrane risk of bias, the strategy’s GRADE certainty of evidence is moderate, due to imprecision. The included study has a moderate quality of economic evaluation.

Electronic prompts

The cost-effectiveness of this strategy was estimated according to a single SWAT.³⁹ With an odds ratio of 1.48 (95% CI: 0.81, 1.96) and an incremental cost of US$0.12, it costs US$0.55 (95% CI: from US$0.28 to US$N/A) to retain an additional participant in a host trial (see Table S8 in Supplemental Material 4 for more details). Given the low sample size of the included study³⁹ and its unclear Cochrane risk of bias, the strategy’s GRADE certainty of evidence is low, due to risk of bias and imprecision. The included study has a low quality of economic evaluation.

Trial-branded pen (before recruitment)

The cost-effectiveness of this strategy was estimated according to a single SWAT.²¹ With an odds ratio of 8.27 (95% CI: 1.04, 66.00) and an incremental cost of US$0.47, it costs US$0.40 (95% CI: from US$0.20 to US$23.50) to retain an additional participant in a host trial (see Table S8 in Supplemental Material 4 for more details). Given the sample size of the included study²¹ and its low Cochrane risk of bias, the strategy’s GRADE certainty of evidence is moderate, due to imprecision. The included study has a high quality of economic evaluation.

In line with Trial Forge Guidance 2,¹⁷ we highly encourage the aforementioned retention strategies to be replicated in future SWATs.

Ranking retention strategies

A summary of the cost-effectiveness rank of the seven retention strategies is provided on Table 5. Providing pens before patient recruitment to an RCT is potentially a very cost-effective strategy, with an ICER of US$0.40. However, as this finding is derived from a single study with a low sample size, more evidence is needed to confirm the figure. A retention strategy, which also seems to be cost-effective with moderate GRADE certainty of evidence, is the provision of £5 up to £10 vouchers; the ICER is relatively low at US$15.89. Providing a trial-branded pen is potentially another cost-effective retention strategy, with its ICER being very low, at US$6.98. However, its lower bound OR = 1, meaning there is still a chance such a strategy is not (cost-) effective. Due to either low GRADE certainty of evidence or wide confidence intervals of the remaining retention strategies, it is inconclusive whether these strategies are cost-effective or not.

Table 5.

Cost-effectiveness rank of different retention strategies.

Cost-effectiveness rank of different retention interventions
Rank	Strategy	Number of SWATs	Sample size	GRADE certainty of evidence	ICER
1	Trial-branded pen versus no trial-branded pen (before recruitment)	1	92	Moderate	$0.40 ($0.20, $23.50)
2	Financial incentive versus no financial incentive	3	5753	Moderate	$15.89 ($10.65,$32.42)
3	Trial-branded pen versus no trial-branded pen	4	9790	Moderate	$6.98 ($3.63, N/A)
4	Nudge intervention versus usual recruitment procedure	3	5276	Moderate	$11.55 ($4.61, N/A)
5	Pre-notification card versus no pre-notification card	1	558	Moderate	$4.86 ($2.76, N/A)
6	Electronic prompts versus no electronic prompts	1	437	Low	$0.55 ($0.28, N/A)
7	Unconditional monetary incentive versus conditional monetary incentive	2	1268	Low	$465.25 ($97.95, N/A)

We encourage trial researchers to consider financial incentives of up to £10 or/and trial-branded pens as retention strategies, while we recommend more SWATs of these strategies be undertaken. Despite the reported lower bound OR, we still encourage pens as a retention strategy due to its low reported ICER and low incremental costs, especially for trials involving postal questionnaires.

Discussion

Summary of findings

Whereas Cochrane reviews have explored the effectiveness of strategies for improving recruitment and retention in RCTs,^2,7 this review additionally appraises the cost-effectiveness of recruitment and retention strategies. The findings demonstrate an uncertainty regarding which strategies are cost-effective for improving participant recruitment and/or retention in RCTs. For both recruitment and retention strategies, the uncertainty of the evidence primarily originates from the evaluation of several potential strategies from single studies, but without any replications. The corresponding Cochrane reviews on recruitment and retention in RCTs suggested that replications of SWATs with strategies having a moderate GRADE certainty of evidence be undertaken, a recommendation we also make for bolstering the evidence on the cost-effectiveness of strategies for improving recruitment or retention in SWATs.

Overall, there is no retention strategy which we would recommend trial teams and researchers adopt with complete statistical certainty. Providing vouchers of up to £10 during follow-up could be a cost-effective retention strategy with an estimated ICER of US$15.89; it costs only US$15.89 for an additional participant to be retained in a host trial. Providing a trial-branded pen may also be a cost-effective strategy, with an ICER of US$6.98, yet not statistically significant since its lower bound OR = 1 (hence its lower bound effectiveness is zero). Also, providing a trial-branded pen before recruitment, may be a cost-effective strategy, with an ICER of US$0.40 which is also statistically significant. However, the GRADE certainty of evidence for both strategies is moderate, meaning that additional SWATs of these strategies could be beneficial for making more certain inferences about their cost-effectiveness. ICERs were derived for further retention strategies; however, it remains inconclusive whether these are cost-effective due to their low or very low GRADE certainty of evidence. Whereas the retention review found the inclusion of self-kits or a diary to be effective strategies,⁷ no data about their cost-effectiveness were available. Therefore, we highly encourage the conduct of future SWATs of these strategies and the inclusion of economic evaluations alongside such SWATs. Similarly to the retention review,⁷ we also encourage the conduct of further SWATs associated with the cost-effectiveness of patient and public involvement (PPI) interventions, since PPI is a key unanswered question about trial retention.⁴⁰ Overall, due to the low ICER and incremental costs we recommend trial teams use trial-branded pens as a retention strategy, especially in trials involving postal questionnaires for follow-up. Providing vouchers of up to £10 could be another beneficial retention strategy for trial-teams.

Also, there is no recruitment strategy which we would recommend trial teams and researchers adopt with complete statistical certainty. Including a branded pen with a trial invitation pack, or a telephone reminder versus an SMS reminder, could be cost-effective strategies, with their ICERs being low, at US$21.41 and US$23.37 respectively. However, as their lower bound ORs are less than 1, their cost-effectiveness is not statistically significant. In addition, their GRADE certainty of evidence is moderate, implying these strategies would benefit from further SWATs to determine their cost-effectiveness with less uncertainty. Another cost-effective strategy could be the provision of primary text message versus primary call and no message, with its ICER estimated at $4.41 (95% CI; $2.45, $23.38); however, its GRADE certainty of evidence is also moderate. Providing financial incentives may be an effective yet a costly strategy, with US$476.57 required to recruit an additional patient. However, there is substantial heterogeneity among the associated SWATs, since very different monetary incentives were present (i.e. from £5 up to £100). Therefore, we encourage the cost-effectiveness of moderate financial incentives (i.e. less than £100 per participant recruited¹⁷) to be evaluated in future SWATs. Moreover, we could not estimate the cost-effectiveness of an open design, compared to a placebo-controlled design, as the associated SWATs did not undertake any relevant economic evaluations or provide costs related to such strategies. Since this strategy appears to be effective at improving recruitment,² economic evaluations of such a strategy alongside future SWATs are welcome. We also encourage the estimation of the cost-effectiveness of incorporating user-testing for improving the participant information leaflet (PIL) in future SWATs.

Recommendations for future economic evaluations alongside SWATs

To minimise the uncertainty regarding the findings from SWATs on the cost-effectiveness of recruitment and/or retention strategies, we highly recommend the application of Value of Information (VoI) analyses. Such an analysis can inform decision makers on whether more trials are needed to minimise the uncertainty of the cost-effectiveness of a strategy. A VoI analysis could be used in line with the Trial Forge Guidance 2¹⁷ to confirm whether a further SWAT associated with a recruitment/retention strategy should be undertaken. For instance, since we concluded trial-branded pens to be a potentially cost-effective retention strategy with moderate GRADE certainty of evidence, and the Cochrane review concluded pens to be a potentially effective strategy with low GRADE certainty of evidence, the GRADE criterion is met in Trial Forge Guidance 2 and hence further SWATs on pens are recommended. However, it seems that such a strategy could be a very cost-effective one for participant retention, and hence it may not be necessary to undertake another SWATs, which would require the financing of an additional SWAT from a constrained allocated budget . To determine whether more SWATs are needed for determining its effectiveness, a VoI analysis for trial-branded pens could be undertaken. A framework of VoI analysis related to SWATs which trial researchers could follow is available in the literature, and applicable after a standard meta-analysis of a recruitment or retention strategy.⁴¹

A concern was that although 139 studies were originally included in the recruitment and retention reviews,^2,7 only 17 of these studies were included in our review. Therefore, economic evaluations were not undertaken alongside the majority of SWATs. Whereas capturing the effectiveness of different recruitment or retention strategies is useful, cost considerations are equally important due to limited availability of financial resources. We highly encourage trialists and researchers to undertake economic evaluations alongside all SWATs in the future. In addition, the costs of obtaining outcome data were not provided in six studies (10 SWATs), meaning that the cost-effectiveness of some strategies may have been overestimated. Therefore, the reporting of costs should be transparent, expressed in unit terms, and stratified into different types of direct and indirect costs, including the costs of obtaining outcome data. Finally, in all cost-effectiveness analyses there is a defined cost-effectiveness threshold to determine whether a given intervention, which is both more effective compared to existing interventions and costlier, is cost-effective. As long as the ICER is less (more) than the threshold, then a strategy is (not) cost-effective. We recommend trial researchers to define such a threshold for determining which recruitment or retention strategies should be considered as cost-effective. In our review, we did not set out a cost-effectiveness threshold, as there has not been any research in this area; instead we presented cost-effectiveness ranks of recruitment and retention strategies for comparisons with respect to their cost-effectiveness to be made.

The perspective all SWATs followed was related to the trial teams. However, poor recruitment into RCTs may also lead to indirect costs through the generation of foregone health benefits to an affected population not experiencing the clinical benefits of a potentially effective intervention. For instance, a study modelled the impact on human lives lost due to poor recruitment in the COVID-19 RECOVERY trial, which showed that over 2800 lives could have been saved in the UK.⁴² Similarly, the financial costs of poor attrition can be significant, with the time costs of researchers dealing with follow-up being dominant.⁴³ When the follow up to a funded RCT is poor, this may generate huge costs for RCT funders, as they could have instead provided funding to trials with better follow-up rates and hence with more statistical accuracy in their results. Therefore, in future SWATs it is recommended that researchers adopt a broader perspective where possible when conducting economic evaluations alongside SWATs, such as the perspective of a national healthcare system or the societal perspective (i.e. through cost-benefit analysis instead of cost-effectiveness analysis).

Strengths and limitations of the review

The major limitations in our study were the differential definitions and computations of cost-effectiveness outcomes among the included studies. These were partially captured through manual conversions of ICERs, or any other secondary economic outcome, into unit incremental costs, by stringently following the definition of ICER (equations (1) or (2)) and the reported recruitment or retention rates. This approach enabled us to obtain cost-effectiveness figures from 20 out of 22 studies, or from 26 out of 29 included SWATs in a homogeneous manner. Another limitation could be our flexible approach towards including studies with high Cochrane risk of bias or low quality of economic evaluation, or studies that have not been peer reviewed or published yet. However, as an appraisal of the cost-effectiveness of strategies for improving participant recruitment and retention in RCTs has not been explored before a, we encouraged this flexible approach during the screening of records and inclusion of studies. Finally, there were differences in the definitions of “recruitment rate” or “retention rate”, especially in terms of the recruitment and retention periods, across the included studies. However, we encouraged flexibility in the definitions of such terms by the same means.

Overall, the review benefits from such flexibility so that the evidence on the cost-effectiveness or recruitment and/or retention strategies is fully captured. In addition, all studies were subject to extensive quality appraisals, including the Cochrane risk of bias and quality of economic evaluation. Moreover, the certainty of the evidence for each recruitment and/or retention strategy was extensively assessed through the GRADE approach and Trial Forge Guidance 2. We believe the use of multiple tools strengthens the reliability of our findings. Finally, our review could motivate the research community to undertake economic evaluations alongside all future SWATs; we have also made recommendations on how such economic evaluations could be undertaken.

Conclusion

There is no recruitment or retention strategy which we would recommend trial teams and researchers adopt with full certainty. Improving recruitment and retention in RCTs is a priority for trial teams, reflected through the emergence of SWATs as a study design to improve trial efficiency. It is of paramount importance for future SWATs to replicate existing recruitment and/or retention strategies, rather than focus on novel strategies. We also recommend that economic evaluations be carried out alongside all future SWATs, costs and benefits be reported clearly and transparently, the cost-effectiveness of existing recruitment or retention strategies be repeatedly evaluated, and broader perspectives be adopted in future SWATs if applicable. Finally, we encourage researchers to undertake VoI analyses for each recruitment and retention strategy, in combination with Trial Forge Guidance 2, to minimise the uncertainty of the evidence.

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material for A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials by Athanasios Nassos Gkekas, Alex Evans, Adwoa Parker, Sarah Ronaldson and David J Torgerson in Research Methods in Medicine & Health Sciences

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Supplemental Material

Supplemental Material - A systematic review of economic evaluations alongside studies within a trial for improving recruitment and retention in randomised controlled trials

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Athanasios Nassos Gkekas

Adwoa Parker

David J. Torgerson

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Fletcher

Gheorghe

Moore

, et al. Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review. BMJ Open 2012; 2: e000496.

Treweek

Pitkethly

Cook

, et al. Strategies to improve recruitment to randomised trials. Cochrane Database Syst Rev 2018; 2(2): MR000013.

Walters

Bonacho Dos Anjos Henriques-Cadby

Bortolami

, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom health technology assessment programme. BMJ Open 2017; 7: e015276.

Torgerson

. Designing randomised trials in health, education and the social sciences: an introduction. Basingstoke: Palgrave MacMillan, 2008.

Bower

Brueton

Gamble

, et al. Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials 2014; 15: 399.

Treweek

Bevan

Bower

, et al.

Trial forge guidance 1: what is a study within a trial (SWAT)?

Trials 2018; 19(1): 139.

Gillies

Kearney

Keenan

, et al. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2021; 3(3): MR000032.

McDonald

Knight

Campbell

, et al. What influences recruitment to randomised controlled trials? a review of trials funded by two UK funding agencies. Trials 2006; 7: 9.

Kitterman

Cheng

Dilts

, et al. The prevalence and economic impact of low-enrolling clinical studies at an academic medical center. Acad Med 2011; 86(11): 1360–1366.

10.

Drummond

Sculpher

Claxton

, et al. Methods for the economic evaluation of health care programmes. 4th ed. Oxford, UK: Oxford University Press.

11.

Page

MJ.

McKenzie

Bossuyt

, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev 2021; 10: 89.

12.

Higgins

Altman

Gøtzsche

, et al. Cochrane bias methods group; cochrane statistical methods group. The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343: d5928.

13.

Centre for Reviews and Dissemination (CRD) . Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: CRD, University of York, 2009.

14.

Chinn

. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med 2000; 19(22): 3127–3131.

15.

Review manager (RevMan) [computer program]. Version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2014.

16.

International Monetary Fund . Purchasing power parity: weights matter [online]. https://www.imf.org/external/pubs/ft/fandd/basics/ppp.htm (accessed 05 August 2021).

17.

Treweek

Bevan

Bower

, et al. Trial forge guidance 2: how to decide if a further study within a trial (SWAT) is needed. Trials 2020; 21(1): 33.

18.

Jennings

MacDonald

Wei

, et al.

Does offering an incentive payment improve recruitment to clinical trials and increase the proportion of socially deprived and elderly participants?

Trials 2015; 16: 80.

19.

Free

Hoile

Robertson

, et al. Three controlled trials of interventions to increase recruitment to a randomized controlled trial of mobile phone based smoking cessation support. Clin Trials 2010; 7(3): 265–273.

20.

Khadjesari

Murray

Kalaitzaki

, et al. Impact and costs of incentives to reduce attrition in online trials: two randomized controlled trials. J Med Internet Res 2011; 13(1): e26.

21.

Whiteside

Flett

Mitchell

, et al. Using pens as an incentive for trial recruitment of older adults: an embedded randomised controlled trial. F1000Res 2019; 8: 315.

22.

Jolly

Sidhu

PSM COPD Group , et al. Improving recruitment to a study of telehealth management for COPD: a cluster randomised controlled ‘study within a trial’ (SWAT) of a multimedia information resource. Trials 2019; 20(1): 453.

23.

Cook

Bongard

, et al. Conditional versus non-conditional incentives to maximise return of participant completed questionnaires in clinical trials: a cluster randomised study within a trial. under review/consideration for Trials, 2021.

24.

Dorling

Hewer

Hurd

, et al. Two speeds of increasing milk feeds for very preterm or very low-birthweight infants: the SIFT RCT. Health Technol Assess 2020; 24(18): 1–94.

25.

Bracken

Hague

Keech

, et al. Recruitment of men to a multi-centre diabetes prevention trial: an evaluation of traditional and online promotional strategies. Trials 2019; 20(1): 366.

26.

Arundel

Jefferson

Bailey

, et al. REFORM study team. A randomized, embedded trial of pre-notification of trial participation did not increase recruitment rates to a falls prevention trial. J Eval Clin Pract 2017; 23(1): 73–78.

27.

Rogers

Flynn

RWV

Mackenzie

, et al. Does the provision of a DVD-based audio-visual presentation improve recruitment in a clinical trial? a randomised trial of DVD trial invitations. BMC Med Res Methodol 2019; 19(1): 24.

28.

Hancocks

King

Gude

, et al. The effect of the method of invitation on recruitment of participants from GP practices to a trial of a smoking reduction intervention: a study within a trial [online]. In: International Clinical Trials Methodology Conference, Brighton, UK, 06–09 October 2019, pp. 143, 2019.

29.

Miller

Markowitz

Kocsis

, et al. Cost effectiveness of screening for clinical trials by research assistants versus senior investigators. J Psychiatr Res 1999; 33(2): 81–85.

30.

Gates

Williams

Withers

, et al. Does a monetary incentive improve the response to a postal questionnaire in a randomised controlled trial? the MINT incentive study. Trials 2009; 10: 44.

31.

Marsh

Kendrick

. Using a diary to record near misses and minor injuries—which method of administration is best? Inj Prev 1999; 5(4): 305–309.

32.

Bell

Clark

Fairhurst

, et al. Enclosing a pen reduced time to response to questionnaire mailings. J Clin Epidemiol 2016; 74: 144–150.

33.

Cunningham-Burley

Roche

Fairhurst

, et al. Enclosing a pen to improve response rate to postal questionnaire: an embedded randomised controlled trial. F1000Research 2020; 9: 577.

34.

Hardy

Bell

Brocklehurst

, et al. Evaluation of the effects of an offer of a monetary incentive on the rate of questionnaire return during follow-up of a clinical trial: a randomised study within a trial. BMC Med Res Methodol 2016; 16: 82.

35.

Kenyon

Pike

Jones

, et al. The effect of a monetary incentive on return of a postal health and development questionnaire: a randomised trial [ISRCTN53994660]. BMC Health Serv Res 2005; 5: 55.

36.

James

Parker

Cockayne

, et al. Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled study within a trial [version 1; peer review: 2 approved with reservations]. F1000Research 2020; 9: 623.

37.

Cochrane

Welch

Fairhurst

, et al. An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial. F1000Res 2020; 9: 154.

38.

Treweek

Gallant

Anderson

. SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance. F1000Res 2021; 10: 84.

39.

Clark

Ronaldson

Dyson

, et al. Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis. J Clin Epidemiol 2015; 68(12): 1446–1450.

40.

Brunsdon

Biesty

Brocklehurst

, et al. What are the most important unanswered research questions in trial retention? a James Lind alliance priority setting partnership: the PRioRiTy II (prioritising retention in randomised trials) study. Trials 2019; 20(1): 593.

41.

Claxton

Griffin

Koffijberg

, et al. How to estimate the health benefits of additional research and changing clinical practice. BMJ 2015; 351: h5987.

42.

Knowlson

Torgerson

. Effects of rapid recruitment and dissemination on Covid-19 mortality: the RECOVERY trial. F1000Res 2020; 9: 1017.

43.

Peterson

Pirraglia

Wells

, et al. Attrition in longitudinal randomized controlled trials: home visits make a difference. BMC Med Res Methodol 2012; 12: 178.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.49 MB

0.09 MB

0.34 MB

0.49 MB

0.16 MB

0.15 MB

0.08 MB