Abstract
Background
Retention is considered the second highest trial methods priority in the UK after recruitment. There is limited evidence on whether notifying trial participants that a follow-up questionnaire will be sent soon (‘pre-notification’) affects retention.
Methods
This Study Within a Trial (SWAT) evaluated whether sending a pre-notification letter or email around 2 weeks before sending a self-report questionnaire increased retention, in terms of provision of the host trial primary outcome. The SWAT was a randomised, two-arm, parallel-design (1:1 allocation ratio) trial, controlled by ‘no pre-notification letter’. It was embedded within the WORKWELL host trial, which evaluated the impact of job retention vocational rehabilitation on work-related and health-related outcomes of employed people with inflammatory arthritis. The SWAT primary outcome was a valid response for the WORKWELL primary outcome.
Results
Two hundred forty-four trial participants took part in the SWAT. All were included in the analysis. Among those sent a pre-reminder, 100/121 (83%) provided a valid response for the WORKWELL primary outcome, compared to 97/123 (79%) of those not sent a pre-reminder. The estimated adjusted odds ratio was 1.28 (95% confidence interval 0.67–2.42), with a risk difference of 3.8% (95% CI -6.1 to 13.6%), favouring the prereminder. The estimated intervention cost per additional participant retained was £53.42, and the total cost per additional participant retained was £46.52.
Conclusion
Researchers may have a small improvement in trial retention by using pre-notification. The cost per additional participant retained is relatively low. However, further evaluations are merited.
Keywords
Introduction
Many trials struggle with participant retention and completion of follow-up questionnaires. A recent UK study found that the median (IQR) retention rate across 151 trials was 89% (79%–97%). 1 Reminders are generally an effective way of increasing response rates to questionnaires, with some evidence that pre-notification (contacting a participant to say that the trial team will be sending a questionnaire out soon) also provides some benefit, although it is not high certainty evidence. 2 However, in the trial setting, the recent Cochrane review of methods to improve retention identified only one randomised evaluation of a pre-notification mailing. 3 In that study, 4 the effect of a postcard, sent around a month prior to face-to-face collection of the primary outcome was evaluated in a Study Within a Trial (SWAT), nested in the ActWELL trial. Whilst it showed a slight increase in attendance at the data collection appointment (231/274 [84.3%] vs 230/284 [81.0%]: risk difference 3.3%), evidence is currently of ‘low certainty’ because the confidence interval was wide and evidence is based on a single trial. It is therefore important to replicate this intervention to increase the evidence base.
WORKWELL is a pragmatic, multi-centre individually-randomised trial of job-retention vocational rehabilitation for employed people with inflammatory arthritis which, unlike the ActWELL trial, uses participant self-report to collect outcome data. 5 The MRC-funded PROMETHEUS project resourced a replication of a pre-notification SWAT nested within WORKWELL. Our intervention differed from the ActWELL SWAT in the pre-notification being sent by letter or e-mail rather than postcard because the host trial gave participants a choice of data collection method (postal or online questionnaire). Pre-notification was sent 2 weeks (rather than 4 weeks) in advance of follow-up outcome assessment, to align with the host trial processes and timelines.
The content of all trial communications may impact on questionnaire return rates. A theory-based cover letter intervention (SWAT 24) 6 was developed using the Theoretical Domains Framework (TDF) 7 and the Behaviour Change Techniques (BCT) Taxonomy. 8 Although a theory-based cover letter has now been compared with the use of a standard cover letter in four SWATs, evidence of its effectiveness is still inconclusive, with a non-significant increase in response rate of 3% (95% CI -2 to 8%) and GRADE evidence classed as ‘very low’. 3 For WORKWELL, the text in the pre-notification communication was informed by the theory and associated text used by Goulao et al. 6 The WORKWELL pre-notification letter or email was personalised to include the (typed) name of the participant because there is some evidence that personalising may improve response rates in surveys, although this is yet to be supported in a trials context (e.g. in reminders for postal questionnaires 3 ).
Whilst many trial process interventions, such as the use of pre-notification, may lead to a marginal gain, there are also cost implications of any such interventions. There has been limited consideration of costs in SWATs. The cost of implementing retention interventions “is an important outcome for evaluators to include in future comparisons so as to provide trial teams with the information they need to make a decision on what will work from an effectiveness and an economic perspective”. 3 There is little methodological literature on how this should be done, and whether and how any cost savings (through, for example, a reduction in the number of reminders or collection of data via telephone interview following unsuccessful reminders) should be included in the economic evaluation. In the evaluation of costs, we therefore performed both a simple analysis of intervention costs and a more detailed analysis of total costs.
We therefore planned to evaluate the effects of a theoretically-informed, personalised pre-notification letter or email on participants providing a valid response to the primary outcome of the WORKWELL trial, with the aim to add to the body of evidence around the effectiveness and cost-effectiveness of pre-notification on trial retention. We followed the PROMETHEUS SWAT Reporting Guidelines 9 which are based on the CONSORT 2010 recommendations. 10
Methods
Design
Parallel-group (1:1 allocation ratio) design randomised controlled trial (RCT) embedded within the WORKWELL RCT (the host trial).
WORKWELL trial PICO
Population
People in work, aged at least 18 years, with rheumatoid arthritis, undifferentiated inflammatory arthritis or psoriatic arthritis classified as at medium or high risk of work instability.
Intervention
WORKWELL Job retention vocational rehabilitation plus a self-help, written, work-advice pack and usual care.
Control
A self-help, written, work-advice pack plus usual care (control intervention);
Outcome
The primary outcome is the Work Limitations Questionnaire–25 (WLQ-25) total score. 11 The WLQ-25 asks about ability to work within the last 2 weeks.
SWAT setting
WORKWELL Trial participants were recruited from hospital Rheumatology and Therapy departments in England, Wales, and Scotland in the United Kingdom (UK). All patients were seen as outpatients and so the setting for the SWAT was the community, with the expectation that questionnaire responses would be made by WORKWELL trial participants from their own homes.
Interventions
Test intervention
Theoretically-informed, personalised, pre-notification communication in advance of a participant’s 6-month follow-up questionnaire mailing. For participants who elected to complete follow-up questionnaires in hard copy form and return by post, the intervention took the form of a letter sent by post 2 weeks prior to the scheduled mailing of the 6-month questionnaire. For participants who elected to complete follow-up questionnaires online, the intervention was in the form of an email sent 2 weeks prior to the scheduled mailing of an email containing a link to the online questionnaire. The letter and e-mail contained the same form of words. Mailing of a participant’s 6-month questionnaire was scheduled to take place 1 week prior to its due date.
Control intervention
No pre-notification communication.
Irrespective of SWAT intervention, participants not responding to the questionnaire mailing within 2 weeks were subject to a standard reminder protocol which involved text, email, reposting and/or telephoning participants, as appropriate. 5
Changes to SWAT following commencement
Due to the Covid-19 lockdown during spring/summer 2020, participants who had chosen to complete paper questionnaires were offered the choice of switching to online completion or via telephone interview if they were not able or willing to complete online. This was necessary as it was not possible for the Clinical Trials Unit (CTU) to receive postal questionnaires during this period. This offer occurs subsequent to the sending of the pre-notification to those randomised to that SWAT intervention arm. As a result, the method of delivery of pre-notification was still the participant’s chosen method of questionnaire response. In September 2020, an alternative Freepost PO Box was set up to receive postal questionnaires. Accordingly, not all participants received their follow up questionnaire via their originally chosen method between March and September 2020.
Also due to the Covid-19 pandemic, a number of employed participants were unable to go to work at the 6-month follow-up period for reasons unanticipated when the SWAT was designed (e.g. participants on the UK Government furlough or self-employed income support (SEIS) schemes, or shielding and unable to work from home (and either on furlough, SEIS or receiving statutory sick pay)) and therefore it was not possible for them to complete the WLQ-25 questionnaire. For this reason, there was a deviation from the SWAT protocol in the definition of the primary outcome measure; a response to the 6-month questionnaire that indicates that a person is ‘not working’ (although still in employment) is now included as a valid response for the primary outcome, with the primary outcome listed in the protocol now being reported as a secondary outcome.
Eligibility criteria
All participants within the WORKWELL trial, except those who had withdrawn or were known to have died prior to the planned mailing of the 6-month outcome questionnaire were eligible.
Ethics
Participants became part of the SWAT without additional recruitment or consent (over and above consent for WORKWELL), as the SWAT was included within the WORKWELL trial protocol. Ethics approval for this SWAT was obtained from the West Midlands – Solihull Research Ethics Committee (18/WM/0327) as part of the WORKWELL approval process.
Outcome measures
Primary
• Valid response for the WORKWELL trial primary outcome (yes/no) (i.e. usable outcome data for the primary outcome measure (either WLQ-25 total score (5)) obtained by any means, or response that indicates that the participant was not working (for whatever reason) at 6 months.
Secondary
• Valid score for the WLQ-25; • Valid response for WORKWELL trial primary outcome without reminder (yes/no) • Questionnaire returned (yes/no) • Number of reminders sent (0–3 reminders); • Time to response [or ceasing follow-up] (days); • Cost of the intervention (£) per participant retained (where “participant retained” was defined as “one who provided a valid response to the WORKWELL trial primary outcome”); • Total cost (£) per participant retained.
The costs were estimated as follows, where staff costs used a Research Assistant grade with 2019–2020 academic year Higher Education Single Pay Spine Point 20 (£25,217).
Test Intervention costs
For those who chose to receive questionnaires by post, the item costs of printing the pre-notification letter (£0.01), envelopes (£0.04) and postage (£0.65) were included (total £0.70), plus the item staff costs for filling and labelling the envelope (estimated item cost £1.61 [7 min @ £0.23/minute]). For those who chose to receive questionnaires electronically, the item staff cost of sending the pre-notification email was also estimated to be £1.61 (7 min at £0.23 per minute). The costs for preparing the content of the pre-notification letter/email were relatively modest and have not been included: it would be anticipated that this would be largely a one-off research cost and that, should the intervention be applied more widely, very similar text would be used. This resulted in a total item cost of £2.31 for those who opted for a postal questionnaire and £1.61 for those who opted for an electronic questionnaire.
Control intervention cost
Zero.
Reminder costs
Item staff costs were: Telephone reminder (£0.92 if no answer; £1.15 if a message was left on voicemail; £1.38 if a call was answered); additional time for a call when answered (£1.15); additional time for collection of the minimum data set (£6.44); additional time of collecting a withdrawal data set (£1.15); additional time for collecting item missing data (£4.14); sending a reminder text (£1.84); sending a reminder email (£1.84); re-sending a questionnaire web-link (£1.61); re-sending a postal questionnaire (£1.61); action unknown - only initialled as action (£1.84). Total direct non-staff costs for postal questionnaire mailings were estimated in a similar way to the direct costs for the pre-notification letter (as this was replaced by a cover letter: total £0.70), to which was added the item cost of professional printing of a questionnaire (£1.50). Actual telephone call and text costs were excluded as these were minimal given that the University central service contracts were planned to be used for their delivery.
Sample size
The SWAT sample size was restricted to a maximum of the size of the host trial, WORKWELL, within which 249 participants were randomised.
Randomisation
Randomisation was stratified by WORKWELL trial arm and chosen method of 6-month questionnaire receipt (postal vs electronic). It was implemented by Lancashire CTU using the online SealedEnvelope system. When a trial participant reached 3 weeks prior to the date when their 6-month outcome was due for collection, a staff member checked whether the participant had withdrawn or was known to have died. If they were still eligible for WORKWELL trial follow-up a CTU staff member entered them into the SWAT and used the randomisation system to generate their allocation, thus ensuring concealment of allocation until a participant was entered into the SWAT.
Blinding
WORKWELL trial participants were not blinded to either their host trial arm or SWAT intervention arm, but were unaware that they were participating in a randomised SWAT. The SWAT statistical team remained blind to SWAT group allocation until the SWAT Statistical Analysis Plan had been approved by the SWAT team and published on Figshare (https://doi.org/10.48420/16854517.v1). The final analysis was, however, performed by the statisticians with knowledge of the SWAT group allocations.
Statistical methods
The main analysis population was all participants randomised as part of the SWAT according to their allocated SWAT treatment group, regardless of any SWAT protocol deviations.
Baseline participant data, and the primary and secondary outcome measures were summarised, using frequency (%), mean (SD) or median (IQR), as appropriate) both overall and by SWAT group allocation.
Comparison of the primary outcome between the pre-notification group and the no pre-notification group used binary logistic regression, including the randomised group factor and adjusting for stratification variables (WORKWELL trial treatment allocation; chosen mode of response). Odds ratios (OR) for the between-groups relative difference in proportions completing the questionnaire were estimated, and presented in conjunction with descriptive statistics of the number and percentage of respondents in each group, and estimates of the risk difference (RD). Both OR and RD are presented as point estimate and 95% confidence interval (CI). Time to response (measured from day first sent) was compared between the groups using Cox regression, adjusted for WORKWELL treatment allocation and chosen mode of response. The date of response was defined by when the response first reached the CTU team. One day was added to all response times to avoid zero survival times. Results are presented as a hazard ratio (HR) and related 95% CI; median time to response in each group is also presented. Kaplan-Meier curves are also presented. Number of reminders by group are summarised as median (IQR) and compared using a Mann-Whitney U test.
For the analysis of the additional cost per additional participant retained we present a crude analysis of the ratio of the average pre-notification intervention cost, divided by the corresponding difference in proportions providing valid response for WORKWELL trial primary outcome. However, as a secondary analysis, we also included the costs of the reminder process, thus performing a fuller cost-effectiveness analysis.
Analysis of the other secondary outcomes was performed using the same method as for the primary outcome.
Missing data
Missing data were not expected for the stratification variables, nor any of the outcome variables. If data were missing on the variable actual method of 6-month questionnaire delivery, this analysis was based on available data only.
Sensitivity analyses
Sensitivity analyses used were 1. Repetition of primary and secondary outcome analysis with no adjustment for stratification variables. 2. Use of the variable ‘Method of 6-month questionnaire delivery’ instead of ‘Chosen method of 6-month Questionnaire receipt’ for adjustment (primary outcome only). 3. As the WORKWELL Trial indicated that participants would normally only be asked for questionnaire responses for a maximum of 8 weeks (56 days), so the analysis of this outcome was repeated with censoring 63 days after the initial posting of the questionnaires (to allow for the posting being 7 days prior to the due date).
Subgroup analyses
Primary outcome and secondary outcomes with subgroups for the chosen response mode, implemented by adding its interaction with SWAT group.
Software
All analyses were performed using Stata IC 14.
Results
Recruitment and Participant flow
Five of the 249 WORKWELL trial participants withdrew prior to the point at which the pre-notification was to be sent, meaning that 244 SWAT participants were randomised between September 2019 and August 2021 Figure 1. For the SWAT, follow-up was completed by 21 October 2021. Flow diagram summarizing the flow of participants through the SWAT evaluation.
Baseline data
Baseline characteristics by SWAT group allocation and overall.
Outcomes and estimation
Outcome data by SWAT group allocation.
aAdjusted for WORKWELL arm and chosen method of delivery.
Kaplan-Meier survival estimates of time (+1 day) to participant return of valid primary outcome data.
The cost of the intervention delivered was an average of £2.02 per person (£2.31 for the 71 sent a postal prompt and £1.61 for the 50 participants sent an electronic prompt), resulting in an estimated cost per additional participant retained of £53.42. A fuller cost-effectiveness analysis, taking into account the costs of the reminders process for those who did not respond to the initial mailing, found a total cost per additional participant retained of £46.53.
Sensitivity analysis
None of the sensitivity analyses showed important sensitivity of the findings to the alternative assumptions made. However, when adjusting the model for the actual method of questionnaire delivery rather than the chosen method, the odds ratios tended to increase slightly (e.g. for the primary outcome, the OR was 1.36 when adjusted for the actual method, compared to 1.28).
Subgroup analysis
Subgroup analysis was performed by chosen mode of questionnaire delivery to investigate whether the effect of the SWAT intervention might differ between those who opted for postal compared with those who opted for electronic questionnaire completion. For the primary outcome, there was some evidence that any intervention effect might be smaller for those who opted for postal than for electronic delivery (OR = 0.30; 95% CI 0.08–1.13, p = 0.075). Similar results were found for key secondary outcomes (Supplemental Table S1). In the electronic choice subgroup, the intervention cost per additional participant retained was £11.48, and total cost per additional participant retained was £6.54; amongst those who opted for postal delivery the percentage of participants retained was slightly lower in the SWAT intervention arm, so no further analysis of costs was performed.
Discussion
Sending participants a theoretically-informed pre-notification prior to sending a self-report outcome questionnaire may result in a small increase in trial retention. This finding is consistent with that recently reported SWAT 4 but this remains GRADE low-certainty evidence as there are no other similar evaluations known in a trial setting. 3 There is one additional SWAT (SWAT77) registered on the SWAT repository (https://www.qub.ac.uk/sites/TheNorthernIreland NetworkforTrialsMethodologyResearch/SWATSWAR Information/Repositories/SWATStore/), so there may be this or other SWATs ongoing to evaluate a pre-notification email, postcard or letter in a trial setting. As recently reported, 4 there is broader evidence from other settings that pre-notification of questionnaires may lead to a marginal improvement in response rates, so our findings are also consistent with the wider literature. Our subgroup analysis suggested that the effect of pre-notification may be greater for electronic than postal delivery; this finding merits further research.
The cost of pre-notification is low, and the estimated intervention cost per additional participant retained was around £50. Compared with the costs of recruiting participants, and the potential bias in estimates of effectiveness due to attrition, this cost appears low. Furthermore, the total cost per additional participant retained may be lower than this due to the lessening of subsequent reminders and the costs inherent in this process.
Although not investigated here, delays in participants responding may impact adversely on effectiveness estimation as it is questionable how relevant their outcome is to the scheduled time-point. As can be seen from the Kaplan-Meier plots, despite over 80% of the remaining WORKWELL participants providing valid primary outcome data responses, only around 50% had provided this information by around 2 weeks after its due date (3 weeks after mailing). There was, however, no evidence that the pre-notification led to earlier questionnaire responses.
Strengths and limitations
We performed a randomised evaluation of a pre-notification letter or email 2 weeks prior to a mailing of a hard-copy questionnaire of, or e-mailing a web-link to, a 6-month outcome questionnaire. This text in the letter (or email) was informed by behaviour-change theory, and was reviewed by the WORKWELL Trial Management Group, including a patient/public member, although it is possible that wider review would have led to improved text. However, as with previous SWATs, our SWAT was performed on a predominantly female and white population of working age. It is unclear how well the results will generalise to more diverse populations. There were also a number of issues caused by COVID: participants were not always able to receive their questionnaires via their original chosen mode, although the pre-notifications were mostly sent using the chosen mode of questionnaire delivery. Costs may not generalise to other trials. However, our SWAT has the strength that, unlike most previous SWATs, in addition to a simple consideration of intervention costs, we performed a full cost-effectiveness analysis. In this, we included estimated reminder costs which will be reduced if the intervention is successful in initiating an early response, in addition to the intervention costs. Similarly, our estimate of the effect of a pre-reminder may not generalise to other trials, as the effect may vary according to the population or control group retention rate. However, our control group retention rate is relatively high (79%) so there would be more potential for a greater absolute effect (risk difference) of the intervention on retention.
Interpretation
Pre-notification may lead to a small increase in response rates to outcome questionnaires. The cost of pre-notification relative to potential gains is low. However, further evaluations of pre-notifications, whether by letter or digital means, and using theoretically-informed text believed likely to maximise any impact, are needed.
Data sharing
Data will be supplied to the University of York as part of the data sharing agreement for the PROMETHEUS project and will be available from the lead author following publication of the WORKWELL trial.
Registration
This SWAT is registered as SWAT86 on the SWAT Repository at https://www.qub.ac.uk/sites/TheNorthernIreland NetworkforTrialsMethodologyResearch/SWATSWAR Information/Repositories/SWATStore/
Supplemental Material
Supplemental Material - Study Within A Trial (SWAT86): Randomised evaluation of pre-notification of trial participants before self-report outcome data collection to improve retention
Supplemental Material for Study Within A Trial (SWAT86): Randomised evaluation of pre-notification of trial participants before self-report outcome data collection to improve retention by Christopher J Sutton, Sarah Cotterill, Denise Forshaw, Sarah Rhodes, Alexandra Haig and Alison Hammond in Research Methods in Medicine & Health Sciences
Footnotes
Acknowledgements
Chris Brown (Lancashire Clinical Trials Unit) for data management support and provision of the data set for analysis; Dr Angela Ching (University of Salford) for project co-ordination support on behalf of the WORWELL Trial team; Dr Antonia Marsden (University of Manchester) for statistical support in relation to the WORKWELL Trial outcome measure definition.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported via the PROMETHEUS project which was funded by the Medical Research Council (MRC) [grant number MR/R013748/1].
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
