Abstract
Background
Randomised Controlled Trials (RCTs) are the most robust method to test new or existing interventions. Recruitment and retention are crucial aspects to their success but are often challenging. Methodological changes to RCTs are often tested by Studies Within A Trial (SWATs). To date there is mixed evidence supporting the use of prenotification newsletters to boost retention. This SWAT aimed to assess this intervention within the PROFHER-2 trial.
Methods
A two-arm parallel group SWAT embedded at the 24-months follow-up for PROFHER-2 – a RCT evaluating treatment methods for a 3- or 4- part proximal humerus fractures in patients over 65 years. Participants were randomised (1:1) to either receive a newsletter 2-to-4 weeks prior to their 24-months questionnaire, or not. The results have been combined in a meta-analysis with existing evidence.
Results
There was no evidence of a difference in retention (OR 2.17, 95% CI 0.21-22.02, p = 0.51). Similarly, there was no statistically significant difference in the completion of the primary outcome of the returned questionnaires (OR 0.72, 95% CI 0.04–12.70, p = 0.82), nor the proximity of completion (HR 1.23, 95% CI 0.74 – 2.05, p = 0.43). This result is similar to that seen in two of the four previous evaluations of this SWAT.
Conclusion
Sending a newsletter as a prenotification to a questionnaire in a RCT does not improve the retention rates. However, the sample size for this piece of work is small, and the retention rates were high (94%). When combined with the existing evidence, there is no evidence of an effect.
Keywords
Introduction
When evaluating a new or existing intervention for a healthcare treatment, or in other research areas, the most robust method is using a Randomised Controlled Trial (RCT). RCTs rely on participant recruitment, and retention to be able to answer their study questions, but these are often challenging.1,2 Methodological changes to elements within RCTs can be tested to improve both recruitment and retention, and one method to test these changes is using a Study Within A Trial (SWAT). 3 SWATs are self-contained research studies, which evaluate alternative ways of delivering elements of a trial process and are embedded within a host trial.
Pre-notification has been shown to increase the return rates of questionnaires, 4 and newsletters are often used to increase engagement with participants in a trial. The impact of their use is however unclear5–7 with insufficient evidence available currently to determine if sending a pre-notification newsletter to participants in advance of their upcoming follow-up postal questionnaire increases retention. This SWAT aims to evaluate the research question: does sending a prenotification newsletter to participants increase the retention rate compared to not sending one, for participants in the PROFHER-2 trial? Individually SWATs are not usually sufficiently powered to detect small differences in retention rates and therefore require replication and collation through meta-analysis. 3 This SWAT will therefore also add to the existing evidence base for newsletters as a participant prenotification aid.
Methods
Trial design
This two-arm parallel group SWAT was embedded within the PROFHER-2 RCT (ISRCTN76296703), 8 which is evaluating the best treatment for older adults (≥65 years old) with a three- or four-part fracture of the proximal humerus. PROFHER-2 compares two surgical treatments (reverse shoulder arthroplasty and hemiarthroplasty) with non-surgical care and is based within hospitals in the United Kingdom. This SWAT is a replication of SWAT 28 on the SWAT repository 9 which evaluated prenotification newsletters in a trial evaluating osteoporosis screening for older female adults. In addition to the replications reported in the recent Cochrane review of retention strategies for RCTs, 4 this SWAT has previously been undertaken in a trial of falls prevention interventions in older adults, 5 and further replications are ongoing in trials of treatments for healing of venous leg ulceration and in a trial of surgical interventions for shoulder arthritis.10,11 Participants were randomised with an allocation ratio of 1:1 to either receive a pre-notification newsletter, or to receive nothing.
This SWAT was approved by the North East – Tyne and Wear South Research Ethics Committee (Reference: 18. NE.0125) as part of a change to the overall trial protocol. The SWAT was embedded with effect from 29th April 2021, and all participants who were fully participating in the trial and were still due their 24-months follow-up postal questionnaire after 13th May 2021 were planned to be included in this SWAT (allowing for the participant prenotification newsletter to be sent 2 weeks prior to the questionnaire). The SWAT was planned to run until the end of the host trial recruitment period; however, following discussion with the trial management group the decision was made to end the SWAT early due to perceived benefit of maintaining participant engagement, and to ensure high response rate. As such, the SWAT was ended on 31st March 2022, and all participant following this point were sent the participant prenotification newsletter before their 24-months follow-up postal questionnaire. Participants data were collected up until 7th May 2022, to allow the last due questionnaire 60days to be returned.
Participants
All participants who were still fully participating in the host trial (i.e., not deceased, and had not withdrawn from follow-up), and were due their 24-months follow-up postal questionnaire, were eligible to be included in the SWAT. SWAT data was collected routinely by York Trials Unit, as part of the monitoring of postal questionnaires being due, sent, and received.
Intervention
This SWAT was testing a participant prenotification newsletter (see Supplement Material), sent with a cover letter, versus nothing.
The design of the newsletter was based on that used in SWAT 28 9 with modifications to reflect the host trial. In SWAT 28, the newsletter was printed as an A5 single sheet and folded into a booklet. The newsletter contained an update on the trials progress and reminded participants about the importance of returning their questionnaires whether they were in the control or intervention group. On the back of the newsletter, there was a brief description, with a photograph, of the local study team, and associated contact details if the participant had any queries or questions. The prenotification participant newsletter was sent approximately 6 weeks before participants were due to receive their 24-months questionnaire.
In this SWAT, the newsletter and accompanying cover letter were sent by post to participants 2-4 weeks prior to the 24-months postal follow-up questionnaire being due. The newsletter was sent in a A5 white envelope using University of York franked mail. The newsletter contained an update on host trial recruitment (which was ongoing as some participants reached 24-months), and number completing the host trial 24-months postal questionnaire follow-up, alongside a photo of, and a message from the Chief Investigator. A seasonal image was also included – such as an image of a daffodil for Spring. It also contained contact details for the trial team, and links to other relevant participant resources: advice on shoulder pain management, study website and where to find additional research participation opportunities. It was two pages long and printed in colour on A4 paper. The PROFHER-2 patient and public advisory group reviewed and approved the newsletter design and content prior to implementation. No changes were made to the SWAT intervention whilst it was under evaluation.
Alongside the implementation of this SWAT, all participants were sent a PROFHER-2 branded ball point pen with their 12- and 24-months follow-up postal questionnaires, to boost retention 12 – however, this was not part of the SWAT, and is not anticipated to interact with the intervention being evaluated here.
Outcomes
The primary outcome of this SWAT was retention rate, defined as the proportion of participants who returned their 24-months follow-up postal questionnaire, out of those who were due to (i.e., still participating, not withdrawn from postal follow-up). This data was collected routinely in the monitoring of the host trial. The primary outcome was combined with the results of similar SWAT evaluations via a fixed effects meta-analysis.
The secondary outcomes were the completeness of primary outcome data (binary outcome) and the proximity of questionnaire completion to the due date. Primary outcome data was considered complete if the 24-months questionnaire was returned with sufficient data present to derive an Oxford Shoulder Score (OSS) – equivalent to 10 or more of the 12 questions being answered. The proximity was measured as the days between due date and completion date. Additionally, intervention costs were provided along with a cost per additional participant retained was calculated, if the intervention was effective.
Sample size
Following SWAT methodology, 3 no formal sample size calculation was undertaken for this SWAT, as the SWAT sample size was constrained by that of the PROFHER-2 trial. The PROFHER-2 trial was designed to recruit 380 participants, however, the SWAT was embedded after some participants had already completed their 24-months postal follow-up questionnaire, and ended whilst the trial was ongoing, so it was not possible for the whole sample to be included.
Randomisation
Participants were randomised using permuted blocks of size 2, 4 and 6, using a 1:1 allocation. The randomisation was stratified by the host trial allocation (3-levels). Allocations were generated by an independent statistician using code written by the PROFHER-2 trial statistician in STATA version 17.0. 13 The randomisation schedule was implemented via the York Trials Unit PROFHER-2 management system, i.e., independently to any person involved in the SWAT.
Blinding
Participants were not aware of their participation in this SWAT. Data for the SWAT was collected routinely, as part of the trial, so there were no data collectors to be blinded. The statistician was not blind to allocation when analysing the SWAT.
Statistical methods
All statistical analyses were conducted in STATA version 17.0. 13 The analysis was conducted on an intention-to-treat basis but included only those participants who had reached the 24-months timepoint within the duration of the SWAT. Participants were analysed according to their randomised group, regardless of whether they received the intervention or not. Two-sided tests were used at the 5% significance level.
Baseline characteristics are summarised descriptively on an “as analysed” and an “as randomised” basis. Ethnicity was collected as per the PROFHER-2 questionnaires, but due to limited variation it is dichotomised to Caucasian or non-Caucasian for reporting in this SWAT. Where dislocation (yes/no) was missing, it was derived using the host trial randomisation stratification (those with a dislocation are only eligible for the two surgical arms). Binary outcome measures (retention rate and completion of primary outcome) were analysed using logistic regression. Proximity of completion was analysed using Cox Proportional Hazards regression, with participants who returned their questionnaire before or on the due date being assigned a return time of 0.1 days, and participants who did not return their questionnaire being censored at 60 days. Where completion date was missing, but received date was not, completion date was set to be 3 days earlier than the received date.
The analysis planned for all models to be adjusted for host trial allocation (3-level), and presence of dislocation (2-level) as a fixed effect and site as a random effect. However, due to a limited sample size, and low variation in the outcomes by stratification variables (i.e., all participants in one of the host trial arms had returned the questionnaire), it was decided to not stratify the models. Model assumptions were checked.
Results
Participant flow
Participant flow through this SWAT is presented in Figure 1. At the time of ending the SWAT (31st March 2022), 230 participants had been randomised into the PROFHER-2, and 174 had been randomised into the SWAT (75.7%). Reasons for non-inclusion in the SWAT can be found in Figure 1. Flow of participants through the embedded trial.
Recruitment
There were 174 participants randomised into the SWAT, 88 were allocated to receive the newsletter, and 86 allocated to not receive it. Of these, only 66 (37.9% of those randomised) were still fully participating and reached the 24-months follow-up postal questionnaire timepoint within the course of the SWAT (27 newsletter, 39 not) – these are included in the analysis. Reasons for exclusion from the analysis can be seen in Figure 1.
Baseline characteristics of those as randomised, and as analysed in the SWAT.
Min = minimum, Max = Maximum, SD = Standard Deviation.
Primary outcome
Results of the primary and secondary analyses.
OR = Odd Ratio, HR = Hazards Ratio, CI = Confidence Interval.
Secondary outcomes
Completion of the primary outcome
Of the 26 participants who returned the 24-months questionnaire in the newsletter group, 25 (96.2%) had provided enough data for the measure to be scoreable. This was similar in the no newsletter group where 35 of the 36 returned questionnaires were scoreable (97.2%) – an increase of 1.0%. This difference was not considered to be statistically significant – full details are in Table 2.
Proximity of completion to due date
The mean return time of the 24-months questionnaire was 7.7 days (SD 11.3) in the newsletter group (range −14 to 49 days – i.e., 14 days early to 49 days late) and 9.3 days (SD 15.0) in the no newsletter group (range −6 to 73 days). There were 4 participants who had not returned the questionnaire (one newsletter, three no newsletter). There was no evidence to suggest a difference in proximity of completion time between the two groups, full details in Table 2.
Associated costs
Although direct costs of the intervention were not collected within this SWAT, the associated tasks and estimated costs having been approximated, these include. - Associated staff time for creating newsletter and cover letter: £336.00 based on 2 days of a mid-tier Grade 6 Researcher’s time using the University of York pay scale. - Printing and paper costs for the newsletter, one A4 double sided, colour: £0.13 per participant, a total of £11.44. - Printing and paper costs for associated cover letter, one side of A4, black and white: £0.025 (2.5 pence) per participant, a total of £2.20. - Envelope cost, which will include two pieces of A4 paper: £0.025 per participant, £2.20 total. - Associated staff time for packing the letters: estimated to be 30 s to prepare each,
14
so a total of 44 min; using a mid-tier Grade 3 salary would equate to a total of £10.28. - Postage cost: second class Royal Mail using Mailmark franking at a cost of £0.41 each – total cost £36.08.
Thus, the total combined cost for this SWAT was estimated to be £398.20, an average of £4.53 per participant in the intervention arm. As there was no evidence of an increase in response rate, the cost per additional participant retained has not been calculated.
Combining with existing evidence
When combining our primary analysis results with the existing evidence (four trials), the overall odds ratio is 1.08 with a 95% CI of 0.91 to 1.28. A forest plot can be seen in Figure 2. This suggests that there is no evidence of an effect on the retention rate when sending a participant prenotification newsletter. This evidence combined the data from 5688 participants, across five trials – one falls prevention, one in osteoporotic fracture screening, one in shoulder fractures, one in arthritis, and one in dental health. Three of the trials have been in over 65-year-olds, and two just in an adult population; with one just in females. Of these evaluations, all those in an older adult population (Mitchell, Rodgers and Coleman) have provided an OR different than 1, which may indicate an effect in this population, whereas those in an adult population have suggested no effect. The GRADE evidence of the four existing trials was one high, one low and two unclear. Therefore, it may be appropriate to consider that sufficient evidence has been accumulated for this research question, when using the Trial Forge Guidance on when further evaluations are needed – however further evaluation in a younger population may be beneficial.
15
Forest plot of the combined results.
Discussion
This SWAT showed that there was no evidence that sending a pre-notification newsletter improved the retention rate, the completion of the primary outcome for those questionnaires that were returned, or how promptly participants complete the questionnaire.
The intervention had an associated financial cost of development, printing and mailing at £4.53 per participant, however cost per additional participant retained was not calculated in the absence of any evidence of effect.
In the most recent Cochrane review 4 for retention strategies (2020), there were four trials which had evaluated this intervention: one found a harmful effect, one a beneficial effect, and two found no evidence of an effect when using a participant prenotification newsletter, compared to none. When combining our results with these, the evidence suggests there is no effect on the retention rate from using a newsletter, so this strategy should be used with caution, as it has associated financial and time costs for a trial, without providing an additional benefit – when other, known to be effective, strategies could be implemented instead.
Strengths and limitations
This SWAT is a replication of previous and so provides additional contribution in relation to accumulating evidence on the effectiveness of this intervention for participant retention.
The intervention used in this SWAT was comparable to that used in the initial SWAT undertaken, 7 albeit with some changes made. These changes were required to ensure the SWAT was relevant to the host trial and as a result the implications of these minor changes are unlikely to influence the overall meta-analysis result.
The demographics of the PROFHER-2 host trial population were comparable to the demographics of the proportion of host trial participants included in the SWAT population. The findings from this study are therefore only generalisable only to this population, however then accumulating evidence via meta-analysis comprises a range of different populations giving greater generalisability. Ethnic diversity is however limited and therefore further replications in more diverse populations may be warranted.
The biggest limitation of this SWAT was its sample size given that this was restricted due to early termination of the SWAT. In addition, the overall retention rate seen in the host trial (93.9%) was already high, as was the completion rate of the host trial primary outcome (96.8%) – meaning observing any difference in this population would have been difficult.
Implications for trial practice and SWAT research
The associated meta-analysis of four studies suggests that use of a newsletter sent in advance of a postal follow up questionnaire does not improve retention rates. The GRADE evidence of the four existing trials was one high, one low and two unclear.
In this study and others in an older population, results do however indicate an effect in this population,5,7 whereas those in an adult population have suggested no effect. Using Trial Forge Guidance, 15 it may therefore be appropriate to consider that sufficient evidence has been accumulated for this research question in an older population, however the value of further information in this context should be assessed. Irrespective it is however likely that further evaluation in a younger population may be beneficial.
Supplemental Material
Supplemental Material - Pre-notification of trial participants to improve response rates retention (SWAT 28); a study within a trial (SWAT) embedded in PROFHER-2
Supplemental Material for Pre-notification of trial participants to improve response rates retention (SWAT 28); a study within a trial (SWAT) embedded in PROFHER-2 by Elizabeth Coleman, Luke Strachan, Catherine Arundel, Natasha Mitchell, Amar Rangan in Research Methods in Medicine & Health Sciences.
Footnotes
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by the host trial that this SWAT was embedded in which was funded by the National Institute for Health Research: Health Technology Assessment (16/73/03).
Declaration of conflicting interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: EC and CA declare that they have previously received funding from the PROMETHEUS programme (MR/R013748/1) and an NIHR CTU infrastructure grant (NIHR132547). All other authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
