Abstract
Background
With attrition common in randomised trials, strategies are needed to minimise this. Many retention strategies include ‘thanks’ elements however there is currently no evidence of the effectiveness of a ‘thank you’ intervention separate to other trial activity or information. This Study Within A Trial (SWAT) sought to assess if a thank you card increases completion of the host trial primary outcome.
Methods
A two arm SWAT, using a 1:1 (intervention:control) allocation ratio, embedded within the DISC trial. The primary outcome was the difference in retention rate at 1 year post-treatment. Secondary outcomes were outcome data completeness, cost, and retention at 2 years post-treatment. Analyses were conducted using logistic regression adjusting for SWAT and host trial allocation.
Results
A total of 358 participants were randomised and included in the SWAT analyses. Completion of the 1-year outcome visit was 89.7% (n = 157) in the intervention group and 90.2% (165) in the control group (adjusted odds ratio (OR) 0.95, 95% CI 0.48 to 1.90, p = .89). There was no evidence of a difference in completeness of key outcome data (adjusted OR 1.84, 95% CI 0.71 to 4.73, p = .20) or retention at 2 years post treatment (adjusted OR 1.13, 95% CI 0.59 to 2.17, p = .72).
Conclusion
It remains unclear if thank you cards increased the rate of primary outcome follow-up completion within the DISC trial. However, as the first evaluation of a distinct ‘thank you’ intervention for improving retention rates, further replications are required to determine effectiveness, ideally in populations other than older, male, Caucasians.
Background
Randomised controlled trials (RCTs) are the gold standard assessment for intervention effectiveness, whereby participants are randomly assigned to receive one of the treatments under evaluation. 1 Testing interventions in this way enables the causality of intervention effectiveness to be assessed, 1 and allows identification of effective and ineffective interventions. 2
Many RCTs experience attrition with 25% of studies reported to experience more than 10% attrition. 3 The impacts of this include research waste due to the need for additional costs, study extensions, reduced statistical power and potential early termination of the research.4,5 There is therefore a need to identify effective interventions to improve retention of participants. One way to do this is to conduct a Study Within A Trial (SWAT). SWATs are a self-contained research study embedded within a host trial to assess or explore trial process delivery or organisation. 6 SWATs may be randomised or non-randomised depending on the question. When randomised, SWATs enable the development and rigorous testing of trial methods (e.g., recruitment, retention, data collection or monitoring) and avoid limitations regarding the practical implementation of findings within further RCTs which can arise if interventions are tested in hypothetical, quasi-randomised and non-randomised settings.
Many RCTs routinely use retention strategies to minimise attrition, for example use of shorter questionnaires, prompts or reminders to complete questionnaires, participant facing study progress newsletters and provision of incentives. 7 None of these interventions however currently have high certainty of effectiveness for increasing RCT retention, and therefore for each further SWAT evaluations are required. 7
Many retention strategies routinely include elements of thanks within them, for example thanking a participant in a cover letter for completing the enclosed questionnaire, in a newsletter for their continued participation or by providing incentives which demonstrate appreciation (e.g., incentives, certificates, pens). The idea of ‘thanks’ as a retention strategy utilises the psychology of positive reinforcement whereby reinforcing a behaviour aims to promote replication at subsequent timepoints. 8 There is currently no evidence as to the effectiveness of a distinct ‘thank you’ intervention (separate to other trial information) 7 and recent evidence suggests that those saying ‘thank you’ often undervalue its effect. 9 The concept of thanking participants fits with two of the PRioRiTy II top 20 retention questions (4 – What are the best ways to encourage participants to complete task; 18 – What strategies make participants feel valued?). 10 Therefore, a need to test the impact of thanks in the context of trial recruitment and retention is warranted.
This paper presents the findings of the first assessment of the effects of providing participants with a distinct ‘thank you’ intervention, separate to other trial information, on completion rates of the primary outcome compared to those participants who do not receive a thank you card.
This publication provides the basis for future SWAT replications of this intervention which will collectively result in a robust and definitive conclusion as to the effectiveness of distinct thanks within a RCT, given it is widely acknowledged that individual SWATs are unable to observe the modest effect sizes associated with most SWAT interventions and therefore replication and meta-analysis of SWATs is required. 6
Methods
Design
A two arm, parallel group randomised controlled SWAT was undertaken with an allocation ratio of 1:1 (thank you card intervention vs no thank you card control).
The SWAT protocol (number 119) can be found at: https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/).
The SWAT was embedded in the DISC trial, the protocol for which has been published previously. 11 The DISC trial recruited adult patients with Dupuytren’s Contracture and randomised them equally (1:1) to receive either collagenase clostridium histolyticum injection (intervention) or limited fasciectomy (control). The primary outcome was the Patient Evaluation Measure, 12 a validated self-report questionnaire to assess hand health, measured at 1 year post treatment.
The SWAT was considered low risk and was approved by the Research Ethics Committee Yorkshire and Humber – Leeds West (17/YH/0120). Participants were not informed of this SWAT and so did not provide informed consent for their involvement. 13
Participants
The SWAT was embedded in the DISC trial formally on 01.08.2019, therefore all participants due to complete the 1 year follow up visit on or after this date were eligible for inclusion in the SWAT. Participants were excluded if they had completed their 1 year follow up at the time of SWAT randomisation, had withdrawn from study follow up (prior to SWAT randomisation) or were deceased (prior to SWAT randomisation).
SWAT data were collected in 31 participating secondary care NHS Trusts via face-to-face clinic appointments for questionnaire completion. Due to the COVID-19 pandemic options of remote (video or postal) follow up was introduced part way through the SWAT, in addition to clinic follow up when or where possible. On the basis that the DISC trial primary outcome was a patient reported outcome measure (the Patient Evaluation Measure 12 ), it was decided to continue the SWAT irrespective of these changes.
Intervention
The SWAT intervention was a thank you card (See Supplementary File 1) sent to participants at the start of the 9th month post treatment delivery. Participants allocated to the control arm did not receive a thank you card.
Outcomes
Primary outcome
- Difference in retention rate at 1 year post treatment of participants who received the thank you card versus those who did not receive a thank you card. Retention rate was defined as the number of participants who completed and returned the questionnaire booklet at the DISC primary outcome timepoint.
Secondary outcome
- Cost per additional participant retained (if effect was positive) calculated as the total SWAT cost divided by the number of additional participants retained. - Completeness of outcome data (Completeness was defined as responses provided to all participant reported outcome questionnaire domains, Incompleteness defined as one or more responses missing from participant reported outcome questionnaire domains).
A post hoc analysis of retention at the subsequent DISC trial timepoint (2 years) was also included. This included participants due for 2 years follow up on or before 31.10.2022, when study follow up completed, and was defined as the number of participants who completed and returned the questionnaire booklet at the DISC 2 years timepoint.
Sample size
The SWAT sample size was dependent on the host trial (DISC), 11 therefore no formal sample size calculation was performed, which is in line with SWAT methodology.6,13
As the SWAT was implemented part way through DISC trial follow up, only those participants currently in follow up and due to complete their 1 year follow up on or after 1st August 2019 were included in the sample.
Randomisation
Participants were randomised in a 1:1 ratio, stratified by DISC trial allocation.
Allocation concealment was achieved through the generation of the allocation sequence on a quarterly basis by the DISC trial statistician based at York Trials Unit, University of York, and who was independent to the trial team.
The randomised allocations were then provided to the central coordinating team for implementation. The statistician was not involved in preparing or sending the thank you cards.
Blinding
It was not possible to blind the central coordinating team to the participant’s allocation. Participants were not informed about the SWAT and so were blind to the SWAT allocation.
Statistical analysis
Analyses were conducted using SPSS (Version 28.0.1.1 (15)). Participants were included in the analyses using the principle of intention to treat, that is, as part of the groups to which they were allocated (with regards to both the SWAT and main trial allocation).
Analyses of the binary primary and secondary outcomes (retention and completeness) were conducted using logistic regression. These models included two binary explanatory variables; SWAT allocation (thank you card or no thank you card) and DISC trial allocation (collagenase injection or limited fasciectomy). No other baseline covariates were conditioned on in the analysis to facilitate pooling of the estimated odds ratios from this study with odds ratios from other SWAT analyses that have not conditioned on patient baseline characteristics (other than host trial allocation). Unadjusted analyses (i.e., without the indicator for DISC trial allocation) were also completed. Treatment effects were reported in terms of odds ratios (conditional on DISC trial allocation), together with two-sided 95% Wald method confidence intervals and p-values for tests of H0: Odds Ratio = 1, with statistical significance based on the 5% significance level.
We calculated the average cost per card as the sum of printing, preparation and postage of the card. Staff time was calculated as the time spent undertaking SWAT activities (e.g., filling and labelling envelopes) multiplied by their associated hourly pay rate, determined using the midpoint of the grade band (University of York) for each member of staff involved in SWAT. Postage costs were calculated using second class Royal Mail Mailmark franking rates. Cost per card was calculated as the total cost for each component, divided by the number of cards. In the event that the primary analyses identified higher retention in the SWAT intervention group (irrespective of statistical significance) the cost per additional participant would be calculated by dividing the total costs by the number of additional participants retained.
Results
The SWAT ran between 01.05.2019 and 30.09.2022. A pause to SWAT activity was required between 01.04.2020 and 30.09.2020 due to COVID-19 restrictions.
Of the 672 participants randomised to the DISC trial, 489 participants were randomised into the SWAT (245 intervention group; 244 control group).
One hundred and 31 participants were excluded from analysis, primarily due to COVID-19 restrictions preventing SWAT activity from being conducted. This resulted in 358 eligible participants included in the analyses (intervention group 175, control group 183). Figure 1 presents the flow of participants through the SWAT. Study flow diagram.
Participant characteristics
Baseline characteristics of participants included in primary analysis by arm.
Primary and secondary outcomes
Adjusted and unadjusted primary and secondary outcome analyses.
aPrimary outcome.
Of those participants in each group who did complete a 1 year assessment, 150 (95.5%) of participants who received a thank you card fully completed all patient reported outcomes compared to 152 (92.1%) of participants who did not receive a card (adjusted odds ratio 1.84, 95% CI 0.71 to 4.73, p = .20).
For the seconday outcome of retention at subsequent time points (2 years), 115 (85.2%) of participants who received a thank you card completed the 2 years follow up compared to 118 (83.7%) of participants who did not receive a card (adjusted odds ratio 1.13, 95% CI 0.59 to 2.17, p = .72).
Secondary outcome – cost per Participant retained
Costs associated with the thank you card SWAT.
aA total of 350 cards were ordered for the SWAT, sufficient to allow all DISC Trial participants to be randomised into the study.
bPreparation and packaging was completed by a University of York Grade 3 member of staff with the salary midpoint of the band used for calculations. Preparation and packaging took approximately 90 min (1.45 h) to complete at a rate of approximately 30 s per card.
cCards were sent to SWAT intervention participants via Royal Mail Mailmark franking at a cost of £0.41 per card.
Discussion
This SWAT found little evidence to suggest that sending a thank you card increases completion of the primary outcome visit, completeness of outcome data or completion of subsequent follow up visits.
Effect size estimates suggest that the control group (no thank you card) were more likely to complete the primary outcome than the intervention group (thank you card) (OR 0.95, 95% CI 0.48 to 1.90, p = .89). Effect estimates did however favour the intervention group (thank you card) for completeness of outcome data (OR 1.84, 95% CI 0.71 to 4.73, p = .20) and completion of subsequent visits (OR 1.13, 95% CI 0.59 to 2.17, p = .72). Given the bounds of the confidence intervals associated with these analyses, although trending in favour of the intervention, there is limited certainty around each of these estimates and hence the result and it’s applicability to other trials must be interpreted with caution at this stage.
The intervention was relatively low cost at £0.79 per card sent, however cost per additional participant retained was not calculated in the absence of any evidence of effect.
Given the low cost of the intervention, the limited certainty around the estimates, and as this is the first evaluation of a distinct thank you intervention, further replication of this SWAT is recommended, to enable subsequent meta-analysis and identification of a robust and definitive conclusion to the effectiveness of thank you cards as a RCT retention strategy.
It is not currently possible to assess the results of this SWAT in the context of existing evidence, due to the lack of other evidence as to the effectiveness of a distinct thank you intervention (separate to other trial information). 7
Strengths and limitations
A limitation of this SWAT was the impact of COVID-19 which resulted in the SWAT being paused between 01.04.2020 and 30.09.2020. This resulted in 122 participants (62 intervention, 60 control) being excluded from the SWAT. While this limits the sample size somewhat, this is unlikely to have a substantial impact given that SWATs are routinely underpowered and require multiple replications and combination via meta-analysis to establish intervention effectiveness.
A further potential limitation to the SWAT was the host trial design whereby follow up timepoints were fixed to treatment delivery rather than randomisation. It was encouraging to note only one participant (intervention) was randomised after the SWAT timepoint however lags in receipt of information at the coordinating centre, meant some participants were not able to be randomised into the SWAT which reduced the sample size. Future studies using follow up fixed to treatment delivery should consider robust methods to ensure receipt of information necessary to ensure participants can be included in retention SWATs.
The timing of the intervention may have also impacted effectiveness as the intervention was delivered to allocated participants at the start of the 9th month post treatment (i.e., 3 months prior to primary outcome timepoint). Future replications may therefore also wish to consider the optimum point at which the intervention is administered.
Finally, the participants included within the SWAT were representative of the overall trial population, given average age, proportion of male versus female participants and ethnicity proportions were not substantially different between the SWAT and host trial population. This likely limits the generalisability of the results to other patient populations. Further replications should therefore be considered in different populations, given this SWAT was applied to an predominantly older, male, Caucasian population. In addition, retention in the DISC trial was already high and evaluation in studies with lower retention rates would also be beneficial.
Implications for trial practice and SWAT research
Thank you cards as a retention method requires further evaluation to confirm or refute intervention effectiveness. As a result, this intervention should currently only be used when being evaluated in a SWAT.
Further replications should be completed in additional (i.e., younger, female, or ethnically diverse) populations or in trials where retention rates are currently lower than anticipated.
Conclusion
From this first evaluation of a distinct ‘thank you’ intervention for improving retention rates, it remains unclear if thank you cards increase the rate of primary outcome follow-up completion. It is however widely acknowledged that individual SWATs are unable to observe the modest effect sizes associated with most SWAT interventions. 6 Further replications are therefore required, ideally in populations other than older, male, Caucasians, to distinguish conclusively whether distinct thanks within a RCT is an effective retention strategy.
Supplemental Material
sj-pdf-1-rmm-10.1177_26320843241229934 – Supplemental Material for Study within a trial 119: The effectiveness of a thank you card to improve trial follow up; a randomised study within a trial
Supplemental Material, sj-pdf-1-rmm-10.1177_26320843241229934 for Study within a trial 119: The effectiveness of a thank you card to improve trial follow up; a randomised study within a trial by Catherine Arundel, Charlie Welch, Puvanendran Tharmanathan, and Joseph Dias in Journal of Research Methods in Medicine & Health Sciences
Footnotes
Acknowledgments
We would like to acknowledge the input from Sally Baker, Emma-Louise Rose Brooks, Sue Heslop and Mo Wakefield at York Trials Unit, University of York for their help and support with administrative coordination of this study.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This SWAT was funded as part of the DISC Trial which was funded the National Institute for Health Research Health Technology Assessment (project number 15/102/04). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Health Technology Assessment programme, NIHR, NHS or the Department of Health.
Ethical statement
Registration
Protocol availability
This SWAT was embedded in the DISC trial, the protocol for which has been published previously. 11
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
