Study within a trial 119: The effectiveness of a thank you card to improve trial follow up;a randomised study within a trial

Abstract

Background

With attrition common in randomised trials, strategies are needed to minimise this. Many retention strategies include ‘thanks’ elements however there is currently no evidence of the effectiveness of a ‘thank you’ intervention separate to other trial activity or information. This Study Within A Trial (SWAT) sought to assess if a thank you card increases completion of the host trial primary outcome.

Methods

A two arm SWAT, using a 1:1 (intervention:control) allocation ratio, embedded within the DISC trial. The primary outcome was the difference in retention rate at 1 year post-treatment. Secondary outcomes were outcome data completeness, cost, and retention at 2 years post-treatment. Analyses were conducted using logistic regression adjusting for SWAT and host trial allocation.

Results

A total of 358 participants were randomised and included in the SWAT analyses. Completion of the 1-year outcome visit was 89.7% (n = 157) in the intervention group and 90.2% (165) in the control group (adjusted odds ratio (OR) 0.95, 95% CI 0.48 to 1.90, p = .89). There was no evidence of a difference in completeness of key outcome data (adjusted OR 1.84, 95% CI 0.71 to 4.73, p = .20) or retention at 2 years post treatment (adjusted OR 1.13, 95% CI 0.59 to 2.17, p = .72).

Conclusion

It remains unclear if thank you cards increased the rate of primary outcome follow-up completion within the DISC trial. However, as the first evaluation of a distinct ‘thank you’ intervention for improving retention rates, further replications are required to determine effectiveness, ideally in populations other than older, male, Caucasians.

Keywords

Study within a trial retention methods embedded randomised controlled trial thanks

Background

Randomised controlled trials (RCTs) are the gold standard assessment for intervention effectiveness, whereby participants are randomly assigned to receive one of the treatments under evaluation.¹ Testing interventions in this way enables the causality of intervention effectiveness to be assessed,¹ and allows identification of effective and ineffective interventions.²

Many RCTs experience attrition with 25% of studies reported to experience more than 10% attrition.³ The impacts of this include research waste due to the need for additional costs, study extensions, reduced statistical power and potential early termination of the research.^4,5 There is therefore a need to identify effective interventions to improve retention of participants. One way to do this is to conduct a Study Within A Trial (SWAT). SWATs are a self-contained research study embedded within a host trial to assess or explore trial process delivery or organisation.⁶ SWATs may be randomised or non-randomised depending on the question. When randomised, SWATs enable the development and rigorous testing of trial methods (e.g., recruitment, retention, data collection or monitoring) and avoid limitations regarding the practical implementation of findings within further RCTs which can arise if interventions are tested in hypothetical, quasi-randomised and non-randomised settings.

Many RCTs routinely use retention strategies to minimise attrition, for example use of shorter questionnaires, prompts or reminders to complete questionnaires, participant facing study progress newsletters and provision of incentives.⁷ None of these interventions however currently have high certainty of effectiveness for increasing RCT retention, and therefore for each further SWAT evaluations are required.⁷

Many retention strategies routinely include elements of thanks within them, for example thanking a participant in a cover letter for completing the enclosed questionnaire, in a newsletter for their continued participation or by providing incentives which demonstrate appreciation (e.g., incentives, certificates, pens). The idea of ‘thanks’ as a retention strategy utilises the psychology of positive reinforcement whereby reinforcing a behaviour aims to promote replication at subsequent timepoints.⁸ There is currently no evidence as to the effectiveness of a distinct ‘thank you’ intervention (separate to other trial information)⁷ and recent evidence suggests that those saying ‘thank you’ often undervalue its effect.⁹ The concept of thanking participants fits with two of the PRioRiTy II top 20 retention questions (4 – What are the best ways to encourage participants to complete task; 18 – What strategies make participants feel valued?).¹⁰ Therefore, a need to test the impact of thanks in the context of trial recruitment and retention is warranted.

This paper presents the findings of the first assessment of the effects of providing participants with a distinct ‘thank you’ intervention, separate to other trial information, on completion rates of the primary outcome compared to those participants who do not receive a thank you card.

This publication provides the basis for future SWAT replications of this intervention which will collectively result in a robust and definitive conclusion as to the effectiveness of distinct thanks within a RCT, given it is widely acknowledged that individual SWATs are unable to observe the modest effect sizes associated with most SWAT interventions and therefore replication and meta-analysis of SWATs is required.⁶

Methods

Design

A two arm, parallel group randomised controlled SWAT was undertaken with an allocation ratio of 1:1 (thank you card intervention vs no thank you card control).

The SWAT protocol (number 119) can be found at: https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/).

The SWAT was embedded in the DISC trial, the protocol for which has been published previously.¹¹ The DISC trial recruited adult patients with Dupuytren’s Contracture and randomised them equally (1:1) to receive either collagenase clostridium histolyticum injection (intervention) or limited fasciectomy (control). The primary outcome was the Patient Evaluation Measure,¹² a validated self-report questionnaire to assess hand health, measured at 1 year post treatment.

The SWAT was considered low risk and was approved by the Research Ethics Committee Yorkshire and Humber – Leeds West (17/YH/0120). Participants were not informed of this SWAT and so did not provide informed consent for their involvement.¹³

Participants

The SWAT was embedded in the DISC trial formally on 01.08.2019, therefore all participants due to complete the 1 year follow up visit on or after this date were eligible for inclusion in the SWAT. Participants were excluded if they had completed their 1 year follow up at the time of SWAT randomisation, had withdrawn from study follow up (prior to SWAT randomisation) or were deceased (prior to SWAT randomisation).

SWAT data were collected in 31 participating secondary care NHS Trusts via face-to-face clinic appointments for questionnaire completion. Due to the COVID-19 pandemic options of remote (video or postal) follow up was introduced part way through the SWAT, in addition to clinic follow up when or where possible. On the basis that the DISC trial primary outcome was a patient reported outcome measure (the Patient Evaluation Measure¹²), it was decided to continue the SWAT irrespective of these changes.

Intervention

The SWAT intervention was a thank you card (See Supplementary File 1) sent to participants at the start of the 9^th month post treatment delivery. Participants allocated to the control arm did not receive a thank you card.

Outcomes

Primary outcome

- Difference in retention rate at 1 year post treatment of participants who received the thank you card versus those who did not receive a thank you card. Retention rate was defined as the number of participants who completed and returned the questionnaire booklet at the DISC primary outcome timepoint.

Secondary outcome

- Cost per additional participant retained (if effect was positive) calculated as the total SWAT cost divided by the number of additional participants retained.

- Completeness of outcome data (Completeness was defined as responses provided to all participant reported outcome questionnaire domains, Incompleteness defined as one or more responses missing from participant reported outcome questionnaire domains).

A post hoc analysis of retention at the subsequent DISC trial timepoint (2 years) was also included. This included participants due for 2 years follow up on or before 31.10.2022, when study follow up completed, and was defined as the number of participants who completed and returned the questionnaire booklet at the DISC 2 years timepoint.

Sample size

The SWAT sample size was dependent on the host trial (DISC),¹¹ therefore no formal sample size calculation was performed, which is in line with SWAT methodology.^6,13

As the SWAT was implemented part way through DISC trial follow up, only those participants currently in follow up and due to complete their 1 year follow up on or after 1^st August 2019 were included in the sample.

Randomisation

Participants were randomised in a 1:1 ratio, stratified by DISC trial allocation.

Allocation concealment was achieved through the generation of the allocation sequence on a quarterly basis by the DISC trial statistician based at York Trials Unit, University of York, and who was independent to the trial team.

The randomised allocations were then provided to the central coordinating team for implementation. The statistician was not involved in preparing or sending the thank you cards.

Blinding

It was not possible to blind the central coordinating team to the participant’s allocation. Participants were not informed about the SWAT and so were blind to the SWAT allocation.

Statistical analysis

Analyses were conducted using SPSS (Version 28.0.1.1 (15)). Participants were included in the analyses using the principle of intention to treat, that is, as part of the groups to which they were allocated (with regards to both the SWAT and main trial allocation).

Analyses of the binary primary and secondary outcomes (retention and completeness) were conducted using logistic regression. These models included two binary explanatory variables; SWAT allocation (thank you card or no thank you card) and DISC trial allocation (collagenase injection or limited fasciectomy). No other baseline covariates were conditioned on in the analysis to facilitate pooling of the estimated odds ratios from this study with odds ratios from other SWAT analyses that have not conditioned on patient baseline characteristics (other than host trial allocation). Unadjusted analyses (i.e., without the indicator for DISC trial allocation) were also completed. Treatment effects were reported in terms of odds ratios (conditional on DISC trial allocation), together with two-sided 95% Wald method confidence intervals and p-values for tests of H₀: Odds Ratio = 1, with statistical significance based on the 5% significance level.

We calculated the average cost per card as the sum of printing, preparation and postage of the card. Staff time was calculated as the time spent undertaking SWAT activities (e.g., filling and labelling envelopes) multiplied by their associated hourly pay rate, determined using the midpoint of the grade band (University of York) for each member of staff involved in SWAT. Postage costs were calculated using second class Royal Mail Mailmark franking rates. Cost per card was calculated as the total cost for each component, divided by the number of cards. In the event that the primary analyses identified higher retention in the SWAT intervention group (irrespective of statistical significance) the cost per additional participant would be calculated by dividing the total costs by the number of additional participants retained.

Results

The SWAT ran between 01.05.2019 and 30.09.2022. A pause to SWAT activity was required between 01.04.2020 and 30.09.2020 due to COVID-19 restrictions.

Of the 672 participants randomised to the DISC trial, 489 participants were randomised into the SWAT (245 intervention group; 244 control group).

One hundred and 31 participants were excluded from analysis, primarily due to COVID-19 restrictions preventing SWAT activity from being conducted. This resulted in 358 eligible participants included in the analyses (intervention group 175, control group 183). Figure 1 presents the flow of participants through the SWAT.

Figure 1.

Study flow diagram.

Participant characteristics

Overall, the average age of SWAT participants (accounting for the exclusions in Figure 1) was 66.3 years (Standard Deviation (SD): 8.75). The majority were male (79.1%) and of white ethnicity (98.3%). Table 1 shows the baseline characteristics of participants included in the analysis overall and by SWAT group.

Table 1.

Baseline characteristics of participants included in primary analysis by arm.

	Intervention (n = 175)	Control (n = 183)	Total (n = 358)
Age: Mean (SD)	66.3 (8.5)	66.2 (9.0)	66.3 (8.8)
Male: n (%)	140 (80.0%)	143 (78.1%)	283 (79.1%)
Ethnicity: n (%)	White 173 (98.8%)	White 179 (97.8%)	White 352 (98.3%)
	Mixed race 0 (0%)	Mixed race 1 (0.6)	Mixed race 1 (0.6%)
	Asian/Asian british 1 (0.6%)	Asian/Asian british 3 (1.6%)	Asian/Asian british 4 (2.2%)
	Missing 1 (0.6%)		Missing 1 (0.6%)

Primary and secondary outcomes

Results for the primary outcome and secondary outcomes are presented in Table 2. In total 157 (89.7%) of participants who were sent a thank you card completed their DISC 1 year outcome assessment, compared to 165 (90.2%) of participants who did not receive a card (adjusted odds ratio 0.95, 95% CI 0.48 to 1.90, p = .89).

Table 2.

Adjusted and unadjusted primary and secondary outcome analyses.

Outcome	Intervention (card)	Control (no card)	Analysis	OR (95%CI)	p Value
Retention rate^a	157/175 (89.7%)	165/183 (90.2%)	Unadjusted	0.95 (0.48 to 1.90)	p = .89
Retention rate^a	157/175 (89.7%)	165/183 (90.2%)	Adjusted	0.95 (0.48 to 1.90)	p = .89
Completeness of outcome data	150/157 (95.5%)	152/165 (92.1%)	Unadjusted	1.83 (0.71 to 4.72)	p = .21
Completeness of outcome data	150/157 (95.5%)	152/165 (92.1%)	Adjusted	1.84 (0.71 to 4.73)	p = .20
Retention at 2 years post treatment	115/135 (85.2%)	118/141 (83.7%)	Unadjusted	1.12 (0.58 to 2.15)	p = .73
Retention at 2 years post treatment	115/135 (85.2%)	118/141 (83.7%)	Adjusted	1.13 (0.59 to 2.17)	p = .72

^aPrimary outcome.

Of those participants in each group who did complete a 1 year assessment, 150 (95.5%) of participants who received a thank you card fully completed all patient reported outcomes compared to 152 (92.1%) of participants who did not receive a card (adjusted odds ratio 1.84, 95% CI 0.71 to 4.73, p = .20).

For the seconday outcome of retention at subsequent time points (2 years), 115 (85.2%) of participants who received a thank you card completed the 2 years follow up compared to 118 (83.7%) of participants who did not receive a card (adjusted odds ratio 1.13, 95% CI 0.59 to 2.17, p = .72).

Secondary outcome – cost per Participant retained

The total cost of sending the thank you cards was £185.32 which equates to £0.79 per card. This was calculated using the parameters detailed in Table 3. As no statistically significant effect of the intervention was identified a cost per additional participant was not calculated.

Table 3.

Costs associated with the thank you card SWAT.

Task	Total cost	No cards involved	Cost per card
Printing cards^a	£94.82	350	£0.27
Preparing cards^b	£18.75	175	£0.11
Postage^c	£71.75	175	£0.41
Total cost	£185.32		£0.79

^aA total of 350 cards were ordered for the SWAT, sufficient to allow all DISC Trial participants to be randomised into the study.

^bPreparation and packaging was completed by a University of York Grade 3 member of staff with the salary midpoint of the band used for calculations. Preparation and packaging took approximately 90 min (1.45 h) to complete at a rate of approximately 30 s per card.

^cCards were sent to SWAT intervention participants via Royal Mail Mailmark franking at a cost of £0.41 per card.

Discussion

This SWAT found little evidence to suggest that sending a thank you card increases completion of the primary outcome visit, completeness of outcome data or completion of subsequent follow up visits.

Effect size estimates suggest that the control group (no thank you card) were more likely to complete the primary outcome than the intervention group (thank you card) (OR 0.95, 95% CI 0.48 to 1.90, p = .89). Effect estimates did however favour the intervention group (thank you card) for completeness of outcome data (OR 1.84, 95% CI 0.71 to 4.73, p = .20) and completion of subsequent visits (OR 1.13, 95% CI 0.59 to 2.17, p = .72). Given the bounds of the confidence intervals associated with these analyses, although trending in favour of the intervention, there is limited certainty around each of these estimates and hence the result and it’s applicability to other trials must be interpreted with caution at this stage.

The intervention was relatively low cost at £0.79 per card sent, however cost per additional participant retained was not calculated in the absence of any evidence of effect.

Given the low cost of the intervention, the limited certainty around the estimates, and as this is the first evaluation of a distinct thank you intervention, further replication of this SWAT is recommended, to enable subsequent meta-analysis and identification of a robust and definitive conclusion to the effectiveness of thank you cards as a RCT retention strategy.

It is not currently possible to assess the results of this SWAT in the context of existing evidence, due to the lack of other evidence as to the effectiveness of a distinct thank you intervention (separate to other trial information).⁷

Strengths and limitations

A limitation of this SWAT was the impact of COVID-19 which resulted in the SWAT being paused between 01.04.2020 and 30.09.2020. This resulted in 122 participants (62 intervention, 60 control) being excluded from the SWAT. While this limits the sample size somewhat, this is unlikely to have a substantial impact given that SWATs are routinely underpowered and require multiple replications and combination via meta-analysis to establish intervention effectiveness.

A further potential limitation to the SWAT was the host trial design whereby follow up timepoints were fixed to treatment delivery rather than randomisation. It was encouraging to note only one participant (intervention) was randomised after the SWAT timepoint however lags in receipt of information at the coordinating centre, meant some participants were not able to be randomised into the SWAT which reduced the sample size. Future studies using follow up fixed to treatment delivery should consider robust methods to ensure receipt of information necessary to ensure participants can be included in retention SWATs.

The timing of the intervention may have also impacted effectiveness as the intervention was delivered to allocated participants at the start of the 9^th month post treatment (i.e., 3 months prior to primary outcome timepoint). Future replications may therefore also wish to consider the optimum point at which the intervention is administered.

Finally, the participants included within the SWAT were representative of the overall trial population, given average age, proportion of male versus female participants and ethnicity proportions were not substantially different between the SWAT and host trial population. This likely limits the generalisability of the results to other patient populations. Further replications should therefore be considered in different populations, given this SWAT was applied to an predominantly older, male, Caucasian population. In addition, retention in the DISC trial was already high and evaluation in studies with lower retention rates would also be beneficial.

Implications for trial practice and SWAT research

Thank you cards as a retention method requires further evaluation to confirm or refute intervention effectiveness. As a result, this intervention should currently only be used when being evaluated in a SWAT.

Further replications should be completed in additional (i.e., younger, female, or ethnically diverse) populations or in trials where retention rates are currently lower than anticipated.

Conclusion

From this first evaluation of a distinct ‘thank you’ intervention for improving retention rates, it remains unclear if thank you cards increase the rate of primary outcome follow-up completion. It is however widely acknowledged that individual SWATs are unable to observe the modest effect sizes associated with most SWAT interventions.⁶ Further replications are therefore required, ideally in populations other than older, male, Caucasians, to distinguish conclusively whether distinct thanks within a RCT is an effective retention strategy.

Supplemental Material

sj-pdf-1-rmm-10.1177_26320843241229934 – Supplemental Material for Study within a trial 119: The effectiveness of a thank you card to improve trial follow up; a randomised study within a trial

Supplemental Material, sj-pdf-1-rmm-10.1177_26320843241229934 for Study within a trial 119: The effectiveness of a thank you card to improve trial follow up; a randomised study within a trial by Catherine Arundel, Charlie Welch, Puvanendran Tharmanathan, and Joseph Dias in Journal of Research Methods in Medicine & Health Sciences

Footnotes

Acknowledgments

We would like to acknowledge the input from Sally Baker, Emma-Louise Rose Brooks, Sue Heslop and Mo Wakefield at York Trials Unit, University of York for their help and support with administrative coordination of this study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This SWAT was funded as part of the DISC Trial which was funded the National Institute for Health Research Health Technology Assessment (project number 15/102/04). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Health Technology Assessment programme, NIHR, NHS or the Department of Health.

Ethical statement

Registration

Host Trial: ISRCTN18254597. Prospectively registered on 11^th April 2017. EudraCT 2016-004251-76. Prospectively registered on 21^st October 2016. SWAT: MRC Hub for Trials Methodology Research SWAT repository #119 Registered 13.04.2020 ().

Protocol availability

This SWAT was embedded in the DISC trial, the protocol for which has been published previously.¹¹

ORCID iD

Charlie Welch

Supplemental Material

Supplemental material for this article is available online.

References

Torgerson

. Designing randomised trials in health, education and the social sciences : an introduction. Basingstoke: Basingstoke: Palgrave Macmillan, 2008.

Bowling

. Research methods in health: investigating health and health services. United Kingdom: McGraw-Hill Education, 2014.

Hewitt

Kumaravel

Dumville

, et al. Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol 2010; 63: 1264–1270. DOI: 10.1016/j.jclinepi.2010.01.010.

Ioannidis

. Why most discovered true associations are inflated. Epidemiology 2008; 19: 640–648.

Moher

Glasziou

Chalmers

, et al.

Increasing value and reducing waste in biomedical research: who’s listening?

Lancet 2016; 387: 1573–1586.

Treweek

Bevan

Bower

, et al.

Trial forge guidance 1: what is a study within A trial (SWAT)?

Trials 2018; 19: 139, DOI: 10.1186/s13063-018-2535-5.

Gillies

Kearney

Keenan

, et al. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2021; 3: MR000032, DOI: 10.1002/14651858.MR000032.pub3.

Scott

Jain

Cogburn

. Behavior modification. Treasure Island (FL): StatPearls Publishing, 2022.

Kumar

Epley

. Undervaluing gratitude: expressers misunderstand the consequences of showing appreciation. Psychol Sci 2018; 29: 1423–1435. DOI: 10.1177/0956797618772506.

10.

questions. NUI Galway , 2021, https://priorityresearch.ie/priority-two-questions/ (accessed 13 02 2023).

11.

Dias

Arundel

Tharmanathan

, et al. Dupuytren’s interventions surgery versus collagenase (DISC) trial: study protocol for a pragmatic, two-arm parallel-group, non-inferiority randomised controlled trial. Trials 2021; 22: 671. DOI: 10.1186/s13063-021-05595-w.

12.

Dias

Bhowal

Wildin

, et al. Assessing the outcome of disorders of the hand. Bone and Joint Journal 2001; 83: 235–240.

13.

Adamson

Hewitt

Torgerson

. Producing better evidence on how to improve randomised controlled trials. BMJ 2015; 351: h4923.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB