Abstract
Background:
Improving outcome data collection rates is an essential part of managing clinical trials and ensures statistical power and generalisability of results are maintained. Studies within a trial (SWATs) provide a robust methodology to investigate the most efficient methods to maximise outcome follow-up.
Methods:
LoTS2Care, a feasibility cluster Randomised Controlled Trial, recruited 269 stroke survivors across 10 services, and incorporated a SWAT to evaluate the effect of questionnaire booklet format (one booklet or two) on follow-up rates for self-reported postal outcomes at 6 and 9 months post-recruitment. Available participants were individually randomised (1 : 1) by the Clinical Trials Research Unit and follow-up rates in the two groups were compared.
Results:
At 6 months post-recruitment, 254 participants were randomised: 126 to receive (125 posted) the single booklet; 128 to receive two booklets. By 9 months post-recruitment, 116 and 123 participants were still available in each group, respectively. For participants randomised to two booklets, return of at least one of the booklets was considered as ‘followed-up’. At 6 months, 114/125 (91.2%) participants sent the single booklet returned it, compared to 108/128 (84.4%) sent two (odds ratio (OR) 1.92, 95% confidence interval (CI) 0.88 to 4.19). By 9 months, 108/116 (93.1%) participants returned the single booklet, compared to 105/123 (85.4%) sent two (OR 2.31, 95% CI 0.97 to 5.55).
Conclusions:
The SWAT was an inexpensive, straightforward way to test how booklet format affected follow-up rates. Larger participant numbers would be required for conclusive results. These initial findings, however, suggest that including all outcome measures in a single booklet may maximise return rates, especially in trials with similar populations, such as those living with brain injury, cognitive or speech impairment or older people.
Keywords
Background
Whilst much focus is placed on participant recruitment in the management of clinical trials, maximising the collection of outcome data is essential in ensuring that statistical power and generalisability of results is maintained. 1 A Cochrane review of 38 clinical trials which implemented ‘randomised or quasi-randomised evaluations of strategies to increase retention’ identified questionnaire format as one of six broad types of strategies to improve retention. 2
LoTS2Care, a feasibility cluster Randomised Controlled Trial (cRCT), part of an National Institute for Health Research programme grant for applied research, recruited 269 stroke survivors across 10 stroke services, along with 85 carers, at approximately 6 months post-stroke.3,4 Participants were followed-up at 3 months, 6 months and 9 months post-registration date using postal questionnaires to collect self-reported outcomes.
Of the 269 registered participants, the mean (SD) in age (years) was 73 (11). The mean (SD) NIH Stroke Scale (NIHSS) score was 4.8 (5.12); although missing data for this score was high. The level of language ability post-stroke varied with 87 recorded as having normal ability, six participants as having dysphasia, 3 recorded as having dysarthria, 170 recorded as ‘not known’ and 3 as missing. Fuller breakdowns of trial population characteristics can be viewed in Tables 13 and 14 of Forster et al. 2021. 4
The booklets comprised several measures, including all primary and secondary self-report outcomes and the health economics questionnaire. To investigate how questionnaire format might impact questionnaire follow-up rates, the LoTS2Care team designed a ‘Study Within a Trial’ (SWAT). The SWAT investigated the use of two different booklet formats. The standard booklet (19 pages) combined all outcome measures and health economic questionnaires into a single but lengthy booklet. The other divided the measures into two shorter booklets (12 and 7 pages each) to be implemented at the follow up points at 6 and 9 months post-recruitment. There was an assumption that shorter booklets may be easier to complete in case the participant became fatigued. By splitting the booklets into two, it was thought that participants might be encouraged to take a break and the return to the second booklet. It was also suggested that participants experiencing questionnaire fatigue may be inclined to return at least one booklet and therefore maximising data collection of the key clinical outcomes in this instance. Thus, the objective of the SWAT was to answer the following question: Does the administration of two shorter questionnaire booklets increase postal follow-up rates compared to one longer questionnaire booklet in longer-term stroke survivors?
The SWAT would provide evidence to answer this key question in order to best inform the method for collection of patient reported outcome data in the LoTS2Care definitive trial.
The SWAT adheres to most of the ‘key features of a typical SWAT’, as highlighted in the Trial Forge Guidance, 5 with the exception of not having a formal protocol; the SWAT was briefly described in the main trial protocol. 3 It addresses several of the top 10 research questions raised in the PRioRiTy II study, such as how to minimise burden on staff and participants, and how to encourage participants to complete tasks by experimenting with mode of delivery of the questionnaire, and assessing how this affects the data returned. 6
Methods
Host trial: LoTS2Care feasibility trial design
LoTS2Care evaluated the ‘New Start’ intervention (with usual care) (NS) versus Usual Care (UC) alone, delivered at 6 months post-stroke and designed to improve quality of life by addressing unmet needs and enhancing participation in stroke survivors. Ten stroke services were allocated to either NS or UC at a 1:1 ratio by minimisation with a random element, using size of stroke service (>300 patients per annum; ≤300 patients per annum) and whether or not the stroke service currently had a 6-month post stroke review service in place, as binary minimisation factors. LoTS2Care was a feasibility study and therefore a formal power calculation was not required as effectiveness was not being assessed.
Participants were recruited by post and registered onto the trial at approximately 6 months post-stroke. Available carers also were recruited by post. Postal recruitment methods may have primed participants to receive and complete questionnaires by post, explaining the high overall follow-up rates achieved in the trial (80.3% at 9 months). Participants within the LoTS2Care trial completed a 28-page baseline booklet at registration. Baseline data was either completed in person or by post and if requested, with researcher assistance. All follow up data (at 3, 6 and 9 months) was collected via self-reported postal questionnaires. At 3 months all participants were sent a 16-page booklet.
Study within a trial design
At 6 months post participant registration, a two arm SWAT was undertaken. Immediately prior to initiation of the 6 month follow-up point, participants were individually randomised (1:1) to receive either the single standard 19-page booklet, or two separate booklets, of 12 and 7 pages. The allocated booklet format was posted to the stroke survivor at 6 and 9 month follow-up points. Participants were allocated via stratified randomisation with fixed blocks of size 4, using the CTRU Gen-24 randomisation system. The two stratification variables were host trial centre allocation (control or intervention) and mode of administration of baseline assessments (postal, face to face or unknown).
Where applicable, the carer was posted the format allocated to the stroke survivor.
The primary objective of the SWAT was to determine whether two shorter questionnaire booklets increased postal follow-up rates compared to one longer questionnaire booklet in longer-term stroke survivors. Booklets that were returned blank were considered as ‘not followed-up’, although partially completed booklets were considered as ‘followed-up’. Participants randomised to two booklets were considered followed-up if they returned at least one non-blank booklet. Item completion rates were also assessed as part of measurement of follow-up rates. All participants who were individually randomised to the SWAT and available (i.e. not died, withdrawn, or lost to follow up) at the respective mailing time points were included in the analysis populations. Logistic regression was used for the analysis and restricted to between-group estimates of follow-up presented as odds ratios with 95% confidence intervals.
As LoTS2Care was a feasibility trial, no formal sample size calculation was performed, nor was one performed for the SWAT, which is in line with SWAT methodology.5,7
The LoTS2Care feasibility trial protocol was previously published and includes a brief description of the SWAT design. 3
Results
The LoTS2Care trial recruited a total of 269 participants. Of these, 254 participants were available (i.e. not died, withdrawn, or lost to follow up) at 6 months post-recruitment for randomisation into the SWAT. 126 participants were randomised to receive a single booklet and 128 participants were randomised to receive two shorter booklets; one of the participants randomised to receive a single booklet withdrew shortly after SWAT randomisation, hence 253 participants in total were sent booklets. By 9 months post-recruitment, 239 participants were available and included in the SWAT analysis population. Figure 1 demonstrates the flow of patients through the SWAT, including the follow-up rates and total figures. For summaries of this study within a trial, only booklets returned via post are considered. Outcomes completed via phone are not used in summarising completion rates but are included in the main study results Flow of participants through study within a trial.
Baseline characteristics shown by SWAT group.
aOther categories of ethnicity were recorded as Black (2(1.6%) one booklet/0 two booklets (0.0%), Asian (1(0.8%) one booklet/2 (1.6%) two booklets), mixed (1 (0.8%) one booklet/0 (0.0%) two booklets) other (0 (0.0%) one booklet/2 (1.6%) two booklets).
Overall, across all participants and over all time-points, follow-up rates were high. At 6 months, participants were more likely to return follow-up questionnaires when administered together as one single booklet than as two shorter booklets, with 114 out of 125 (91.2%) participants sent the single booklet returning it, compared to 108 out of 128 (84.4%) sent two booklets (OR 1.92, 95% confidence interval (CI) 0.88 to 4.19). Similar trends were observed at 9 months, with 108 (93.1%) of the 116 participants in the one booklet group returning their booklets, compared with 105 out of 123 (85.4%) in the two booklet group returning both booklets (two (1.6%) returned a single booklet containing the outcome measures only), corresponding to an odds ratio (OR) of completing one single booklet compared to completing two booklets of 2.31 (95% confidence interval 0.97 to 5.55).
Stroke survivor follow-up rates by SWAT allocation.
For summaries of this study within a trial, only booklets returned via post are considered. Outcomes completed via phone are not used in summarising completion rates but are included in the main study results.
Carer follow-up rates by SWAT allocation.
For summaries of this study within a trial, only booklets returned via post are considered. Outcomes completed via phone are not used in summarising completion rates but are included in the main study results.
Overall, these results suggest outcome follow-up rates are maximised when sending all required questionnaires within a single, longer booklet, rather than splitting across two, shorter booklets.
Discussion
The results of the SWAT show increased odds of stroke survivors returning questionnaires when sent one booklet, as opposed to two booklets. Numbers within the SWAT were small and therefore these results did not reach statistical significance. This finding is contrary to that reported in a 2009 Cochrane review looking at methods to increase response rates in postal and online questionnaires, 8 which found that shorter questionnaires had increased odds of being returned. However, this comparison does not take into account the fact that we had two shorter booklets, which together contained the same measures (and therefore length) as the single booklet.
Carers were also more likely to return the longer, single booklet, although the result should be interpreted with caution due to low number of available carers assessed.
It is also of note that the questionnaire return rates for the study were already high, leaving limited room for improvement.
The study involved a small sample, limiting generalisability of the results; a much larger sample would be required for conclusive findings. Moreover, the study could have benefitted from recording participant-reported preference for booklet format. Nonetheless, the format of the booklets was developed and tested in a target population and the results inform assessment in further stroke trials in order to maximise follow-up rates.
Most SWATs are implemented in full-scale trials and often attempt to answer broader questions in order to improve the evidence base for trial conduct. Only one SWAT on the SWAT-SWAR repository is listed currently as a feasibility trial SWAT.
For example, one current trend emphasises use of standard SWATs in a larger number of host trials to increase generalisability of SWAT results and maximise the impact of SWAT research. Although these steps are essential in informing best and most effective practice in running large-scale trials, it is also fruitful to explore implementation of SWATs in smaller, more specific trial contexts. SWATs can be used as a low-cost way to test trial processes and conduct for a very specific population, disease area, or type of trial. Feasibility studies (provided they are of reasonable size) make natural host trials for SWATS. Small-scale SWATs implemented in feasibility trials can be an efficient way to test aspects of trial conduct in order to optimise methods for the large-scale trial. We would therefore urge trialists to explore ways to test key trial processes at feasibility stage in order to maximise trial efficiency and ensure success in definitive trial processes.
Conclusions
The SWAT was a simple and straightforward way to test how booklet length and number of booklets sent at one time affected follow-up rates in an older, stroke survivor population. Whilst the findings are not conclusive, they do suggest that in this particular setting for this population, including all self-reported questionnaires in one booklet was most likely to maximise follow-up rates. These initial findings may also be relevant in trials with similar populations, such as those living with brain injury, cognitive or speech impairment or older people.
Study within a trials embedded in feasibility trials have the ability to inform the best methods for trial conduct in a specific trial target population and can therefore offer a unique opportunity to maximise definitive trial efficiency.
Footnotes
Acknowledgements
We would like to thank all clinical teams participating in the LoTS2care trial and the participants. We are pleased to acknowledge the LoTS2Care Programme Management Group for their contributions to and support of the work. We would also like to acknowledge the wider teams based at the Academic Unit for Ageing and Stroke Research and Leeds Institute for Clinical Trials Research, Clinical Trials Research Unit for their contributions to the LoTS2Care study, including Richard Brindle, Lorna Barnard and Marie Chenery.
Author contributions
Lauren Moreau was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Ivana Holloway was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Seline Ozer was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Anne Forster, was Principal Investigator for the LoTS2care feasibility cluster Randomised Controlled Trial (cRCT), was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Claire Hulme was involved in the design of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Suzanne Hartley was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Amanda Farrin was involved in the design and implementation of the LoTS2Care SWAT study and contributed to drafts of the manuscript, critically reviewed and revised different versions of the manuscript and approved the final version.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Grant Reference Number RP-PG-0611-20010). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Abbreviations
Confidence Intervals (CI), Odds Ratio (OR), Study within a trial (SWAT).
REC Approval
Ethical approval has been obtained through the Yorkshire & The Humber - Leeds East Research Ethics Committee (REC) (Ref: 16/YH/0068) for the LoTS2Care feasibility trial. This SWAT was submitted as a substantial amendment to the REC dated 09 November 2016 and approved by the REC on 13/12/2016.
Availability of data and materials
Any data requests should be sent to the corresponding author and would be subject to review by a subgroup of the trial team, which will include the data guarantor, Professor Farrin. All data-sharing activities would require a data-sharing agreement.
Patient data statement
This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives
