Improving the Development,Monitoring and Reporting of Stroke Rehabilitation Research: Consensus-Based Core Recommendations from the Stroke Recovery and Rehabilitation Roundtable *

Abstract

Recent reviews have demonstrated that the quality of stroke rehabilitation research has continued to improve over the last four decades but despite this progress, there are still many barriers in moving the field forward. Rigorous development, monitoring and complete reporting of interventions in stroke trials are essential in providing rehabilitation evidence that is robust, meaningful and implementable. An international partnership of stroke rehabilitation experts committed to develop consensus-based core recommendations with a remit of addressing the issues identified as limiting stroke rehabilitation research in the areas of developing, monitoring and reporting stroke rehabilitation interventions. Work exploring each of the three areas took place via multiple teleconferences and a two-day meeting in Philadelphia in May 2016. A total of 15 recommendations were made. To validate the need for the recommendations, the group reviewed all stroke rehabilitation trials published in 2015 (n=182 papers). Our review highlighted that the majority of publications did not clearly describe how interventions were developed or monitored during the trial. In particular, under-reporting of the theoretical rationale for the intervention and the components of the intervention call into question many interventions that have been evaluated for efficacy. More trials were found to have addressed the reporting of interventions recommendations than those related to development or monitoring. Nonetheless, the majority of reporting recommendations were still not adequately described. To progress the field of stroke rehabilitation research and to ensure stroke patients receive optimal evidence-based clinical care, we urge the research community to endorse and adopt our recommendations.

Introduction

Recovery and a return to a full and meaningful life following stroke are the main goal for stroke survivors, their families and the health professionals who strive to provide the best care possible. Stroke rehabilitation research has informed our understanding of how we enhance recovery following stroke and consequently has helped shape our clinical practice. However, rigorous evaluation methodologies and complete reporting of such are essential if we are to provide evidence that is robust, meaningful and implementable. Recent reviews have demonstrated that the quality of stroke rehabilitation research has continued to improve over the last four decades ¹ but despite this progress, there are still many barriers in moving the field forward.

In 2015, an international partnership of stroke recovery and rehabilitation experts committed to addressing four key areas that require attention from the stroke rehabilitation research community. ² These were: the first translational gap; recovery biomarkers; intervention development, monitoring and reporting; measurement in clinical trials. This paper addresses the theme of developing, monitoring and reporting interventions. It highlights issues identified as requiring improvement by an international working group and the recommendations made following the first International Stroke Rehabilitation and Recovery Roundtable (SRRR) meeting which took place in Philadelphia in May 2016.

Development of Interventions

Stroke rehabilitation interventions are complex and commonly multi-faceted, containing many inter-related components with the ultimate aim of addressing specific post-stroke problems.

In order to have confidence in any research findings, it is essential that we fully understand the theoretical and/or the biological rationale that underpins each intervention. The rationale for an intervention is required if we are to clearly understand how it will effect change in the population studied. In order to achieve an understanding of the likely process of change, researchers should have previously drawn on biological, physical and/or psychological rationales, existing evidence from the literature, and from other related theories.

Sequential intervention development, which is not necessarily linear, is also important to ascertain the optimal patient group to target, the essential core and flexible elements of the intervention, how (by whom) it should be delivered, and the optimum dose and frequency, which all ultimately inform fidelity assessment.

Several intervention development frameworks exist to aid researchers and clinicians achieve this. One such example would be the Behaviour Change Approach. ³ The most commonly referenced framework used in the global stroke community is the MRC Framework for the development and evaluation of complex interventions, originally published in 2000, ⁴ updated in 2006⁵ and is currently undergoing a further update.

Redfern et al. ⁶ used the MRC framework ⁴ to conduct a systematic review of how complex interventions in stroke care had been developed. In this review, 67 studies were identified of which 39 were randomized controlled trials. This work concluded that “theoretical grounding to support intervention choice was reported in 40 studies but only 14 were theoretically well developed.” The authors went on to conclude that few complex interventions in stroke care had been sufficiently developed or evaluated using appropriate methodology and that this may explain the failure of many studies to demonstrate efficacy.

During the Stroke Recovery and Rehabilitation Roundtable discussions, the consensus of the international group was that the findings of Redfern and colleagues were still largely applicable. The theoretical and/or biological rationale and sequential development of many stroke rehabilitation interventions does not follow a recommended framework for development or if a framework is followed then it is not reported in the main study publications.

Intervention Monitoring

Another problem highlighted by the SRRR working group was the lack of intervention fidelity and reporting, i.e. whether interventions were delivered as intended, whether their use in the context of a particular study or setting was modified to suit the local context (setting or therapist) or individually tailored to the person being treated, or of the methods used to maintain intervention fidelity during the course of a trial.

As the pre-development work for larger trials often involves multiple, smaller, single centre studies, sometimes in different countries, it becomes increasingly important to understand the extent to which an intervention delivered in one setting differs from the same intervention delivered elsewhere and how it might have been modified to suit the local context or person. Similarly, the extent to which the training or skills of the therapist differed between settings and therefore how it might have influenced the outcome also needs to be known. The template for intervention description and replication (TIDieR) checklist (items 5 and 11) ⁷ prompts researchers to describe these elements where possible. However, if researchers fail to recognize potential factors of influence at the design stage, they may be not collected at all, poorly reported or absent from reports.

Implementation frameworks and intervention fidelity models prompt researchers to consider these factors more broadly.^8–14 The consolidated framework for implementation research (CFIR) ¹⁰ adopts the metaphor of an arena to highlight factors in different contextual areas that can affect fidelity. Examples of factors in the “outer” setting include resources, external policies and incentives, and those in the “inner” setting include culture, networks and communication and at an individual level, patients and therapists knowledge and beliefs and personal attributes. The CFIR also recognizes the importance of “process” factors including the extent to which a particular site is engaged and willing to execute delivery of the intervention, the competency and skills of people involved in its delivery and the characteristics of the interventions itself, such as the strength and quality of evidence to support it, its cost and complexity.

Adopting an implementation fidelity model or framework at the design stage may prompt stroke rehabilitation researchers to consider which aspects of fidelity are most important, which to monitor, and how to do this during the trial. Moreover, they may inform the theory by which the intervention works and help to identify mechanisms by which it will have its desired effect. ¹⁵ As Gitlin ¹⁵ suggests, the question, “why would this intervention work?”, is frequently overlooked. Many feasibility trials now include mixed methods or process evaluations ¹⁶ to better understand not only “what” to monitor and “how,” but to identify mechanisms of success, and factors likely to affect the outcome in a definitive trial.

Reporting of Interventions

Incomplete intervention reporting is problematic for all evaluations of interventions, but particularly so for non-drug interventions, such as stroke rehabilitation. ¹⁷ These complex interventions have long been described (inadequately) as “black boxes”. ¹⁸ The consequences of this are numerous, with users of the research having to guess the components of the intervention. As a consequence, clinicians are unable to reliably replicate effective interventions and patients may miss out on receiving interventions in a manner that is faithful to what has been evaluated in trials. Researchers who wish to replicate evaluation of the intervention may be unable to do so. Also, interpretation of the effectiveness of interventions in syntheses such as systematic reviews is hampered, and the useability of systematic reviews and clinical guidelines is compromised when original trials lack crucial details. In an analysis of a random sample of 60 systematic reviews of nonpharmacological stroke interventions, most reviews were missing intervention information for the majority of items. ¹⁹ Incomplete intervention descriptions hinder research useability, replicability, and development of standardized rehabilitation interventions, which has been identified as priority for the world stroke agenda. ²⁰

Method

SRRR Working Group Activities

Identification of the intervention development, monitoring and reporting working group membership was conducted using a snowball technique by the SRRR steering committee with the aim of ensuring a global perspective. Efforts were made to incorporate stroke research leaders from Europe, North America, and Asia who had an interest and expertise in this area. Prior to the Roundtable meeting in Philadelphia, multiple teleconferences took place where the scope of the work was discussed and agreed, reading materials were shared and pre-workshop activities were delegated to working group members. Two co-chairs (Walker and Hoffmann) of the group each led a strand of work and facilitated teleconferences and email discussions in preparation for a productive meeting in Philadelphia.

In Philadelphia, the group dedicated time to the discussion of each of the three areas: intervention development, monitoring, and reporting. Relevant literature and evidence were further shared and issues that were identified as limiting the progress of rehabilitation research were presented and discussed with the wider group membership. Discussions were recorded by a scribe and audio-taped. Two primary outputs were achieved: provisional group recommendations in each of the three areas; agreement on the validation work required to examine the extent to which contemporary stroke rehabilitation trials were aligned with the recommendations.

During the discussions, recommendations addressing issues identified as limiting stroke rehabilitation research were proposed.

Exploration of Recommendations in RecentlyPublished Papers

The SRRR working group acknowledged that many of the recommendations made had been documented elsewhere^4,5,7 but despite this, the group felt that the stroke rehabilitation research community would benefit from a stroke-specific output which collated previous and new recommendations made by the group. To validate the need for our recommendations, it was agreed that all stroke rehabilitation randomized controlled trials conducted over a recent 12- months period would be reviewed to ascertain if our recommendations had already been addressed by the stroke community. Our hypothesis was that a significant proportion of papers would not have addressed our recommendations.

Identification of the Sample

We searched MEDLINE and CINAHL for studies published in 2015 using the terms “cerebrovascular accident” OR “cerebrovascular disease” OR “stroke.” The search results were then paired using AND with keywords pertaining to any adult rehabilitation treatment. While it was expected that the term “rehabilitation” would find the majority of treatments, other keywords were generated and reviewed by X experts in stroke rehabilitation to be as inclusive as possible. To select only randomised controlled trials (RCTs), we also paired using AND random* OR control*.

The intervention terms were as follows:

Activit* of Daily Living OR Acupuncture OR ADL OR Aerobic OR Antidepress* OR Ataxi* OR Aphasi* OR Balanc* OR Botox OR Botulin* OR Caregiver OR Cerebrolysin OR Chronic Disease Management OR Cogniti* OR Community Integration OR Conditioning OR Continence OR Contrain* OR Direct Current OR Dopamine OR Electric* Stimulation OR Exercis* OR FES OR Hand OR Homecare OR Incontinence OR Language OR Leisure OR Mental Practice OR Mirror OR Mobility OR Music OR Occupational Therapy OR Orthoses OR Percept* OR Physical Therapy OR Physio Therapy OR Postural Control OR Recreation OR Rehab OR Rehabilitation OR Respirat* OR Robot* OR Selective Serotonin Reuptake Inhibitor* OR Selegiline OR Self-Management OR Sensory OR Shoulder OR Speech OR SSRI OR Supported Discharge OR Tele Rehab* OR Telehealth OR Telerehab* OR TMS OR Training OR Transcranial Magnetic Stimulation OR Transfer* OR Upper limb OR Vibrat* OR Virtual Reality OR Visu* OR Vocation* OR VR OR Wheelchair

English language and human studies were used as restrictions. Abstracts and conference posters were excluded. Titles, abstracts, and full-texts were screened by two independent reviewers. The original search resulted in 1437 articles from PubMed and 387 articles from CINAHL. After removing duplicates and those that did not meet the eligibility (i.e. RCT, rehabilitation intervention, published in 2015, adult stroke population), there were 182 studies remaining.

Procedure

Each member of the SRRR group was assigned an equal number of articles to assess the content of the paper against the recommendations in Box 1. A structured rating template was developed and for each article, each recommendation was assessed as: fully described; partially described; not described; or not applicable, with space for any additional comments.

Box 1.

Recommendations.

Intervention development • Follow a structured framework for intervention development. This enhances accountability/acts as aide memoire. The particular framework to be used is dependent on the purpose and specific target/population/setting of the research • Co-design research where appropriate with key stakeholders (e.g. health care providers) and incorporate strong patient involvement • Underpin development with theoretical rationale (e.g. the biological basis for the intervention; behaviour change theory) • Describe routine practice in sufficient detail to facilitate in-depth understanding of “usual care” (see reporting recommendations below). Methods to capture this may include: videos, non-participant observation, surveys and ideally at more than 1 center • Provide a clear rationale for each component of the intervention. This should include details on: why the specific dose has been chosen, how often it should be administered, for how long, who delivers it, training required of person delivering it, participant selection (e.g. type of stroke, severity etc.), time-frame post-stroke, stage of recovery post-stroke • Provide a clear description of core intervention components that must be delivered (see reporting recommendations) • Describe the potentially variable contexts in which the intervention can be delivered • Consider and address issues which might affect the implementation of the intervention into clinical practice • Use iterative modeling prior to definitive evaluation (e.g. dose escalation studies). Intervention monitoring • Build an assessment of fidelity into the trial protocol (described in Item 11 of TIDieR) • Describe method for measuring fidelity that is trial and intervention-specific. No fidelity framework ideal for all situations. • Routinely report training of health professionals in delivering the intervention, as well as whether (and if so, how) their level of competence was assessed (described in Item 5 of TIDieR). Intervention Reporting • Follow the appropriate reporting guideline for study design type (e.g. SPIRIT for trial protocols; CONSORT for randomized trials) and the appropriate extension (e.g. extension for Cluster trials).   - Refer to EQUATOR Network site (www.equator-network.org) for reporting guidelines • Provide the trial registration number in the article (preferably in the abstract) to facilitate linkage of related articles • Use TIDieR (an extension to SPIRIT and CONSORT) as a guide to ensure complete reporting of interventions:  - Item 1: Brief name or phrase that describes the intervention  - Item 2: Describes any rationale, theory, or goal of the elements essential to the intervention  - Item 3: Describe physical or informational materials used in the intervention (provided to participants and/or used in training of providers) and where and how these can be accessed (and any requirements on their access and use)  - Item 4: Describe each of the procedures, activities, and/or processes used in the intervention (including if it needs to be delivered in sequence/ step wise fashion/ staged)  - Item 5: For each category of intervention provider (e.g. OT, psychologist, nurse), describe their expertise, background, any specific training given (& if employed by trial funding or health service)  - Item 6: Describe the modes of delivery (e.g. face-to-face or remote contact such as telephone phone, internet, etc.) and whether provided individually or in a group  - Item 7: Describe the type/s of location/s where the intervention occurred, including any necessary infrastructure or relevant features  - Item 8: When and how much of the intervention: describe the number of times the intervention was delivered and over what period of time, including the number of sessions, their schedule, and their duration, intensity or dose  - Item 9: Tailoring: If the intervention was planned to be personalized, titrated or adapted, describe what, why, when, and how  - Item 10: Modifications – If the intervention was modified during the course of the study, describe the changes (what, why, when, and how)  - Item 11: Fidelity plans: If intervention adherence or fidelity was assessed, describe how and by whom, and if any strategies were used to maintain or improve fidelity, describe them  - Item 12: Actual fidelity: If intervention adherence or fidelity was assessed, describe the extent to which the intervention was delivered as planned  - Clearly describe (using TIDieR items 3–9) “usual care” or the comparator in the trial to enable interpretation of trial results.

Analysis

The completed assessment of all 182 papers was entered into three Excel tables (intervention development, monitoring, and reporting) and descriptive statistics applied. Data were summarized in bar charts.

Results

Figure 1 shows the proportion of trials which addressed the recommendations for intervention development and monitoring. As hypothesized, the majority of the recommendations were not described in most of the publications. In terms of intervention development, only 8% of studies fully or partially reported co-design of the research with stakeholders. Consideration of implementation issues during intervention development was addressed in only 12% of trials. The majority of trials described when to give the intervention and the dose required to be administered, yet the iterative modeling required to optimize the intervention was scarcely reported with only 13% partially or fully reporting this. This brings into question if the phases required to fully understand the optimal dosage for stroke survivors was addressed before the onset of the trial. Over 80% of trials reported, the theoretical rationale underpinning the trial intervention and a clear description of the core intervention components, but these aspects were rated as fully described in only 41% and 42% of trials, respectively.

Figure 1.

Intervention development and monitoring.

Intervention monitoring was particularly poorly described across all three recommendations in this category. Thus, for the majority of trials, it is unclear how interventions were monitored while the trials were in progress.

Overall, a higher proportion of trials addressed the reporting recommendations than the intervention development and monitoring recommendations, as shown in Figure 2. However, most papers addressed only one aspect of fidelity, such as adherence to a defined element of the protocol, for example acceptability or a measure of dose, intensity or duration. Few considered factors such as program drift, program differentiation or competence. Many different factors can potentially influence how an intervention is delivered and its outcome success.

Figure 2.

Reporting.

Discussion

The SRRR working group were cognizant of the limitations of the work undertaken which involved significant time constraints and resources and a limited pool of rehabilitation experts. Nonetheless, it was felt that the consensus from an international group of stroke rehabilitation research experts was an important first step in trying to address the perceived shortfall in methodological and reporting weaknesses in rehabilitation research.

Our review of recently published stroke rehabilitation RCTs has highlighted that the majority of publications did not clearly describe how interventions were developed or monitored during the trial. In particular, under-reporting of the theoretical rationale for the intervention and the components of the intervention call into question many interventions that have been evaluated for efficacy.

With regard to intervention reporting, more trials were found to have addressed these recommendations than those related to intervention development and monitoring. This may be due in part to the publication of the TIDieR guideline ⁷ and increased awareness and emphasis on intervention reporting. Notwithstanding this, the majority of reporting recommendations were still not fully described in the vast majority of the published studies. Of particular concern is the lack of detail provided around intervention fidelity and usual care. Without knowing exactly what intervention was provided, and exactly what it was compared to (these factors were fully described in less than 10% of studies), the validity of results about the effectiveness of the interventions is clearly questionable. This should be of concern to the professional integrity of researchers working in the area of stroke rehabilitation.

Funders of research generally provide limited support for developmental work, instead moving directly to feasibility phase II trials, with minimal reassurance that the essential intervention development phases have been completed, and that a rigorous intervention is ready to be evaluated for effectiveness. One potential solution to address this shortfall is by having MPhil and PhD studies focus more on the biological, and or theoretical basis for interventions followed by rigorous modeling to understand the optimal dosage and subgroups of patients who would benefit most.

We acknowledge that the lack of detailed reporting regarding intervention development, monitoring, and reporting may not always mean that certain steps were not undertaken. Reporting omissions could instead be due to prioritization regarding what to report within a manuscript with a finite word limit. However, as many journals accept additional detail as on-line supplements, our opinion is that this is no longer acceptable. It may also be that this information may have been described elsewhere. We did not search for secondary/related articles. Complete reporting is often low on the priority list for authors, reviewers, and editors, and many do not know what constitutes complete reporting. Moher and Altman ²¹ have proposed the development of core competencies and additional training for authors, peer reviewers, and editors with the aim of raising the reliability of published outputs in the medical literature.

There appears to be little academic reward in detailing intervention fidelity, as considerably more publication space is dedicated to details regarding clinical outcomes following evaluation of an intervention. We know from our search that very few papers are published regarding intervention fidelity. However, we are not in a position to know whether papers dedicated to intervention development or trial fidelity are being written, submitted and not accepted for publication, or whether these aspects of clinical trials are either not being investigated or not written up. Questions remain about: Is intervention development being conducted systematically yet not reported, or are interventions that do not have sound biological or theoretical rationale for efficacy being evaluated? Also, we cannot be clear if fidelity is being monitored closely and not being reported, or is it sometimes overlooked?

Fidelity is a crucial part of learning about efficacy of an intervention—and for positive trials, lessons learned through monitoring fidelity can often guide how an intervention can be applied in clinical practice, or can raise questions about mechanisms of action in a positive trial where lack of fidelity has been demonstrated. Accurate monitoring of intervention fidelity can require intensive resourcing, yet there seems to be little academic reward (in terms of high impact publications) in documenting this process. An exception to this rule are grants funded by the National Institute for Health Research (NIHR) in the UK, where researchers are required to create a 50,000 word monograph on completion of the study, which includes documenting a detailed description of intervention fidelity. Given the importance and complexity of ensuring intervention fidelity, it would be to the benefit of the academic community to create incentives to report fidelity, such as journals including papers dedicated to reporting intervention fidelity, or at a minimum, increasing the word limit or including in the on-line supplement and insisting intervention fidelity is reported for every clinical trial.

In conclusion, we call to action the stroke rehabilitation research community to endorse our recommendations and to ensure that we develop, monitor and report the best evidence thereby maximizing the chance of best care for our stroke patients. They deserve no less.