Abstract
Background
The generalizability of treatments examined in rehabilitation randomized controls trials (RCTs) partly depend on the similarity between trial subjects and a stroke rehabilitation inpatient population. The aim of this study was to determine the proportion of stroke rehabilitation inpatients that would have been eligible or ineligible to participate in published stroke RCTs.
Methods
This was a secondary analysis of chart review data collected as part of an independent quality improvement initiative. Data pertaining to the characteristics of stroke rehabilitation inpatients (e.g. age, cognitive impairment, previous stroke, comorbidities) were extracted from the medical charts of patients consecutively admitted to an inpatient stroke rehabilitation unit at a large urban rehabilitation hospital in Canada. Using the exclusion criteria categories of stroke RCTs identified from a systematic scoping review of 428 RCTs, we identified how many stroke rehabilitation inpatients would have been eligible or ineligible to participate in stroke RCTs based on their age, cognitive impairment, previous stroke and presence of comorbidities.
Results
In total, 110 stroke rehabilitation inpatients were included. Twenty-four percent of patients were 80 years of age or older, 84.5% had queries or concerns regarding patient cognitive abilities, 28.0% had a previous stroke, and 31.8% had a severe stroke. Stroke rehabilitation inpatients had six comorbidities on average. Based on these factors, most stroke rehabilitation inpatients could have been excluded from stroke RCTs, with cognitive impairment the most common RCT exclusion criteria.
Conclusions
Changes to the design of RCTs would support the development of clinical practice guidelines that reflect stroke rehabilitation inpatient characteristics, enhancing equity, diversity, and inclusion within samples and the generalizability of results.
Background
Health systems are implementing clinical practices based on the best available evidence with the aim of improving the quality of care for patients and in the context of stroke services specifically, health system leaders are transforming services to align with stroke best practice recommendations. 1 This widespread adoption of evidence-informed health care has happened against the backdrop of increasing numbers of people living with multiple concurrent chronic conditions 2 which have substantive implications for health care design and delivery.3,4 People who have experienced a stroke commonly have comorbidities; in fact, less than 6% of patients with stroke experience a stroke in isolation (i.e., with no other chronic comorbidities).2,5
Providing care for people with multiple conditions is challenging 6 ; despite the development of multimorbidity focused guidelines it is well recognized that the majority of clinical guidelines are still organized around single conditions7–10 which can be impractical or even harmful.10,11 In previously published work, stroke rehabilitation clinicians have questioned the applicability of stroke rehabilitation clinical guidelines to their ‘real-world patients,’ as the high quality evidence that underpins these guidelines often exclude patients on the basis of age, history of previous stroke, comorbidities and cognitive impairment.2,12
Although randomized clinical trials (RCTs) are considered the highest level of evidence, there has been widespread recognition that inclusion criteria may be overly restrictive, 13 which can limit the external validity of the results (i.e. the ability to extend “the results from a sample to the population from which the sample was drawn)”.13,14 A 2017 systematic scoping review by Nelson & colleagues determined that of 428 inpatient stroke rehabilitation RCTs they reviewed, 83% excluded patients with comorbidities, 54% excluded patients with cognitive impairment, 36% excluded patients with prior stroke and 5% excluded patients based on age. 15 However, the clinical question regarding the generalizability of the stroke RCT evidence to the ‘real world’ patients in a stroke rehabilitation program remains unaddressed within existing research.
Methods
Aim
The purpose of the current study was to ascertain what proportion of patients admitted to an inpatient rehabilitation program may have been ineligible to participate in the previously synthesized stroke rehabilitation RCTs based on common exclusion criteria.13,15,16 This would allow us to address the question posed by clinicians, ‘if age, previous stroke, comorbidities, and cognition are common exclusion criteria, how many of our actual patients may have been eligible to participate in the stroke rehabilitation trials’.
Design
A secondary analysis of retrospective chart review data was undertaken.
Setting of the study
The study sample was drawn from an inpatient stroke rehabilitation unit within a large rehabilitation hospital located in an urban Canadian city. The stroke unit has 28 high intensity neuro-rehabilitation beds, which provide specialized rehabilitation for approximately 120 patients with moderate to severe strokes per year. The rehabilitation team is comprised on an interdisciplinary team which includes physicians, nurses, rehabilitation therapists, social workers, dieticians. In high-intensity rehabilitation, patients receive 180 minutes of individualized therapy (occupational/physical/speech) therapy per day. The average length of stay is roughly 28.5 days, which is in accordance with provincial length of stay targets. The organization committed to the delivery of evidence informed care and has regularly achieved stroke distinction status 17 – an external recognition of the fidelity of clinical practices to best practice recommendations. 18 Within this Canadian context, the proportions of patients with severe, moderate, or mild stroke are prescribed by the provincial stroke system to ensure patient severity distribution across the broader system.
Data collection
A secondary analysis of a previously completed retrospective chart review dataset was undertaken. As part of an independently conducted quality improvement initiative, an inpatient stroke rehabilitation unit within a large urban Canadian hospital collected the following data from a chart review of all patients (i.e. ‘ stroke rehabilitation inpatients’) admitted to and subsequently discharged from the stroke rehabilitation unit between June 1, 2012, and June 30, 2013: demographic information, type of stroke, admission date, discharge date, Stroke Severity Index score [based on their Rehabilitation Patient Group (RPG)], length of inpatient rehabilitation stay, admission Functional Independence Measure 19 score, discharge Functional Independence Measure score, and Functional Independence Measure change (Discharge Functional Independence Measure minus Admission Functional Independence Measure). All comorbidities identified in the patient’s admission notes, history, or discharge summary were included, as well as selected complications that were a focus of their project – pneumonia, venous thromboembolism, gastrointestinal bleeding, hemorrhagic conversion, pressure injuries or urinary tract infections. This original quality improvement project did not assess the patients’ eligibility to participate in stroke rehabilitation trials; that was outside the scope of the project, which created the opportunity for the presented work.
The data were collected and collated as part of a program quality improvement initiative that did not seek research ethics board review. As this study is a comparative project of the previously collected and de-identified data with the results of a published scoping review, research ethics board review for this study was not obtained.
Data analysis
The analysis for this comparative study was conducted by members of our research team, who were not involved in the quality improvement project that completed the primary data collection. Duplicate entries and records with insufficient information were removed. Data were analyzed using SPSS (version 23) and Microsoft Excel using descriptive statistics. To ascertain the degree to which the included cohort of patients was representative of the larger stroke population, we identified relevant characteristics of patients admitted to the inpatient stroke rehabilitation unit compared to the broader stroke population. The most common comorbidities were calculated by counts of patient presentations. Data for each patient were then compared to the RCT exclusion criteria categories as previously identified in the Nelson et al. 15 scoping review: age, cognitive impairment, previous stroke, and the presence of comorbidities (determined using the list of comorbidities outlined in the Charlson Comorbidity Index20,21 or the presence of conditions other than stroke).
Results
stroke rehabilitation patient characteristics.
Comorbid conditions of the stroke rehabilitation patients.
Discussion
The purpose of the current study was to examine the generalizability of stroke RCTs to stroke rehabilitation inpatients on a clinical unit by identifying comorbidities and other clinical characteristics and then determining what proportion of these ‘real-world’ patients would have been ineligible to participate in the previously synthesized stroke rehabilitation RCTs. This study determined that all patients in this real-world clinical setting had at least one comorbid condition, nearly a fourth were over 80 years of age or had a prior stroke, a third were diagnosed as having a severe stroke, and most patients had queries regarding cognition in their medical records. These results align with other studies examining the demographics and characteristics of people who have experienced a stroke requiring rehabilitation. 22 By comparing the results of the current study with the results of a previously conducted scoping review of RCTs which intended to identify the extent and nature of RCT stroke rehabilitation literature that included patients with multimorbidity, 15 the results of this study provides practice-level insights into the potentially limited generalizability of stroke RCT research to practice settings through incongruence between clinical trials subjects and ‘real-world’ patients. While the study was conducted in a single site, we believe that the results could be transferrable to other similarly structured clinical units and stroke rehabilitation patient populations.
Limited generalizability of stroke research to practice settings
Based on our comparative analysis, many of the stroke rehabilitation inpatients included in this study may have been excluded from relevant stroke rehabilitation RCTs based on one or more of the following: the presence of comorbid conditions, prior stroke (28.0%), stroke severity (31.8% had severe strokes) and/or age (24% were 80 years of age). In addition, more than 80% of patient charts in the stroke rehabilitation inpatient sample contained queries regarding cognition, therefore it is reasonable to extrapolate that many of these people would have been ineligible to participate in the RCTs). These results align with a substantial pool of published work examining common exclusion criteria within RCTs,23–29 and provide stroke rehabilitation-specific data, a population that has not been extensively examined. Prior research regarding the generalizability of RCT results has compared baseline characteristics between study participants and the stroke rehabilitation inpatients using a single study exclusion criterion (age, comorbidity, etc.) to assess representativeness or lack thereof between stroke rehabilitation inpatients and RCT samples rather than a clinical population as a whole. Additionally, consistent with the findings of our study, a recent review suggested that most RCTs evaluating a specific physical condition excluded more than half of patients living with that condition, often because comorbid conditions are an exclusion criterion. 23
Reasons for excluding patients with multimorbidity from clinical trials are complex 30 and advocates of greater inclusion of those living with multimorbidity in research studies propose a multi-pronged approach. 31 Previous work has identified ageism in the clinical management of older adults with stroke, 32 and the design and conduct of stroke research 33 . Of the 428 RCTs reviewed by Nelson & colleagues 15 20% excluded patients over 80 years of age. Yet, among the stroke rehabilitation patients included in this study, almost one in four patients were 80 years of age or older. Furthermore, a multi-site Canadian study by Saposnik & colleagues reported an even higher proportion of patients with stroke 80 years and older, finding that 38% of patients admitted with stroke to 606 hospitals in Canada were aged 80 years and older. 34 While the underrepresentation of older adults in RCTs generally is problematic, it is a critically important issue to address in stroke research as the global population is aging,35,36 stroke prevalence has been increasing among older adults, and they tend to experience worse outcomes (e.g. higher risk-adjusted fatality, longer period of hospitalization and lower likelihood of return to post-hospital discharge location). 34
Thirty six percent of RCTs examined by Nelson et al. excluded patients with a previous stroke. In the stroke rehabilitation inpatient sample, over 25% had a previous stroke/transient ischemic attack. Nelson & colleagues 15 also determined that 24% of the reviewed RCTs excluded participants based on having conditions included in the Charlson Comorbidity Index, while 83% of the trials excluded patients based on other comorbid conditions. However, stroke rehabilitation inpatients in the study sample had an average of six comorbid conditions, which is consistent with prior literature examining multimorbidity and stroke.5,37,38 Nelson & colleagues 15 also determined that over half (55%) of the stroke rehabilitation RCTs excluded patients who were cognitively impaired. Given the comparison of aggregated data, we were unable to retrieve individual patient data on their cognitive status or to determine the precise number of stroke rehabilitation inpatients that would have been excluded based on cognitive impairment. However, given the large number of queries regarding cognition in the stroke rehabilitation inpatient sample, it is possible that many of these people would have been ineligible to participate in the RCTs. Since the majority of RCTs excluded participants reflective of the stroke rehabilitation inpatients, the findings raise questions about the justification of study inclusion and exclusion criteria. While certain criteria may be well justified, a prior study found that RCTs published within high-impact medical journals poorly justified their exclusion criteria 39 ; if the exclusion criteria result from requirements for ethical research conduct (e.g. ability to provide written informed consent) or age limits with no intervention-focused rationale, then a large group of patients may be unjustifiably excluded from clinical research interventions from which they may benefit. 23 This is of particular relevance to this study as patients deemed ineligible to participate in RCTs are also those who are less likely to experience positive outcomes due to increased age, prior stroke, comorbidities, and cognitive impairment.40–42 Strategies have been developed to support the inclusion of older adults and those with cognitive impairments within clinical trials, reducing cognitive ableism and ageism as a potential bias35,43,44; adopting these could enhance the generalizability of future stroke RCTs.
The presented study results contribute to a long-standing academic conversation regarding the generalizability of RCTs and applicability of clinical practice guidelines in patient populations with comorbidities.28,39 Although explanatory RCTs seek to determine the efficacy of an intervention, they must be relevant and definable to be clinically useful patient populations. 34 Study results support clinicians’ previously stated concerns regarding the applicability of clinical guidance documents in real-world patient populations when they are based primarily on RCTs. 12 Therefore, one strategy to support clinicians in managing multimorbidity is to generate clinical guidelines that take multimorbidity into account, as outlined by Uhlig and colleagues. 11 A second strategy lies in altering the process of evidence generation. Arguments for more inclusive stroke rehabilitation research are plentiful, most advocating for pragmatic approaches.45–47 While these pragmatic approaches may support greater generalizability of RCT results, researchers must also account for multimorbidity in the study design, implementation, and analysis. 31 To support this, Kennedy-Martin et al. (2015) developed a summary of recommended research strategies to manage external validity issues. 16
Study limitations
This study reports a secondary analysis of one-year retrospective chart review data collected from referral notes, rehabilitation assessments, and discharge summaries from a single institution. Many of the patients’ records did not contain specific information regarding the assessment of cognitive impairment, making it difficult to align with the criteria used in the RCTs directly. In addition, we did not have data on equity variables, such as patient ethnicity, socioeconomic status, or caregiver presence. Analysis of this information may reveal valuable insights regarding intersectionality and should be explored within future research. While the data captured all admitted and subsequently discharged patients from stroke rehabilitation, it was not possible to determine the number of patients who did not complete the course of rehabilitation. Finally, the comparison of aggregate level data from the previous research activities did not allow for comparison at the individual patient level to the 428 RCTs included in the scoping review, which limited our ability to determine a precise proportion of patients who would have been excluded from any specific trial.
Conclusion
Our study revealed that a high proportion of rehabilitation stroke inpatient could have been excluded from stroke rehabilitation RCTs. Yet, most clinical practice guidelines are developed based on RCT evidence; this mismatch may contribute to the challenge of clinical practice guidelines uptake in clinical practice and persisting stroke care disparities. Changes to research design and the development of clinical practice guidelines that reflect stroke rehabilitation patient characteristics are required.
Ethical statement
Ethical approval
This study is a secondary analysis of a de-identified dataset collected through a program quality improvement initiative; research ethics approval for this secondary analysis was not sought or obtained.
Footnotes
Acknowledgements
We wish to acknowledge the hospital staff who completed the original quality improvement initiative.
Author’s contributions
MN and JN conceptualized the study, MN, JN and RU designed the study. All authors participated in analysis and/or interpretation of the data. MN, HS and JN drafted the manuscript and all authors reviewed and substantively revised the manuscript. All authors have approved the submitted version.
Funding
The data for this secondary analysis study was collected as part of previously conducted quality improvement (QI) project. Neither the original QI project, nor this secondary analysis study, were grant funded. HS holds the March of Dimes Paul J.J. Martin Early Career Professorship.
