Abstract
Background
The use of prediction tools in stroke rehabilitation research and clinical practice is increasing, but it is not clear whether these prediction tools out-perform clinician predictions.
Objective
This study aimed to compare physiotherapist predictions for independent walking with the Time to Walking Independently after STroke (TWIST) prediction tool.
Methods
Adults with new lower limb weakness and unable to walk independently (Functional Ambulation Category [FAC] < 4) were recruited. At 1 week post-stroke, the treating physiotherapist was asked to predict whether their patient would achieve independent walking by 4, 6, 9, 12, 16, or 26 weeks, or remain dependent. Predictions were also made using the TWIST prediction tool, but not shared. Binary logistic regressions were conducted with the time independent walking was achieved as the dependent variable and independent variables were the physiotherapist and TWIST predictions.
Results
Ninety-one participants were included (median age 71 years, 36 [40%] female). Most participants (67 [74%]) were non-ambulatory (FAC = 0) at 1-week post-stroke. Thirty-seven physiotherapists were recruited. Physiotherapists made accurate predictions for time taken to achieve independent walking for 39 participants (43%). Prediction accuracy was not related to physiotherapist confidence or years of stroke-specific experience. TWIST out-performed physiotherapist predictions (Physiotherapists 76%-77%, TWIST 86%-88% accurate) for participants who achieved independent walking by 4, 6, and 9 weeks post-stroke. Accuracy of physiotherapist and TWIST predictions was similar for 16 and 26 weeks post-stroke.
Conclusions
The TWIST prediction tool is more accurate than physiotherapists at predicting whether a patient will achieve independent walking by 4, 6, or 9 weeks post-stroke, but not for 16 or 26 weeks post-stroke. TWIST may be useful to inform early rehabilitation and discharge planning. Clinical Trial Registration-URL: www.anzctr.org.au Unique Identifier: ACTRN12617001434381.
Introduction
Recovering the ability to walk independently is an important goal for most people early after stroke. 1 Patients and family report a lack of knowledge about their stroke and potential for recovery, which affects their ability to engage effectively in goal setting. 2 Stroke clinicians can be reluctant to provide prognostic information to their patients due to concerns about providing inaccurate information and the perceived detrimental effects of discussing the possibility of a poor outcome. 3 Despite these concerns, recovery prognosis is routinely discussed within the interdisciplinary team and prognosis for functional outcome is a key factor in determining location and type of rehabilitation offered after stroke.4,5 This means that, whether a prognosis is shared with the patient or not, clinician predictions influence patient care and rehabilitation, and need to be as accurate as possible.
The accuracy of physiotherapist predictions for walking recovery has been reported previously, with the most recent study published over 20 years ago.6,7 Physiotherapists were very accurate in predicting whether a patient would achieve independent walking or not, but less accurate when predicting timing of recovery.6,7 With subsequent advances in stroke recovery knowledge, and turnover in the therapy workforce, it is not known how accurately current physiotherapists can predict recovery of independent walking.
Prediction tools may improve prognostic accuracy, reduce the effects of clinician bias, and aid in rehabilitation planning.8 -10 Clinicians’ perceptions of prediction tools range from enthusiasm to scepticism.3,11 Those experienced in the use of prediction tools generally report that the tool provides meaningful information and informs the rehabilitation strategy. 11 Other clinicians believe that prediction tools are only beneficial for new graduates as more senior clinicians can draw on their years of experience to provide accurate prognoses. 3 This suggests many clinicians have greater confidence in their own intuition than a prediction tool and expect the accuracy of their prognoses to improve with years of clinical experience.
The aim of this study was to compare the accuracy of physiotherapist predictions for recovery of independent walking with Time to Walking Independently after STroke (TWIST) tool predictions. The TWIST tool was developed in a sample of 93 people to predict whether and when a patient is likely to achieve independent walking post-stroke. 12 It is a simple clinical tool that combines age, 1 week Berg Balance Test 13 score (<6, 6-15, or ≥16) and 1 week knee extension strength (Medical Research Council 14 [MRC] grade ≥3) to predict the likelihood of the patient achieving independent walking by 4, 6, 9, 16, or 26 weeks post-stroke, or remaining dependent at 26 weeks (Figure 1). 12 TWIST predictions are at least 83% accurate for each post-stroke timepoint, but have not been directly compared with physiotherapist predictions. 12 The second aim was to understand factors associated with accuracy of physiotherapist predictions, including confidence in their predictions and their years of stroke-specific clinical experience. We hypothesized that TWIST tool predictions would be more accurate than physiotherapist predictions.
TWIST score and probabilities. (A) TWIST prediction tool. Age, knee extension strength and berg balance scale are combined for a TWIST score out of 4. (B) Probabilities of achieving independent walking with confidence intervals for each TWIST score (0-4) by 4, 6, 9, 16, and 26 weeks post-stroke.
Methods
Participants were recruited between 19/02/2018 and 16/03/2020 from two acute hospitals in a regional network. The primary aim of the study was to develop and internally validate the TWIST prediction tool and this has been reported. 12 The present study analyses secondary data. Adults over 18 years were included within 5 days of ischemic or hemorrhagic stroke if they had new lower limb (LL) weakness and were unable to walk independently (Functional Ambulation Category 15 [FAC] score < 4). Exclusion criteria were cerebellar or bilateral stroke, dependent pre-stroke mobility (FAC < 4), walking frame use pre-stroke, significant cognitive or language impairment precluding supported informed consent, pre-existing neurological or orthopedic issues affecting mobility, unwell at time of recruitment, and palliative or not expected to survive for 6 months post-stroke. Previous stroke was allowed if any LL weakness had resolved and the patient was independently mobile prior to the current stroke. The physiotherapist treating each participant on day 7 post-stroke was also consented into the study.
Written informed consent was obtained from people with stroke and their treating physiotherapists. The study was approved by the National Ethics Committee and conducted under the principles of the Declaration of Helsinki.
Baseline Clinical Assessments
Demographic and stroke characteristics were obtained on day 3 to 5 post-stroke. Stroke severity was assessed using the National Institutes of Health Stroke Scale (NIHSS). 16 At 1 week post-stroke, walking capacity (FAC score out of 5) 15 and LL strength (LL Motricity Index out of 100) 17 were assessed by a research physiotherapist not involved in patient care. Balance (Berg Balance Test out of 56) 13 and knee extension strength (MRC grade out of 5) 14 were also assessed in order to calculate a score for the TWIST prediction tool. 12
The FAC is a six-point scale of walking capacity and can be dichotomized to indicate independent walking (FAC ≥ 4) or dependent (FAC < 4). A score of 0 indicates the participant is unable to walk or requires at least 2 people. A score of 1 indicates that 1 person is needed to provide moderate-maximal assistance for walking. A score of 2 indicates light assistance is required for walking. A score of 3 indicates supervision is required. FAC 4 indicates independent indoor mobility and FAC 5 indicates independent outdoor mobility, including on stairs, slopes, and different surfaces. Participants were allowed to use an ankle-foot orthosis, walking stick, or quad stick, but were not allowed to use a walking frame for FAC assessment. Participants who achieved independent walking (FAC ≥ 4) by 1-week post-stroke were excluded from the analysis. All participants underwent usual care rehabilitation with no influence from the research team.
Physiotherapist Questionnaires
At 1-week post-stroke, the treating physiotherapist for each participant with stroke was asked to complete a questionnaire about expectations for their patient’s walking recovery. Questionnaires were completed by physiotherapists on either acute or rehabilitation wards depending on the location of the participant with stroke. Physiotherapists were asked whether they thought their patient would achieve independent walking (FAC ≥ 4) within 26 weeks, and if so, how long it would take. They were provided with options of 4, 6, 9, 12, 16, 20, or 26 weeks post-stroke. Physiotherapists were also asked to rate their confidence in their predictions on a 6-point Likert scale (completely unsure, moderately unsure, a little unsure, a little sure, moderately sure, completely sure). Location of the physiotherapist (acute or rehabilitation ward), and years of stroke-specific clinical experience were recorded.
TWIST Score
The TWIST 12 score out of 4 was calculated by the research team using age, 1 week Berg Balance Test score, and 1 week knee extension strength (Figure 1). Physiotherapists participating in the study were not aware of assessments required to calculate the TWIST score. TWIST prediction information was not shared with the clinical team, participant with stroke, or their family.
Follow-Up Assessments
Walking capacity (FAC) was assessed at 4, 6, 9, 12, 16, 20, and 26 weeks post-stroke. Independent walking was defined as FAC ≥ 4. FAC assessments were conducted by a research physiotherapist in person while the participant was in hospital, and over the phone using a standardized questionnaire once discharged. Information was confirmed with family, the community rehabilitation team, or aged-care facility staff. The research physiotherapist conducting follow-up assessments was blinded to the TWIST score.
Statistical Analysis
Each physiotherapist prediction was categorized as correct or incorrect for prediction of independent walking by 26 weeks post-stroke. Prediction of time taken to achieve independent walking was also categorized as correct or incorrect. Incorrect predictions were further categorized as too optimistic (recovered independent walking later than predicted), or too pessimistic (recovered independent walking earlier than predicted). Absolute weeks in error were calculated.
To compare physiotherapist predictions with TWIST tool predictions, physiotherapist predictions were grouped into 4, 6, 9, 16, and 26 weeks post-stroke and cumulatively categorized. This matches the timepoints and scoring method for the TWIST prediction tool. 12 Participants were allocated a score of 1 (predicted independent) or 0 (predicted dependent) for each time point. Once a participant was predicted to have achieved independent walking, they were also scored 1 at each subsequent timepoint. This means independent walking is predicted to be achieved “at or before” a specific timepoint post-stroke, rather than “at” a specific time point. Separate multivariate binary logistic regressions, using forward stepwise with likelihood ratio, were conducted for each timepoint (4, 6, 9, 16, and 26 weeks). Critical P values for entry and exit were ≤.05 and ≥.10 respectively. The dependent variable for each regression was actual achievement of independent walking (independent, dependent). Independent variables for each regression were physiotherapist predictions (independent, dependent) and TWIST score (0-4). Accuracy, sensitivity, specificity, positive and negative predictive values were calculated for physiotherapist predictions and TWIST predictions for each time point. Level of significance for all statistical tests was set at P < .05.
To identify factors associated with accuracy of physiotherapist predictions, separate binary logistic regressions were conducted with dependent variables: accuracy of prediction of independent walking (correct, incorrect); and accuracy of predicted time taken to achieve independent walking (correct, incorrect). Forward stepwise logistic regression with likelihood ratio was used with critical P values for entry and exit of ≤.05 and ≥.10 respectively. Independent variables for both regression analyses were grouped into physiotherapist factors and patient factors. Physiotherapist factors were confidence (1-6), stroke-specific experience (years) and location (acute or rehabilitation ward). Patient factors were age, NIHSS (stroke severity), 1-week LL Motricity Index (LL strength) and 1-week FAC (walking ability). Patient factors were a priori selected based on established predictors in literature12,18 -24 and assessments commonly used in clinical care. Berg Balance Test and 1-week MRC knee extension grade were not included in the regression analyses due to collinearity with FAC score and LL Motricity Index.
Results
We recruited 106 participants with stroke. Six participants achieved independent walking by 1-week post-stroke and two had no physiotherapist prediction and were removed from analysis. Seven participants were lost to follow-up or died before achieving independent walking (Figure 2). A total of 91 participants were included in the final analysis (median age 71 years [range 38-97 years], 36 [40%] female). Sixty-seven participants (74%) were non-ambulatory with an FAC score of 0 at 1-week post-stroke and 55 (60%) had moderate to severe stroke (NIHSS ≥ 7) (Table 1).
Study flowchart.
Demographic and Stroke Characteristics.
Abbreviations: CCI, Charlson Comorbidity Index; NIHSS, National Institutes of Health Stroke Scale; LL, lower limb; MRC, Medical Research Council strength grading; FAC, Functional Ambulation Categories.
Thirty-seven physiotherapists were recruited with 16 (43%) based on an acute stroke ward and the remainder on a rehabilitation ward. Median years of stroke-specific experience was 2 (range 0-14 years). The median number of predictions made by each physiotherapist was 2 (range 1-7) (Table 2). Physiotherapists rated themselves either completely, moderately or a little sure for 66% of predictions (60/91).
Walking Outcomes and Physiotherapist Predictions.
Most participants (74/91, 81%) achieved independent walking (FAC ≥ 4) by 26 weeks post-stroke (Table 2). A third (n = 30, 33%) of participants achieved independent walking by 4 weeks, 26 (29%) participants by 9 weeks, and 18 (19%) between 9 and 26 weeks post-stroke. Seventeen (19%) remained dependent at 26 weeks post-stroke (Table 2).
Physiotherapist Prediction of Independent Walking or Remaining Dependent
Physiotherapists correctly predicted whether their patient would achieve independent walking by 26 weeks post-stroke for 80 (88%) of participants. Positive predictive value (95%) and sensitivity (91%) were also high. However, specificity (59%) and negative predictive value (71%) were low indicating that physiotherapists made more accurate predictions for participants who did achieve independent walking than for those who remained dependent. Physiotherapists made correct predictions for 10 of the 17 participants who remained dependent.
In multivariate binary logistic regression, no physiotherapist or patient factors explained the variance in accuracy of physiotherapists’ overall walking prediction (independent or not at 26 weeks post-stroke) (all P > .05).
Physiotherapist Prediction of Time Taken to Achieve Independent Walking
Physiotherapists were asked to predict time taken to achieve independent walking (4, 6, 9, 12, 16, 20, or 26 weeks post-stroke). Their predictions were accurate for fewer than half of participants (39/91, 43%). Twenty-four predictions (26%) were too pessimistic (achieved independent walking earlier than predicted) and 28 (31%) were too optimistic (achieved independent walking later than predicted). When predictions were incorrect, the median number of weeks in error was ±5 weeks (range 2-14 weeks).
In multivariate binary logistic regression, physiotherapist location (acute or rehabilitation ward) and week 1 FAC score explained some of the variance in accuracy of predicted time taken to achieve independent walking (physiotherapist location P = .004, β = 1.37, Exp (β) = 3.94, 95% CI = 1.56-9.99; week 1 FAC P = .015, β = .63, Exp (β) = 1.9, 95% CI = 1.13-3.10). Physiotherapists on rehabilitation wards correctly predicted time taken to achieve independent walking for 29 out of 50 participants (58%) while physiotherapists on acute stroke wards made accurate predictions for only 11 of 41 participants (22%). Physiotherapists were more likely to make correct predictions for participants with higher FAC scores at 1-week post-stroke. Participant age, NIHSS, and LL Motricity Index were not significant (all P > .1). There were no relationships between physiotherapist confidence or years of stroke-specific experience and accuracy of predictions (both P > .4).
Physiotherapist Predictions Versus TWIST Score Predictions
When physiotherapist predictions and TWIST score were included in binary logistic regression, TWIST score was the only surviving predictor for achievement of independent walking by 4, 6, and 9 weeks post-stroke (all P < .001). Both physiotherapist and TWIST predictions survived as predictors for achievement of independent walking by 16 or 26 weeks (16 weeks Physiotherapist P = .003, TWIST P = .001; 26 weeks Physiotherapist P = .003, TWIST P = .003) (Table 3).
Binary Logistic Regression Results.
Binary logistic regression with physiotherapist prediction and TWIST score prediction for each timepoint (4, 6, 9, 16, and 26 weeks post-stroke).
Accuracy of TWIST predictions was higher than physiotherapist predictions for participants who achieved independent walking by 4, 6, and 9 weeks post-stroke (Physiotherapist 76%-77%, TWIST 86%-88%). Specificity, sensitivity, positive and negative predictive value were 6%-15% higher for TWIST than physiotherapist predictions at all three time points (Table 4).
Accuracy of Physiotherapist and TWIST Predictions.
At 16 and 26 weeks post-stroke, accuracy of physiotherapist and TWIST predictions was similar (Physiotherapist 86%-88%, TWIST 80%-86%). Physiotherapist predictions had higher sensitivity and positive predictive value than TWIST predictions at both timepoints. TWIST predictions had higher specificity and negative predictive value than physiotherapist predictions (Table 4). This means TWIST score is better able to identify those participants who are not likely to achieve independent walking by 16 and 26 weeks post-stroke than physiotherapists.
Discussion
This is the first study, to our knowledge, to directly compare the accuracy of physiotherapist predictions for walking outcomes with those made by a published prediction tool. This study also explored the relationships between accuracy of physiotherapist walking predictions and years of stroke-specific experience, confidence, and physiotherapist location (acute or rehabilitation ward).
Physiotherapists accurately predicted whether patients would achieve independent walking within 26 weeks post-stroke (80/91, 88% correct) but were poor at predicting how long it would take each patient to achieve independent walking (39/91, 43% correct) with a median error of ±5 weeks. There was no relationship between physiotherapists’ prediction accuracy and their years of stroke-specific experience or confidence in their prediction. The TWIST prediction tool was more accurate than physiotherapist predictions for participants who achieved independent walking at early timepoints post-stroke (4, 6, and 9 weeks). For 16 and 26 weeks, TWIST and physiotherapist predictions had similar accuracy, with physiotherapists better at predicting those who would achieve independent walking and TWIST better at predicting those who would not.
The finding that physiotherapists are more accurate at predicting independent walking or not than predicting time taken to achieve independent walking supports previous work.6,7 However, despite high positive predictive value (95%), physiotherapists correctly identified only 10 of 17 participants (59%) who did not achieve independent walking, suggesting that physiotherapists have an optimistic bias.
Physiotherapists based on rehabilitation wards were more accurate at predicting time taken to achieve independent walking (29/50 accurate, 58%) than those based on acute stroke wards (11/41 accurate, 22%). It is unlikely that these differences in accuracy are driven by participant characteristics as baseline characteristics were similar between participants on the acute and rehabilitation wards (Table 1) and there was no relationship between prediction accuracy and age, stroke severity or LL strength. Healthcare systems are rarely set up with an opportunity for feedback on prognostic expectations. 8 Typically, different allied health teams work with the patient in acute, inpatient, and community rehabilitation settings. This means that clinicians in the acute setting may not see their patients once they leave the acute service, and therefore clinical expectations are neither affirmed nor corrected. Physiotherapists with opportunities to observe recovery trajectories over several weeks appear able to more accurately predict when a patient will achieve independent walking, though prediction accuracy was still relatively poor.
Physiotherapists leaned towards confidence in their predictions, rating themselves at least a little bit sure for 66% of their predictions. The mismatch between confidence and accuracy of predictions reported in this study is concerning, given the influence of clinical expectations on patient care and discharge destination.4,5,25 Contrary to physiotherapist perceptions, 3 prediction accuracy did not improve with increasing years of experience. The experience gained from years of clinical practice develops clinical skills and may influence how clinicians arrive at their decisions, 26 but did not result in more accurate prognoses. This supports previous work demonstrating that years of stroke-specific experience is not related to prediction accuracy for upper limb outcomes. 27
Making predictions can be uncomfortable and challenging for clinicians. 3 Much can be learned about working with prognostic information from the field of palliative care. Concern about the effects of delivering inaccurate information leads to reticence in sharing prognostic information with patients and families in both stroke and palliative care.3,28 Even clinicians experienced in delivering life-changing information report issues with confidence and that confidence influences the likelihood of delivering prognostic information. 28 The TWIST prediction tool improves prognostic accuracy over clinical intuition, and provides a structured approach to form the prediction, which in turn may increase clinician confidence and lower the barriers to using and sharing prognostic information.
The TWIST tool adds the most value when predicting recovery timeframes of up to 9 weeks. Predictions of whether a patient will achieve independent walking or not by 9 weeks post-stroke will have a greater influence on early discharge planning than predictions for later timepoints. After 9 weeks post-stroke, TWIST predictions tended towards optimistic. Several participants predicted by TWIST to achieve independent walking by 16 and 26 weeks post-stroke did not attain independent walking in this timeframe. Patient variables not included in the TWIST prediction tool may influence walking recovery for this more severely impaired group. Arguably a prediction tool that tends towards optimism is preferred to an overly pessimistic tool.
Prediction tools should not be used to ration care or rehabilitation.3,8,12,23 There are many important mobility-related goals that can reduce care needs and support safe discharge home, such as independent transfers, transfers without a full hoist, or independent sitting balance. The TWIST prediction tool gives no information about whether a person will achieve these mobility goals. The TWIST tool should also not be used to predict that a person with stroke will never achieve independent walking as follow-up did not extend beyond 26 weeks post-stroke. 12 Instead it is a time-based tool that can be used early after stroke to guide expectations about how long assistance will be needed for walking, thus giving patients and families more time to prepare sustainable support options. Importantly, careful consideration needs to be given to how to deliver TWIST predictions, ensuring clinicians are appropriately trained in delivering prognostic information, and avoiding mixed messages between members of the clinical team.
One limitation of this study is that the cohort of participants is the same as that used to develop and internally validate the TWIST tool. Another limitation is that qualitative data were not collected about physiotherapists’ perceptions, or barriers and facilitators to using a walking prediction tool. This warrants further study, as a prediction tool is of little use if clinicians do not see value in it. 29 Finally, the whole interdisciplinary team is involved in working with prognostic information after stroke. Future studies could investigate accuracy of predictions and perception of prediction tools in a wider range of clinician groups delivering stroke care.
Conclusions
The TWIST prediction tool is more accurate than physiotherapists at predicting whether a patient will achieve independent walking by 4, 6, or 9 weeks post-stroke. Physiotherapists can predict whether a patient will achieve independent walking by 26 weeks post-stroke, but accuracy is poor when predicting time taken to achieve independent walking. Physiotherapist accuracy isn’t improved by years of stroke-specific experience and there is no relationship between confidence and accuracy of predictions. The TWIST prediction tool is more accurate than clinical intuition and may be particularly useful to inform early rehabilitation and discharge planning.
Footnotes
Acknowledgements
The authors thank Gemma Nolan and Christine Mangold for assistance with data collection.
Author Contributions
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Neurological Foundation of New Zealand 1735-PG; Auckland Academic Health Alliance 3715213.
