Abstract
Objective:
To evaluate the acceptability, test–retest reliability and validity of the Longer-term Unmet Needs after Stroke monitoring tool.
Design:
A questionnaire pack was posted to stroke survivors living at home three or six months after stroke. A second pack was sent two weeks after receipt of the completed first pack.
Setting:
Stroke survivors living at home across England.
Subjects:
Stroke survivors were recruited from 40 hospitals across England, in two phases. The first with an optimal cohort of patients, the second to capture a broader post-stroke population, including those with communication and/or cognitive difficulties. Patients were excluded if they required palliative care or if permanent discharge to a nursing or residential home was planned.
Main measures:
The questionnaire pack included the Longer-term Unmet Needs after Stroke tool, General Health Questionnaire-12, Frenchay Activities Index, and Short Form-12.
Results:
Interim analysis of phase 1 data (n = 350) indicated that the tool was sufficiently robust to progress to phase 2 (n = 500). Results are reported on the combined study population. Of 850 patients recruited, 199 (23%) had communication and/or cognitive difficulties. The median age was 73 years (range 28–98). Questionnaire pack return rate was 69%. For the new tool, there was 3.5% missing data and test–retest reliability was moderate to good (percentage item agreement 78–99%, kappa statistic 0.45–0.67). Identification of an unmet need was consistently associated with poorer outcomes on concurrent measures.
Conclusions:
The Longer-term Unmet Needs after Stroke tool is acceptable, reliable, can be self-completed, and used to identify longer-term unmet needs after stroke.
Introduction
Allocation of resources according to need is a key objective of health and social care service provision. Needs can be defined as ‘the ability to benefit from health (or social care)’, 1 and unmet needs defined as ‘expressed needs that are not satisfied by current service provision’. 2 These definitions incorporate some value judgement and will be influenced by previous experience, expectations, personal factors and the environment. 3 Many research and consumer reports indicate that stroke survivors may experience a range of problems including incontinence, depression, lack of information, pain, and restrictions in social and household activities. 4 –8 Such problems can be prevalent and long lasting, 9 and often translate into unmet needs. 10
Needs can be assessed ‘top down’ by professionals using their judgement and clinical assessment tools (for example, depression screening tools) or ‘bottom up’ in which patients are involved in self-assessment of their needs. 11 Self-identification of unmet need, although necessarily subjective, facilitates a more patient-centred approach to care. Existing stroke tools tend to be ‘top-down’ instruments, or are focussed on one clinical area and therefore unable to describe the range of domains important to stroke survivors. Others, which cover a range of domains, can be long with poor response rates, 12,13 or not fully psychometrically evaluated. 10,14 Similar difficulties exist in other clinical areas, with multiple tools used to measure social care unmet needs of older people 15 and limited psychometric testing of tools used to measure unmet need in cancer. 11
We developed a 22-item monitoring tool for identifying Longer-term Unmet Needs after Stroke (LUNS) (Appendix 1, online), predicated on patient- and carer-identified post-stroke unmet needs. This article reports the evaluation of acceptability, test–retest reliability and validity of this new tool.
Methods
A multistage, iterative process was used to develop, refine and test the LUNS tool (Appendix 2, online). We have reported findings regarding the longer term stroke experience (Stage I), 16,17 and semi-structured interviews (Stage II), 18 elsewhere. The style of questionnaire packs used in the pilot and multicentre study were informed by previous work. 19 Recorded completion time of the tool (in Stage VIII, Appendix 2, online) was a median of six minutes (range 2–12). The final 22-item tool encompasses the domains of information, services, social and emotional consequences, and health related problems. Each item has a yes/no response, where the ‘no’ option applies to either no need or the need is met. This article reports the findings of a multi centre study (Stage IX, Appendix 2, online) to fully evaluate the new 22-item tool.
The study was conducted in two phases: the first with an optimal cohort of patients; the second with a broader post-stroke population. In both phases, patients were recruited by Stroke Research Network 20 staff from stroke units in England. Eligibility criteria were purposefully broad: a diagnosed new stroke; aged 16 or over; and discharge to their own home or that of their carer anticipated. Patients were excluded if they had subarachnoid haemorrhage, required palliative care, or if permanent discharge to a nursing or residential home was planned. In phase one, a selected population of English-speaking patients without communication or cognitive impairment (assessed following verbal consent using the six-item Cognitive Impairment Test, 21 or the Frenchay Aphasia Screening Test, 22 respectively) were recruited. These patients required a minimum hospital stay of 72 hours (to avoid recruiting a sample likely to have low or no unmet needs). Interim analysis was conducted to ensure acceptability, test–retest reliability and validity of the LUNS tool in this optimised group before extending recruitment in the second phase to a sample with a broader range of stroke impairments. To facilitate this, during the second phase of recruitment, minimum hospital stay was increased to 14 days (i.e. patients likely to have more stroke-related impairments) and a target set to recruit 50% of patients with communication or cognitive impairment (assessed as above 21,22 ). In this phase, patients who were unable to read and understand English were also recruited if they had an English-speaking relative or friend to help with questionnaire completion.
Stroke Research Network staff approached patients (or carers of patients who lacked capacity) prior to their discharge home to obtain written informed consent or consultee declaration. The Information Sheet and recruiting staff made it clear to patients that their care would not be affected and their answers used only to test the questionnaire. Following consent, network staff collected baseline data, including demographic information, stroke details, pre- and post-stroke Barthel Index 23,24 and comorbidity information. 25 Recruited patients who became ineligible were withdrawn.
After confirming patient contact details and survival status through General Practitioner (Family Physician) or NHS database records, the new tool was administered by post in two self-completion questionnaire packs at three or six months after stroke (determined by participant date of recruitment into the study, for timing of study management). Proxy responses were allowed. Participants were asked to indicate what level of help (from carers/friends) they had received in completing the questionnaires. The first pack (Pack 1) contained the new LUNS tool and other questionnaires. As no measure of unmet need exists with which we could evaluate concurrent validity, the following were chosen to reflect health related needs.
Impairment Manikin (on which respondents identify complete recovery or indicate problems by marking relevant areas on a diagram of a manikin). 26
General Health Questionnaire-12 (GHQ-12, emotional health and wellbeing). 27
Frenchay Activities Index (extended activities of daily living). 28
Short Form-12 v2 (SF-12, health-related quality of life, including mental and physical component scores). 29
To assess test–retest reliability, a second pack (Pack 2) was sent approximately one week after the fully completed Pack 1 was received. Pack 2 contained the new LUNS tool, the SF-12 and a question asking whether they had experienced any health changes since completing Pack 1, to provide contextual information when assessing reliability.
Patients who had not returned packs to the study team within two weeks were telephoned to check receipt and if necessary the pack was re-sent. If questionnaires were missing less than 50% of items, patients were telephoned to complete the missing fields. Questionnaires missing 50% or more data were re-sent to the patient.
The analysis in this study was designed to evaluate the new LUNS tool. Baseline data were summarised using descriptive statistics, and comparisons between Pack 1 responders and non-responders made using Mann-Whitney U, Chi square and independent t-test. Acceptability of the new tool was determined by the percentage of items completed on initial receipt of Pack 1. Percentage of missing data for each of the questionnaires was reported for comparison purposes. Mann-Whitney was used to investigate differences in health status (GHQ-12, Frenchay Activities Index, SF-12) between the group who identified unmet need, and the group who identified no unmet need, on each item of the new tool. Test–retest reliability of the new tool was assessed by comparing Pack 1 and Pack 2 percentage agreement and kappa statistics for individual items. 30 Patients who indicated a change in their health status between Packs 1 and 2 were excluded from test–retest reliability assessment. Strength of agreement for Kappa was defined as ranging from poor (0–0.20) to very good (0.81–1). 31 SF12 test–retest was calculated in order to provide context for test–retest of the new tool.
The study was approved by the Bradford Research Ethics Committee and adopted onto the National Institute for Health Research (UK) portfolio.
Results
Interim analysis of phase 1 data (n = 350) found adequate acceptability, test–retest reliability and validity to progress to the broader phase 2 study. Results are reported on the combined study population.
Study population
A total of 850 patients from 40 hospitals were recruited (phase 1 n = 350, phase 2 n = 500) (Figure 1). The data available indicate two-thirds of eligible patients approached were consented into the study. The median age was 73 years (range 28–98), and 199 (23%) had communication or cognitive impairments or did not speak English (n = 2). Responders had less cognitive and physical impairment compared with non-responders (Table 1). Median time in hospital was 12 days in phase 1 and 42 days in phase 2. Pack 1 was sent three or six months post-stroke, the median time between stroke onset and completion was three months (range 3–6) (n = 115) and six months (range 5–10) (n = 414), respectively. The time between sending and receiving pack 1 was a median of 12 days (range 3–120).
Study flowchart.
Baseline characteristics.
Significant differences (p < 0.01) were observed for number of previous strokes, ethnicity, communication or cognitive impairment, and post-stroke Barthel score. No other significant differences were found when comparing responders with non-responders.
A total of 213 patients (40% of those who returned Pack 1) reported that they received help to complete the questionnaires. For the majority of these someone helped to tick the boxes (161, 76%), helped to read out the questions (153, 73%) or helped to discuss the questions with them (145, 68%). A minority had someone translate the questions (2, 1%) or received other help (10, 5%). For 14 responders (6% of those who received help, or 2.6% of those who returned Pack 1) someone had completed the pack without consulting them (proxy completion). The median number of unmet needs identified on the new LUNS tool was 4 (range 0–19) with the prevalence of each unmet need item ranging from 3% to 58%.
Acceptability
Response rate to Pack 1 was 69% (Figure 1). Of those who returned pack 1, 61% did so with no reminder (phase 2 data only). On initial receipt, 85% of the returned LUNS questionnaires had all items completed, and overall 3.5% of items were not completed. Table 2 shows acceptability of the new tool was comparable to the concurrent measures.
Acceptability, missing data.
Test–retest reliability
Of the Pack 2 responders, 336 (73%) reported no health changes between completing Pack 1 and Pack 2. Test–retest was assessed in this group only. The median time between completion of Packs 1 and 2, was 14 days (range 3–96). For the new tool, individual item percentage agreement was between 78–99%, with 19 items having over 85% agreement (Table 3). Kappa values were between 0.45–0.67, with 14 items showing moderate agreement (kappa 0.45–0.59), and eight items with good agreement (kappa 0.61–0.67). As a comparison, SF-12 test–retest of individual items resulted in kappa scores between 0.34–0.57 and percentage agreement between 44–73%.
Test–retest reliability.
Comparison with concurrent measures
For each item on the LUNS tool, the group of responders who identified unmet need on this item had significantly poorer health scores on the GHQ-12 (p < 0.05) than those who identified no unmet need. This significant difference in health status between those who did or did not identify unmet need was also observed for 21 items with the SF-12 Mental Component, 15 items with the SF-12 Physical Component and 14 items with the Frenchay Activities Index. Table 4 shows those items for which the group with unmet need had significantly poorer health than the group with no unmet need (p < 0.05) on the concurrent measures.
Comparison of unmet need with concurrent measures. Showing those concurrent measures where the group identifying an unmet need had significantly poorer health (p < 0.05) than the group with no unmet need.
Discussion
This large multicentre study evaluated the acceptability, test–retest reliability and validity of a new 22-item tool (LUNS) for identifying longer-term unmet needs after stroke. Response rates and minimal missing data indicate that the tool was acceptable to the target group. Individual item test–retest reliability was good and compares favourably to SF12.
Setting a target for the proportion of patients with communication and cognitive impairments, and allowing proxy responses, facilitated robust evaluation of the new tool on a sample with a range of stroke impairments. Comparison with data from the UK National Stroke Audit 32 indicates that this study sample is reasonably representative of the wider UK stroke population (age, gender, time in hospital, post-stroke Barthel Index).
The test–retest reliability results are reassuring for a questionnaire that attempts to assess unmet needs that might be considered subjective and therefore potentially unstable. Each item has been worded to ask whether the respondent has a need and whether the need is met, and potential complexity of double-barrelled questions was managed by providing instructions to tick ‘no’ if there is no need or if the need has been met. Use of dichotomous responses is considered a strength as it is simple and provides an unambiguous response. 33 Results show that for the majority of items responders most commonly tick ‘no’. As a result, the new tool may be vulnerable to a ‘floor effect’. However, it was shown to have a response rate and median number of unmet needs similar to that in a UK Stroke Association needs survey (n = 1251, response rate 60–78%; median number of unmet needs three). 10 While only 61% of responders returned the questionnaire pack without a reminder in Phase 2, it should be noted that this is in a population deliberately selected to have poorer health (data not available for Phase 1). In addition, participants in this study returned a 10-page pack containing multiple questionnaires, whereas in practice they will only be completing the two-page LUNS tool.
A weakness of the study is that we could not compare our tool with a benchmark measure of unmet need to assess concurrent validity. Therefore, we selected validated measures of health status as proxy measures of need: lower health status as an indicator for increased need. Results of this comparison showed that identification of an individual unmet need was consistently associated with poorer health status on the concurrent measures. This supports our plausible link between low health status (emotional wellbeing, activities of daily living, health related quality of life) and unmet need, and goes someway to establishing concurrent validity of the LUNS tool.
This new tool is based on patient- and carer-identified post-stroke problems, as well as evidence from the literature, and was refined with the assistance of over 150 stroke survivors or carers. Although some topics do not specifically map onto the LUNS tool (e.g. vision, communication), the ‘bottom-up’ approach to development should ensure the tool is capable of capturing all areas relevant in the lives of stroke survivors. For example, ‘communication’ may warrant a specific item when considered from a clinical or ‘top-down’ perspective, however, this may be captured by the tool in terms of the effect on using public transport or daily social occupations. However, identification of unmet needs alone would not be sufficient to improve an individual’s outcomes, further investigation would be required to establish the cause, significance or service response to identified unmet needs.
There have been a number of attempts to capture unmet needs in the post-stroke population. The Stroke Association survey 10 was developed to identify unmet needs in stroke survivors, but the survey tool has not been psychometrically evaluated. The Southampton Needs Assessment Questionnaire was designed to assess rehabilitation needs for people with physical disability, but has not been evaluated for test–retest reliability 14 and postal response rate was low (49%). 13 The Stroke Impact Scale is long (59 multiple choice items) and has low postal response rates of 41% 19 and 63%. 12 The Greater Manchester Stroke Assessment Tool (GM-SAT) has been designed for an assessor to work with the stroke survivor to identify and address unmet needs using algorithms. 34 Therefore, to assess stroke survivor unmet need, the LUNS tool is alone as a self-completion questionnaire with well described acceptability, validity and reliability.
The LUNS tool is quick for stroke survivors to complete and easy to interpret. Within the stroke population it can be used pragmatically to identify unmet needs relating to either stroke or comorbidities. Policy recommends that stroke patients are regularly reviewed and their unmet needs assessed in the longer term. 35 This new tool offers a simple and validated method to profile an individual’s unmet needs, which can be integrated into existing practice. Furthermore, it could be used as a survey tool to assess the extent to which community services are meeting the needs of their stroke patients, for example measuring the percentage of the local stroke population whose mobility needs are not met. Such data could be used to compare different services, promote service development or guide commissioning.
Clinical messages
The LUNS tool is acceptable, reliable and identification of an individual unmet need is associated with poorer outcomes on concurrent measures of health status.
The LUNS tool offers a simple and validated method for identifying longer-term unmet needs of stroke survivors.
Footnotes
Acknowledgements
Major contributors: Anne Forster, Jenni Murray, John Young, Rosie Shannon, Kirste Mellish, Mike Horton, Alan Tennant, Rachel Breen, Natasha Alvarado, Bipin Bhakta.
Scientific contributors: Sue Bogle (Aysgarth Statistics), Jenny Hewison, Allan House.
Other contributors: We are grateful to staff in the UK stroke research networks, study participants, contributors from stroke groups and the Consumer Research Advisory Group, Jane Smith, Chung Fu and Nick Preston. We thank Dr Kersten for providing a copy of the Southampton Needs Assessment Questionnaire
Conflict of interest
The author declares that there is no conflict of interest.
Funding
This publication presents independent research commissioned by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research programme (Grant Reference Number RP-PG-0606-1128). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. This work was also supported by the Stroke Association for the development of LUNS (reference TSADRC 2006/01).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
