Abstract
Objective:
A psychometric evaluation of the Needs and Provision Complexity Scale (NPCS).
Main measure:
The NPCS is designed to evaluate both needs for health and social support (NPCS-Needs) and services provided to meet those needs (NPCS-Gets).
Design:
A consecutive cohort of patients were recruited from nine specialist neurorehabilitation units in London.
Subjects:
Four hundred and twenty-eight patients were assessed at discharge (63.1% males; mean age 49 years) of whom 73.6% had acquired brain injury (49.5% stroke/subarachnoid, 14.7% traumatic brain injury, 9.3% ‘other acquired brain injury’), 8.9% spinal cord injury, 6.1% peripheral neuropathy, 4.9% progressive neurological and 6.3% other neurological conditions.
Results:
The NPCS-Needs was completed by the clinical team at discharge and 212 patients reported NPCS-Gets after six months. NPCS-Gets repeatability was tested in a subsample (n = 60). Factor analysis identified two principal domains (‘Health and personal care’ and ‘Social care and support’) accounting for 66% of variance, and suggested a large general factor underpinning the NPCS. Internal consistency was high (alpha = 0.94) and repeatability acceptable. Intraclass coefficients for domain scores were healthcare 0.67 (95% confidence interval (CI) 0.48–0.80); personal care 0.83 (0.73–0.90); rehabilitation 0.65 (0.45–0.78); social/family support 0.66 (0.46–0.79) and environment 0.84 (0.74–0.90). Linear-weighted kappas for item-by-item agreement ranged from 0.42 to 0.83. Concurrent validity was demonstrated through correlations with measures of dependency and community integration.
Conclusions:
Notwithstanding a 50% response rate after six months, the NPCS has good internal consistency, a robust two-factor structure, acceptable test–retest reliability and initial evidence of concurrent validity.
Keywords
Introduction
The needs of patients with long-term disabling conditions often change variably, and often unpredictably, over time. 1 At present there is no standard method for quantifying and documenting an individual patient’s needs and the extent to which rehabilitation services actually meet these needs. For example, the Southampton Needs Assessment Questionnaire (SNAQ) is a structured interview that is designed to assess a patient’s perceptions of met and unmet need, but it does not attempt to assess the service provider’s perspective. 2 Moreover, it has 77 items, and while it is a valuable research tool it may be impractical for inclusion in national datasets.
Following publication of the UK National Service Framework for Long Term Conditions, 3 the NHS Information Centre developed a dataset to monitor implementation which included a new Needs and Provision Complexity Scale (NPCS).
The NPCS is a simple, pragmatic tool to measure both the needs an individual has for rehabilitation and support – and the extent to which those needs are met through service provision. 4 Subtracting NPCS-Needs scores from NPCS-Gets scores provides a measure of ‘degree that needs are met’ and a simple costing algorithm provides a generic estimate of the likely cost implications of addressing the unmet needs. Its development was initiated in 2008 by the Long Term Neurological Conditions (LTnC) Dataset Development Group. As well as service users and carers, the Dataset Development Group consisted of commissioners and healthcare professionals from a range of disciplines, all of whom were experienced in care and planning of services of people with long-term neurological conditions. The instrument evolved through an iterative process of testing and refinement before publication in 2010. 5 However, it has yet to be subjected to formal psychometric testing.
In this paper, we report the first psychometric evaluation of the NPCS using cross-sectional and longitudinal data from a community-based study of patients with complex neurological disability. In particular we aimed to examine the dimensionality or factorial structure of the NPCS, its internal consistency and test–retest reliability, and concurrent validity through correlations with measures of related constructs.
Methods
Measures
The NPCS is a 15-item measure with five subscales. It has two parts:
Part A (NPCS-Needs) is completed by the treating clinician(s) to evaluate each patient’s needs for health and social care in any given period.
Part B (NPCS-Gets) is subsequently self-completed by the patient (or carer on their behalf) to evaluate the level of provision in relation to those needs over the same period.
Total scores range from 0 to 50 and cover ‘low’ to ‘high’ levels of needs.
Full details on the NPCS, including the tool itself, are obtainable from the UK Rehabilitation Outcomes Collaborative (UKROC) website at: http://www.csi.kcl.ac.uk/npcs.html.
A consecutive cohort of patients discharged over a 12-month period from the nine tertiary (Level 1) neurorehabilitation units in London were approached to participate. All patients meeting the eligibility criteria were given a verbal explanation about the study, provided with an information sheet and given the opportunity to ask any questions. If the patient, or their carer by proxy, agreed to take part, consent was taken by the discharge clinician. Participants who consented to participate were rated on the NPCS by their treating clinical team clinician at the time of discharge from inpatient rehabilitation. As part of a longitudinal study of community rehabilitation services, participants then completed the NPCS at four weeks, six and twelve months post discharge. The present study focuses on clinician-completed NPCS-Needs data at discharge and patient-completed NPCS-Gets data from six months post discharge. Where necessary, a caregiver assisted with or completed the NPCS-Gets for the patient. Patient participants had the options of completing the NPCS-Gets in pencil-and-paper format (i.e. postal survey), online or over the telephone. Research assistants made telephone contact with participants soon after their return to the community to ensure their participation in the study. At this time participants were also invited to take part in a study of the repeatability of the NPCS-Gets, which required them to complete it twice at a one-week interval. While not formally assessed, our extensive contact with clinicians, patients and carers throughout the study indicated that, in general, both NPCS-Needs and NPCS-Gets were easily understood and completed by clinicians and patients or their carers.
In the absence of any ‘gold standard’ measure for met and unmet need, initial evidence for the concurrent validity of the NPCS-Gets was explored through correlation with measures of dependency and community integration at six months after discharge. These were:
the Neurological Impairment Scale 6 – impairment;
the Barthel Index 7 – independence in activities of daily living;
the Northwick Park Dependency Scale (NPDS) 8 subdivided into its 13-item physical (NPDS-P) and 3-item cognitive behavioural (NPDS-CB) subscales;
the Community Integration Questionnaire. 9
We expected to find higher levels of service provision (NPCS-Gets) in patients who were:
more impaired (i.e. those with higher impairment scores),
more dependent (i.e. those with a higher NPDS score and a lower Barthel Index score),
less well integrated (i.e. those with lower community integration questionaire scores).
Statistical analysis was undertaken using SPSS version 19, except for linear weighted kappa coefficients, which were computed in Stata version 10.
Missing data: No data were imputed. Cases were excluded if the patient had missing values for variables included in the analysis.
Bartlett’s test of sphericity and the Kaiser–Meyer–Olkin test were first applied to ensure the correlation matrix was suitable for factor analysis.
Principal components analysis with Varimax rotation was used to examine the factor structure of both the NPCS-Needs and NPCS-Gets.
Cronbach’s α was calculated as a measure of internal consistency for the full NPCS scale and the scale minus each individual item. Cronbach’s α should be at least 0.70 for group comparisons. 10 Corrected item-total correlations were also computed.
Test–retest repeatability of the NPCS-Gets was assessed by calculating absolute agreement and linear weighted kappa coefficients (κw) for individual item scores, interpreted according to Landis and Koch. 11 Intraclass correlation coefficients (ICCs) were calculated for domain scores.
Correlation with other measures was tested by Spearman rank correlation tests.
Results
A consecutive cohort of patients (n = 467) discharged over a 12-month period from the nine tertiary neurorehabilitation units within the London area were approached to participate, of whom 428 (92%) were recruited. Despite strenuous attempts to contact all patients at six months post discharge, only 212 (49%) responded to provide NPCS-Gets data. Responses were mostly by pencil-and-paper questionnaire (82%) followed by online (12%) and telephone interview (6%).
Test–retest reliability of patient-reported NPCS-Gets was examined in a subgroup of n = 60 participants, who completed a second NPCS-Gets form a week after returning the first.
Demographic, diagnostic and impairment data for the 428 participants at discharge and 212 at six months are reported in Table 1. Chi-square tests did not show any significant differences in the distributions of these variables when the two samples were compared.
Demographic characteristics, impairment and dependency of recruits in the whole sample and respondents at six months.
SAH, sub-arachnoid haemorrhage; TBI, traumatic brain inury; ABI, acquired brain injury; LTNC, long-term neurological condition; NIS, Neurological Impairment Scale; NPDS, Northwick Park Dependency Scale.
Results of the factor analyses and individual item–total correlations for both the clinician-completed NPCS-Needs and patient-completed NPCS-Gets data are presented in Table 2. For clinician-completed ratings at discharge, 404 cases remained in the dataset after exclusion of cases with missing data. Bartlett’s test of sphericity was significant (P < 0.01) and the Kaiser–Meyer–Olkin measure of sampling adequacy at 0.92 was well above the minimum value for factor analysis. Two components had eigenvalues greater than 1.0 (8.6 and 1.2) and the scree plot also indicated a two-factor solution was appropriate. The first two principal components accounted for approximately 66% of total variance. There was evidence of a large general factor with all 15 items loading high (i.e. >0.50) on the first unrotated principal component. Results of the two-factor Varimax rotation of clinician ratings show evidence for two distinct factors – one corresponding to the ‘Health and personal care’ domain and the other to the ‘Social care and support’ domain. Item–total correlations were consistently high and positive (0.59–0.86). Cronbach’s α was also high for the full 15-item scale at 0.94.
Results of two-Factor Varimax rotation with corrected item–total correlations for clinician-completed NPCS-Needs at discharge and patient-completed NPCS-Gets after six months.
For patient-completed ratings at six months post discharge, 179 cases remained in the dataset. Sphericity (P < 0.01) and sampling adequacy (0.70) were still acceptable for factor analysis. Five components had eigenvalues above 1.0 but this resulted in a fragmented and uninterpretable pattern of factor loadings. Consequently we extracted and rotated two components to permit comparison with the solution from the clinician-completed NPCS-Needs data. Thirteen items had loadings above 0.30 on the first unrotated principal component and two items (Vocational rehabilitation, Residential respite) had near zero loadings (these services were rarely provided). This reveals good support for a two-factor structure that is quite similar to that observed for the clinician-completed NPCS-Needs. At the same time item–total correlations were generally lower, with seven items having a correlation with the total score below 0.30. Interestingly, Personal enablement now loaded more strongly with Social care and support, while Equipment and accommodation, now loaded with Health and personal care. Cronbach’s α was also somewhat lower for the NPCS-Gets, although still within the acceptable range at α = 0.75.
Table 3 shows the analysis of agreement for patient-reported NPCS-Gets ratings completed approximately seven days apart. These were all completed within the first six-month period. The ICCs for domain scores range from 0.66 to 0.84 and the linear-weighted kappas for item-by-item agreement ranged from 0.42 to 0.83. Interpreted according to Landis and Koch, agreement was ‘moderate’ for five of the 15 items, but ‘substantial’ or ‘excellent’ for the remaining 10. Nevertheless, the confidence intervals were quite wide.
Test–retest reliability statistics for patient-completed NPCS-Gets items with seven-day retest interval (N = 60).
Table 4 presents correlations of the two principal domains of the NPCS-Gets (Health and personal care and Social care and support) with other measures of impairment, dependency and community integration.
Spearman rho correlations between NPCS-Gets domains with other measures of dependency and community integration at six months after discharge.
P < 0.01; **P < 0.001.
NIS, Neurological Impairment Scale; NPDS, Northwick Park Nursing Dependency Scale; CIQ, Community Integration Questionnaire.
Moderate correlations were seen in the expected directions with other measures of disability. The relationship with community integration was weaker but still significant.
Discussion
This paper presents the first reported psychometric analysis of the Needs and Provision Complexity Scale (NPCS). Our results demonstrate acceptable test–retest repeatability and the expected relationships were seen with other measures. Factor analysis suggests scale structure in two principal domains (Health and personal care and Social care and support), but also provides support for a single general factor underpinning the full NPCS with good overall internal consistency.
Some differences were observed in the factor structure of the clinician-completed NPCS-Needs and the patient-completed NPCS-Gets at six months. For the latter, internal consistency was lower (although still in the acceptable range) and there was less evidence of a single general factor underpinning the full NPCS. Indeed the two unrotated principal components solution was quite similar to the two-factor rotated solution – suggesting two quite independent factors.
These differences need to be interpreted with some caution, as they may in part reflect the smaller numbers in the factor analysis for the NPCS-Gets version. However, one possible interpretation is that clinicians see both the healthcare and the social support items as a single bundle of the person’s needs at discharge, whereas in the community the actual provision of healthcare and social support are quite independent. This resonates with experience in that community-based health and social services are provided by separate bodies in the UK (primary care trusts and local authorities, respectively), often with little evidence of coordination between them. Similarly, the loading of items such as equipment and accommodation with health and personal care may reflect the fact that, while provided by social services, the route for referral is typically through the community-based rehabilitation services that are funded by healthcare in the UK. Therefore, from this preliminary evaluation, it appears that the NPCS may provide quite a good reflection of the reality of service provision as it is currently offered in the community.
The authors recognize a number of limitations to this study.
The study population was a multicentre cohort recruited from nine different units, but the sample was drawn from the London area and the findings need to be duplicated in other, more rural, settings.
Despite our best efforts to trace patients and encourage participation in the follow-up evaluation, only about 50% of the original recruited sample provided data on service provision at six months post discharge. This degree of attrition is not uncommon in acquired brain injury populations. Wade et al. reported a 60% rate in their trial of follow-up after traumatic brain injury. 12 Although no significant demographic differences were demonstrated between the recruited and responder groups in demographics or impairment, this high attrition could still have introduced a degree of sample bias, especially as dependency data was not available for the recruited sample (see Table 1).
Although the repeatability results were generally encouraging, the confidence intervals were wide, possibly due to the relative small sample size. Moreover, the repeatability results were only obtained for patient-completed NPCS-Gets data and remain to be established for clinician-completed NPCS-Needs data.
The study only captured clinicians’ perceptions of need at discharge (i.e. ‘needs’) and patients’ reports of actual services received at six months (i.e. ‘gets’). Ideally we would have measured needs at six months as well. This would have permitted a comparison of clinicians and patients in the way they conceptualize needs and the extent to which they are met. However this was beyond the scope and funding of the study.
In this evaluation we have focused on the findings of traditional psychometric methods. Further evaluations using other psychometric techniques such as Rasch analysis will be presented separately as the datasets enlarge.
Despite these limitations, our findings provide support for the NPCS as a useful overall measure of an individual’s requirements for health and social services. It has also undergone further development to derive a costing algorithm, now available on the website that details the tool itself: http://www.csi.kcl.ac.uk/npcs.html.
In addition, it has been used most recently to examine met and unmet needs in the community. 2 Although the NPCS was designed specifically for inclusion in the UK National Health Service (NHS) dataset, it could be easily adapted to suit other countries and their health systems. Further evaluation is now required in other populations and settings.
Clinical messages
This study provides evidence for the validity and reliability of the Needs and Provision Complexity Scale in patients with neurological disability.
Factor analysis suggests that items fall into two subscales: ‘Health and personal care’ and ‘Social care and support’.
Nevertheless, summation to a single overall scale of needs or provision is acceptable.
Footnotes
Acknowledgements
The authors are most grateful to the all patients and caregivers who participated in the study. We would also like to thank the clinical staff from the nine services comprising the London Neurorehabilitation Consortium who recruited participants.
Special thanks are due to Helena Morris, Sasha Wade, Bernadette Khoshaba, Christine O’Connell, Amanda Fahy, Ralph Crook and Ali Price for their roles in data collection and administrative support for the project.
Ethics approval
Approval for the study was granted by Bromley Research Ethics Committee (Ref no: 09/H0805/25) and subsequently R&D centres of the seven participating NHS trusts. The remaining two recruiting centres were in the independent sector - approval for recruitment was obtained through their internal clinical and research governance processes.
Conflict of interest
Outcome measurement is a specific research interest of our centre. The NPCS was developed through this department, but is disseminated free of charge. Neither of the authors has any personal financial interests in the work undertaken or the findings reported.
Funding
This project was funded by the National Institute for Health Research Health Services and Delivery Research (NIHR HS&DR) programme (project number 08/1809/235). Visit the HS&DR website for more information. The views expressed therein are those of the authors and do not necessarily reflect those of the NHS, the NIHR or the Department of Health.
Financial support for the preparation of this manuscript was also provided by the Dunhill Medical Trust, the Luff Foundation.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
