Abstract
The diagnostic assessment of patients with cognitive impairment relies on the synthesis of information collected from the medical history, the physical and mental status examination, informants and relevant investigations [1]. While neuropsychological assessment using tools such as the Wechsler Adult Intelligence Scale [2] remains the ‘gold standard’ of cognitive assessment, this depth of assessment is not always necessary and is of limited availability, except within specialized services. In routine clinical practice, the clinicians' goal should be to elicit a broad assessment of cognitive function within the shortest possible time and to incorporate this information into the diagnostic process. The differing needs of clinicians and their patients have led to the development of a number of clinical or bedside cognitive assessment tools such as the Mini-Mental State Examination (MMSE), the Neurobehavioural Cognitive Status Examination [3] and the Addenbrooke's Cognitive Examination (ACE) [4], which each have their own strengths and limitations regarding their breadth, portability and brevity [1].
The development of the Neuropsychiatry Unit Cognitive Assessment Tool (NUCOG) was driven by the specific requirements of a tertiary referral inpatient, outpatient and consultation–liaison neuropsychiatric service. Patients are referred from mental health, neurological or general medical services with complex presentations which may include combinations of cognitive impairment, psychosis, movement disorder, behavioural disturbance, affective disturbance, somatoform symptoms, epilepsy, head injury and other neuropsychiatric symptoms.
Based on three decades of clinical experience with this clinical population, we collated the bedside tests found to be of greatest clinical utility into a single scored instrument. The tool needed to be suitable for bedside or office use, be able to be administered by junior as well as senior clinicians, require minimal instrumentation, provide a breadth of cognitive assessment items and be brief in administration. The result is the NUCOG, a 21-item tool that tests a broad range of cognitive functions and requires no instrumentation. It comprises bedside cognitive tests in established use, with items similar to those found in other cognitive screening tools. It was designed to be administered in approximately 20 minutes or less by trained clinicians. Administration of the NUCOG provides a total score for a subject out of a possible 100, and scores out of 20 on five cognitive domains: attention, memory, executive, visuoconstructional function and language. Each of these domains is tested by a group of items that aim to ensure breadth of coverage while ensuring a balance between depth of coverage and time (Table 1).
Items in the five domains of the NUCOG
NUCOG, the Neuropsychiatry Unit Cognitive Assessment Tool.
When initially piloted the NUCOG was shown to have high face validity, to correlate with MMSE scores and to provide significant scoring differences between patients with dementia and those without dementia [5]. The current study reports our further experience with the NUCOG and its psychometric properties in a larger group of patients, together with data regarding the performance of control subjects on the tool.
Method
Subjects
The research protocol was approved by the local Research and Ethics Committee. The
study evaluated a group of healthy subjects (n = 82) and a group of patients seen in
the Neuropsychiatry Unit (n = 265). The patient group comprised of inpatients (n =
104), outpatients (n = 145) and consultation–liaison patients (n = 16). These were an
unselected group of patients assessed through the normal intake processes of the
unit. The patient group was divided into three groups depending on the primary
discharge diagnosis and a consensus agreement between the authors DV and MW. The
three diagnostic groups were dementia (n = 65), diagnosed according to DSM-IV
criteria for dementia [6], neurological disorders not presenting with dementia (n = 44) and
psychiatric illness (n = 156). The dementia patient group consisted of patients with
dementia of the Alzheimer's disease (AD, n = 18), diagnosed according to NINCDS–ADRDA
criteria [7];
frontotemporal dementia patients (FTD, n = 17), diagnosed according to consensus
criteria of Neary
Control subjects (n = 82) were recruited from relatives and carers of patients of neurology and neuropsychiatry outpatient clinics (n = 22), university students and staff (n = 20), and respondents to community advertisements (n = 40). Individuals with active or past psychiatric or neurological illness as detected by the Mini International Psychiatric Interview [11], physical and neurological examination were excluded.
Instruments
As part of the standard clinical assessment, all subjects were administered the NUCOG and the MMSE [12]. The patient assessments were undertaken by neuropsychiatry consultants, neuropsychiatry registrars or medical officers who had been trained in the use of the NUCOG by one of the senior authors (MW, DV) and control subjects were assessed by RS. A subset of the patient group (n = 22), in whom neuropsychological assessment was clinically indicated, underwent a thorough battery of neuropsychological tests including items from the Wechsler Adult Intelligence Scale – Third Edition [13], Wechsler Memory Scale – Revised [14], Trailmaking Test [15], Rey Figure [16], Controlled Oral Word Association Test [17], Stroop Test [18], and Rey Auditory Verbal Learning Test [19]. This was generally completed within a week of NUCOG assessment.
Statistical analysis
Demographic variables were compared using Kruskal–Wallis test for age and education, chi-squared tests for gender proportion, and oneway anova for test cognitive data. Criterion validity was performed with ROC (receiver operator characteristic) curves. The influence of demographic variables upon the NUCOG scores was calculated by using Pearson's correlation coefficient. Convergent validity was assessed using partial correlations between the NUCOG and neuropsychological test scores. Interrater reliability was assessed with the intraclass correlation coefficient, and internal consistency was measured with Cronbach's α. Comparison of the performance of the MMSE, NUCOG and its subscales across the sample was undertaken using repeated-measures anova, controlling for age and years of education, with within-subjects effects tested using the Greenhouse– Geisser test and post-hoc comparisons undertaken with the Games– Howell procedure. Statistical analyses were undertaken with SPSS 12.0 software for Windows (SPSS Inc., Chicago, IL, US).
Results
Table 2 summarizes the demographic characteristics of the population sample (age, gender and years of education), the performance of each group on the MMSE, NUCOG total score, subscale scores and time taken to complete the NUCOG.
Comparison of control and patient group on demographic variables, MMSE score, NUCOG total and sphere scores, and time taken to complete the NUCOG
All figures other than male gender are given in the format: mean (SD). †Kruskal–Wallis test, p < 0.001; ‡Pearson χ2 test, p < 0.05; §One-way ANOVA, p < 0.001. MMSE, the Mini-Mental State Examination; NUCOG, the Neuropsychiatry Unit Cognitive Assessment Tool; SD, standard deviation.
Demographic data
Age and years of education was significantly different (p < 0.001) across groups, with the psychiatric and neurological groups being the youngest and the dementia group the oldest, and the control group being more educated than the patient groups. There was an excess of women in the control group (p = 0.043), compared with a slight excess of men in all patient groups, which did not differ in gender balance (p = 0.179).
Time taken
The mean time to complete the NUCOG for the patient group as a whole was approximately 24 minutes, with significant differences across the patient groups (F3,347 = 33.85, p < 0.001).
Relationship between age, education and the NUCOG/MMSE scores
Control group
The MMSE scores significantly correlated with subject age (r = −0.346, p = 0.002) and years of education (r = 0.344, p = 0.002). The NUCOG scores showed a similarly strong correlation with age (r = −0.516, p < 0.001) and education (r = 0.552, p < 0.001).
Patient groups
The MMSE scores significantly correlated with age (r =−0.303, p < 0.001) and education (r = 0.261, p < 0.001) across the patient groups. There was a significant difference between the three patient groups (F3,347 = 33.24, p < 0.001) with the dementia group scoring lowest and the neurological and psychiatric groups scoring similarly. The NUCOG scores significantly correlated with age (r =−0.331, p < 0.001) and education (r = 0.387, p < 0.001) and differed significantly, but in the same fashion as the MMSE, across groups (F3,347 = 67.915, p < 0.001).
To determine if the MMSE and NUCOG performed differently, repeated-measures anova was undertaken in the control and patient groups. There was a strong interaction of test by diagnostic group in favour of the NUCOG (F3,1 = 65.272, p < 0.001), although its capacity to differentiate psychiatric from neurological patients was not significantly different to the MMSE (Games–Howell test, p = 0.851).
Dementia and psychiatric diagnostic groups
Dementia diagnostic groups
The NUCOG scores (F5,65 = 2.971, p = 0.019), but not the MMSE scores (F5,65 = 1.323, p = 0.270), differed across the dementia subtypes (Table 3). A repeated-measures anova revealed a significant test by diagnostic group interaction (F3,1 = 3.058, p = 0.017), with the NUCOG differentiating MCI from other forms of dementia. When the MCI group was removed, there was no significant test by group interaction (p = 0.177). When the dementia subgroups were limited to the FTD and senile dementia, Alzheimer's type (SDAT) groups, the NUCOG but not the MMSE was able to differentiate the groups (F1,1 = 5.720, p = 0.023).
Comparison of different diagnostic subtypes in the dementia and psychiatric patient groups on total MMSE and NUCOG scores
Data provided in the format: mean (SD). FTD, frontotemporal dementia; MCI, mild cognitive impairment; MMSE, the Mini-Mental State Examination; NUCOG, the Neuropsychiatry Unit Cognitive Assessment Tool; ODT, other dementia type; SDAT, senile dementia, Alzheimer's type; SD, standard deviation; VaD, vascular dementia.
Psychiatric diagnostic groups
There was no difference between different psychiatric subtypes in the MMSE scores or the NUCOG scores.
Performance of individual NUCOG scales
Repeated-measures anova identified a significant scale by main group effect (F4,3 = 14.086, p < 0.001). Visual inspection of the data suggests that all patients tend to score better on scale E (language) compared with other scales, whereas controls score similarly across all scales. A similar analysis of the dementia and psychiatric diagnostic groups revealed no scale by group effects.
Criterion validity
In the patient population, the capacity of the NUCOG to detect dementia was assessed
using receiver-operating characteristic (ROC) methods. The area under the ROC curve
in this population was significantly higher for the NUCOG than for the MMSE (0.912
compared with 0.885, p = 0.024), suggesting greater diagnostic performance for the
NUCOG over the MMSE (see Fig. 1). Taking a score of two standard deviations below the mean control
score (NUCOG = 83) as a cut-off, the sensitivity of the NUCOG for the detection of
dementia Receiver-operating characteristic (ROC) curve showing cut-off points for
NUCOG scores of 80 and 83. (–––) NUCOG score; (—) MMSE score; (······)
Reference line. NUCOG, the Neuropsychiatry Unit Cognitive Assessment
Tool.
Convergent validity
NUCOG and MMSE
The NUCOG and MMSE were strongly correlated within the entire patient group (τ = 0.689, two-tailed p < 0.001) and remained significant when age and years of completed education were controlled for (Pear-son's partial r = 0.894, p < 0.001). The correlation was strongest in the dementia-only group (τ = 0.762, p < 0.001), and weakest in the control group (τ = 0.437, p < 0.001).
NUCOG and neuropsychological measures
Correlations between the NUCOG scale scores and meaningfully selected subitems of the various neuropsychological tests were undertaken (Table 4). When neuropsychological tests were organized into the five general domains of the NUCOG, more than half correlated at a significance level of <0.01.
Relationship between neuropsychological test variables and NUCOG scale scores
COWAT, Controlled Oral Word Association Test; NUCOG, the Neuropsychiatry Unit Cognitive Assessment Tool; RAVLT, Rey Auditory Verbal Learning Test; WAIS-III, Wechsler Adult Intelligence Scale – Third Edition; WMS-R, Wechsler Memory Scale – Revised.
Reliability
Internal consistency of the NUCOG by Cronbach's α was 0.915. Interrater reliability was measured by the intraclass correlation coefficient on 10 consecutive subjects scored by two raters, and was 0.910.
Combination of the NUCOG scales
In routine use of the NUCOG clinically, it was noted that patients with dementia tended to score poorly on at least two scales. The number of scales scoring less than two standard deviations below the control group mean differed significantly between groups (one-way anova, p < 0.001) (see Table 3).
Discussion
We examined the validity, reliability and performance of the NUCOG in a large neuropsychiatric population and control group. The NUCOG scores were lowest in the dementia group but not significantly different in the psychiatric and neurological patient groups, although it showed a greater capacity than the MMSE to differentiate these groups. Unlike the MMSE, the NUCOG differentiated MCI from dementia, and showed a trend towards the differentiation of depression and psychosis. In a subanalysis, the NUCOG was significantly better in differentiating the FTD from SDAT patients, which could be expected given its executive subscale. This differentiation was one of the original aims of the ACE [4] and it appears that the NUCOG also shares this capacity, which would be expected as it contains even more items rating executive function than the ACE. The subscale by group effect in the four main subject groups appears attributable to the relative ease of the language subscale, and no clear subscale profile in these groups, or the dementia and psychiatric subgroups, emerged. However, the number of subscales scoring less than two standard deviations differed significantly between the patient groups, and in psychiatric subgroups a greater number of subscales scoring in this fashion were suggestive of a psychotic rather than affective disorder.
The performance of the NUCOG in this large but heterogeneous sample is notable, particularly considering the atypical nature of the dementia population seen in the unit. The ability of the NUCOG to differentiate a dementia population (with a mean MMSE of <20) from other neurological and psychiatric populations suggests it is sensitive to early cognitive change and may be useful in other settings where early presentations of dementing illness are likely to be encountered. Of this population, 17 patients scored 24 or higher on the MMSE. Of these, only 6/17 scored higher than the 83-point cut-off for the NUCOG, suggesting it is more likely to correctly classify this group as impaired. For this select group of patients, however, a brief cognitive screening test like the NUCOG alone is not sufficient to characterize cognitive impairment and more sensitive neuropsychological tests are indicated.
Many brief cognitive instruments, including the MMSE, have been developed to assist with the differentiation of ‘organic’ or neurologic from ‘functional psy chiatric’ illness, and most have shown limited ability to achieve this satisfactorily. In this population, it was notable that patients with psychiatric disorders generally scored similarly on the NUCOG to patients with a range of neurological disorders. Rather than indicating a weakness of the NUCOG and other instruments, this could be seen to be more indicative of the fact that a number of psychiatric disorders such as psychosis and major affective disturbance are often associated with subtle but significant impairments of cognitive function. In larger samples, tools such as the ACE have demonstrated the capacity to more reliably differentiate psychiatric disorders from other early organic disease [20] and it may be that larger and more homogeneous groups are required to demonstrate differentiation between these groups using total NUCOG or subscale scores.
There are several limitations of this study which may limit its generalizability to other populations. The patients seen in the Neuropsychiatry Unit are generally referred for tertiary opinion and as such have complex presentations and symptoms, and may carry multiple psychiatric, medical and neurological diagnoses. The discrete diagnostic categories used for the purposes of statistical analysis oversimplify the diagnoses in individual patients. The diagnosis given to patients for the purposes of statistical comparisons was that diagnosis considered to be the primary diagnosis for that patient. For example, if a patient with FTD also had a depressive syndrome then the primary diagnosis of FTD was made. A further limitation of this study is the clinical nature of the assessments and the contribution of performance on the NUCOG to the diagnostic process, that is, clinicians were not blind to the NUCOG or MMSE score when deciding on a diagnosis, and on a minority of assessments the diagnostician also completed the NUCOG.
Additionally, while a number of individuals with progressive cognitive decline have been rated over time with the NUCOG, this was not formally assessed during this study, so we cannot comment on its capacity to detect change. Finally, the 22 patients who had been administered neuropsychological testing did not all undergo the same battery of neuropsychological tests as the assessments were clinical in nature. As not every patient had had the same individual neuropsychological tests, the NUCOG domains were compared with individual neuropsychological tests, rather than composite scores of neuropsychological tests relevant to each domain.
The NUCOG has been conceptualized and developed as a tool which slots into the routine clinical assessment of patients and which contributes to the diagnostic process, rather than a diagnostic tool in itself (Box). Despite a wealth of attempts to develop a quick diagnostic cognitive tool, the reality remains that clinical diagnosis must be based on a multiple sources of evidence from the history, the mental state examination, the physical examination and the results of investigations.
Footnotes
Acknowledgements
The authors kindly thank North Western Mental Health for their support of this project, and Stephen Wood for his guidance on statistical issues.