INTRODUCTION
While screening for cognitive impairment is advocated by some [1–3], there remains limited evidence supporting routine cognitive screening in clinical practice [4, 5]. This is in part because most cognitive screening instruments (CSI) lack sufficient sensitivity and specificity to differentiate normal cognition from mild cognitive impairment (MCI) and dementia [2, 6], but also because of the effects of age and education on scoring variability [7, 8]. Misclassifying may be harmful, as treatments indicated at one stage may not beneficial at another [9–11].
The utility of screening instruments is defined by two conditional probabilities, sensitivity and specificity, that rule in (sensitivity) or rule out (specificity) disease [12]. In general, sensitivity and specificity are inversely related and vary with the threshold [13]. Ideally, screening tests should strike a balance between both, so they can confirm or exclude a disease, although sensitivity is particularly important for short screens to reduce the rate of false negatives. Cut-off scores, transition points along the spectrum of cognitive impairment, may be useful in busy clinical practice although their utility is limited by context and uncertainty regarding the optimal method of establishing test cut points [14]. Two of the most widely used methods are the maximal accuracy [15] and Youden’s Index [16] approaches. The latter often provides higher cut-offs and greater sensitivity, and has been suggested as the optimal method for developing cut-offs for short CSIs [15]. Although age and educational cut-offs, based upon population-based norms, are available for some CSIs including the Mini-Mental State Examination (MMSE) [7] and the Montreal Cognitive Assessment (MoCA) [17, 18], few studies have published usable cut-off scores for patients presenting with symptomatic memoryloss [19].
The objectives of this study are to (1) examine and compare the effects of using different methods of defining cut-off scores on two short CSIs, the widely used MMSE (Standardized MMSE), and a short screen for MCI and early dementia called the Quick Mild Cognitive Impairment (Qmci) screen [20]; (2) to investigate the extent that these cut-off scores require adjustment for age and education; and (3) develop cut-off scores for the Qmci screen.
MATERIALS AND METHODS
Participants and data collation
Data from three sources: the Geriatric Assessment Tool (GAT) [21] and original Qmci validation databases [20], and the DARAD (Doxycycline And Rifampicin for Alzheimer’s Disease) [22] clinical trial database were analyzed. Participant characteristics and recruitment from all three studies have been published previously [20–22] and are presented in Fig. 1 and Table 1 below. In summary, the GAT database was collected in outpatient geriatric clinics in two university hospitals in Ontario, Canada between 1999 and 2010. The GAT is a geriatric outpatient database including patients aged 41–104 years [21]. The Qmci validation database includes patients referred for assessment of cognition, aged ≥55 years, recruited from four outpatient memory clinics in Ontario, Canada [20]. The DARAD was a multi-center, blinded randomized controlled trial conducted between 2006 and 2010, comparing the effect of rifampicin and doxycycline to placebo on the progression of Alzheimer’s disease (AD) [22]. The DARAD database includes patients aged ≥50 years with mild to moderate AD recruited from 14 Canadian geriatric outpatient clinics.
All data were collected under the supervision of the same principle investigator (WM) in a manner consistent with the Helsinki Declaration of 1975 and were approved by the local research ethics committee at the time. No additional approval was required for this secondary analysis. Each participant underwent similar comprehensive work-up including laboratory investigations, neuropsychological assessment, and neuroimaging where appropriate [20–22]. Dementia was diagnosed using the Diagnostic and Statistical Manual of Mental Disorders (4th-edition) [23]. MCI was diagnosed using Petersen’s Criteria [24] in patients presenting with subjective and objective memory loss without loss of function. Those with active depression, screened using the Geriatric Depression Scale [25], were excluded. All participants were English literate. Persons attending clinic with patients, mainly caregivers (Qmci database, n = 630), and patients without memory loss attending geriatric clinics (GAT database, n = 245), recruited by convenience sampling, were available as normal controls. No controls were available from the DARAD database.
Participants were included in this analysis if paired SMMSE and Qmci scores were available. To ensure a consistent cross sectional approach and to reduce heterogeneity between the different samples, only patients with AD, vascular dementia (VaD), or mixed AD-VaD were included. This is consistent with the original validation of the Qmci screen [20] and recruitment for the DARAD trial [22]. Patients with missing demographic data (age or education) were also excluded. In total, 4,565 subjects were available. Of these, 3,387 were included. The majority (n = 1,081) were excluded from the GAT database: 46 with Lewy body dementia, 15 with Parkinson’s disease dementia, 33 with frontotemporal dementia, 70 with other dementia subtypes (post-traumatic, anesthetic, alcohol, or other mixed non AD-VaD), and 571 with depression (195 with isolated depression, 94 with concomitant MCI, and the remainder with active co-morbid depression and established dementia). For 346 subjects, the diagnosis was unclear. A further 97 were excluded from the Qmci and DARAD data sets due to missing data. Selection for this analysis is presented as a flow chart in Fig. 1.
Measures
The SMMSE and Qmci were available in all three databases. The SMMSE is a standardizedversion of the MMSE, scored out of 30 points, which improved inter-rater reliability by the inclusion of explicit administration and scoring guidelines [26, 27]. The Qmci screen is a short (median time 4.24 minutes) [28], valid, and reliable CSI with six subtests, covering five cognitive domains: orientation, working memory, sematic memory (verbal fluency for animals), visuospatial/executive function (clock drawing) and two tests of episodic memory (delayed recall and logical memory, the immediate verbal recall of a short story) [28]. Originallyvalidated in English, it has now been validated in Dutch [29] and Turkish [30]. It is scored from 0 (indicating severe cognitive impairment) to 100 points (normal cognition). It has comparable accuracy to the MoCA [31, 32] and similar responsiveness to the standardized Alzheimer‘s Disease AssessmentScale-cognitive section, Clinical Dementia Rating scale, and Lawton-Brody Activities of Daily Living scale in clinical trials [33]. Alternative, validated forms of the Qmci subtests were used to minimise learning effects [34].
Analysis
Pooled data were grouped according to age (≤75 or >75 years) and education (<12 or ≥12 years), to create four subgroups. An age cut-off of 75 years provided the best balance in numbers between the four subgroups and represents the accepted age cut-off for cognitive impairment using the MMSE [35], above which it is suggested that scores must be adjusted to account for age. The cut-off for education was 12 years, the average length of formal schooling in North America [36].
Optimal cut-off scores were estimated using two methods: Youden’s Index [16] and maximal accuracy calculated from receiver operating characteristic (ROC) curves. The cut-point derived by Youden’s Index optimizes a tests ability to differentiate when equal weight is given to sensitivity and specificity. It is defined mathematically as: J = Sensitivity + Specificity –1 [16]. Diagnostic accuracy was calculated from the area under the curve (AUC) of ROC curves with scores >0.8 regarded as good and >0.9 as excellent. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for candidate cut-off points. The maximal accuracy method is based on maximizing the overall classification accuracy (OCA) such that OCA = true positives + true negatives. Likelihood ratios for the probability of having cognitive impairment (MCI and dementia) were also determined. A Positive Likelihood Ratio (PLR) of between 2–5 suggests a small chance, 5–10 moderate, and >10 a large chance of having cognitive impairment. A Negative Likelihood Ratio (NLR) of 0.2–0.5 suggests a small chance, 0.1–0.2 moderate, and <0.1 a large chance of not having the condition. The Mann–Whitney U test compared non-parametric data and student’s t-tests compared parametric data. The Kruskal-Wallis H test was used for comparisons between three or more non-normally distributed groups. ROC curves were compared using the Hanley method [37]. Binary logistic regression was used to adjust ROC curve analysis for the effects of age and education.
RESULTS
Characteristics
In all, 3,387-paired assessments from 2,074 individuals were included. The median age of all participants at assessment was 75 years, interquartile range (IQR) ±13.5. The median years in education were 12 (±5) and 46.9% were male. The median age of those with dementia was 78 (±8) years, older than normal controls 70 (±14) (p < 0.001) and those with MCI, 76 (±10) (p < 0.001). The median number of years in education was 12 (±5) for those with dementia compared with 13 (±4) (p < 0.001) for controls and 12 (±4) (p = 0.46) for MCI. The characteristics of those included in each database and comparisons between databases are presented in Table 1. The characteristics of participants according to their age and educational level are presented in Supplementary Table 1. In summary, the median SMMSE score for controls was 29/30 (±2) points, compared to 28/30 (±3) for MCI and 23/30 (±8) for dementia. SMMSE scores were significantly different between these three diagnostic groups, χ2 (2)= 1,262 (p < 0.001). The median Qmci scores were 74/100 (±15) points for controls, 57/100 (±20) for MCI, and 37/100 (±26) for dementia. Qmci scores were also significantly different in the three groups, χ2 (2)= 1,379 (p < 0.001). Median age, gender, education, SMMSE, and Qmci screen scores, though similar, were significantly different between the three databases; participants in the dementia trial database (DARAD) were, as would be expected by the results above, older, had spent less time in formal education and had lower median SMMSE and Qmci scores.
Effects of age and education on median cognitive screen scores
Median SMMSE scores were significantly higher for normal controls than for patients with MCI, irrespective of age or education, though clinically the results appeared similar, particularly for older participants (>75) with less time in education (<12) both with a median SMMSE score of 28/30, reflecting the known ceiling effects of the instrument.Differences between median Qmci scores for controls and those with MCI were all statistically significantly different, irrespective of age and education but the differences were more clinically meaningful (minimum difference >10 points), for example the median Qmci screen score for younger patients (≤75) with more education (≥12) was 78/100 and 59/100 for normal controls and MCI respectively, a median 19-point difference. Median SMMSE and Qmci screen scores for patients with established dementia were the lowest for younger patients with less education. These results are presented in Supplementary Table 1.
Cut-off scores using the maximal accuracy method
Both instruments had good to excellent accuracy in differentiating patients with cognitive impairment (MCI or dementia) from controls. Scores for both instruments were highest for younger (≤75), more educated (>12) participants. The Qmci was, however, significantly more accurate than the SMMSE in identifying cognitive impairment, with an AUC score of >0.90, irrespective of participant age or education (p < 0.001). Its superior accuracy was particularly evident in younger (≤75) and more educated (>12) patients (AUC of 0.84 for the SMMSE versus 0.92 for the Qmci, p < 0.001). As expected the Qmci was also statistically significantly more accurate than the SMMSE in detecting MCI, irrespective of age or education, while there was no difference for differentiating dementia from MCI. The optimal SMMSE cut-off for cognitive impairment, using the maximal accuracy approach was <28/30 (AUC 0.86), which produced a sensitivity of 74% and specificity of 88%. The optimal Qmci cut-off was <63/100 (AUC 0.93), which produced a sensitivity of 85% and specificity of 85%. Comparisons between AUC scores and the cut-off scores generated using the maximal accuracy method, for both instruments and according to their age and education, are presented in Table 2. The sensitivity and specificity obtained at different cut-off points including that found using the maximal accuracy approach are presented in Supplementary Table 2.
Cut-off scores using Youden’s Index
Using Youden’s Index, the optimal cut-off score for the SMMSE for cognitive impairment (MCI or dementia) was <28/30. This provided a sensitivity of 74% and a specificity 88%. The optimal cut-off for the Qmci was <62/100, producing a sensitivity of 83% and specificity of 87% for cognitive impairment. Examining age and education specific cut-offs again showed that scores for both instruments were highest for younger (≤75), more educated (>12) subjects. The Qmci was more sensitive (83%) and specific (73%) for MCI than the SMMSE (59% and 72%, respectively). Cut-offs for the Qmci screen calculated using Youden’s Index are presented in Fig. 2. Detailed tables comparing the sensitivity and specificity of both CSIs, for a range of cut-off scores, calculated from Youden’s Index are provided in Supplementary Table 3.
Overall, SMMSE cut-offs were similar whether using Youden’s Index or the maximal accuracy approach, while Qmci cut-offs varied depending upon the method used and by the age and education defined subgroups. These comparisons are presented in Table 3.
Likelihood ratios
Likelihood ratios were calculated for several cut-off scores, for both instruments, for the total population and by age and education. The optimal SMMSE cut-off for cognitive impairment (<28/30), calculated using Youden’s Index, provided a moderate chance of cognitive impairment (PLR 5.97) and a small chance of normal cognition (NLR 0.29). The cut-off using the maximal accuracy method was the same. Likelihood ratios also confirmed the choice of cut-off scores for each subgroup according to age and education. For example, using Youden’s Index or the maximal accuracy approach, the cut-off for older participants (>75) with less education (<12) (i.e.,<27/30) had a moderate chance of cognitive impairment (PLR 5.25) and a small chance of being classified as normal (NLR 0.29).
At the optimal Qmci cut-off for cognitive impairment, calculated using Youden’s Index (<62/100), participants also had a moderate chance of having cognitive impairment (PLR 6.22) and a small chance of having normal cognition (NLR 0.20). At <63/100 points, the cut-off calculated by the maximal accuracy approach, there was also a moderate chance of having CI (PLR 5.65) but a moderate chance of being normal (NLR 0.18). Complete results for likelihood ratios according to diagnostic classification (normal cognition, MCI or dementia) and age (≤75 or >75 years) and education (<12 or≥12 years) are presented in the Supplementary Table 4.
DISCUSSION
This paper presents cut-off scores for two short CSIs, the SMMSE and Qmci screen, adjusted for age and education, based upon a large population of participants including patients with symptomatic memory loss, in three pooled databases, using two derivation methods. While data on cut-off scores have been partially presented for the MoCA [19], to our knowledge, this is the first study to investigate in depth the differential effects of age and education on cut-off scores, comparing two short CSIs in participants with symptomatic cognitive impairment, one designed specifically to screen for MCI (Qmci). It is also one of only a few studies to investigate the effects of different methods of calculating cut-offs [15], despite the regularity that they are used in clinical practice.
The results show small, albeit significant, differences in median SMMSE, and large and significant differences in median Qmci scores, which clearly separate normal controls, MCI, and dementia. This study found that cut-off scores are influenced by several factors including their derivation method as well as age, education and diagnostic classification, and that these effects also vary between instruments. Comparing cut-off scores calculated with Youden’s Index and the maximal accuracy method showed that theoverall scores for the SMMSE and Qmci were similar irrespective of diagnostic classification. For example, the optimal cut-offs for cognitive impairment (i.e., either MCI or dementia) were <28/30 for the SMMSE and <62/100 for the Qmci using Youden’s Index compared with <28/30 and <63/100, respectively, using the maximal accuracy approach, more or less unchanged. Likelihood ratios, however, suggested that the cut-off scores created using Youden’s Index were better able, albeit marginally, to classify patients with confidence.
The study also showed that age and education affect the cut-off scores for all diagnostic categories with differences evident between scores depending on the cut-off method used. This was more obvious for the Qmci. For example, the optimal SMMSE cut-off score for detecting cognitive impairment was <28/30 for young participants (≤75) with more education (≥12) using Youden’s Index and <29/30 using maximal accuracy, versus <60/100 and <67/100 respectively for the Qmci screen. This suggests that it requires greater correction for age and education than the SMMSE, which appeared to be less influenced by age and education than the Qmci screen.
The effects of age and education on patients presenting with symptomatic memory complaints are complex. The median scores of the total sample were lowest for older patients with less education and highest for younger patients with the most education. Median scores for patients with established dementia were, however, lowest for younger patients with less education, possibly reflecting more severe pathology in young patients, without the protective effects of education. The influence of education on cognition remains unclear [38] with some but not all studies suggesting that lower levels of education are associated with an elevated risk for dementia [38]. Recent longitudinal data suggests education may delay onset but does not affect the rate of decline in those with established dementia [39]. Likewise, education does not clearly affect decline in cognitive scores for persons without dementia [40].
Comparing the diagnostic accuracy of both CSIs showed again that the Qmci was significantly more accurate in differentiating MCI from normal cognition [20, 32], while both instruments had similar accuracy for distinguishing dementia. This reflects that the MMSE and its standardized version are poorly sensitive to early cognitive changes, particularly in patients with MCI [42]. Likewise, irrespective of age, education, or method of derivation, all but one of the SMMSE cut-offs (differentiating MCI from dementia in young patients with lower levels of education) were above the often-used cut-off of <24/30 [27]. The cut-off for older patients with high levels of education found in this study (<26/30) is, however, similar to that found in corresponding samples in North America [43]. For patients with MCI, the cut-off ranged between <29/30 and <30/30 suggesting that any failure to fully complete the test should prompt more detailed assessment.
Thus, the SMMSE, while less accurate, requires less adjustment for age, education, and derivation method than the more accurate Qmci screen, which is an advantage. However, it could be argued that the narrow range of SMMSE cut-off scores, clustered together, limits their utility in clinical practice, confirming the ceiling effects characteristic of this instrument [7, 41]. The argument is therefore that clinicians should have a suite of short CSIs available (i.e., be trained to score) and make their selection of which to use based on the diagnostic pre-test probability, with MCI specific screens used where MCI is suspected. Given that MoCA cut-offs for symptomatic cognitive impairment also require adjustment for age and education [19], suggests that there may be a trade-off between diagnostic accuracy and a requirement to adjust cognitive screen scores for all these factors. Being aware of this is particularly important in order to avoid misclassification in busy clinical practice.
Strengths and weaknesses
Strengths of this study include the inclusion of a large sample of patients presenting with memory loss in a clinic setting, allowing analysis of subgroups according to age and education. Another strength is that cut-offs were confirmed as being clinically meaningful using likelihood ratios. The study includes patients with different dementia subtypes all of whom underwent a comprehensive assessment using standardized diagnostic criteria across multiple sites. This paper has a number of limitations. This study included pooled data from three sources, a clinical trial and two outpatient databases, potentially resulting in spectrum bias. Baseline characteristics were significantly different between the three data sets, although the same principle investigator, using similar approaches, supervised participant assessment. The findings are drawn from a single country, potentially reducing the generalizability of results. However, Canada is ethnically diverse with similar demographics to many other industrialized nations and data were drawn from multiple clinics spread over a large geographic area. The age and educational level cut-offs chosen could also limit the external validity of these results. Although 12 years is the average number of years of education for older adults in North America, this figure is lower in other countries. Another potential limitation is that the efficiency of the maximal accuracy method, used to develop cut-offs, varies with disease prevalence.
In summary, this study provides usable cut-off scores for the SMMSE and Qmci and illustrates the effects of age and education on cut-off scores for patients presenting with symptomatic memory loss. The results suggest that the type of test selected also determines the extent to which adjustments in age and education are required. The SMMSE is less accurate than the Qmci but may also be less affected by age and education. The results also suggest that cut-offs are influenced by the method of derivation, which varies by age, education, and between instruments. Based upon this analysis, the authors recommend the use of cut-offs derived by Youden’s Index (presented in Fig. 1): an overall SMMSE cut-off score of <28/30 for cognitive impairment (MCI and dementia) and a Qmci cut-off of <62/100. However, although cut-offs are useful in clinical practice, age and education impact upon these and while providing a valuable guide, they should not replace clinical judgment [44]. Further study is required to confirm these findings in different cultural groups and with different cognitive tests.
