Abstract
Background: Traditional evaluation of mild cognitive impairment (MCI) can be costly, time consuming, and impractical for widespread screening. DETECT is a portable device developed to rapidly perform cognitive testing in diverse settings. This study compares DETECT with formal clinical assessment.
Introduction
Mild cognitive impairment (MCI) is an intermediate stage between normal aging and dementia. 1 –5 Dementia, whether due to Alzheimer’s disease (AD) or other causes, is a pathological condition that increases the morbidity and mortality of its victims and negatively impacts their families and society. Patients with amnestic MCI often progress to AD with an annual rate of conversion at approximately 10 times that of the normal elderly population. 1,5
In 2006, 37.3 million or 12.4% of Americans were 65 years of age or older. This number is expected to double by 2030 to about 71.5 million, accounting for 20% of the US population. 6 As this number grows, cases of MCI, AD, and other forms of dementia will rise. 7 In 2000, there were approximately 2.2 to 4.8 million cases of AD in the United States alone. Experts expect that this total will increase 4-fold by 2050. 8 The global burden of AD is even more striking; over 26.6 million people worldwide are currently affected by this condition. 9 The number of persons with MCI is thought to be double or triple this figure. 10
Mild cognitive impairment often goes undetected. As a result, patients are frequently not identified until significant loss of cognitive function has occurred or they have progressed to frank dementia. 11,12 In one study, up to 63% of patients in an assisted living facility had undiagnosed dementia. 12 In another study, primary care physicians in a geriatric clinic did not recognize dementia in 67% of their patients. 13
Few diseases produce as dramatic an impact on the patient, his or her family, and society as AD. The average family caregiver of a patient with AD spends 16.1 hours a week providing care. 8 The economic toll of AD exceeds $100 billion annually in the United States—more than heart disease, cancer, and stroke. 14 The cost of care increases dramatically when an individual is placed in assisted living or a skilled nursing facility. 8
Many professional organizations advocate early screening to detect the first signs of AD. In the Alzheimer’s Foundation of America (AFA) 2008 report, Memory Matters, states, “the screening of at-risk populations for dementia should become a cornerstone for early treatment or prevention of cognitive decline.” Interventions that can delay symptom progression and defer the need for long-term care services (LTC) are a major focus of research. 11,15 Early diagnosis permits the best opportunity to understand disease progression, begin long-term planning, and ultimately implement a treatment regimen.
The adoption of new screening tools into clinical care environments is lagging. Studies have sited barriers to screening such as inadequate staff time to administer screening tests, the need for dedicated space, lack of training and other resources, and lingering concerns over the accuracy of these tests. 16,17
DETECT is a simple, self-contained device that administers a battery of cognitive tests in 7 to 10 minutes. In an earlier study, 18 DETECT accurately identified patients with cognitive impairment (MCI) and discriminated between MCI and dementia. The primary objectives of this current study are 2-fold—to determine the degree of agreement between DETECT subtests and formal neuropsychological testing and to use these results to formulate a predictive model capable of generating estimated probabilities of impairment in an at-risk elderly population.
Methods
Participants and Sample
Participants were recruited from patients of the Wesley Woods Primary Care Clinic in Atlanta, Georgia and other local retirement facilities. To be eligible, a participant had to be ≥65 years of age without a previous diagnosis of MCI or dementia and be physically capable of testing on both the DETECT and neuropsychologist’s assessment (NPA). Participants were prescreened with the Mini-Mental State Examination (MMSE) and/or Mini-Cog Assessment tool to reduce the pool of severely impaired patients. 26,27 Therefore, participants with a score <18 on the MMSE, or <4 on the Mini-Cog score were excluded. Patients with significant visual impairments (blindness or noncorrectable vision loss), severe hearing impairments (unable to hear even with a hearing aid), physical impairments (not able to push the buttons on the controller), claustrophobia, non-English speaking, illiterate, unable to provide informed consent, or had an acute medical illness that warranted immediate attention were excluded. The Emory University’s Institutional Review Board approved the study protocol.
Measurements
Each participant was first given the DETECT test in a familiar setting (ie, clinic, retirement facility, etc). DETECT took an average of 10 minutes to administer (including instructions). Following automated assessment with the DETECT device, each participant was scheduled for a second assessment, within 1 month of the DETECT testing, to complete the standard battery of neuropsychological (NP) tests. The NP battery was scored by a professional neuropsychologist who was blinded to each participant’s DETECT score.
The DETECT device was developed by modifying the elements of a battery of NP tests that, in their full form, have been previously used in MCI (ie, Simple & Complex Choice Reaction Time, 19,20 Selective Reminding Memory Test, 21 Go/No-Go Executive function, 22,23 and N-back Working Memory 24 ). The heads up display and noise-canceling headphones create a portable immersive environment for the testing (see Figure 1). The other components include a dedicated computer with the DETECT software and a handheld input unit with 2 buttons. Depending on the test, the buttons signal “Yes” and “No” or “Right” and “Left.” The software automatically administers a short series of four NP tests that evaluate information processing speed, episodic memory, working memory, and executive function. Results from each patient are stored in a database for later analysis. A description of the DETECT subtests is contained in Table 1.
The DETECT system.
NP Testing Incorporated in DETECT Software
The formal NPA included a 90-minute neuropsychological test battery (NP), 10-item Functional Assessment Questionnaire (FAQ), 25 and an expert neuropsychologist’s judgment. This NPA constituted the reference or “gold” standard. The NP battery chosen is well validated, routinely administered at our institution, and widely employed by other experts. The NP tests are manually administered using paper and pencil. The results are converted to z scores using published norms. A summary of the tests used in the NP battery is found in Table 2.
Neuropsychological Battery of Tests Administered to Each Patient
Using the NPA, participants were diagnosed as normal, MCI, or demented. The neuropsychologist classified a participant as “normal” if no cognitive scores were less than 1.5 standard deviations (SDs) below the mean, and the participant performed instrumental activities of daily living (IADLs) without assistance. Mild cognitive impairment was defined as cognitive scores more than 1.5 SDs below the mean, but a participant was able to perform the IADLs. Mild cognitive impairment was further subdivided into 2 categories, possible MCI (only 1 score within the cognitive domain was impaired—such as memory or language) and probable MCI (when 2 or more scores, either within a domain or across domains, were impaired). The category of dementia was defined as 3 or more cognitive test scores more than 1.5 SDs below the mean and participant requiring assistance with IADLs.
Scoring of the DETECT tests fall into three categories—accuracy, response time, and nonresponse (hereafter called time-outs). When measuring accuracy, 4 outcomes are possible with every click of the device’s handheld unit—true positive (correct), false positive, (incorrect), true negative (correct), and false negative (incorrect). In each round of each DETECT test, the accuracy percentage was tallied as a summary measure.
For response time, participants were instructed to answer each question as rapidly as possible. For each prompt inside each test, a response time was measured. Participants must respond to each question/prompt within 2.5 to 3 seconds (depending on the specific test). When a response was not registered, the system automatically moved to the next prompt/question. Nonresponses are recorded as time-outs. In each test/round, the percentage of responses resulting in a time-out was calculated. For each DETECT test/round, the median, first and third quartiles of response time were calculated, as well as the SD of response time.
Analysis
Each of the DETECT summary measures described above were examined to determine how strongly they were associated with NPA. For each DETECT test, separate univariate logistic regression models were constructed that related the participant’s impairment as a function of the summary measures (accuracy, response time, and time-outs). Tests of significance of the beta parameter associated with each DETECT test summary measure were performed and the associated P values were calculated. The c-index (similar to a receiver–operating characteristic [ROC]) was calculated as a measure of discrimination, that is, how well the summary measurement discriminated between those who are versus those who are not impaired.
We sought to develop a multivariable predictive ordinal regression model that would use DETECT summary measures to generate the estimated probabilities of each level of impairment (normal/no impairment, possible MCI, probable MCI, and definite impairment). We employed 3-model selection procedures—forward, backward, and stepwise selection—to develop these predictive models. Each model was then evaluated on the basis of discrimination (c-index) and model calibration—the degree of agreement between observed and expected probabilities—was gauged via the Hosmer-Lemeshow (H-L) statistic.
A comparison between the relative predictive abilities of DETECT and MMSE was also performed. Two different classifications of the outcome were considered. For the first classification, all patients were classified as either demented or not demented. Second, the 4-level ordinal outcome (normal/no impairment, possible MCI, probable MCI, and definite impairment) was compared between MMSE and DETECT. For each outcome, the respective c-indices of the scores were calculated and tested for differences.
A 10-fold cross-validation approach was used to measure the degree to which subsamples of the study could predict MCI in other subsamples. This is an internal validation algorithm designed to test the performance of our model in subsets of the data set that did not contribute to the formulation of the model itself. The validation consists of 10 steps. First, the participants were randomly divided into 10 approximately equal groups of 40 patients each. Second, Groups 1 to 9 were used to calculate the model parameters of the model we chose to validate. Third, we used these estimated model parameters to calculate the predicted probabilities of impairment for the participants in group 10, the held-out group. Fourth, we repeated this process 9 more times, each time holding out a different group and applying model estimates from the other groups to obtain predicted probabilities. In the end, all patients have predicted probabilities of impairment that were computed from other cases, thus avoiding the “contamination” that can come with estimating probabilities for a participant that are functions of that participant’s own data. Once probabilities are obtained for each participant, the c-index was calculated and compared to the c-index of the same model calculated on all of the patients. This validation exercise was primarily designed to avoid model “overfitting,” that is, having a model that is too large or too specific to one group of patients to be widely applicable to other cases.
Results
Of the 425 participants enrolled in the study, 2 could not complete DETECT due to anxiety during testing and 18 did not return for NP testing. The remaining 405 participants (95%) completed both forms of assessment and are included in our analysis. Participants determined to have normal cognition through formal NP testing were slightly younger (76.3 vs 79.4), better educated (16.2 vs 14.9 years), and disproportionately Caucasian (90.7% vs 75.0%) compared to their nonnormal peers (P < .001). These demographic variables were considered as possible confounders/predictors in our multivariable model. There was no association between gender and cognitive impairment (P = .33).
The accuracy of each DETECT test is shown in Table 3. The discrimination is highest for the selected reminding first round (c-index = 0.760) and delayed recall tests (c-index = 0.769) which are essentially the same test given at different points during the DETECT administration. Response time and time-out statistics were examined to assess their association with the clinical diagnosis (Tables 4). The c-indices for the NP measures were generally in the moderate-to-low moderate range for both response time and time-outs. The selective reminding test had the highest c-index for both response time (0.728) and time-outs (0.723).
Estimates of Accuracy of DETECT Subtests, Mean (SD)
Abbreviations: MCI, mild cognitive impairment; SD, standard deviation.
aComparisons between normal and nonnormal patients are statistically significant at the α < .001 for all DETECT variables
Estimates of Median Response Time of DETECT Subtests, Mean (SD) a
Abbreviations: MCI, mild cognitive impairment; SD, standard deviation.
a Comparisons between normal and nonnormal patients are statistically significant at the α <.01 for all DETECT tests.
Using model selection procedures (forward, backward, and stepwise), 1 common model was recommended from each algorithm, a model containing 7 of the DETECT test summary measures. This model provided good discrimination (c = 0.85) of the 4-level diagnosis, and excellent discrimination for nondemented versus dementia (c = .99). In addition, the DETECT model was capable of discriminating between the 2 subgroups of patients with MCI having varying severity (possible vs probable) as evidenced in both univariate (Tables 3-4) and multivariable analyses of the predicted probability of any impairment (see Figure 2). This multivariable model showed acceptable calibration (P = .8428, which means we fail to reject the H-L null hypothesis of agreement between observed and predicted probabilities).
Box plots of DETECT-generated probabilities of impairment by neuropsychological diagnosis.
When comparing DETECT and MMSE (perhaps the most widely used NP screening test) DETECT showed superior discrimination for both outcome types with respect to the dementia (yes or no). The DETECT c-index (0.995) was significantly higher than the MMSE c-index (0.901). Similarly, for the 4-level ordinal outcome, the c-index for DETECT (0.85) was significantly higher than the MMSE c-index (0.739). Both comparisons were statistically significant (P < .001).
To assess the internal validity of DETECT 7-term multivariable model, we employed the 10-fold cross-validation procedure, in which various subsamples of the study population were used to predict impairment in other subsamples. The model showed itself to be internally valid with a c-index of 0.837, only slightly (and expectedly) smaller than the c-index calculated from the entire sample (0.850).
Discussion
The DETECT system demonstrated good correlation with our reference standard NPA and was able to correctly stratify participants into 4 categories of cognition: normal, possible MCI, probable MCI, and dementia. The tool was able to discriminate between normal and demented in 99% of the cases. There was also a strong correlation for the MCI group (ROC .85).
Patients with known and probable dementia were excluded to avoid artificially increasing the utility of the test. To minimize the training effects, a delay of 1 to 4 weeks was placed between testing with the device and the subsequent NPA assessment. This allowed any learned behavior to “washout” before the reference standard test was administered. The neuropsychologist who performed the NPA evaluation and scored the pencil-and-paper tests was blinded to the results of the DETECT assessments. High rates of follow-up were achieved, minimizing the potential for nonresponse bias.
The diagnosis of MCI, a frequent precursor to AD, is difficult to make in the best of circumstances. Even in our study, 32 patients (8%) who were classified as “demented” by NPA were not previously known to be impaired. In addition, these patients all passed prestudy screening (MMSE >17 or Mini-Cog test >3). This observation underscores the difficulty of identifying dementia, much less MCI. Unfortunately, clinicians rarely have the time, training, or suitable facilities to administer a standard battery of NP tests. The cost of performing a formal NP screening on every patient >65 years of age would be prohibitive. As a result, many patients are misdiagnosed or undiagnosed until their symptoms are quite advanced.
The most commonly used screening instrument for dementia, the MMSE, 26 has been available for decades, is inexpensive, and relatively easy to administer. However, it has not been widely adopted for screening. The limitations of the MMSE include relatively poor discrimination, especially for milder forms of cognitive impairment, the time required for health care staff to perform the test, and the need for an appropriate setting to administer. 27 –29 Other brief screens, like the Mini-Cog, may be more sensitive than the MMSE 30 but provide limited information about the type of cognitive deficit and are also poor at detecting milder forms of the condition. Newer testing paradigms that incorporate computerized NP batteries have been developed and have the advantage of providing greater detail about the cognitive domains affected and can measure reaction time, an important factor impacted by cognitive impairment. 27-28, 31–34 However, these and other currently available tests still require a conducive environment (free from external distractions) in order to ensure reliability.
Limitations
The NPA composite (NP battery, FAQ, and neuropsychologist judgment) was used as the reference standard in this study. Although fairly comprehensive, our NPA did not include all the elements of a complete dementia workup. Participants did not undergo a full clinical evaluation to rule out other potential organic causes for dementia or cognitive impairment. The NP battery we employed was restricted to a selection of tests considered to be most important by our expert neuropsychologist but is consistent with what most experts and institutions use. The participants in this study were drawn from diverse settings (retirement homes, geriatrics clinics), but the demographics of the study population were fairly uniform. Therefore, the generalizability of our findings to minority populations and seniors living in other parts of the country is unknown. The adoption of new screening tools into clinical care environments must be done with caution. Misdiagnosing an individual as impaired could provoke needless anxiety for a patient and his or her family. 29 As a result, it is important to carefully study a test’s accuracy, reliability, and precision before disseminating it to clinical practice.
Conclusion
DETECT was superior to the MMSE and compared favorably with a formal neuropsychological evaluation for detecting MCI and early dementia in elderly patients. The DETECT device has many features that make it more practical than current screening tools. If additional testing produces equally encouraging results, it may represent a more feasible way for clinicians to screen for MCI.
Footnotes
DWW, ML, and FCG, conceived and designed the study, participated in study operations, assisted with data interpretation, and contributed significantly to manuscript preparation. JRB, HN, AR, and SF assisted with study development and conduct, and participated in manuscript preparation. PK performed data analysis, assisted in data interpretation, and contributed significantly to manuscript preparation.
Acknowledgments
We would like to thank the Wallace H. Coulter Foundation for funding this study and the development of the DETECT device. We would also like to thank the Wesley Woods Clinic at Emory University, Park Springs Retirement Community, St George Village, Waynesville First United Methodist Church, Auburn Neighborhood Senior Center, and Lenbrook for the use of their facilities.
Drs. Wright and LaPlaca are co-inventors of the DETECT technology. After the completion of this study, a patent was filed and is pending. A start-up company, Zenda Technologies, has formed around the technology. Dr Wright, Dr LaPlaca, and Mr Brumfield are not employees of the company but do retain financial interest.
The Wallace H. Coulter Foundation provided funding for this study.