Sage Journals: Discover world-class research

Abstract

Background:

The mobile screening test system for mild cognitive impairment (mSTS-MCI) was developed and validated to address the low sensitivity and specificity of the Montreal Cognitive Assessment (MoCA) widely used clinically.

Objective:

This study was to evaluate the efficacy machine learning algorithms based on the mSTS-MCI and Korean version of MoCA.

Method:

In total, 103 healthy individuals and 74 patients with MCI were randomly divided into training and test data sets, respectively. The algorithm using TensorFlow was trained based on the training data set, and then its accuracy was calculated based on the test data set. The cost was calculated via logistic regression in this case.

Result:

Predictive power of the algorithms was higher than those of the original tests. In particular, the algorithm based on the mSTS-MCI showed the highest positive-predictive value.

Conclusion:

The machine learning algorithms predicting MCI showed the comparable findings with the conventional screening tools.

Keywords

mild cognitive impairment machine learning TensorFlow MoCA

Introduction

With the importance of early diagnosis of dementia, early detection of mild cognitive impairment (MCI), which is a prodromal stage of dementia, has gained significant attention in public health and clinical environments.¹ Indeed, most studies have developed a variety of neuropsychological tests for screening of MCI in clinical settings, and their clinical usefulness has been identified.^2,3

The Montreal Cognitive Assessment (MoCA), developed as a brief screening test to distinguish patients with MCI from healthy individuals, is one of the most frequently used clinical tools for this condition.³ However, several studies have suggested that the MoCA is critically limited in detecting MCI, despite its high reliability and validity.^2,4,5 The most obvious characteristic of patients with MCI is memory impairment compared to healthy individuals. However, test items for memory only represent 5 of 30 points in the MoCA. Therefore, it is difficult to sensitively detect patients with MCI using the MoCA.^4,5

To overcome this issue, a mobile screening system for mild cognitive impairment (mSTS-MCI) was developed and standardized.^4,5 In previous studies, the mSTS-MCI showed higher sensitivity (99.0%) and specificity (97.0%) compared to the Korean version of MoCA (MoCA-K) with a high degree of reliability and validity, suggesting that the mSTS-MCI can be used clinically as a screening tool for MCI.^4,5

Meanwhile, machine learning algorithms have been developed to analyze large, complex data sets in medical settings and clinical environments.⁶ Indeed, machine learning algorithms have been used to detect a variety of diseases.^7

-10 Machine learning is used to quickly and accurately screen patients with diseases based on previous data with machine learning algorithms. A machine learning algorithm shows a high level of accuracy by differentiating subtle differences that are generally difficult to distinguish.¹¹ In particular, for the diagnosis of MCI, machine learning algorithms have been used to analyze the use of brain functional magnetic resonance imaging and electroencephalography.¹² Accordingly, it is also possible to analyze scores of screening tools including the mSTS-MCI and the MoCA using machine learning algorithms. Indeed, Youn and colleagues reported the usefulness of the machine learning algorithm to predict patients with cognitive impairments based on the use of various screening tools with high accuracy.¹³

Although the validity of the mSTS-MCI and the MoCA-K was investigated in previous studies, the number of patients was relatively small, which limited the optimal predictive power of the mSTS-MCI and the MoCA-K.⁵ Thus, it is necessary to confirm whether the machine learning algorithms based on the mSTS-MCI and the MoCA-K were more effective than the mSTS-MCI and the MoCA-K for screening of MCI.

The objective of this study was to evaluate and compare the clinical efficacy of machine learning algorithms and the original tests for detection of MCI. The author hypothesized that the machine learning algorithms using the mSTS-MCI increase the accuracy of evaluation compared to the conventional mSTS-MCI.

Methods

The machine learning algorithm was trained using TensorFlow to differentiate patients with MCI from healthy individuals, which was partially based on the data obtained from the author’s previous studies that investigated the cutoff score, concurrent validity, and test–retest reliability of the mSTS-MCI.^4,5 Some of the data were tested using this algorithm for the accuracy. TensorFlow is a software library for machine learning created by Google based on Python computer language.¹⁴ The study was approved by the institutional review board of Yonsei University (1041849-201611-BM-060-01).

Participants

The original data set from the author’s previous study consisted of 103 healthy patients and 74 patients with MCI. Patients were older than 65 years of age and were recruited from communities and senior welfare centers in South Korea.⁵ In accordance with Petersen, based on the author’s previous study, the inclusion criteria for MCI were as follows: (1) a subjective memory complaint; (2) memory impairment relative to age- and education-matched healthy elderly individuals confirmed by performance on neuropsychological assessments (below 1.5 standard deviation); (3) intact general cognitive function confirmed by the Korean version of Mini-Mental State Examination scores ≥ 24; (4) intact activities of daily living; and (5) without dementia confirmed by a physician.¹⁵ The exclusion criteria were as follows: (1) psychiatric disorders such as depression or schizophrenia and (2) neurological disorders such as stroke or traumatic brain injury. These criteria were based on amnestic-MCI, which is one of the subtypes of MCI.¹⁵ The author previously investigated the cutoff score, concurrent validity, and test–retest reliability of the mSTS-MCI.

One-sixth of the healthy group and the MCI group were randomly extracted and assigned to the test data set. Consequently, at this stage, the test data set consisted of 17 healthy patients and 12 patients with MCI. The trained model was adapted to the rest of the patients.

Model Training

The first step in data modeling involved the processing step entailing the standardization of cardinal variables. When using logistic regression with machine learning, data are generally normalized to prevent data divergence. Accordingly, the scores of the mSTS-MCI and the MoCA-K were normalized as follows:

χ' = \frac{χ - μ}{σ},

where χ′ means standardized cardinality variables, χ refers to raw cardinality of variables, μ denotes mean, and σ denotes standard deviation.

In the second step, the data set was randomly divided into the training or the test data set, and each set was developed to create feature (x_data) and outcome (y_data) variables. The third step included model training with the train data set via a logistic regression model using TensorFlow. A logistic regression analysis was used to calculate the cost, which is the difference between the actual data and the value predicted by the logistic regression, and the initial value of the gradient descent was set to 0.5. The machine learning was conducted until the cost value was stabilized, and the number of learning steps was 10 000. The fourth step involved calculation of the accuracy based on the test data set (Supplemental Table 1).

Compared to the conventional logistic regression analysis, it is possible to develop the optimal model by adjusting the attributes of the hypothesis function with the TensorFlow algorithm, which is one of the advantages of machine learning algorithms. Furthermore, the calculation of the TensorFlow algorithm is faster than the conventional logistic regression analysis.¹⁶ Taken together, the TensorFlow algorithm is more effective than the conventional regression analysis.

Outcome Measures

The Mobile Screening Test System for Mild Cognitive Impairment

The mSTS-MCI, developed and standardized by Park and colleagues, comprises 8 test items for memory, 1 item for attention, and 4 items for executive function. In effect, the total scores of the mSTS-MCI range from 0 to 28 points, and the high scores indicated high cognitive functions. In the previous study with 19 cutoff scores, the sensitivity and specificity of the mSTS-MCI were 99.0% and 93.2%, respectively.⁵

The Korean version of Montreal Cognitive Assessment

The MoCA-K, translated and standardized by Lee et al for the demographic group of Korean elderly individuals, consists of visuospatial/executive function, naming, memory, attention, language, abstraction, and orientation. It utilized scores ranging from 0 to 30, where higher scores represent better cognition in the participant. The cutoff score for MCI was 23, and one point was added for patients with less than 6 years of education.¹⁷ In the previous study with a cutoff score of 23, the sensitivity and specificity of the MoCA-K were 94.2% and 40.5%, respectively.⁵

Statistical Analyses

The accuracy of the machine learning algorithm based on the mSTS-MCI and the MoCA-K in predicting MCI was based on the frequency of accurate estimations by the trained algorithm in the test data set. Therefore, the ratio of correct estimations in the test data set was used to determine the accuracy of the test. Meanwhile, the accuracy of the original tests was calculated according to the ratio of correct estimations in the original data set. The sensitivity and specificity were determined using the receiver–operating characteristics curve. All data were analyzed by using the Microsoft Excel version 20.0 and SPSS version 22.0.

Results

General Characteristics of Patients

There were no significant differences in age, sex ratio, and education between the MCI and the healthy groups of patients. In contrast, the healthy group showed significantly higher scores of the mSTS-MCI and the MoCA-K than the MCI group, indicating the discriminative validity of the mSTS-MCI and the MoCA-K (Table 1).

Table 1.

General Characteristic of Participants.^a,b

	Healthy group	MCI group	t/χ²	P
Age, years	74.93 (6.96)	74.45 (6.51)	.471	.639
Sex, male/female	45 / 58	33 / 41	.014	.905
Education period, years	5.83 (4.52)	6.14 (4.54)	.450	.654
MoCA-K score	25.74 (2.10)	22.51 (2.20)	9.894	<.001
mSTS-MCI score	22.85 (2.44)	15.28 (2.41)	20.499	<.001

Abbreviations: mSTS-MCI, mobile screening test system for mild cognitive impairment; MoCA-K, Korean version of Montreal Cognitive Assessment; MCI, mild cognitive impairment.

^a N = 177.

^b Shown are mean value (standard deviation).

Comparison Between Machine Learning Algorithm Based on the mSTS-MCI and the Original mSTS-MCI

The accuracy of the algorithm predicting MCI based on the mSTS-MCI and the original mSTS-MCI was 96.67% and 96.61, respectively. The algorithm based on the mSTS-MCI had the 91.7% sensitivity and the 100.00% specificity. However, the sensitivity and specificity of the original mSTS-MCI were 99.00% and 93.20%, respectively (Figure 1A; Table 2). With respect to the predictability of MCI, the positive-predictive value (PPV) meaning the proportion of MCI among MCI group was 100.00% for the algorithm and 98.57% for the original mSTS-MCI score. The negative-predictive value (NPV) indicating the proportion of non-MCI patients in the healthy group was 94.44% for the MoCA-K score, and 95.32% for the original mSTS-MCI score (Tables 3 and 4).

Figure 1.

Receiver–operating characteristic (ROC) curve of machine learning algorithms and the original tests. (A) Mobile screening test system for mild cognitive impairment (mSTS-MCI); (B) Korean version of Montreal Cognitive Assessment (MoCA-K).

Table 2.

Sensitivity and Specificity of MCI Detection via Machine Learning Algorithms Compared to Original Tests.

		Sensitivity	Specificity	Youden index	AUC
mSTS-MCI	Machine learning algorithm	.917	1.000	.917	.958^a
mSTS-MCI	Original test	.990	.932	.922	.985^a
MoCA-K	Machine learning algorithm	.750	.944	.694	.846^b
MoCA-K	Original test	.942	.595	.347	.848^b

Abbreviations: AUC, Area under the curve; MCI, mild cognitive impairment; MoCA-K, Korean version of Montreal Cognitive Assessment; mSTS-MCI, mobile screening test system for mild cognitive impairment.

^a P < .001.

^b P < .01.

Table 3.

Predictability of Machine Learning Algorithms Based on the mSTS-MCI and the MoCA-K to Distinguish Between Healthy Participants and Patients With MCI.^a

Machine learning algorithms		Classification		Predictability
Machine learning algorithms		MCI group (n = 12)	Healthy group (n = 17)	Predictability
mSTS-MCI	MCI	11	0	PPV: 100.0%
mSTS-MCI	Healthy	1	17	NPV: 94.44%
MoCA-K	MCI	9	1	PPV: 90.00%
MoCA-K	Healthy	3	16	NPV: 84.21%

Abbreviations: MCI, mild cognitive impairment; MoCA-K, Korean version of Montreal Cognitive Assessment; mSTS-MCI, mobile screening test system for mild cognitive impairment; NPV, negative-predictive value; PPV, positive-predictive value.

^a N = 29.

Table 4.

Predictability of the Original mSTS-MCI and the MoCA-K to Distinguish Between Healthy Participants and Patients With MCI.^a

Original tests		Classification		Predictability
Original tests		MCI group (n = 74)	Healthy group (n = 103)	Predictability
mSTS-MCI	MCI	69	1	PPV: 98.57%
mSTS-MCI	Healthy	5	102	NPV: 95.32%
MoCA-K	MCI	44	6	PPV: 88.00%
MoCA-K	Healthy	33	97	NPV: 76.37%

^a N=177.

Comparison Between Machine Learning Algorithm Based on the MoCA-K and the Original MoCA-K

The accuracy of the algorithm for predicting MCI based on the MoCA-K and the original MoCA-K was 86.20% and 79.66%, respectively. The sensitivity and specificity of the algorithm based on the MoCA-K were 75.00% and 94.10%, respectively. However, the sensitivity and specificity of the original MoCA-K were 94.20% and 59.50%, respectively (Figure 1B; Table 2). With respect to the predictability of MCI, the PPV was 90.00% for the algorithm, and 88.00% for the original MoCA-K. The NPV was 84.21% for the algorithm, and 76.37% for the original MoCA-K (Tables 3 and 4).

Discussion

This study compared the accuracy of the machine learning algorithms based on the mSTS-MCI and the MoCA-K and the original tests to screen for amnestic-MCI and verify its utility. As a result, the accuracy of the algorithms was higher than that of the original tests. Although the sensitivity and specificity of the machine learning algorithms were relatively lower than those of the original tests, the PPV and NPV of the algorithms were higher than those of the original tests. These findings suggested that the machine learning algorithm based on the screening test for amnestic-MCI is as effective in screening amnestic-MCI as the original test.

In terms of predictability, the machine learning algorithms showed higher PPV and NPV than the original tests. Specially, the algorithm based on the mSTS-MCI showed the highest PPV. A high PPV is essential to avoid missed screening of patients with MCI,¹⁸ which means that the algorithm based on the mSTS-MCI is more effective compared to the other algorithm. Consequently, it was confirmed that the machine learning algorithm yields higher predictive power when the sample size is relatively small. Nevertheless, the low sensitivity and specificity of the machine learning algorithms were attributed to the lack of patients in the test data set.

The accuracies of the 2 machine learning algorithms in this study were higher than those in the previous study investigating accuracy of a machine learning algorithm, which was used to detect people with cognitive impairment including MCI and dementia.¹³ Since the algorithm was based on the Korean Dementia Screening Questionnaire that was originally developed to screen dementia, it is possible that patients with MCI were not properly screened, which might affect the accuracy of the algorithm. However, another previous study reported that the accuracy of the algorithm based on a variety of neuropsychological tests was 96.6%.¹⁹ Considering that the accuracy of the algorithm based on several tests is similar that of the algorithm based on only the mSTS-MCI, it is more meaningful in reducing the test duration.

In the current study, the sensitivity and the specificity of the algorithm based on the mSTS-MCI was higher than that of the MoCA-K, suggesting that the mSTS-MCI was more effective when used in screening amnestic-MCI. The primary factor underlying the comparative effectiveness is based on a review of the differences between the components of the 2 tools.⁴ The mSTS-MCI includes 8 memory items with 18 of 28 points assigned, which indicates that the mSTS-MCI is focused on memory tests, when compared to the MoCA-K. Indeed, the memory items in the MoCA-K represent 5 of 30 points.^5,17 Several studies have reported that patients with amnestic-MCI show impaired episodic memory when compared to healthy patients.^5,20 Although a few studies indicated that patients with amnestic-MCI showed a decline in executive function as well as in the early stage of amnestic-MCI, the decline can occur especially when episodic memory decline is impaired, suggesting that a decline in episodic memory is a general hallmark of amnestic-MCI.⁵ Given these characteristics, in order to distinguish amnestic-MCI from normal aging, the most appropriate screening tools for amnestic-MCI should be based on the assessment of episodic memory.¹⁵ Accordingly, the algorithm based on the mSTS-MCI, which reflects the cognitive characteristics of amnestic-MCI, has high sensitivity and predictive power when compared to the MoCA-K.

In conclusion, the machine learning algorithm based on the mSTS-MCI facilitates the screening of patients with amnestic-MCI in clinical settings. Additionally, in order to discriminate amnestic-MCI, a screening tool needs to focus on items that assess episodic memory. Generally, the manual diagnosis of amnestic-MCI is time consuming in a clinical setting, and therefore, expensive medical tests represent a major challenge.⁷ However, it is possible to automate the analysis of medical data by using a machine learning algorithm with a set of attributes, such as the use of the mSTS-MCI scores and demographic information. Once the machine learning algorithm is established, practitioners can determine whether or not a patient is cognitively impaired based on data input, which enables clinicians to identify amnestic-MCI accurately and rapidly in the clinical settings.¹³

This study has some limitations. First, since the study limited to patients with amnestic-MCI, the findings cannot be generalized. Nevertheless, given that most previous studies of MCI diagnosis were based on criteria for amnestic-MCI because of minimal cognitive bias of amnestic-MCI compared to other types of MCI such as non-amnestic-MCI and multidomain MCI,^5,15,21,22 the results of this study are clinically meaningful. Second, the machine learning algorithm used in this study adopted a single model of logistic regression. Other models such as the use of support vector machine and decision tress are effective in discriminating MCI.^23,24 Therefore, in order to investigate the usefulness of machine learning algorithms in clinical settings, it is necessary to analyze machine learning algorithms using a variety of accessible and available models for screening MCI. Third, fewer data sets were utilized in this study compared to other studies investigating the effectiveness of machine learning algorithms, which might limit the implications of the algorithm based on the mSTS-MCI. Notably, in order to improve the accuracy of machine learning algorithms, it is necessary to collect additional data for evaluation. Finally, the imbalance in data set used in this study was a limitation, since a 1.5-fold higher number of healthy patients were included. Therefore, in the future, it is necessary to incorporate a balanced data set comprising a large number of healthy patients and patients with MCI for a more balanced review.

Conclusions

Machine learning algorithms based on the mSTS-MCI and the MoCA-K are effective in distinguishing patients with amnestic-MCI from healthy patients, when compared to the original tests. This result suggests that the machine learning algorithms are comparable to the traditional screening tools for amnestic-MCI.

Supplemental Material

Supplementary_Table_1 - Machine-Learning Algorithms Based on Screening Tests for Mild Cognitive Impairment

Supplementary_Table_1 for Machine-Learning Algorithms Based on Screening Tests for Mild Cognitive Impairment by Jin-Hyuck Park in American Journal of Alzheimer's Disease & Other Dementias

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Soonchunhyang University Research Fund. This work was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIT) (Number 2019R1F1A1060719).

ORCID iD

Jin-Hyuck Park

Supplemental Material

Supplemental material for this article is available online.

References

Cummings

Vinter

Cole

Khachaturian

ZSd

. Alzheimer’s disease: etiologies, pathophysiology, cognitive reserve, and treatment opportunities. Neurology. 1998;51(suppl 1):S2–S17. doi:10.1212/WNL.51.1_Suppl_1

Nasreddline

Phillips

Bédrian

, et al. The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53(4):695–699. doi:10.1111/j.1532-5415.2005.53221.x

Shin

Lee

. Comparing clinical usefulness of cognitive function tests (CDT, K-MMSE, K-3MS, CDR) in dementia patients. Korean J Rehabil Nurs. 2008;11(2):90–98.

Park

Heo

. Mobile screening test system for mild cognitive impairment: concurrent validity with the Montreal cognitive assessment and inter-rater reliability. J Korean Soc Cog Rehabil. 2017;6(2):25–42

Park

Jung

Kim

Park

Kim

Park

. Validity of a novel computerized screening test system for mild cognitive impairment. Int Psychogeriatr. 2018;30(1):1455–1463. doi:10.1017/S1041610218000923

Kononenko

. Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med. 2001;23(1):89–109. doi:10.1016/S0933-3657(01)00077-X

Chen

Herskovits

. Machine-learning techniques for building a diagnostic model for very mild dementia. Neuroimage. 2010;52(1):234–244. doi:10.1016/j.neuroimage.2010.03.084

Bryan

. Machine learning applied to Alzheimer disease. Radiology. 2016;281(3):665–668. doi:10.1148/radiol.2016162151

Mathotaarachchi

Pascoal

Shin

, et al. Identifying incipient dementia individuals using machine learning and amyloid imaging. Neurobiol Aging. 2017;59:80–90. doi:10.1016/j.neurobiolaging.2017.06.027

10.

Tohka

Moradi

Huttunen

. Alzheimer’s disease neuroimaging I. comparison of feature selection techniques in machine learning for anatomical brain MRI in dementia. Neuroinformatics. 2016;14(3):279–296. doi:10.1007/s12021-015-9292-3

11.

Lao

Shen

Xue

Karacali

Resnick

Davatzikos

. Morphological classification of brains via high-dimensional shape transformations and machine learning methods. Neuroimage. 2004;21(1):46–57. doi:10.1016/j.neuroimage.2003.09.027

12.

Khodayari-Rostamabad

Hasey

MacCrimmon

Reilly

de Bruin

. A pilot study to determine whether machine learning methodologies using pre-treatment electroencephalography can predict the symptomatic response to clozapine therapy. Clin Neurophysiol. 2010;121(12):1998–2006. doi:10.1016/j.clinph.2010.05.009

13.

Youn

Choi

Shin

, et al. Detection of cognitive impairment using a machine-learning algorithm. Neuropsychiatr Dis Treat. 2018;14(1):2939–2945. doi:10.2147/NDT.S171950

14.

Rampasek

Goldenberg

. Tensorflow: biology’s gateway to deep learning? Cell Syst. 2016;2(1):12–14. doi:10.1016/j.cels.2016.01.009

15.

Petersen

. Mild cognitive impairment as a diagnostic entity. J Intern Med. 2004; 256(3):183–194. doi:10.1111/j.1365-2796.2004.01388.x

16.

Yuan

Zhao

, et al. A convolutional neural network based on TensorFlow for face recognition. 2017 IEEE 2nd IAEAC. 2017:525–529. doi:10.1109/IAEAC.2017.8054070

17.

Lee

Cho

, et al. Brief screening for mild cognitive impairment in elderly outpatient clinic: validation of the Korean version of the Montreal cognitive assessment. J Geriatr Psychiatry Neurol. 2008;21(2):104–110. doi:10.1177/0891988708316855

18.

Isella

Mapelli

Siri

. Validation and attempts of revision of the MDS-recommended tests for the screening of Parkinson’s disease dementia. Parkinsonism Relat Disord. 2014;20(1):32–36. doi:10.1016/j.parkreldis.2013.09.008

19.

Kang

Kim

, et al. Prediction of cognitive impairment via deep learning trained with multi-center neuropsychological test data. BMC Med Inform Decis Mak. 2019;19(1):231. doi: 10.1186/s12911-019-0974-x

20.

Vliet

Manly

Tang

Marder

Bell

Stern

. The neuropsychological profiles of mild Alzheimer’s disease and questionable dementia as compared to age-related cognitive decline. J Int Neuropsychol Soc. 2003;9(5):720–732. doi:10.1017/S1355617703950053

21.

Troyer

Murphy

Anderson

, et al. Item and associative memory in amnestic mild cognitive impairment: performance on standardized memory tests. Neuropsychology. 2008;22(1):10–16. doi:10.1037/0894-4105.22.1.10

22.

Karrasch

Sinervä

Grönholm

, et al. CERAD test performances in amnestic mild cognitive impairment and Alzheimer’s disease. Acta Neurol Scand. 2005;111(3):172–179. doi:10.1111/j.1600-0404.2005.00380.x

23.

Zhang

Wang

Zhou

Yuan

Shen

. Multimodal classification of Alzheimer’s disease and mild cognitive impairment. Neuroimage. 2011;55(3):856–867. doi:10.1016/j.neuroimage.2011.01.008

24.

Dyrba

Barkhof

Fellgiebel

, et al. Predicting prodromal Alzheimer’s disease in subjects with mild cognitive impairment using machine learning classification of multimodal multicenter diffusion-tensor and Magnetic resonance imaging data. J Neuroimaging. 2015;25(5):738–747. doi:10.1111/jon.12214

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB