Diagnostic Utility of the Shortened Version of the Wisconsin Card Sorting Test in Patients With Sporadic Late Onset Alzheimer Disease

Abstract

Background:

The classic version of the Wisconsin Card Sorting Test (WCST) consists of correctly sorting 128 cards according to changing sorting criteria. Its application is costly in terms of the time employed, with all the negative consequences this entails (decrease in motivation, frustration, and fatigue).

Method:

The main objective of this study was to test the usefulness of the shortened version of the WCST as compared to the full test by analyzing the equivalence between the two decks comprising the full 128-card version on a sample of patients diagnosed with sporadic late onset Alzheimer disease (SLOAD) and to check its clinical usefulness.

Results:

The variables showed equivalence between the two decks and their ability to differentiate between the control group (CG) and the Alzheimer disease (AD) group.

Conclusion:

The scores obtained suggest equivalence between decks and that the application of only the first deck is sufficient.

Keywords

Wisconsin Card Sorting Test sporadic late onset Alzheimer disease executive function neuropsychological examination convergent validity shortened version

Introduction

The term “executive function” (EF) is frequently utilized to refer to a series of high-level cognitive activities performed by a complex and multifaceted mental system with its neurological basis in the prefrontal cortex, particularly the cingulate and anterior neocortical areas. It is a system that deals with the operation of “controlled” processes with any type of information.¹ A great variety of skills has been included within the so-called EFs, such as the ability to set goals, develop plans of action, flexible thinking, inhibition of automatic responses, self-regulation of behavior, and verbal fluency.^2,3 Thus, any disturbance of these functions can limit an individual’s ability to lead an independent and productive life, even if other cognitive abilities remain intact.⁴

In Alzheimer disease (AD), the EFs are altered prematurely such that the patient shows an inability to plan and execute goal-oriented actions as well as an important lack of cognitive flexibility.

Generally, the first indications of AD are cognitive complaints, particularly in relation to memory. But problems are also detected in EFs, which must also be taken into account.⁵ Some studies even suggest that executive impairments are present in the predementia stages of the disease.⁶ This type of deficit in the frontal functions is one of the ones having the greatest impact on an individual’s functioning and his or her ability to remain independent. The capacity of patients with AD for abstraction is also affected, making it hard for them to understand metaphorical or figurative language. Also typical in the more advanced stages is the lack of inhibition of motor and verbal responses, which can give rise to socially inadequate behavior, such as sexual disinhibition or vulgar language.^7
-9

The literature has made available many instruments for measuring the EFs, and one of them is the Wisconsin Card Sorting Test (WCST). The main objective of this test is to assess mental flexibility. It was developed by Berg and Grant^10,11 to evaluate a person’s capacity for abstraction in response to a change in environmental contingencies. But it was not until 1981 that Heaton utilized it as a clinical instrument and the first normative data and rules for correction were published in 1993.

Performing the test correctly requires strategic planning, organized searching, the ability to utilize feedback from the environmental surroundings to make changes on the cognitive test, goal-oriented behavior, and the ability to modulate impulsive answers.¹²

The classic version of this test consists of correctly sorting 128 cards according to changing sorting criteria (color–form–number). Although the test has been widely implemented in the clinical setting, its application is costly in terms of the time employed, with all the negative consequences this entail (decrease in motivation, frustration, fatigue, etc).¹³

Authors such as Nelson¹⁴ and Vayalakkara¹⁵ proposed several shortened versions. Among them, the one most highlighted is a 64-card version (WCST-64), whose application is similar to that of the standard version (WCST-128) and allows clinicians to obtain the same information from the neuropsychological point of view, with the advantages entailed in using only half of the cards.¹⁶

Different studies have shown that performance on the full version of the test is much worse in patients with a diagnosis of dementia than in controls without dementia and furthermore that the WCST-64 version is sensitive to the impairments in EF typical of dementia, thus supporting the usage of the short version.¹⁷ Among the different versions, the 64-card version stands out because its application is similar to the standard version, and it allows clinicians to obtain the same neuropsychological information with the advantage of using only 1 deck.^18

-21 Other authors, such as Sherer,²² found a high association between the WCST-128 and the WCST-64, as well as high agreement in classifying patients as “normal” or “pathological,” although the WCST-128 turned out to be more sensitive in detecting impairments, since the patients showed greater difficulty in performing with the second deck of cards.

In this study, we aim to compare the usage of both of these versions by comparing the 2 decks that form the complete version of the test and thus assess the diagnostic effectiveness of the shortened version in a sample of patients diagnosed with sporadic late onset Alzheimer’s disease (SLOAD).

Participants and Methods

Participants

The sample utilized in this study comprised 213 patients who took part in the study voluntarily and were divided into 2 groups: 1 of 141 patients diagnosed with SLOAD and another of 72 patients who, at the time of examination, showed no type of neurological or psychiatric disorder, neither did they have any record of alcoholism or drug addiction. They were thus utilized as the control group (CG).

The patients with SLOAD were selected in the Neurology Service of the University Clinical Hospital of Salamanca (Spain). Selection of the patients in the CG began when the clinical group was already underway; the patients included in this group were chosen from among the relatives of the patients attending this Service.

In order to be included in the clinical group, the patients had to satisfy the following criteria: (1) neurological and neuropsychological diagnosis of AD according to Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria ²³ and (2) cutoff age of 60 in order to differentiate between early and late onset according to parameters established by the researchers and based on the observations of early and late presentation in the population studied by the group. For the patients with SLOAD, diagnostic tests were utilized in order to exclude those individuals with possible secondary dementia. Included in these tests were thyroid function, luetic serology, and the levels of vitamin B12 and folic acid Table 1 shows the descriptive data of the socioeconomic and clinical factors in both groups.

Table 1.

Descriptive and Comparative Data of the Socioeconomic and Clinical Variables of the 2 Groups.

Quantitative Variable	Group	N	Mean	SD	Statistic Test; P Value
Age	Control	72	69.96	9.99
Age	Alzheimer	141	75.06	7.31	t _{exp = −4.27}; <.0001
Years of schooling	Control	70	8.33	2.84
Years of schooling	Alzheimer	141	7.18	2.33	t _{exp = 2.93}; .001
Mini-Mental State Examination	Control	72	28.58	1.38
Mini-Mental State Examination	Alzheimer	141	22.07	4.67	t _{exp = 11.56;} .0001
Overall IQ	Control	70	105.72	16.95
	Alzheimer	140	84.42	19.49	t _{exp = 7.79;} <.0001
Qualitative variable		N	Category	%
Gender	Control	70	Female	55.6	χ²=3.24; .07
Gender	Alzheimer	140	Female	68.1	χ²=3.24; .07
Educational level	Control	70	Low	40.3	χ² = 5.12; .163
			Medium	29.2
			High	30.6
	Alzheimer	140	Low	49.6
			Medium	31.2
			High	19.1
Economic level	Control	70	Low	3.1	χ² = .07; .995
			Medium	78.1
			High	18.,8
	Alzheimer	140	Low	3.0
			Medium	77.8
			High	19.3

Abbreviations: SD, standard deviation; t _exp, t-test; χ², Chi-square test.

Significant differences were found between the groups in the following factors: age, years of schooling, mental status, and overall IQ.

No significant differences were detected in relation to gender (P = .072), socioeconomic level (P = .995), or educational level (P = .163).

Procedure

The evaluation of each patient in the clinical group consisted of a full neurological examination performed at the Neurology Service of the Clinical Hospital of Salamanca, a semistructured interview on sociodemographic and clinical aspects and a neuropsychological evaluation with a battery of tests designed for this study.

Sociodemographic and clinical evaluation

A sociodemographic and clinical questionnaire was applied to each patient and to the patients in the CG. This questionnaire included, according to the patient’s own self-report, information on his or her sociodemographic profile as well as medical, neurological, and psychiatric variables. The information thus obtained was contrasted and expanded by consulting the hospital clinical record where the patient had been selected. The main purpose of this register was to establish a control over these variables and at the same time guarantee a thorough description of the sample.

Neuropsychological evaluation

The tests comprising the neuropsychological examination battery designed for this study were selected for their validity and because they had been shown to be sensitive to neuropsychological deterioration in patients with AD.

Neuropsychological testing

Mental Status: Mini-Examen Cognoscitivo (the Spanish version of the MMSE).²⁴

Intelligence: Wechsler Adult Intelligence Scale (WAIS-III).²⁵

EFs: WCST.²⁶

The neuropsychological evaluation, both the administering of the tests and their correction, was performed according to the norms for each of the tests. In the case of the WCST, the complete WCST of 128 cards was administered to all the patients according to the norms set out in the manual by Heaton.²⁶

Data analysis

In the descriptive analysis, the means and standard deviations of sociodemographic and clinical variables were calculated. One-way analysis of covariance (ANCOVA) was utilized to analyze the differences in the global WCST scores to observe what best differentiated test performance between the patients in the AD group and the CG. Once the variables that best differentiated test performance between CG and AD were identified, those variables that did not differentiate were eliminated from the analysis. The percentage of nonperseverative errors, failure to maintain the category, and learning to learn were eliminated from the analysis. It is customary to utilize a smaller number of the scores obtained on this test, such as perseverative responses, errors, and complete categories.²⁷ Only the significant variables were utilized and in cases where the item was in terms of percentages, we worked only with the percentages. Mixed between–within patients ANCOVA was conducted to analyze these significant variables. This technique analyzes whether there would be differences between the decks of 64 cards (within-patients factor), study groups (between-groups factor), and the interaction between both factors. If the interaction was significant, the differences in scores between AD and CG groups would be analyzed by 1-way, between-patients ANCOVA for each deck of cards separately. Accordingly, the differences in scores between the decks of cards would be analyzed by 1-way within-patients ANCOVA for each study group. If the interaction was not significant, the main effects of factor would be interpreted. The chosen covariables for all the analyses were age and years of education, the same as occurs with the standardized scores in the Heaton manual.²⁶ Cohen effect size (d) was used as a measure of magnitude of effect between the AD and CGs and between the first and second decks of cards. The level of significance chosen was 5%. IBM-SPSS 23 was used to analyze the results.

Results

Table 2 shows the variables in which significant differences were found, the patients with CG having performed better than the patients with AD.

Table 2.

Descriptive Statistics From the Study Groups and P Value From 1-Factor ANCOVA Analysis.

Variables	Group	N	Means	SD	P Value
Number of hits	CG	72	66.32	1.94	<.0001
Number of hits	AD	141	57.53	1.34	<.0001
Errors	CG	72	57.02	2.23	<.0001
Errors	AD	141	68.91	1.55	<.0001
Percentage of errors	CG	72	45.63	1.73	<.0001
Percentage of errors	AD	141	53.61	1.20	<.0001
Perseverative responses	CG	72	37.36	3.16	<.0001
Perseverative responses	AD	141	52.14	2.19	<.0001
Percentage of perseverative responses	CG	72	29.91	2.48	<.0001
Percentage of perseverative responses	AD	141	41.03	1.72	<.0001
Perseverative errors	CG	72	32.89	2.29	<.0001
Perseverative errors	AD	141	44.41	1.59	<.0001
Percentage of perseverative errors	CG	72	26.20	1.76	<.0001
Percentage of perseverative errors	AD	141	34.13	1.22	<.0001
Nonperseverative errors	CG	72	23.92	1.72	.495
Nonperseverative errors	AD	141	25.38	1.19	.495
Percentage of nonperseverative	CG	72	18.57	1.39	.274
Percentage of nonperseverative	AD	141	20.47	.96	.274
Conceptual-level responses	CG	72	50.37	2.38	<.0001
Conceptual-level responses	AD	141	34.43	1.65	<.0001
Percentage of conceptual-level responses	CG	72	41.86	2.08	<.0001
Percentage of conceptual-level responses	AD	141	27.91	1.44	<.0001
Sorting category	CG	72	3.45	.21	<.0001
Sorting category	AD	141	1.78	.15	<.0001
Trials to complete first category	CG	72	27.02	4.78	.01
Trials to complete first category	AD	141	42.62	3.39	.01
Failures to maintain	CG	72	.98	.16	.421
Failures to maintain	AD	141	1.14	.11	.421
Learning to learn	CG	72	−1.55	1.68	.223
Learning to learn	AD	141	−4.25	1.35	.223

Abbreviations: ANCOVA, analysis of covariance; AD, Alzheimer disease; CG, control group; SD, standard deviation.

Based on these results, the variables selected to analyze the differences in mean scores between the 2 decks were as follows: number of hits, percentage of errors, percentage of perseverative responses, percentage of perseverative errors, percentage of conceptual-level responses, sorting category, and trials to complete first category.

With these variables, we set out to see how the patients in both groups behaved with each of the versions and to observe whether the scores obtained with deck 1 differed from those obtained with deck 2 or whether the 2 decks provided the same information, thus showing that the usage of only 1 deck is sufficient for obtaining the scores in this test.

Number of hits

Analysis of variance of this variable showed that there was no interaction between the 2 groups, meaning that behavior in the 2 decks is similar in both groups (P = .948; F _1,207 = .004). There were statistically significant differences between the decks (P = .02; F _1,207 = 5.35; d = .49), and the differences between the CG and the AD were indeed significant (P < .001; F _1,207 = 49.93; d = −.41). These findings indicate that for the “number of hits” variable, although the behaviour is similar in the two groups, there is a significant decrease in the number of hits with the second deck. This decrease is the same in both groups although the scores are lower in the AD group as can be observed in Figure 1. In this variable, the hits tend to decrease as the test progresses, affecting both groups in the same way.

Figure 1.

Differences between the decks in the “number of hits” variable.

Percentage of errors

Interaction was detected in the case of percentage of errors (P = .015; F _1,207 = 6.008), that is, the difference between groups is different in each deck. In the first deck, the standardized mean difference is .55 (P = .004, F _1,207 = 8.53), and for the second deck, it is .89 (P < .0001; F _1,207 = 22.23). The differences were observed above all in the scores of the second deck as can be seen in Figure 2.

Figure 2.

Differences between decks in the percentage of errors.

Analysis of these differences between the 2 decks for each group showed that the difference was not significant in either group (CG: F _1,67 = .48, P = .490, d = −.05; AD: F _1,138 = .57, P = .452; d = .39), indicating that both decks contribute the same information in both groups, despite the between-group differences, the difference in means, and the variability being higher in AD than in CG.

Percentage of perseverative responses

No interaction was found, and therefore, the behavior in both groups was similar (F _1,207 = 2.16, P = .144). There were no differences between the decks (F _1,207 = .65, P = .423; d = .07), but there were indeed significant differences between the 2 groups of patients (F _1,207 = 14.14, P < .0001, d = −.46). Thus, the information offered by both decks of the test for this variable is equivalent, and both groups show similar behavior; furthermore, the scores between the groups are very different. This is a good indicator of the alterations in EF in patients with AD.

Percentage of perseverative errors

This same pattern is found in the percentage of perseverative errors. No interaction was detected (F _1,207 = 1.23, P = .269). There were no significant differences between the decks (F _1,207 = 1.58, P = .211, d = −.12), but there were indeed significant differences between AD and CG (F _1,207 = 14.29, P < .0001; d = .46). As in the previous variable, both decks are equivalent, offering the same information and differentiating the control patients from the patients with AD.

Percentage of conceptual-level responses

Interaction was not significant for this variable either (F _1,207 = .33, P = .569). No significant differences were found between the decks, although in this case the differences were close to significance (F _1,207 = 3.35, P = .069, d = .28). Significant differences were detected between the 2 groups of patients (F _1,207 = 30.18, P < .0001, d = −.61).

Sorting category

No significant interaction was detected in the case of the number of categories (F _1,207 = 1.27, P = .260); and there were no significant differences found between the decks (F _1,207 = .37, P = .54; d = .11), although they were indeed found between the groups (F _1,207 = 24.43; P < .0001; d = −.6), indicating that the control patients showed higher values than the patients with AD.

Trials to complete the first category

This pattern is also repeated in trials to complete the first category, there being no interaction (F _1,207 = 2.69, P = .103); the decks show equivalence (F _1,207 = .004, P = .953, d = −.21) and the groups are differentiated according to diagnosis, making it a good tool to differentiate between groups (F _1,207 = 38.58, P < .00001, d = .75).

Discussion

The main objective of this study was to compare the complete version of the WCST-128 with the shortened 64-card version by analyzing the equivalence of the 2 decks of 64 cards each in the complete version of the test on a sample of patients diagnosed with SLOAD and to test its clinical utility.

After analysis of variance, the results indicate that the scores obtained suggest equivalence between deck 1 and deck 2 and that the application of only the first deck is sufficient. This would be compatible with the statistical justification given by the WCST-128 test manual,²⁶ which states that there is good reliability based on the coefficient of homogeneity in the 2 halves of the test, thus justifying the usage of only 1 deck, given the equivalence between the 2 decks.

Our findings are in line with many studies that have demonstrated the utility of the short version (WCST-64), both in the clinical population^13,17,20,28 and in the population without the pathology.^16,18

In our study, we selected the items that showed significant differences between the performance of the CG and the patients diagnosed with AD on the complete 128-card test. The variables that showed differences between these 2 groups were the following: number of hits, percentage of errors, percentage of perseverative responses, percentage perseverative errors, percentage of conceptual-level responses, sorting category, and trials to complete first category. These results are in line with other studies of similar characteristics.^16,29 Furthermore, all the variables adequately differentiated the AD group from the CG, this being a good indicator of alterations in the EFs.

As regards the comparison of performance between patients with CG and AD in both decks, the results obtained suggest the equivalence between the decks for most of the variables mentioned above, in which no significant differences were found between the 2 decks in terms of the performance of either group. These findings are compatible with those published in a study comparing the validity of the 2 versions of the test, the shortened and the complete versions, in a sample of patients who had undergone cranioencephalic trauma; in 3 of the variables in which we found equivalence between the decks, in that study, there was a correlation in the scores between the 2 versions greater than .7 (perseverative responses, sorting category, and trials to complete first category).¹³

In the case of the “number of hits” variable, significant differences were found in the scores between the 2 decks; it was observed that the number of hits in the second deck decreased significantly in both the groups. A possible explanation for this is the long duration of the test which places an increased cognitive demand on the patients and therefore the lower scores.¹³ “Percentage of errors” increases in the second deck, but only in the AD group, although the differences are not significant when they are analyzed for each of the groups. This decrease in performance as the test progresses could be explained by the long time it takes to administer (30–60 minutes), being particularly complicated for the patients with AD³⁰ or CG participants.³¹ This situation justifies the need to utilize the shortened version of the WCST for a reliable test to evaluate the EFs in elderly populations or populations with neurodegenerative diseases.

The WCST has been shown to be a good tool for measuring EF¹⁶, and furthermore, there seems to be a direct relation between performance on this test and a patient’s clinical condition, contributing relevant information for a diagnosis of AD.³⁰ As can be seen in our findings, this test is a tool that is sensitive to AD and allows clinicians to differentiate patients with the pathology from patients without it. The main objective of our study was to demonstrate the utility of the shortened version in patients with AD; a similar study aimed at learning the utility of the WCST-64 in patients with AD and PD¹⁷ concluded that the usage of the shortened version is advisable, and our findings can be associated with the ones found in that study.

Other shortened versions of the test have also been used, such as Nelson modified version.¹⁴ Some authors express caution when comparing the results between the 2 versions, emphasizing that it is important to have specific data available for each pathology¹⁹ and not to generalize the results obtained solely from the nonclinical population. This supports the need to prove the validity of the test for each pathology, which is why we have focused on AD in our study.

Furthermore, both from the neuropsychological context and the research perspective, an increasing need has been observed to have shorter protocols that facilitate the task, since a protocol that is too long can consume resources that in some cases are very limited.³¹

By way of conclusion, we can say that our findings indicate that the use of only 1 deck (the WCST-64) provides the same information in patients with AD as the use of both decks (the full version), as we found the 2 decks to be equivalent for all the items with the exception of “number of hits,” which decreases with the second deck, possibly as a result of the long duration of the test.¹³ It is also important to underscore the sensitivity of the test, since it found significant differences between the CG and the experimental group for all the test items. This shows that it is a good indicator of AD, which can be extremely useful in clinical practice when accompanied by other diagnostic tests.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflict of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Badre

Wagner

. Selection, integration, and conflict monitoring: assessing the nature and generality of prefrontal cognitive control mechanisms. Neuron. 2004;41(3):473–487.

Anderson

. Assessment and development of executive function (EF) during childhood. Child Neuropsychol. 2002;8(2):71–82.

Rosselli

Ardila

Lopera

Pineda

. Neuropsicología Infantil. Medellín: Prensa Creativa; 1997.

Lezak

. The problem of assessing executive functions. Int J Psychol. 1982;17:281–297.

Rainville

Amieva

Lafont

Dartigues

Orgogozo

Fabrigoule

. Executive function deficits in patients with dementia of the Alzheimer’s type: a study with a Tower of London task. Arch Clin Neuropsychol. 2002;17(6):513–530.

Perry

Watson

Hodges

. The nature and staging of attention dysfunction in early (minimal and mild) Alzheimer’s disease: relationship to episodic and semantic memory impairment. Neuropsychologia. 2000;38(3):252–271.

Doody

Stevens

Beck

. Practice parameter: management of dementia (an evidence-based review) report of the quality standards subcommittee of the American Academy of Neurology. Neurology. 2001;56(9):1154–1166.

Martínez Lage

Láinez Andrés

. El Alzheimer: Teoría y Práctica. Madrid, Spain: Aula médica ediciones; 2000.

Martínez Lage

Martínez-Lage Álvarez

. Concepto, criterios diagnósticos y visión general de las demencias. In: López-Pousa

Villalta

Llinás

, eds. Manual de demencias. Barcelona, Spain: Prous Science; 1996.

10.

Berg

. A simple objective technique for measuring flexibility in thinking. J Gen Psychol. 1948;39(1):15–22.

11.

Grant

Berg

. A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card sorting problem. J Exp Psychol. 1948;38(4):404–411.

12.

Strauss

Sherman

EMS

Spreen

. A Compendium Of Neuropsychological Tests: Administration, Norms, and Commentary by, Sherman. 3rd ed. New York, NY: Oxford University Press; 2006.

13.

Del Valle-del Valle

Puerta-Cuestas

Renau-Hernández

Utilidad clínica de la versión de 64 cartas del test de clasificación de cartas de Wisconsin en pacientes que han sufrido un traumatismo craneoencefálico. Rev Neurol. 2008;46(3):142–146.

14.

Nelson

. A modified card sorting test sensitive to frontal lobe deficits. Cortex. 1976;12(4):313–324.

15.

Vayalakkara

Devaraju-Backhaus

Bradley

JDD

Simco

Golden

. Abbreviated form of the Wisconsin Card Sort Test. Int J Neurosci. 2000;103(1-4):131–137.

16.

Greve

Stickle

Love

Bianchini

Stanford

. Latent structure of the Wisconsin Card Sorting Test: a confirmatory factor analytic study. Arch Clin Neuropsychol. 2005;20(3):355–364.

17.

Paolo

Tröster

Blackwell

Koller

Axelrod

. Utility of a Wisconsin card sorting test short form in persons with Alzheimer’s and Parkinson’s disease. J Clin Exp Neuropsychol. 1996;18(6):892–897.

18.

Axelrod

. Are normative data from the 64-card version of the WCST comparable to the Full WCST? Clin Neuropsychol. 2002;16(1):7–11.

19.

Basso

Lowery

Ghormley

Bornstein

. Practice effects on the Wisconsin Card Sorting Test-64 card version across 12 months. Clin Neuropsychol. 2001;15(4):471–478.

20.

Donders

Wildeboer

. Validity of the WCST-64 after traumatic brain injury in children. Clin Neuropsychol. 2004;18(4):521–527.

21.

Merrick

Donders

Wiersum

. Validity of the WCST-64 after traumatic brain injury. Clin Neuropsychol. 2003;17(2):153–158.

22.

Sherer

Nick

Millis

Novack

. Use of the WCST and the WCST-64 in the assessment of traumatic brain injury. J Clin Exp Neuropsychol. 2003;25(4):512–520.

23.

American Psychiatric Association, ed. Guía de consulta de los criterios diagnósticos del DSM-5. Arlington, VA: American Psychiatric Publishing; 2014:438.

24.

Lobo

Saz

Marcos

Día

De la Cámara

Ventura

. Revalidación y normalización del miniexamen cognoscitivo (primera versión en castellano del Minimental State Examination) en la población general geriátrica. Med Clin (Barc). 1999;112(20):767–774.

25.

Weschler

. Weschler Adult Intelligence Scale-Third Edition (WAIS-III) Administration and Scoring Manual. San Antonio, TX: The Psychological corporation; 1997.

26.

Heaton

Chelune

Talley

Kay

Curtiss

. Wisconsin Card Sorting Test manual: Revised and Expanded. Odessa, FL: Psychological Assessment Resources; 1993.

27.

Nyhus

. The Wisconsin Card Sorting Test and the cognitive assessment of prefrontal executive functions: a critical update. Brain Cogn. 2009;71(3):437–451

28.

Lin

Kwan

Guo

. Construct validity of the Wisconsin Card Sorting Test-64 in patients with stroke. Clin Neuropsychol. 2008;22(2):273–287.

29.

Bowden

Fowler

Bell

. The reliability and internal validity of the Wisconsin Card Sorting Test. Neuropsychol Rehabil. 1998;8(3):243–254.

30.

Silva-Filho

Pasian

, Vale FAC. Typical performance of elderly patients with Alzheimer disease on the Wisconsin Card Sorting Test (WCST). Dement Neuropsychol. 2007;1(2):181–189.

31.

Chao

Hao

Chao

Shi

Chao

IHC

. Utility of nelson’s modified card sorting test in patients with Alzheimer’s disease or vascular dementia. Open J Prev Med. 2013;3(2):172–177.