A review of the validation of the Brief Cognitive Assessment for Multiple Sclerosis in Latin America

Abstract

Background

The present review aimed to identify published studies that reported the validation of the Brief Cognitive Assessment for Multiple Sclerosis (BICAMS) in Latin America (LATAM).

Methods

To compile a comprehensive list of available validation studies, we performed a systematic review of the literature via an electronic search of PubMed and Web of Science via the keywords “Validation,” “Brief Cognitive Assessment for Multiple Sclerosis,” “BICAMS,” and “Latin America.”

Results

Twenty-seven sources of validation studies for the BICAMS were identified. Of the 27 citations identified, only four provide validation of the BICAMS in LATAM. These studies include a comparison of cognitive performance between multiple sclerosis (MS) patients and healthy controls (HCs) across all three BICAMS tests. Overall, the studies included a greater proportion of patients with RRMS and middle-aged adults and included participants with wide ranges of education levels.

Conclusion

We provide a detailed description of the BICAMS validation currently available for people living in LATAM. Although the validation of tests in diverse populations has gained interest in the field, there is still a need for more studies among people from LATAM countries.

Keywords

Review BICAMS validation studies Latin America multiple sclerosis

Introduction

Multiple sclerosis (MS) is an immune-mediated disease of the central nervous system (CNS) that primarily affects young adults (mean onset, 20–40 years).^1,2 It is more common in women than in men.³ According to a 2020 study conducted by Walton et al., an estimated 2.8 million people worldwide are living with MS.⁴ Several studies have examined the prevalence of MS in Latin America (LATAM). In Argentina, the estimated prevalence ranges from 14 to 19.8 cases per 100,000 inhabitants.⁵ Another study in the same country reported that the crude prevalence rate of MS in Rosario was 48.3/100,000 inhabitants (95% confidence interval [CI]: 48.28–48.35), 63.7/100,000 for females and 28.3/100,000 for males.⁶ A study conducted in Brazil reported a prevalence of 15.54/100,000 inhabitants.⁷ In 2021, another study in Brazil reported that the point-prevalence rate for MS was 26.4/100,000 people (95% CI, 19.7 to 34.6/100,000).⁸ In Colombia, the prevalence rate was 4.41/100,000 inhabitants.⁹ Studies conducted in Mexico reported that MS affects between 11% and 20% of the Mexican population.^10,11

The clinical symptoms and signs of MS include sensory, motor, fatigue, depression, anxiety, pain and visual problems.¹² In addition to their physical symptoms, MS patients present with cognitive impairment (CI).¹³ Several studies have reported that approximately 50%–70% of MS patients have CI.^14–18 CI occurs independently of the course and stage of the disease.^19,20 It interferes with the performance of activities of daily living (work, school, social functioning and quality of life).^21–26 CI is characterized by alterations in information processing, memory, attention, verbal fluency, visuospatial perception and executive functioning.^27–30

The cognitive assessment in MS patients provides information regarding baseline conditions and cognitive changes and allows for the planning of early cognitive interventions.^15,31 Some studies have demonstrated that identifying cognitive dysfunction provides greater benefits to the quality of life of MS patients.^32,33 The diagnosis of cognitive impairment in MS patients requires neuropsychological batteries. Several cognitive tools have been proposed for identifying cognitive deficits in patients with MS. The Rao Brief Repeatable Battery of Neuropsychological Tests (BRB-N),³⁴ the Minimal Assessment of Cognitive Function in MS (MACFIMS),²⁷ and the Brief International Cognitive Assessment for MS (BICAMS) are neuropsychological tools commonly and widely used to measure CI in MS patients.³⁵ The BICAMS has been validated in several countries.³⁶ It is a tool for monitoring cognitive function and should be incorporated into clinical practice as a cognitive battery for assessing cognitive function.³⁷

A number of studies highlight how appropriate norms improve the sensitivity and specificity of neuropsychological tests.^38–40 Normative data can control for the influence of demographic variables that have been shown to affect cognitive performance on neuropsychological tests.^41–43 Despite the recognition that cognitive tools are needed for cognitive assessment, there is a lack of available neuropsychological test normative data for Spanish-speaking patients with MS. In 2020, a study conducted by Matias-Guiu et al. validated the Neuronorma battery for neuropsychology assessment in MS, which included 280 MS patients and 280 HCs. Their findings indicated that most tests showed medium or large effect sizes, which suggests adequate criterion validity.⁴⁴ In 2022, Rivera et al. validated the Norma Latina neuropsychological assessment battery in patients with MS in Mexico. The sample included 100 MS patients and 100 HCs. The participants were subjected to a cognitive assessment with a neuropsychological battery, and their findings indicated that the neuropsychological battery is sensitive for discriminating cognitive deficits in MS patients.⁴⁵ However, there is a paucity of available normative neuropsychological test data for MS patients in LATAM.

Validation studies conducted in countries other than the U.S. have been valuable resources for clinicians and researchers. However, to our knowledge, this is the first publication providing a review of available studies on the validation of BICAMS in LATAM. Our objective was to identify studies that have validated the BICAMS specifically in adults living in LATAM countries.

Methods

To compile a comprehensive list of available validation tests for the BICAMS, we used a systematic review of the literature to identify published validation studies for LATAM countries. We conducted a search on PubMed and Web of Science via the keywords “Validation,” “Brief Cognitive Assessment for Multiple Sclerosis,” “BICAMS,” and “Latin America.” This search yielded 27 citations from which relevant studies were selected. From that search, four citations contained validation of the BICAMS, specifically for adults in LATAM.

Cognitive instrument

The administration of the BICAMS can be completed in approximately 15 minutes and does not require any special equipment or specific assessor training.²⁷ This instrument mainly assesses processing speed (the Symbols-Digit Modalities Test-SDMT), learning and verbal memory (the California Verbal Learning Test-CVLT), and visuospatial memory (the Brief Visuospatial Memory Test-BVSMT-R). These subtests are reliable and sensitive for identifying cognitive impairment in MS patients.³⁶

This instrument consists of three different tests: (1) The SDMT has good psychometric properties (a sensitivity of 82% and a specificity of 60%).^27,43 This instrument involves two key rows, where the upper row contains a series of nine abstract geometric symbols and the lower row contains corresponding numbers (1–9). The SDMT can be completed within 5 minutes and includes instructions, tests and responses. The participants are instructed to verbally associate each symbol with its corresponding number as quickly as possible within a 90-second time frame. The score for this test is determined by the number of correct substitutions completed within the given time limit, with a maximum score of 110.^28,46,47 The CVLT-II has excellent sensitivity and good predictive validity.^48,49 (2) The CVLT-II requires the examiner to read a list of 16 words in five trials. All trials, including instructions, tests and responses, can be completed in 5–10 minutes. The participants are then asked to recall as many words as they can remember (in any order) a list given by the psychometrist of the study. The score for this test is calculated on the basis of the total number of words correctly recalled across the five trials.⁵⁰ (3) The BVMT-R has good psychometric properties.²⁷ In the BVMT-R, participants are shown a sheet of paper containing six abstract geometric designs. There are three learning trials, each with a duration of 10 seconds. After the stimulus is removed, the participants are given paper and pencil to draw the geometric designs from memory. Scores are determined by summing the scores from three trials (each drawing received a score of 0–2 points).⁵¹

Results

Table 1 includes a summary of all validation studies for the BICAMS (including studies from LATAM and other countries). These studies compared cognitive performance between MS patients and healthy controls (HCs) via all three BICAMS tests.^52–78 Below is a description of each study (four in total) that provides validation of the BICAMS for adults living in LATAM. Information on the other countries (23 in total) is available in Table 1 but will not be further described.

Table 1.

Study design and participant demographics.

Author Year	Geographic Location	Sample size	Age	Education	Gender n % Female	BICAMS results
			Mean ± SD	Mean ± SD		Mean ± SD
			Median [R]	Median [R]		Median [Range]
			Median [R]	Median [R]		Tests	MS	HCs	P
Valdivia-Tangarife et al. 2024	Mexico LATAM	100 MS	43.48 ± 5.96	12.71 ± 2.29	63%	SDMT	42.23 ± 9.60	55.65 ± 8.91	<.001
		10 HCs	43.39 ± 6.03	12.55 ± 2.52	63%	CVLT-II	45.55 ± 7.86	55.17 ± 3.91	<.001
						BVMT-R	21.15 ± 7.47	23.51 ± 4.93	.009
Alarcon et al. 2020	Colombia LATAM	50 MS	41.44 ± 10.99	14.76 ± 2.61	64%	PAMCL	45.34 ± 10.1	48.78 ± 8.45	.04
		100 HCs	37.75 ± 12.63	14.73 ± 3.57	48%	BVMT-R	21.64 ± 6.91	25.67 ± 6.81	<.001
						SDMT	46.47 ± 14.2	54.11 ± 12.19	<.001
Vanotti et al. 2016	Argentina LATAM	50 MS	43.42 ± 10.17	14.86 ± 2.78	74%	SDMT	45.14 ± 16.0	56.71 ± 10.85	<.001
		100 HCs	42.37 ± 10.07	14.94 ± 2.49	75%	CVLT-I	50.92 ± 12.3	60.88 ± 10.46	<.001
						BVMT-R	20.70 ± 7.74	23.44 ± 5.84	.017
Spedo et al. 2015	Brazil LATAM	58 MS	41.2 ± 12.2	12.7 ± 5.2	69%	SDMT	35.9 ± 16.1	47.5 ± 13	<.001
		58 HCs	40.3 ± 11.9	12.5 ± 3.6	55.2%	CVLT-II	42.1 ± 12.4	53.4 ± 10.8	<.001
						BVMT-R	19.9 ± 8.6	23.8 ± 7.7	.013
Darwish et al. 2022	Lebanon	43 MS	36.06 ± 12.37	14.63 ± 3.17	81.4%	SDMT	53[66]	60[42]	.003
		180 HCs	45.01 ± 19.36	15.13 ± 3	60%	VMAT	52[38]	54[29]	.583
						BVMT-R	21[30]	25[25]	.018
Drulovic et al. 2022	Serbia	500 MS	39.9 ± 9.4	14.0 ± 2.9	70.2%	SDMT	45.9 ± 16.7	56.3 ± 12.9	<.001
		69 HCs	40.3 ± 11.5	14.1 ± 3.4	63.77%	CVLT-II	50.0 ± 11.7	52.7 ± 9.6	.063
						BVMT-R	18.8 ± 7.4	22.6 ± 5.8	<.001
Evdoshenko et al. 2022	Russia	98 MS	38.44 ± 11.47	15.12 ± 2.79	70.4%	SDMT	49.16 ± 13.42	58.34 ± 11.52	<.001
		86 HCs	38.17 ± 13.29	16.26 ± 3.02	63.95%	CVLT-II	61.5[40–73]	65.5[48–75]	.02
						BVMT-R	26.5[13–34]	28[16–35]	.03
Souissi et al. 2022	Tunisia	104 MS	33.3 ± 9.8	14–16 years	75%	SDMT	36 ± 13	47 ± 15	<10⁻³
		104 MS	33.3 ± 9.4	= 14.4%	75%	TVLT	42 ± 7	46 ± 6	<10⁻³
				14–16 years = 14.4%		BVMT-R	23 ± 9	27 ± 7	.002
Betscher et al. 2021	Poland	61 MS	39 [28–49]	13 [12–17]	74%	CVLT	51.7 ± 10.9	56.1 ± 9.2	.02
		61 HCs	37 [29–50]	13 [12–17]	75%	BVMT-R	24.0 ± 7.7	27.1 ± 5.7	.03
						SDMT	48.8 ± 12.1	57.2 ± 9.7	<.001
Botchorishvili et al. 2021	Georgia	68 MS	39.2 ± 9.9	14.3 ± 2.1	71%	SDMT	35.5 ± 12.7	46.0 ± 11.8	<.001
		68 HCs	38.5 ± 9.9	14.5 ± 1.9	68%	CVLT-II	51.0 ± 11.8	58.5 ± 8.2	<.001
						BVMT-R	22.0 ± 8.0	25.6 ± 6.8	<.001
Farghaly et al. 2021	Egypt	90 MS	30.8 ± 6.7	14.5 ± 2.6	77.78%	SDMT	39.2 ± 13.3	50.9 ± 10.8	<.001
		85 HCs	30.5 ± 7.9	14.3 ± 3.3	70.59%	CVLT-II	53.7 ± 10.5	59.6 ± 8.5	<.001
						BVMT-R	19.7 ± 9.2	25.4 ± 8.7	<.001
Hämaäläinen et al. 2021	Finland	65 MS	50.9 ± 8.8	13.8 ± 9.8	71%	SDMT	41.9 ± 11.8	54.6 ± 8.3	<.001
		45 HCs	49.4 ± 12.6	14.0 ± 2.1	71%	CVLT-II	43.0 ± 11.5	51.3 ± 10.7	<.001
						BVMT-R	19.2 ± 8.0	24.7 ± 6.8	<.001
Maubeuge et al. 2021	France	123 MS	49.69 ± 9.41	≤6 years = 46.3%	63.4%	SDMT	50.31 ± 11.12	58.55 ± 8.44	<.001
		276 HCs	43.84 ± 12.42	≤6 years = 41.3%	57.3%	FVLT	49.72 ± 12.77	57.78 ± 8.67	<.001
		276 HCs	43.84 ± 12.42	≤6 years = 41.3%	57.3%	BVMT-R	22.89 ± 7.26	26.73 ± 5.67	<.001
Marstrand et al. 2020	Denmark	65 MS	37.2 ± 8.8	15.2 ± 2.4	41	SDMT	61.0 ± 10.0	66.0 ± 9.6	<.005
		65 HCs	36.8 ± 9.6	15.9 ± 2.1	41	CVLT-II	65.4 ± 9.9	68.6 ± 6.4	<.05
						BVMT-R	27.4 ± 5.8	29.6 ± 3.7	<.05
Estiasari et al. 2019	Indonesia	40 MS	31[20–61]	>12 years =	82.5%	SDMT	40.9 ± 14.8	64.8 ± 16.2	<.001
		66 HCs	29[22–51]	75.0%	72.7%	CVLT-II	52.0 ± 12.8	61.5 ± 9.7	<0.001
				>12 years = 89.4%		BVMT-R	22.2 ± 7.7	29.3 ± 5.6	<0.001
Skorve et al. 2019	Norway	65 MS	37.02 ± 0.40	14–16 years =24	64.6%	SDMT	54.65 ± 10.79	58.52 ± 10.53	.039
		65 HCs	38.13 ± 11.40	14–16 years =31	66.2%	CVLT-II	54.55 ± 10.86	60.32 ± 7.75	.001
						BVMT-R	26.55 ± 5.76	29.03 ± 4.01	.005
Filser et al. 2018	Germany	172 MS	43.33 ± 11.64	10.74 ± 1.56	68%	SDMT	47.43 ± 11.67	56.07 ± 11.64	<.001
		100 HCs	43.04 ± 15.59	10.77 ± 1.58	71%	VLMT	55.35 ± 11.43	55.16 ± 10.27	>.05
						BVMT	24.44 ± 7.59	27.37 ± 5.96	<.05
Sousa et al. 2018	Portugal	105 MS	38.26 ± 11.03	13.55 ± 3.71	66.7%	SDMT	51.77 ± 11.20	58.68 ± 10.02	.000
		60 HCs	36.17 ± 12.01	14.62 ± 3.47	58.3%	CVLT-II	55.05 ± 11.84	60.47 ± 10.12	.003
						BVMT-R	21.72 ± 7.27	24.68 ± 5.52	.004
Costers et al. 2017	Belgium	97 MS	45.42 ± 9.24	14.28 ± 1.86	68%	SDMT	52.11 ± 13.11	60.95 ± 10.21	<.001
		97 HCs	43.52 ± 12.69	14.69 ± 1.61	75%	CVLT-II	64[54–70	63[56–68]	.573
						BVMT-R	28[21–31]	29[25–32]	.007
Ozakbas e t al. 2017	Turkey	173 MS	37.5 ± 10.7	13.9 ± 7.3	71.7%	SDMT	43.2 ± 12.5	53.5 ± 9.5	<.001
		153 HCs	36.9 ± 8.9	15.4 ± 8.8	71.2%	CVLT-II	45.7 ± 11.3	53.9 ± 7.7	.002
						BVMT-R	16.9 ± 8.5	22.5 ± 9.2	.002
Niino et al. 2017	Japan	156 MS	41.4 ± 9.3	14.1 ± 1.9	69%	SDMT	47.9 ± 14.0	61.0 ± 9.5	<.001
		126 HCs	39.3 ± 11.9	14.3 ± 1.6	72%	CVLT-II	48.6 ± 12.6	55.7 ± 10.5	<.001
						BVMT-R	23.5 ± 8.4	28.3 ± 5.4	<.001
Polychroniadou et al. 2016	Greek	44 MS	40.2 ± 9.9	13.9 ± 4.2	27	SDMT	45.0 ± 17.2	61.4 ± 13.1	<.001
		79 HCs	36.2 ± 10.6	15.6 ± 5.5	48	GVLT	55.5 ± 12.3	60.5 ± 10.7	.02
						BVMT-R	18.5 ± 8.3	22.1 ± 6.5	.009
Walker et al. 2016	Canada	57 MS	45.4 ± 9.9	15.44 ± 44	80%	SDMT	49.65 ± 10.78	59.11 ± 8.46	<.001
		51 HCs	41.9 ± 10.8	16.31 ± 2.1	86%	CVLT-II	51.63 ± 10.05	57.72 ± 7.92	.08
		51 HCs				BVMT-R	24.60 ± 6.54	29.80 ± 3.63	<.001
Giedraitienè et al. 2015	Lithuania	50 MS	38.8 ± 10.2	15.9 ± 2.8	47%	SDMT	42.7 ± 13.9	57.7 ± 11.5	<.001
		20 HCs	36.7 ± 16.4	17.5 ± 3.5	33%	CVLT-II	55.9 ± 10.0	65.7 ± 5.9	<.001
						BVMT-R	23.1 ± 7.0	29.6 ± 4.1	<.001
ÒConnell et al. 2015	Ireland	67 MS	43.9 ± 12.1	13.6 ± 2.7	73%	SDMT	46.0 ± 12.9	56.1 ± 10.6	<.001
		66 HCs	42.7 ± 12.8	14.1 ± 3.1	68%	CVLT-II	53.6 ± 9.1	53.6 ± 9.1	<.001
						BVMT-R	20.9 ± 6.5	20.9 ± 6.5	.013
Sandi et al. 2015	Hungary	65 MS	41.9 ± 8.9	$\leq$ 12 years = 31	49	SDMT	44.31 ± 11.76	54.8 ± 10.32	<.001
				$\geq$ 13 years = 34		CVLT-II	55.37 ± 10.71	66.8 ± 12.44	<.001
				$\leq$ 12 years = 31		BVMT-R	22.54 ± 8.54	26.6 ± 5.56	<.001
		65 HCs		$\geq$ 13 years = 34	49
Dusankova et al. 2012	Czech Republic	369 MS	34 ± 10	14 ± 3	68%	SDMT	50 ± 13	65 ± 9	<.001
		134 HCs	34 ± 9	14 ± 2.5	71%	CVLT-II	52 ± 11	60 ± 8	<.001
						BVMT-R	23 ± 7	29 ± 4	<.001

Note: PAML: Prueba de Aprendizaje y Memoria con Codificacion Libre; BVMT-R: Brief Memory Test Revised; SDMT: Symbols Digit Modalities Test; GVLT: Greek Verbal Learning Test; FVLT: French Verbal Learning Test; VMAT: Verbal Memory Test; TVLT: The Tunisian Verbal Learning Test.

Spedo et al. validated the BICAMS for adults living in Brazil. The sample consisted of 58 MS patients and 58 HCs. The HCs were selected from community sources affiliated with the MS center in Brazil. The groups did not differ in terms of age (MS patients = 41.2 ± 12.2 years; HCs group = 40.3 ± 11.9; P = .689) or education level (MS patients = 12.7 ± 5.2 years; HCs group = 12.5 ± 3.6, P = .803). Student's t test revealed statistically significant differences in cognitive performance between MS patients and HCs across all three neuropsychological test scores. These significant differences were observed on the Symbol Digit Modalities Test (SDMT; t = 4.28; P < .001), the California Verbal Learning Test (CVLT-II; t = 5.21; P < .001), and the Brief Visuospatial Memory Test (BVMT-R; t = 2.52; P = .013). A sample of MS participants (n = 49) underwent follow-up assessments (within a two-week interval) of the BICAMS, using the same test forms as previously taken to enable test‒retest reliability analysis. The test-retest reliability for each test was high (SDMT [r = .86; P < .001], CVLT-II [r = .84; P < .001], and BVMT-R [r = .77; P < .001]). Additionally, the diagnostic validity of the BICAMS revealed that age was negatively correlated with all BICAMS tests for MS patients in the SDMT (r = −.30; P < .05), CVLT-II (r = −.30; P < .05), and BVMT-R (r = −.29; P < .05). Furthermore, education was positively correlated with all BICAMS tests for MS patients (CVLT-II, r = .18, P < .05; BVMT-R, r = .27, P < .05; and SDMT, r = .29, P < .05). Moreover, the study controlled for the main effects of age, education, Hospital Anxiety and Depression Scale-Depression (HADS-D), and Hospital Anxiety and Depression Scale-Anxiety (HADS-A) scores in an ANCOVA test but did not eliminate the significance of the group effect on any of the BICAMS tests: CVLT-II (F_1,110 = 28.99; P < .001), BVMT-R (F_1,110 = 7.77; P < .01), and SDMT (F_1,110 = 21.09; P < .001). This study utilized mixed-factor ANCOVAs to model the learning curves of MS samples and HCs on the CVLT-II and BVMT-R. First, on the CVLT-II, HCs performed significantly better than did MS samples in all trials (F_1,110 = 29.03; P < .001) and presented a significantly steeper learning curve (F_1,111 = 10.82; P < .01). Second, for the BVMT-R, HCs performed significantly better than did the MS sample in all trials (F_1,110 = 7.816; P < .01), with the interaction effect of time and learning curve demonstrating significance (F_1,111 = 7.747; P < .01).⁷⁵

Vanotti et al. validated the BICAMS for adults living in Argentina. The sample consisted of 50 MS patients and 100 HCs. Participants were recruited from the general population of Buenos Aires City, Argentina. The groups did not differ significantly with respect to age (MS patients = 43.42 ± 10.17 years; HCs=42.37 ± 10.07 years; P = .55), education level (MS patients = 14.86 ± 2.78 years; HCs=14.94 ± 2.49 years; P = .86) or sex. A greater proportion of patients had relapsing-remitting MS (RRMS = 78%), followed by secondary progressive MS (SPMS=18%) and primary progressive MS (PPMS=4%). The mean Expanded Disability Status Scale (EDSS) score was 3.29 ± 2.55. Student's t test revealed statistically significant differences in cognitive performance between MS patients and HCs across all three BICAMS test scores. These significant differences were observed in the SDMT (P < .001), CVLT-II (P < .001), and BVMT-R (P = .017). The SDMT and CVLT-II tests (Cohen's d = 0.85) had large size effects (Cohen's d = 0.87), whereas the BVMT-R was small (Cohen's d = 0.40). A sample of MS patients (n = 25) underwent follow-up assessments (within a two-week interval) of the BICAMS, using alternate forms to mitigate practice effects (SDMT; CVLT-I = form 4 was used, and BVMT-R = form 4 was used to enable test‒retest reliability analysis). The test-retest reliability for each test was high (SDMT [r = .95], CVLT-I [r = .87] and BVMT-R [r = .82]).

Additionally, this study presented regression-based normative models for the BICAMS, controlling for demographic variables (Age, Age², Sex, and Education). For the SDMT (Constant=10.9231; Age=−0.2218; Age² = 0.0013; Sex=0.2850; Education=0.3714; and SD of residuals=2.5842); CVLT-II (Constant=4.8721; Age=−0.0603; Age² = 0.001; Sex=1.079; Education=0.3819; and SD of residuals=2.7956); and BVMT-R (Constant=6.5162; Age=0.1438; Age²=−0.0029; Sex=−0.1892; Education=0.2111; and SD of residuals=2.8341).⁷⁶

Alarcón et al. validated the BICAMS for adults living in Colombia. The sample consisted of 50 MS patients and 100 HCs. HCs were recruited from the general population in Bogotá, Colombia. The groups did not differ significantly with respect to age (MS patients = 41.44 ± 10.99 years; HCs=37.75 ± 12.63 years; P = .089), education level (MS patients = 14.76 ± 2.61 years; HCs=14.73 ± 3.57 years; P = .57) or sex. All patients had RRMS, and the EDSS score was 1.33 ± 1.54. Student´s test revealed statistically significant differences in cognitive performance between MS patients and HCs across all three BICAMS test scores (SDMT, PAMCL = Prueba de Aprendizaje y Memoria con Codificación Libre, and BVMT). These significant differences were observed in the SDMT (P < .001), PAMCL (P = .04), and BVMT-R (P < .001). The effect sizes for the SDMT (Cohen's d = 0.59) and the BVMT-R were medium (Cohen's d = 0.58) and small for the PAMCL (Cohen's d = 0.38). A subsample of MS patients (n = 16) underwent follow-up assessments within a two-week interval using the same test forms (SDMT; PAMCL; and BVMT-R) used in session one, enabling test‒retest reliability analysis. The test‒retest reliability for each test was high (SDMT [r = .88], PAMCL [r = .86], and BVMT-R [r = .93]). Additionally, this study presented regression-based normative models for the BICAMS, controlling for demographic variables (Age, Education, and Sex) as follows: SDMT (Constant=7.901; Age=−0.107; Education=0.315; Sex=1.114; and SD of residuals=2.13463); PAMCL (Constant=8.05; Age=−0.055; Education=0.205; Sex=−; and SD of residuals=2.62747); and BVMT-R (Constant=8.367; Age=−0.116; Education=0.397; Sex=−; and SD of residuals=2.66933).⁵²

Valdivia-Tangarife et al. validated the BICAMS for adults living in Mexico. The sample consisted of 100 MS patients and 100 HCs. HCs were recruited from the general population in Guadalajara, Mexico. The groups did not differ significantly in terms of age (MS patients = 43.48 ± 5.96 years; HCs=43.39 ± 6.03 years; P = .916), education level (MS patients = 12.71 ± 2.29 years; HCs=12.55 ± 2.52 years; P = .640) or sex. A greater proportion of patients had RRMS (94%), followed by SPMS (4.0%) and PPMS (2.0%). The mean EDSS score was 3.75 ± 1.92. Student's t test revealed statistically significant differences in cognitive performance between MS patients and HCs across all three BICAMS test scores. These significant differences were observed in the SDMT (P < .001), CVLT-II (P = .002), and BVMT-R (P = .009). The SDMT (Cohen’s d = 0.58) and CVLT-II (Cohen’s d = 0.61) had medium-sized effects, whereas the BVMT-R had a small-sized effect (Cohen’s d = 0.18). A subsample of MS patients (n = 30) underwent follow-up assessments (within a two-week interval) of the BICAMS, using alternate forms to mitigate practice effects (SDMT; CVLT-II = form 4, and BVMT-R = form 4 enabling test‒retest reliability analysis). The test–retest reliability for each test was considered high (SDMT r = .95, CVLT-I r = .84, and BVMT-R r = .81). Additionally, this study presented regression-based normative models for the BICAMS, controlling for demographic variables (Age, Age², Sex, and Education) as follows: SDMT (Constant=7.621; Age=0.242; Age²=−0.003; Sex=−0.837; Education=−0.118; and SD of residuals=2.95957); CVLT-II (Constant=14.083; Age=−0.590; Age² = 0.006; Sex=−0.493; Education=−0.086; and SD of residuals=2.95203); and BVMT-R (Constant=7.758; Age=−0.681; Age² = 0.008; Sex=0.274; Education=0.124; and SD of residuals=2.95367).⁷⁷

Additional details about the aforementioned four validation studies in LATAM can be found in Table 1, including information on the validation of BICAMS in other countries.

Discussion

While considerable efforts have been undertaken over the last few years to validate neuropsychological tests for assessing cognitive function in MS patients living in LATAM, these data are limited, particularly compared with the available data in the U.S. and Europe, especially for the BICAMS.

BICAMS has been validated in different countries (the Czech Republic, Hungary, Ireland, Brazil, Lithuania, Canada, Argentina, Greece, Japan, Turkey, Belgium, Portugal, Germany, Norway, Indonesia, Denmark, France, Colombia, Finland, Egypt, Georgia, Poland, Tunisia, Russia, Serbia, Lebanon, and Mexico) (Table 1). Among the 27 sources of validation identified by the BICAMS, only four provide validation data for adults living in LATAM. Regression-based norms have been recommended for BICAMS validation.^27,79 However, the findings from the present literature review indicate a relative paucity of validated neuropsychological tests for detecting cognitive impairment in MS patients in LATAM.

Considering the studies of BICAMS validation conducted in LATAM. Highlights, the study was conducted in Brazil. Their objective was to investigate the reliability and validity of a Brazilian-Portuguese adaptation of the BICMAS. In this study, the translation of the SDMT, CVLT-II, and BVMT-R was necessary. They followed the published consensus opinion guidelines for a new language validation of BICAMS.²⁷ Conversely, the studies conducted in Argentina, Colombia, and Mexico used a Spanish adaptation of the BICAMS battery.²⁷ On the other hand, these studies included 100 HCs to develop normative data.^52,76,77 In contrast, the study conducted in Brazil included 57 HCs.⁷⁵ According to the international standard for validating BICAMS, a sample size of at least 65 healthy volunteers is recommended for the development of normative data. This minimum sample size should provide enough power to detect a medium effect size in a two-group comparison.

In the test-retest reliability analysis, we identified several methodological characteristics. For example, (1) the BICAMS forms used for cognitive assessment at Time 1 and Time 2 (Time 1; Test, and Time 2; retest) in the studies conducted in LATAM. The study conducted in Argentina used an alternate form of the BICAMS in Time 1 (SDMT, CVLT-I, and BVMT-R) and Time 2 (SDMT, form 4 was used; CVLT-I, list B was used; and for BVMT-R, form 4 was used). The study conducted in Brazil used the same form of the BICAMS (CVLT-II, BVMT-R, and SDMT) at both Time 1 and Time 2. In Colombia, the same form of the BICAMS (SDMT, PAMCL, BVMT-R) was used at both Time 1 and Time 2. In Mexico, an alternate form of the BICMAS was used at Time 1 (SDMT, CVLT-II, and BVMT-R) and Time 2 (SDMT, form 4 was used; CVLT-I, list B was used; and for BVMT-R, form 4 was used).

Several studies have recommended alternate methods to mitigate practice effects. In a study conducted by Benedict and Zgaljardic,⁸⁰ 30 healthy participants were included and divided into two groups: the alternative form (AF) group with 15 participants and the same form (SF) group with 15 participants. These groups were matched for age, education, and baseline memory test performance. The study investigated practice effects during repeated administrations of verbal and nonverbal memory tests.

They employed the Hopkins Verbal Learning Test-Revised and the BVMT-R.^51,81 The results showed that participants in the SF group, who were tested with the same form every two weeks, improved significantly over four sessions. Participants in the AF group, who completed alternate forms of the nonverbal memory test, produced a small practice effect. This study revealed that verbal memory was resistant to practice effects when alternate forms were employed. Additionally, a study conducted by Benedict^82,83 examined test‒retest effects in MS patients randomly assigned to the alternative form (AF) group (17 MS patients) or the same form (SF) group (17 MS patients). They employed the Minimal Assessment of Cognitive Function in MS, which includes the following tests with alternate forms: the CVLT-II, the BVMT-R, the Paced Auditory Serial Addition Test (PASAT),⁸⁴ the Controlled Oral Word Association Test (COWAT),⁸⁵ and the Sorting Test from the Delis-Kaplan Executive Functions System.⁸⁶ The author concluded that the use of alternative forms of the CVLT-II and BVMT-R helps preserve test validity without compromising test‒retest reliability.

(2) In the BICAMS validation studies conducted in LATAM, a subsample of MS patients (Argentina, n = 25; Brazil, n = 49; Colombia, n = 16; and Mexico, n = 30) underwent cognitive assessments at two different times: Time 1 (test) and Time 2 (retest), separated by 2 weeks. Benedict et al. recommended a test‒retest interval ranging from 1 to 3 weeks.²⁷ Charter mentioned that test–test reliability can vary depending on the sample assessment and amount of time between test and retest.⁸⁷

(3) When the reliability values were evaluated, the studies in which the BICAMS was validated in LATAM reported a Pearson correlation coefficient >.70 for all three neuropsychological tests. Some authors have reported r values for test‒retest correlation of .70‒.79 as “adequate” and .80‒.89 as “high.”^27,88–90 Additionally, several studies mentioned that test‒retest reliability is one psychometric property that neuropsychologists can use when selecting a test battery and evaluating test scores.^91,92

The majority of studies on BICAMS available specifically for LATAM are based on adult MS patients living in Argentina, Brazil, Colombia, and Mexico.^52,75–77 Only three studies included norms based on a multiple regression model that allows for obtaining an individuaĺs predicted score while controlling for the influence of demographic variables (age, sex, and education).^52,76,77 The norms based on multiple regression equations are important, as they increase the clinical utility of results and their applicability.^93–96 Regression equations permit the prediction of an individuaĺs level of performance on a cognitive instrument at retest from their score at initial testing.⁹⁷

The validation of BICAMS is necessary in other LATAM countries. Several studies indicate that cultural differences and variations in educational systems can influence cognitive performance.^98,99 Although normative data for the BICAMS have been developed for Argentina, Brazil, Colombia and Mexico, these normative data are not necessarily interchangeable due to cultural differences between these countries. In summary, we argue that local norms are essential for accurately identifying the presence or absence of cognitive impairment.

Conclusion

The present review provides a list of twenty-seven published studies on BICAMS validation worldwide. Of the 27 citations identified, only four validate BICAMS in LATAM. The BICAMS battery is a reliable tool for assessing cognitive function in MS patients and can be used in clinical practice. The BICAMS consists of three different tests: the SDMT, CVLT-II, and BVMT-R. These instruments have strong psychometric properties. This review aims to advance the field of neuropsychological assessment in MS patients in LATAM and promote the clinical application of BICAMS validation in Argentina, Brazil, Colombia, and Mexico.

Footnotes

Acknowledgments

This study was conducted by members of the Research Group for the Development and Validation of Cognitive Instruments, led by Teresita J. Villaseñor-Cabrera. The group comprised a multidisciplinary team of researchers dedicated to advancing the field of cognitive assessment.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Edgar R Valdivia-Tangarife

Jazmin Marquez-Pedroza

Teresita Villaseñor-Cabrera

References

Dobson

Giovannoni

. Multiple sclerosis–a review. Eur J Neurol 2019; 26: 27–40.

McGinley

Goldschmidt

Rae-Grant

. Diagnosis and treatment of multiple sclerosis: a review. Jama 2021; 325: 765–779.

Jobin

Larochelle

Parpal

, et al. Gender issues in multiple sclerosis: an update. Women’s Health 2010; 6: 797–820.

Walton

King

Rechtman

, et al. Rising prevalence of multiple sclerosis worldwide: insights from the Atlas of MS. Mult Scler J 2020; 26: 1816–1821.

Cristiano

Patrucco

Rojas

, et al. Prevalence of multiple sclerosis in Buenos Aires, Argentina using the capture-recapture method. Eur J Neurol 2009; 16: 183–187.

Luetic

Menichini

. Prevalence of multiple sclerosis in Rosario, Argentina. Mult Scler Relat Disord 2021; 55: 103212.

Fragoso

Peres

. Prevalence of multiple sclerosis in the city of Santos, SP. Rev Bras Epidemiol 2007; 10: 479–482.

Passos

GRD

Becker

Varela

, et al. Prevalence of multiple sclerosis in key cities of Brazil: a study in Passo Fundo, Southern Brazil. Arq Neuropsiquiatr 2021; 79: 692–696.

Toro

Sarmiento

Díaz del Castillo

, et al. Prevalence of multiple sclerosis in Bogotá, Colombia. Neuroepidemiology 2007; 28: 33–38.

10.

Arrambide

De la Maza

Perez-Zuno

, et al. The prevalence of multiple sclerosis in the city of San Pedro GG. Mexico. Arq Neuro-Psiquiatr 2004; 62: 7–10.

11.

Velazquez

Marquez

Castano

. Características epidemiológicas de la esclerosis múltiple en un estado fronterizo con los Estados Unidos de Norteamérica. Arch Neurocien 2002; 7: 147–150.

12.

Garg

Smith

. An update on immunopathogenesis, diagnosis, and treatment of multiple sclerosis. Brain Behav 2015; 5: e00362.

13.

Lugosi

Engh

Huszár

, et al. Domain-specific cognitive impairment in multiple sclerosis: A systematic review and meta-analysis. Ann Clin Transl Neurol 2024; 11: 564–576.

14.

Duquin

Parmenter

Benedict

. Influence of recruitment and participation bias in neuropsychological research among MS patients. J Int Neuropsychol Soc 2008; 14: 494–498.

15.

Meca-Lallana

Gascón-Giménez

Ginestal-López

, et al. Cognitive impairment in multiple sclerosis: diagnosis and monitoring. Neurol Sci 2021; 42: 1–11.

16.

Ruano

Portaccio

Goretti

, et al. Age and disability drive cognitive impairment in multiple sclerosis across disease subtypes. Mult Scler J 2017; 23: 1258–1267.

17.

Heitmann

Andlauer

Korn

, et al. Fatigue, depression, and pain in multiple sclerosis: how neuroinflammation translates into dysfunctional reward processing and anhedonic symptoms. Mult Scler J 2022; 28: 1020–1027.

18.

White

Russell

. Invisible and visible symptoms of multiple sclerosis: which are more predictive of health distress? J Neurosci Nurs 2008; 40: 85–95.

19.

Chiaravalloti

DeLuca

. Cognitive impairment in multiple sclerosis. Lancet Neurol 2008; 7: 1139–1151.

20.

Johnen

Landmeyer

Bürkner

, et al. Distinct cognitive impairments in different disease courses of multiple sclerosis—A systematic review and meta-analysis. Neurosci Biobehav Rev 2017; 83: 568–578.

21.

Benedict

Rodgers

Emmert

, et al. Negative work events and accommodations in employed multiple sclerosis patients. Mult Scler J 2014; 20: 116–119.

22.

Cattaneo

Lamers

Bertoni

, et al. Participation restriction in people with multiple sclerosis: prevalence and correlations with cognitive, walking, balance, and upper limb impairments. Arch Phys Med Rehabil 2017; 98: 1308–1315.

23.

Kavaliunas

Danylaite Karrenbauer

Gyllensten

, et al. Cognitive function is a major determinant of income among multiple sclerosis patients in Sweden acting independently from physical disability. Mult Scler J 2019; 25: 104–112.

24.

Kordovski

Frndak

Fisher

, et al.

Identifying employed multiple sclerosis patients at-risk for job loss: when do negative work events pose a threat?

Mult Scler Relat Disord 2015; 4: 409–413.

25.

Rao

Leo

Ellington

, et al. Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology 1991; 41: 692–696.

26.

Ruet

Deloire

Hamel

, et al. Cognitive impairment, health-related quality of life and vocational status at early stages of multiple sclerosis: a 7-year longitudinal study. J Neurol 2013; 260: 776–784.

27.

Benedict

Amato

Boringa

, et al. Brief international cognitive assessment for MS (BICAMS): international standards for validation. BMC Neurol 2012; 12: 1–7.

28.

Benedict

DeLuca

Phillips

, et al. Multiple Sclerosis Outcome Assessments Consortium. Validity of the symbol digit modalities test as a cognition performance outcome measure for multiple sclerosis. Mult Scler J 2017; 23: 721–733.

29.

Carotenuto

. Look beyond the door, not through the keyhole: evidence from a cognitive assessment including social cognition evaluation in multiple sclerosis. Eur J Neurol 2018; 25: 205–206.

30.

Kalb

Beier

Benedict

, et al. Recommendations for cognitive screening and management in multiple sclerosis care. Mult Scler J 2018; 24: 1665–1680.

31.

Cheng

Crandall

Bever

Jr , et al. Quality indicators for multiple sclerosis. Mult Scler J 2010; 16: 970–980.

32.

Arnett

Higginson

Voss

, et al. Depressed mood in multiple sclerosis: relationship to capacity-demanding memory and attentional functioning. Neuropsychology 1999; 13: 434.

33.

Chiaravalloti

DeLuca

. The influence of cognitive dysfunction on benefit from learning and memory rehabilitation in MS: a sub-analysis of the MEMREHAB trial. Mult Scler J 2015; 21: 1575–1582.

34.

Rao

. A manual for the brief repeatable battery of neuropsychological tests in multiple sclerosis. Milwaukee: Medical College of Wisconsin 1990; 1696.

35.

Corfield

Langdon

. A systematic review and meta-analysis of the brief cognitive assessment for multiple sclerosis (BICAMS). Neurol Ther 2018; 7: 287–306.

36.

Potticary

Langdon

. A systematic review and meta-analysis of the brief cognitive assessment for multiple sclerosis (BICAMS) international validations. J Clin Med 2023; 12: 703.

37.

Artemiadis

Bakirtzis

Chatzittofis

, et al. Brief international cognitive assessment for multiple sclerosis (BICAMS) cut-off scores for detecting cognitive impairment in multiple sclerosis. Mult Scler Relat Disord 2021; 49: 102751.

38.

Drebing

Van Gorp

Stuck

, et al. Early detection of cognitive decline in higher cognitively functioning older adults: sensitivity and specificity of a neuropsychological screening battery. Neuropsychology 1994; 8: 31.

39.

Morlett Paredes

Gooding

Artiola i Fortuny

, et al. The state of neuropsychological test norms for Spanish-speaking adults in the United States. Clin Neuropsychol 2021; 35: 236–252.

40.

Testa

Winicki

Pearlson

, et al. Accounting for estimated IQ in neuropsychological test performance with regression-based techniques. J Int Neuropsychol Soc 2009; 15: 1012–1022.

41.

Crawford

Howell

. Regression equations in clinical neuropsychology: an evaluation of statistical methods for comparing predicted and obtained scores. J Clin Exp Neuropsychol 1998; 20: 755–762.

42.

Heaton

Ryan

Grant

, et al. Demographic influences on neuropsychological test performance. 1996.

43.

Parmenter

Testa

Schretlen

, et al. The utility of regression-based norms in interpreting the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). J Int Neuropsychol Soc 2010; 16: 6–16.

44.

Matías-Guiu

Sánchez-Benavides

Rivera-Àvila

, et al. Validation of the Neuronorma battery for neuropsychological assessment in multiple sclerosis. Mult Scler Relat Disord 2020; 42: 102070.

45.

Rivera

Usuga

Mendoza

EMF

, et al. Validation of the Norma Latina neuropsychological assessment battery in individuals with multiple sclerosis in Mexico. Mult Scler Relat Disord 2022; 59: 103685.

46.

Smith

. Symbol digit modalities test (SDMT). Manual (revised). Los Angeles: Western Psychological Services, 1982.

47.

Vanotti

Smerbeck

Eizaguirre

, et al. BICAMS In the Argentine population: relationship with clinical and sociodemographic variables. Appl Neuropsychol: Adult 2018; 25: 424–433.

48.

Fink

Eling

Rischkau

, et al. The association between California Verbal Learning Test performance and fibre impairment in multiple sclerosis: evidence from diffusion tensor imaging. Mult Scler Journal 2010; 16: 332–341.

49.

Stegen

Stepanov

Cookfair

, et al. Validity of the California Verbal Learning Test–II in multiple sclerosis. Clin Neuropsychol 2010; 24: 189–202.

50.

Delis

Kramer

Kaplan

, et al. California verbal learning test. 2nd ed. San Antonio, TX, USA: Psychological Corporation, 2000.

51.

Benedict

. The Brief Visuospatial Memory Test Revised (BVMT-R). Lutz, FL, USA: Psychological Assessment Resources Inc., 1997.

52.

Alarcon

Ayala

García

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in a Colombian population. Mult Scler Relat Disord 2020; 42: 102072.

53.

Betscher

Guenter

Langdon

, et al. Polish validation of the brief international cognitive assessment for multiple sclerosis (BICAMS battery): correlation of cognitive impairment with mood disorders and fatigue. Neurol Neurochir Pol 2021; 55: 59–66.

54.

Botchorishvili

Shiukashvili

Mikeladze

, et al. Validity and reliability of the Georgian-language brief international cognitive assessment for multiple sclerosis (BICAMS). BMC Neurol 2021; 21: 218.

55.

Costers

Gielen

Eelen

, et al. Does including the full CVLT-II and BVMT-R improve BICAMS? Evidence from a Belgian (Dutch) validation study. Mult Scler Relat Disord 2017; 18: 33–40.

56.

Darwish

Zeinoun

Farran

, et al. The brief international cognitive assessment in multiple sclerosis (BICAMS): validation in Arabic and Lebanese normative values. J Int Neuropsychol Soc 2022; 28: 94–103.

57.

Dusankova

Kalincik

Havrdova

, et al. Cross cultural validation of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS) and the brief international cognitive assessment for multiple sclerosis (BICAMS). Clin Neuropsychol 2012; 26: 1186–1200.

58.

Drulović

Tončev

Nadj

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in a large cohort of relapsing-remitting MS patients. Acta Clin Croat 2022; 61: 62–68.

59.

Estiasari

Fajrina

Lastri

, et al. Validity and reliability of brief international cognitive assessment for multiple sclerosis (BICAMS) in Indonesia and the correlation with quality of life. Neurol Res Int 2019; 2019: 4290352.

60.

Evdoshenko

Laskova

Shumilina

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in the Russian population. J Int Neuropsychol Soc 2022; 28: 503–510.

61.

Farghaly

Langdon

Shalaby

, et al. Reliability and validity of Arabic version of the brief international cognitive assessment for multiple sclerosis: Egyptian dialect. The Egyptian Journal of Neurology, Psychiatry and Neurosurgery 2021; 57: 1–8.

62.

Filser

Schreiber

Pöttgen

, et al. The brief international cognitive assessment in multiple sclerosis (BICAMS): results from the German validation study. J Neurol 2018; 265: 2587–2593.

63.

Giedraitienė

Kizlaitienė

Kaubrys

. The BICAMS battery for assessment of Lithuanian-speaking multiple sclerosis patients: relationship with age, education, disease disability, and duration. Med Sci Monit Int Med J Exp Clin Res 2015; 21: 3853.

64.

Hämäläinen

Leo

Therman

, et al. Validation of the Finnish version of the brief international cognitive assessment for multiple sclerosis (BICAMS) and evaluation of the applicability of the multiple sclerosis neuropsychological questionnaire (MSNQ) and the fatigue scale for motor and cognitive functions (FSMC). Brain Behav 2021; 11: e02087.

65.

Marstrand

Østerberg

Walsted

, et al. Brief international cognitive assessment for multiple sclerosis (BICAMS): a danish validation study of sensitivity in early stages of MS. Mult Scler Relat Disord 2020; 37: 101458.

66.

Maubeuge

Deloire

Brochet

, BICAFMS Study Investigators , et al. French Validation of the brief international cognitive assessment for multiple sclerosis. Rev Neurol (Paris) 2021; 177: 73–79.

67.

Niino

Fukazawa

Kira

, et al. Validation of the brief international cognitive assessment for multiple sclerosis in Japan. Mult Scler J–Exp Transl Clin 2017; 3: 2055217317748972.

68.

O’Connell

Langdon

Tubridy

, et al. A preliminary validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) tool in an Irish population with multiple sclerosis (MS). Mult Scler Relat Disord 2015; 4: 521–525.

69.

Ozakbas

Yigit

Cinar

, et al. The turkish validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) battery. BMC Neurol 2017; 17: 1–6.

70.

Polychroniadou

Bakirtzis

Langdon

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in Greek population with multiple sclerosis. Mult Scler Relat Disord 2016; 9: 68–72.

71.

Sandi

Rudisch

Füvesi

, et al. The Hungarian validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) battery and the correlation of cognitive impairment with fatigue and quality of life. Mult Scler Relat Disord 2015; 4: 499–504.

72.

Sousa

Rigueiro-Neves

Miranda

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in the Portuguese population with multiple sclerosis. BMC Neurol 2018; 18: 1–7.

73.

Souissi

Mrabet

Ferchichi

, et al. Tunisian Version of the brief international cognitive assessment for multiple sclerosis: validation and normative values. Mult Scler Relat Disord 2022; 58: 103444.

74.

Skorve

Lundervold

Torkildsen

, et al. The Norwegian translation of the brief international cognitive assessment for multiple sclerosis (BICAMS). Mult Scler Relat Disord 2019; 36: 101408.

75.

Spedo

Frndak

Marques

, et al. Cross-cultural adaptation, reliability, and validity of the BICAMS in Brazil. Clin Neuropsychol 2015; 29: 836–846.

76.

Vanotti

Smerbeck

Benedict

, et al. A new assessment tool for patients with multiple sclerosis from Spanish-speaking countries: validation of the brief international cognitive assessment for MS (BICAMS) in Argentina. Clin Neuropsychol 2016; 30: 1023–1031.

77.

Valdivia-Tangarife

Morlett-Paredes

Villaseñor-Cabrera

, et al. Validation of the brief international cognitive assessment for multiple sclerosis (BICAMS) in individuals with multiple sclerosis from Mexico. Mult Scler Relat Disord 2024; 83: 105451.

78.

Walker

Osman

Berard

, et al. Brief international cognitive assessment for multiple sclerosis (BICAMS): Canadian contribution to the international validation project. J Neurol Sci 2016; 362: 147–152.

79.

BICAMS Committee. US norms for BICAMS. Retrieved February 12, 2013, from. 2013 http://bicams.net/data/us-norms-for-bicams/ .

80.

Benedict

Zgaljardic

. Practice effects during repeated administrations of memory tests with and without alternate forms. J Clin Exp Neuropsychol 1998; 20: 339–352.

81.

Brandt

Benedict

RHB

. The Hopkins Verbal Learning Test Revised: Professional manual. Odessa, FL: Psycho- logical Assessment Resources, Inc, 2001.

82.

Benedict

. Effects of using same-versus alternate-form memory tests during short-interval repeated assessments in multiple sclerosis. J Int Neuropsychol Soc 2005; 11: 727–736.

83.

Benedict

. Effects of using same-versus alternate-form memory tests during short-interval repeated assessments in multiple sclerosis. J Int Neuropsychol Soc 2005; 11: 727–736.

84.

Gronwall

DMA

. Paced auditory serial-addition task: a measure of recovery from concussion. Percept Mot Skills 1977; 44: 367–373.

85.

Benton

Sivan

Hamsher

, et al. Contributions to neuropsychological assessment. 2nd ed. New York: Oxford University Press, 1994.

86.

Delis

Kaplan

Kramer

. Delis-Kaplan Executive Function System. San Antonio, TX: Psychological Corporation, 2001.

87.

Charter

. A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability. J Gen Psychol 2003; 130: 290–304.

88.

Anastasi

. Psychological Testing. 6th ed. Macmillan Publishing Company, 1988.

89.

Cohen

West

, et al. Applied multiple regression/correlation analysis for the behavioral sciences. Routledge, 2013.

90.

Polit

. Getting serious about test–retest reliability: a critique of retest research and some recommendations. Qual Life Res 2014; 23: 1713–1720.

91.

Calamia

Markon

Tranel

. The robust reliability of neuropsychological measures: meta-analyses of test–retest correlations. Clin Neuropsychol 2013; 27: 1077–1105.

92.

Straus

Sherman

EMS

Spreen

. A compendium of neuropsychological tests: administration, norms, and commentary. Am J Hypn 2006: 267–270.

93.

Berrigan

Fisk

Walker

, et al. Reliability of regression-based normative data for the oral symbol digit modalities test: an evaluation of demographic influences, construct validity, and impairment classification rates in multiple sclerosis samples. Clin Neuropsychol 2014; 28: 281–299.

94.

Parmenter

Weinstock-Guttman

Garg

, et al. Screening for cognitive impairment in multiple sclerosis using the symbol digit modalities test. Mult Scler J 2007; 13: 52–57.

95.

Vanderploeg

Axelrod

Sherer

, et al. The importance of demographic adjustments on neuropsychological test performance: a response to reitan and wolf son (1995). Clin Neuropsychol 1997; 11: 210–217.

96.

Zachary

Gorsuch

. Continuous norming: implications for the WAIS-R. J Clin Psychol 1985; 41: 86–94.

97.

Schretlen

Testa

Winicki

, et al. Frequency and bases of abnormal performance by healthy adults on neuropsychological testing. J Int Neuropsychol Soc 2008; 14: 436–445.

98.

Messinis

Nasios

Mougias

, et al. Age and education adjusted normative data and discriminative validity for Rey’s auditory verbal learning test in the elderly Greek population. J Clin Exp Neuropsychol 2016; 38: 23–39.

99.

Stricker

Christianson

Lundt

, et al. Mayo normative studies: regression-based normative data for the auditory verbal learning test for ages 30–91 years and the importance of adjusting for sex. J Int Neuropsychol Soc 2021; 27: 211–226.