Abstract
Background:
The need for more robust outcomes in multiple sclerosis (MS) clinical trials has been a main priority of the field for decades. Dissatisfaction with existing measures has led to several consensus meetings and initiatives over the past few decades in hopes of defining and gaining acceptance of measures that are valid, reliable, sensitive to change and progression, and most importantly, relevant to those living with MS. The Multiple Sclerosis Outcome Assessments Consortium (MSOAC) was formed for this purpose.
Objective:
The objective of this paper is to describe the results of the MSOAC plan to obtain qualification for a cognitive performance measure that meets these requirements.
Methods:
Using data from 14 MS disease-modifying registration trials, we completed a comprehensive examination of the psychometric qualities of the Symbol Digit Modalities Test (SDMT) and the Paced Auditory Serial Addition Test (PASAT) with the goal of compiling evidence to support the utilization of one of these measures in future clinical trials.
Results and conclusion:
Consistent with the published literature, the SDMT proved superior to the PASAT. The SDMT should be considered the measure of choice for MS trials in assessing cognitive processing speed.
Introduction
The work reported in this paper is part of a larger effort by the Multiple Sclerosis Outcome Assessments Consortium (MSOAC) 1 to obtain qualification of a new clinical outcome assessment measure that is clinically validated, sensitive, reliable, practical, cost-effective, and reflective of changes that are meaningful to persons with multiple sclerosis (PwMS). 2 MSOAC was established to (1) develop a methodology to assess the impact of new treatments intended to slow or stop the worsening of multiple sclerosis (MS)-related disability and (2) explore the possibility of obtaining regulatory approval for a multiple component measurement of disability for use as an endpoint for MS clinical trials. The creation of MSOAC was driven by the need to improve the assessment of MS-related disability, which is largely dependent upon the Expanded Disability Status Scale (EDSS). 3 The EDSS has been criticized for lacking measures of cognition, now recognized to be an important dimension of MS. An alternative approach, the Multiple Sclerosis Functional Composite (MSFC), 4 has not been generally accepted, in part, because of the difficulty interpreting the clinical meaning of z-score change, and the inclusion of the Paced Auditory Serial Addition Test (PASAT), a cognitive test that exhibits a marked learning effect and has been shown to be aversive to PwMS as it provokes anxiety.5–7 MSOAC has focused on qualifying a better measure of cognition and a visual measure, and placing a greater emphasis on the limitations in daily activities and fulfillment of life roles of PwMS. A full description of the MSOAC approach to qualification of a new multi-faceted endpoint can be found in LaRocca et al. 2
For the purposes of the present investigation, we report on one of the main goals, which is the qualification of a cognitive performance outcome measure, the Symbol Digit Modalities Test (SDMT), a widely used measure of information processing speed (IPS). 8 This paper describes the research completed to further examine the psychometrics and clinical utility of the SDMT and whether it could serve as a superior measure for use in MS given the long-standing encouragement from the field to consider replacing the PASAT with the SDMT in the MSFC.9,10
At the time that the MSFC was developed, the PASAT was the most likely candidate for inclusion. 11 PASAT was chosen given its sensitivity to change, brevity, ease of administration, lack of reliance on visual or motor function, and reliability. However, work during the ensuing years has revealed an aversion to the PASAT on the part of PwMS. In addition, the SDMT requires less time than the PASAT to complete, requires less assessor expertise, and requires no special equipment since the SDMT is a simple paper and pencil task while the PASAT requires a CD or tape and a CD or tape player, respectively. As a result, it has been suggested that the PASAT be replaced by the SDMT.9,10 The SDMT is a simple, brief measure of IPS. The SDMT does not demonstrate significant ceiling effects, the test–retest reliability is quite sound, and practice effects are less of an issue with the SDMT as there are alternate forms available. 10 Given these qualities, the SDMT has been increasingly used in clinical trials, in some cases alongside the PASAT, and in some trials replacing the PASAT. The SDMT was also recently chosen over the PASAT for use in the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) battery. 12
Recently, literature related to the validity of SDMT further supported the use of SDMT as the best candidate for a cognition outcome measure. 13 Herein, we present a detailed analysis of the psychometric qualities of the SDMT, its sensitivity to change and clinical meaningfulness and its performance compared to the PASAT.
Methods
The MSOAC Defining Disability Workgroup identified several domains relevant to individuals with MS. 2 Among the criteria was the ability to quantify the domain using performance measures that could be operationalized in a multi-center study. The MSOAC selected ambulation, manual dexterity, vision, and cognition as domains of interest and assessed commonly used measures of these domains—namely, the Timed 25-Foot Walk (T25FW), Nine-Hole Peg Test (9HPT), Low-Contrast Letter Acuity (LCLA), and the SDMT. Clinical trial data (active and comparator arms) were shared by sponsors and mapped to the Clinical Data Interchange Standards Consortium (CDISC) standard. 2 For the present investigation, the following additional measures of interest were included: EDSS, PASAT, Health Status Questionnaire (SF-36) as a measure of health-related quality of life (HRQOL), and the Beck Depression Inventory–Second Edition (BDI-II) as a measure of psychological functioning.
Statistical analyses
The first aim was to examine the psychometric qualities of the SDMT, including score distribution, validity, and reliability. We conducted descriptive analyses of SDMT and PASAT at various time points, examined Spearman correlations with relevant clinical measures, and examined intraclass correlation coefficients of SDMT and PASAT. Practice effects on SDMT and PASAT were assessed using a mixed linear regression model using multiple measures from the same PwMS during a period in which PwMS were expected to be clinically stable as defined by an unchanged EDSS score during a period no later than 6 months after baseline. The number of times the PwMS had completed the test and the interval since the previous assessment (categorized as 0–15 days, 16–30 days, 31–90 days, and >90 days) were included as fixed effects, as categorical variables, and subject was included as a random effect.
For analyses of cognitive disability worsening, survival estimates for the percentage of patients with disability worsening at various time points were examined and presented in graphical form in Kaplan–Meier plots.
The second aim was to assess sensitivity to change and clinical meaningfulness of the SDMT. Sensitivity to change was evaluated using paired t-tests to compare SDMT scores before and after events expected to result in improvement or worsening and examining the association of SDMT with changes in HRQOL (SF-36). Finally, to examine the relationship of SDMT scores to known-group differences, an analysis of variance (ANOVA) was used to compare SDMT performance of various disability groups. PwMS were classified as having longer (⩾10 years) versus shorter (<10 years) disease duration; and higher (4–10) versus lower EDSS (0–3.5). An ANOVA was conducted adjusting for age in 5-year bands. To adjust for the effect of age, least squares (LS) mean scores are presented. All analyses were also conducted with the PASAT–3 second administration for comparison.
Results
Baseline characteristics
Total sample size was 12,776 and three datasets were compiled: one that included both the PASAT and the SDMT (Both Set; N = 1512), one that included the SDMT (SDMT Set; N = 2586), and one that included the PASAT (PASAT Set; N = 11,702) (See Table A1 in Supplementary Appendix).
Descriptive statistics
The SDMT had a near normal distribution, but the PASAT had a substantially negatively skewed distribution (Figure 1(a) and (b)), suggesting a pronounced ceiling effect. Locally weighted scatterplot smoothing was applied to the data to examine scores over time (See Figures A1 and A2 in Supplementary Appendix). Scores on both the SDMT and PASAT were found to increase over time, despite the fact that MS tends to worsen over time, suggesting a practice effect. SDMT scores were only available for up to 2 years in duration, while PASAT scores were available for up to 5 years (Table 1).
(a) Distribution of Symbol Digit Modalities Test (SDMT) at baseline. The histogram of SDMT scores at baseline, representing 2583 PwMS, is presented. (b) Distribution of Paced Auditory Serial Addition Test (PASAT) scores at Baseline. The histogram of PASAT scores at baseline, representing 11,609 PwMS, is presented.
Participant characteristics.
M: male; F: female; RR: relapsing remitting; SP: secondary progressive; PP: primary progressive; SDMT: Symbol Digit Modalities Test; PASAT: Paced Auditory Serial Addition Test; SD: standard deviation; EDSS: Expanded Disability Status Scale.
Construct validity analyses
Spearman correlations comparing the SDMT and physical clinical measures were performed to examine construct validity (Table 2). Results suggest that the SDMT has good construct validity given its modest associations with measures of physical disability with rs ranging from 0.34 to 0.47. The strongest correlations were with the 9HPT (r = 0.47) and the T25FW (r = −0.42). PASAT demonstrated significantly weaker associations with all clinical measures (rs = 0.18 to 0.34).
Spearman correlation coefficients (CC) for the Symbol Digit Modalities Test (SDMT) and Paced Auditory Serial Addition Test (PASAT) with clinical measures of disability at baseline and change from baseline at endpoint.
EDSS: Expanded Disability Status Scale; 9HPT: Nine-Hole Peg Test; T25FW: Timed 25-Foot Walk; LCLA: Low-Contrast Letter Acuity; PASAT: Paced Auditory Serial Addition Test.
Convergent validity analyses
SDMT and PASAT were significantly correlated (r = 0.54), supporting convergent validity and indicating that both measures assess a similar cognitive construct. However, the measures are clearly not strongly correlated, suggesting that each measures one or more aspects of cognition independent of the other (Table 2).
Further evidence that the SDMT assesses domains not captured by physical markers is the relative lack of correlation between SDMT and change scores of disease measures. More specifically, these associations are typically less than 0.1 and not always significant. This finding suggests that changes in cognitive status may occur independently of changes in physical markers. These correlations are also lower with the PASAT (Table 3).
Correlation between changes in SDMT and PASAT with change scores in clinical measures.
SDMT: Symbol Digit Modalities Test; PASAT: Paced Auditory Serial Addition Test; EDSS: Expanded Disability Status Scale; 9HPT: Nine-Hole Peg Test; T25FW: Timed 25-Foot Walk; LCLA: Low-Contrast Letter Acuity.
Reliability analyses of test–retest reliability and practice effects
Both SDMT and PASAT were found to have good reliability with an intraclass correlation of 0.85 for the SDMT and 0.86 for the PASAT. Scores on both improved over time, presumably due to practice effects (Figures A1 and A2 in Supplementary Appendix). Effect sizes were calculated from regression coefficients for test number from mixed effects models. Data in Table 4 demonstrate that improvement was 2–3 times greater for the PASAT than for the SDMT. For example, the effect size for SDMT improvement after three administrations was 0.10, compared with 0.27 for the PASAT. After six administrations, the effect size for SDMT was 0.27, compared with 0.50 for PASAT.
Regression coefficients for practice effects of the Symbol Digit Modalities Test (SDMT) and Paced Auditory Serial Addition Test (PASAT) expressed as effect sizes.
CC: correlation coefficient; CI: confidence interval.
For SDMT, the no. of observations = 8567 and the number of individuals = 2094.
For PASAT, the no. of observations = 24,327 and the number of individual = 7962.
Known-group analyses
Findings suggest that individuals with a longer disease duration perform, on average, 3.31 points lower on the SDMT. When conducting a similar ANOVA with EDSS, a larger difference between the two groups was observed. As expected, individuals with a higher EDSS had lower SDMT scores. On average, this difference was 8.60 points, suggesting that the SDMT is more closely related to disease severity as measured by EDSS than it is to disease duration. These findings further confirm the validity of the SDMT given the expectation that individuals with higher EDSS and longer disease duration would have greater cognitive deficits and therefore lower scores on the SDMT. Similar findings were noted with the PASAT, although the magnitude of the differences was smaller (See Table 5).
Difference in scores on the Symbol Digit Modalities Test (SDMT) and Paced Auditory Serial Addition Test (PASAT) among differing disease durations and Expanded Disability Status Scale (EDSS) scores.
CI: confidence interval.
Sensitivity to change
Analyses of cognitive disability worsening, as calculated by the survival estimates for the percentage of PwMS worsening at various time points, are shown in the Kaplan–Meier plots (Figure 2). Time to worsening on EDSS is included as a reference for each plot. As expected, the Kaplan–Meier curves differ for the various definitions of worsening in cognition as measured by SDMT (e.g. 20% deterioration and 15% deterioration). The definition of disability worsening for the SDMT that most closely mirrors the pattern of EDSS disability worsening is a worsening in SDMT score of 15% from baseline. While similarities in the pattern of worsening were observed for SDMT and EDSS, individuals worsening on the EDSS were different from individuals worsening on the SDMT. Kappa coefficients were calculated to quantify agreement between EDSS and SDMT worsening. Kappa coefficients ranged from −0.02 to +0.03, depending on the definition of SDMT worsening (data provided in Figure 2 legends), indicating that correspondence between EDSS and SDMT worsening was low. This suggests that the SDMT is capturing MS-related changes not detected by the EDSS.
Kaplan–Meier graphs of disability worsening by various SDMT definitions.
Relationship of cognitive scores to HRQOL
We found that while the SDMT has modest associations with physical performance measures, it also has a significant association with measures assessing HRQOL and psychological functioning (Table 6). Moreover, the association between the SDMT and the Physical Component Summary (PCS) of the SF-36 was significantly greater than between the PASAT and PCS.
Spearman correlation coefficients of the Symbol Digit Modalities Test (SDMT) and Paced Auditory Serial Addition Test (PASAT) with measures of health-related quality of life and psychological functioning.
PCS: Physical Component Summary of the SF-36; MCS: Mental Component Summary; BDI-II: Beck Depression Inventory–Second Edition.
We utilized changes in the PCS of the SF-36 as an indicator of change relevant to PwMS for two reasons. First, SF-36 was the one patient-reported outcome measure that was common across all clinical trials in the MSOAC database. Second, there is agreement that a 5-point worsening on the PCS is considered clinically meaningful. 14 We compared the proportion of individuals with and without worsening on various definitions of worsening on the SDMT to clinically meaningful worsening on the PCS (>5 points) to determine what magnitude of SDMT worsening would correlate with a meaningful worsening on the PCS (Table 7). More specifically, we defined populations of PwMS who worsened on the SDMT by the following definitions—20%, 15%, 10%, 3 points, or 4 points from baseline. We then compared the risk of worsening on PCS with worsening on SDMT.
Proportion of patients with a 5-point worsening on the PCS among patients with and without worsening on various clinical measures and the Symbol Digit Modalities Test (SDMT).
PCS: Physical Component Summary of the SF-36.
Findings suggest that a decline in performance on the SDMT using any of the definitions was associated with worsening on the PCS, though in most cases these relationships were not statistically significant. A 4-point or greater worsening on the SDMT was the strongest predictor of a concomitant worsening on PCS. Notably, this degree of change in the SDMT has been reported to be clinically meaningful, using relapses and employment as clinical anchors.15,16
Discussion
One of the MSOAC’s goals is to qualify a cognition performance measure that is clinically validated, sensitive, reliable, practical, cost-effective, and reflective of meaningful changes to PwMS. The SDMT was chosen as such a measure. Previously, the SDMT has been found to be associated with disease progression as measured by various magnetic resonance imaging markers of MS disease severity,17–20 with these associations being greater for SDMT than PASAT. 21 SDMT has also been shown to be a significant predictor of several functional, patient-related outcomes in MS including employment, driving, and instrumental daily activities.22–24 Performance on the SDMT is also predictive of future cognitive decline. 25 For these reasons, the SDMT has been included in most neuropsychological test batteries designed specifically for MS26,27 and was deemed the only core common data element measure of cognition for use in MS by the National Institute of Neurological Disorders and Stroke. Thus, in many instances, the SDMT has been found to be the most sensitive individual cognitive measure for use in MS. Due to its predictive validity, high sensitivity and specificity, ease of administration, and patient-friendliness, the SDMT is frequently included in clinical practice to help identify PwMS at greatest risk for cognitive impairment, poor outcomes such as unemployment, and disease progression. This paper provides a detailed statistical analysis of the psychometric qualities and clinical utility of the SDMT compared to PASAT, adding to the existing literature on this subject. By every measure, SDMT proved superior to PASAT and should therefore be considered the measure of choice for MS trials for assessing cognitive processing speed.
For instance, the SDMT had a normal distribution of scores with little or no skewness. In contrast, the PASAT showed a severely negatively skewed distribution, indicative of pronounced ceiling effects. Both instruments demonstrated practice effects which persisted over the course of the trials; however, the practice effects shown by the PASAT were larger. In addition, both instruments showed moderate but significant correlations with physical measures such as the EDSS and the T25FW; however, the SDMT showed higher values. Moreover, while the SDMT correlated moderately with the physical measures, the modest values of these correlations indicate that the SDMT is capturing facets of MS-related disability not encompassed by the physical measures. Furthermore, the association between the SDMT and EDSS over time indicated a divergence, further suggesting that changes in cognition over time may be occurring independently of physical impairment(s). Such findings further support the contention that a cognitive measure is imperative for use in clinical trials and in the long-term assessment of individuals with MS, as it captures functional change that may be even more meaningful than the physical assessment.
Related to this, of particular importance was the relationship between the SDMT and the PASAT with HRQOL. Both showed significant correlations with the PCS. However, the SDMT correlated with the PCS more strongly than the PASAT. We also examined the alignment between changes in the PCS and cognitive measures. This analysis revealed that a 4-point worsening on the SDMT was correlated with clinical decline as evidenced by a 5-point worsening on the PCS. It is important to note that in the literature, a 4-point change in the SDMT and a 5-point change in the PCS are both considered to be clinically meaningful, as reviewed in Benedict et al. 13 In contrast, neither the SDMT nor the PASAT was strongly related to change in the physical measures. This finding is particularly important because it points to the fact that cognition, which is adversely affected in more than half of individuals with MS, is not captured by commonly used physical measures. The implication is that in order for clinical trials to adequately address MS-related disability, a cognitive measure such as the SDMT must be included.
Data suggest that a worsening from baseline SDMT scores, defined as a 10%, 15%, or 20%, or a 3-point or 4-point change sustained for 3 months occurs in a proportion of PwMS similar to the proportion worsening using the traditional EDSS definition (Figure 2), providing evidence for sensitivity to change of the SDMT. More importantly, worsening on the SDMT occurs independently from worsening on the EDSS. These findings further support the inclusion of a cognitive measure to fully capture MS disability progression.
While the SDMT and the PASAT did not correlate strongly with the short-term changes observed for physical measures in trials, the cognitive measures did correlate well with long-term changes as evidenced by their strong relationship in the known-group analyses. Both the SDMT and the PASAT showed robust relationships with lower versus higher EDSS scores and with shorter versus longer duration. Those known groups represent, in dichotomous form, the effects of MS over time periods longer than is possible in a clinical trial. These findings provide strong support for the idea that the SDMT and, to a lesser extent, the PASAT, are sensitive to clinically meaningful change, although to a limited extent when measured over a short period of time.
To summarize, our analyses have shown that the SDMT is a valid and reliable method to assess clinically meaningful change in PwMS and captures variance in MS-related disability not encompassed by physical measures. These findings suggest that the most common endpoints used in disease modifying therapy (DMT) registration trials, annualized relapse rate and sustained EDSS progression, miss an important source of MS-related disability, namely, decline in cognitive function. Moreover, utilization of the endpoint known as no evidence of disease activity (NEDA) has significant shortcomings if it does not include cognition. Future DMT registration trials need to expand their horizons to include cognition if they are to fully assess MS-related disability in ways that are meaningful to PwMS. This study has provided strong support for the use of the SDMT to fulfill this need.
Supplemental Material
MSJ808204_appendix – Supplemental material for Symbol Digit Modalities Test: A valid clinical trial endpoint for measuring cognition in multiple sclerosis
Supplemental material, MSJ808204_appendix for Symbol Digit Modalities Test: A valid clinical trial endpoint for measuring cognition in multiple sclerosis by Lauren Strober, John DeLuca, Ralph HB Benedict, Adam Jacobs, Jeffrey A Cohen, Nancy Chiaravalloti, Lynn D Hudson, Richard A Rudick and Nicholas G LaRocca in Multiple Sclerosis Journal
Footnotes
Acknowledgements
The authors acknowledge Gary Lundstrom for expert project management and Alicia West for assistance with the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Multiple Sclerosis Outcome Assessments Consortium (MSOAC) is funded largely through the National Multiple Sclerosis Society grant number RG 4869-A-1 to the Critical Path Institute. Annual dues from sponsors supplement the NMSS grant.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
