Abstract
Background:
Apathy is a common debilitating symptom of myotonic dystrophy type 1 (DM1). The Apathy Evaluation Scale (AES) has been identified as a promising measurement instrument to be used in DM1 but its metrological properties must be further documented.
Objective:
To determine the internal consistency of the Self (AES-S), Informant (AES-I), and Clinician (AES-C) versions of the AES and to assess the test-retest reliability, standard error of measurement, and minimal detectable change of the AES-S and AES-I in a sample of DM1 patients and their related informants.
Results:
All scales showed good internal consistency (Cronbach’s alpha: 0.83–0.87) and the AES-S and AES-I showed good test-retest reliability (ICC = 0.79–0.91). Additionally, clinicians and informants had a tendency to overestimate DM1 patients’ level of apathy compared to patients’ self-ratings, suggesting potentially impaired self-awareness in DM1 patients.
Conclusions:
The present results advocate the use of the AES-I as a reliable instrument to estimate apathy in DM1 patients for either clinical or research purposes, and support the relevance to pursue the assessment of metrological properties of the AES as a tool of great value for the development of outcomes for clinical trial readiness in DM1.
HIGHLIGHTS
The 3 versions of the AES present good internal consistency. The AES-I is a reliable scale to assess apathy in DM1. The AES-S shows poor reliability in DM1.
INTRODUCTION
Myotonic dystrophy type 1 (DM1), an autosomal dominant disorder, is the most common adult form of muscular dystrophy [1, 2]. DM1 results from an unstable CTG repeat expansion in the 3’ untranslated region of the chromosome 19q13.3 [3]. DM1 is not only a muscle disease but a multisystemic disorder affecting the muscular, central nervous, ocular, respiratory, cardiovascular, digestive, endocrine and reproductive systems [4, 5].
Adult DM1 patients often experienced emotional blunting, apathy, and an inability to anticipate pleasure (anhedonia), which are not related to the presence of anxious or depressive symptoms that tends to appear early in the progression of the disease [6]. Also, apathy is highly prevalent (40%) in DM1 patients compared to matched healthy controls [7]. While the reliability and validity of daytime sleepiness and fatigue scales have been largely documented in DM1 [8–11] no study has yet established metrological properties of apathy rating scales in the DM1 patient population. Hobart (2003) earlier underlined that the soundness of clinical evidence substantially depends upon the applicability of an instrument to the population of interest [12].
Globally, a barrier to the assessment of the efficacy of treatments for DM1 patients is the paucity of data concerning the validity and the reliability of key outcome measures for this condition. Launched in 2011, the Outcome Measures in Myotonic Dystrophy type 1 (OMMYD) international initiative aimed to address and resolve this problem [13, 14]. As part of the OMMYD initiative, the special interest group on daytime sleepiness, fatigue, and apathy identified the need to determine metrological data for rating scales addressing these symptoms in the DM1 population [14]. In this regard, the widely used Apathy Evaluation Scale (AES) [15] has been identified as promising since it is one of the most robust measure for assessing apathy for several diseases [16]. The AES consists of 3 versions: a clinician version (AES-C) based on clinical observations, an informant version (AES-I) such as a family member, friend or caregiver whose estimations are based on direct observation of the patient, and finally, a self-rated version (AES-S).
The objectives of this study were to document the internal consistency of the 3 AES versions and to assess the test-retest reliability, standard error of measurement (SEM), and minimal detectable change of the AES informant and self-rated versions. A secondary objective was to analyze the agreement between the 3 versions of the AES.
METHOD
Participants
A total of 50 patients with DM1 were randomly selected from a registry of adult patients followed at the Saguenay Neuromuscular Clinic (Québec, Canada). Inclusion criteria were 1) being 18 years of age or older, 2) having a molecular confirmation of DM1 diagnosis, 3) having the juvenile, the adult or the late-onset phenotype of the disease (respectively age of disease onset >10, >20, and >40 years old), and 4) providing written informed consent. Patients with congenital and infantile phenotypes were excluded from this study. Two participants abandoned during the project due to time constraints. Of the 48 participants, 26 had an informant accepting to fulfill the AES-I version. The study was approved by the Ethics Review Board of the Centre intégré universitaire de santé et de services sociaux (CIUSS) du Saguenay–Lac-Saint-Jean (Québec, Canada).
Measures
Information was gathered regarding age, sex, and educational level for both participants and informants. Participants were also asked about their level of independence (independent; need partial assistance; need complete assistance) and the use of walking aid in their home (no walking aid; cane/walker; wheelchair). The muscular impairment was assessed by the Muscular Impairment Rating Scale (MIRS), an ordinal five-point rating scale [17]. CTG repeat number was taken from patients’ medical record. The protocol required filling out the AES-S and AES-I on 2 occasions (T1, T2) within a 2-week interval. In order to improve the feasibility of the study, participants were first asked to come at the Neuromuscular clinic to be administered the clinician version of the AES. One week later, the AES-S and AES-I were administered by phone. As for the AES-C, it was administered by a trained graduate student in psychology under the supervision of a psychologist with extensive experience with neuromuscular patients (B.G.).
AES
All 3 versions of the AES contain the same 18 core items and are based on the subject’s thoughts, emotions, and actions over the previous 4 weeks (Supplementary Material) [15]. The AES-I and AES-S are administered as paper-and-pencil self-administered questionnaires. The clinician version of the AES is rated based on a semi-structured interview starting with questions probing for the subject’s interests, activities, and daily routines. Moreover, clinician rating is based on both verbal and nonverbal information, which means that the interview is viewed as a “motivational laboratory” that “provides a valid sample of subject’s overall motivation in other situations”.
Clinicians therefore use patient’s verbal and non-verbal behaviours during the interview to rate the level of patient’s apathy. Items address the affective, behavioral, and cognitive domains of apathy and are rated on a 4-point scale where 1 reflects a characteristic “not at all present” and 4 reflects a characteristic which is present “a lot”. Total scores vary between 18 and 72 points, a higher score indicating a higher level of apathy (after recoding some items).
Statistical analysis
Statistical analyses were performed in accordance with COSMIN guidelines for methodological quality of outcome measures [18]. Internal consistency, i.e. the extent to which all items in a scale are interrelated, was assessed with Cronbach’s alpha reliability coefficient from the first questionnaire completion. The desired range for the alpha to be considered as good is between 0.70 and 0.90 [19]. For the AES-S and AES-I versions, the test-retest reliability was determined using the intraclass correlation coefficient (ICC) to compare both results (T1 vs T2) [20]. An ICC above 0.75 was considered as indicative of good reliability [21]. The SEM was calculated by multiplying the baseline SD of each outcome measure by the square root of one minus the reliability coefficient; SEM = SDbaseline×√(1 – ICC). The minimal detectable change (MDC), which is defined as the amount of change in a variable that must be achieved to reflect a true difference [21], was calculated by multiplying 1.96 by the square root of 2 times the SEM; MDC95 = 1.96×√2×SEM. The MDC was further expressed as a percentage of the baseline mean.
For the inter-version agreement comparisons, the strength and direction of the relationships were assessed using Pearson r correlation coefficients. Bland and Altman plots [22] were also used to evaluate the agreement between the 2 assessments of the test-retest and between the 3 versions of the AES (AES-C, AES-S, AES-I). Difference between scores at the 2 time points (T2-T1) were plotted against the mean scores of T1 and T2. The 95% limits of agreement (mean±1.96SD), which illustrates the magnitude of the systematic difference, were also added to see the level of agreement, i.e. how much the score of the second assessment is below or above the score of the first assessment. The same procedure was performed between the total scores of the 3 versions by pairing AES-C vs AES-S, AES-C vs AES-I, and AES-S vs AES-I; the limits of agreement then represent how much the score of one version is below or above the score of the other version. The mean difference between the 2 assessments must be as close as possible to zero (i.e., group level of interpretation) and the 95% limits of agreement must be as small as possible (i.e., individual level of interpretation). In addition, a cut-off score of 34 was used to classify participants as apathetic (AES score≥35 indicating apathy) or not [23–26]. All data were analysed using IBM SPSS Statistics for Windows, Version 24.0 (Armonk, NY: IBM Corp).
RESULTS
Sociodemographic and clinical characteristics of participants
Sociodemographic data of participants and informants are shown in Table 1. Participants’ age ranged from 20 to 64 years and 66.7% of participants had less than 12 years of schooling. Most participants did not have a walking aid inside (85.4%) and qualified themselves as independent in their daily activities (87.5%). Three out of 5 DM1 patients (60.4%) had a MIRS score of 4 or 5, indicating proximal weakness. Mean CTG repeat number was 715 (SD 480). Informants were mainly spouses (65.4%) and were aged 17 to 79 years. The mean level of fatigue/somnolence was 9.4, according to the FDSS.
Sociodemographic characteristics of the 48 participants and 26 informants
*Corresponds to 7 years of schooling or less. **Corresponds to 8 to 12 years of schooling. CTG: Cytosine-thymine-guanine; MIRS: Muscular impairment rating scale; FDSS: Fatigue and daytime sleepiness scale.
Internal consistency and test-retest reliability
The 3 versions of the AES showed a good internal consistency (AES-C = 0.87; AES-S = 0.83; AES-I = 0.86), indicating that items are strongly correlated to each other. Also, the AES-S and AES-I questionnaires showed good test-retest reliability (ICC > 0.75) (Table 2), although the ICC was smaller for the AES-S than for the AES-I (0.79 < 0.91). Considering the maximum score of each questionnaire, the AES-I also presented a smaller variability of the measure attributable to the measurement error than the AES-S, with a SEM respectively of 3.2 vs. 4.1. It also had smaller estimates of clinically important change, as defined by the use of the MDC (AES-I MDC = 8.9 <AES-S MDC = 11.4). This result indicates that the AES-I score must change of at least 8.9 points (24% of the baseline mean score) to reflect a true difference, while the AES-S score must change of 11.4 (33% of the baseline score) to reflect a true difference.
Reliability and standard error of the measurement (n = 48)
SD: standard error, ICC: intraclass correlation coefficient [2,1, 2,1], CI: confidence interval, SEM: standard error of the measurement, MDC95: minimal detectable difference (at 95% CI), % MDC95: ratio MDC95 / baseline mean in percent.
Using the Bland and Altman (BA) plot method at the group level, Fig. 1 shows that the repeated measurements in the AES-S and AES-I versions are sufficiently similar. Indeed, the average of paired differences is very close to zero (mean AES-S = 0.71; mean AES-I = 1.08), and the zero line is comprised within the confidence interval of the mean differences, revealing no or negligible systematic bias between the test-retest scores. However, at the individual level, the 95% CI of agreement limits are higher in the AES-S than in the AES-I, ranging respectively from –10.4 to 11.8 and from –7.7 to 9.8.
Bland and Altman plots for test-retest agreement of the A) self- and B) informant-rated versions of the AES. The mean of differences with its confidence intervals (doted lines) are illustrated. Limits of agreement are represented by the Upper and Lower limit lines (mean±1.96SD).
Comparisons between the AES-C, AES-I, and AES-S
Results obtained with the 3 versions of the AES were compared to determine their relationship with each other. Results show that the AES-C was significantly correlated with the AES-S (r = 0.52; p < 0.001) and the AES-I (r = 0.45; p = 0.02), but no significant correlation was found between the AES-S and AES-I versions (r = 0.29, p = 0.15).
Using the BA plot method to compare the AES-C and the AES-S versions (Fig. 2A), it was noted that the average of paired differences is quite important (mean = 6.5), suggesting a tendency for the clinician to score apathy on average 6.5 points above scores given by patients themselves. Additionally, the zero line is not comprised within the confidence interval of the mean differences, revealing a significant systematic bias between the 2 versions. The 95% confidence interval (CI) of agreement limits are also high (range from –9.9 to 22.9), showing an important discrepancy between the 2 versions at the individual level.
Bland and Altman plots for inter-versions agreement of the A) AES-C versus AES-S, B) AES-C versus AES-I, and C) AES-I versus AES-S. The mean of differences with its confidence intervals (doted lines) are illustrated. Limits of agreement are represented by the Upper and Lower limit lines (mean±1.96SD).
The comparison of the AES-C and AES-I versions (Fig. 2B) revealed that the average of paired differences is less important (mean = 3.9) than that of the AES-C and AES-S versions, the zero line being comprised within the confidence interval of the mean differences reveals no or negligible systematic bias between those 2 versions. However, at the individual level, the 95% CI of agreement limits range from –15.7 to 23.5, which seems substantial.
Finally, when comparing the AES-I and the AES-S versions (Fig. 2C), the average of paired differences appears quite small (mean = 2.5). Also, the zero line being comprised within the confidence interval of the mean differences reveals no or negligible systematic bias between the 2 versions. Both the considerable range of the 95% CI of agreement limits (from –21.0 to 26.0) as well as the points cloud shape clearly show that the disagreement between informants and patients increases as the latter report to be more apathetic (threshold observable for mean scores over 30).
Assessment of agreement of diagnosis established with AES versions
As shown in Table 3, 9 of the 26 participants for whom the 3 versions were completed obtained the same diagnosis (2 non-apathetic and 7 apathetic; see Table 3). The other 17 patients obtained different diagnoses depending on the version used.
Comparison of the apathy diagnosis obtained with the 3 AES versions (n = 26)
Clinician = AES-C; Self = AES-S; Informant = AES-I.
DISCUSSION
Test-retest reliability
All 2 AES versions have shown good test-retest reliability (ICC > 0.75) in a sample of 48 DM1 patients. However, the 95% CI of agreement limits reflected relatively high intra-individual variability as regards the AES-S and hence limited usefulness of measures for individual comparisons. Even though the mean differences are negligible for the AES-S and AES-I versions, DM1 patients self-assessed mean apathy scores on the AES-S varied from –10.4 to 11.8 points within a 2-week period. Such changes in scores may potentially lead to a different diagnosis (apathetic vs non-apathetic, based on cut-off scores) from one assessment to another, in a short period of time. This may be due to the lack of awareness observed clinically in this patient population, as recently described by Baldanzi et al. [27]. Since the AES-S refers to patients’ point of view, it requires adequate abilities in self-criticism and mental shifting that is often impaired in DM1 due to poor judgement and executive impairments [28, 29]. For example, when answering questions, DM1 patients may focus on one or 2 events that happened in their lives very recently, whereas they are asked to refer to a 4-week period and to their “general” habits. Also, the present reliability study supports the relevance to pursue the assessment of the AES-I as a tool that can potentially detect change in DM1 patients.
Comparisons between the 3 versions of the AES
Despite significant correlations between the AES-C and the other AES versions, a tendency for clinicians to overrate apathy compared to patients and caregivers was noted. As mentioned above, the presence of disease unawareness in DM1 [27] may explain the tendency of patients to give themselves lower apathy scores than clinicians and informants do. Since the AES-S version is based on their own point of view about activity, participation, interests, and emotions, this is self-explanatory that AES-S scores may be influenced by patients’ level of self-awareness. Similar results were found in patients with brain injury whose family members were more likely to rate patients higher on the AES-I than the patients’ self-report on the AES-S [23]. Authors interpreted this discrepancy as a result of diminished self-awareness observed following head injury due to frequent frontal lobe injury.
Another factor that may result in lower scores of apathy as reported by patients, compared to that reported by clinicians and informants, is that apathy evaluated by means of the AES should be considered relatively to the norms for patients’ age and culture [30]. The level of expected activity and motivation of individuals experiencing chronic and slowly progressive disease may be modified by a process of coping to illness. As a result, patients with DM1 may have norms that are in disagreement with expected norms from caregivers or clinicians.
Again, those results confirmed the importance of using the BA plots procedure for the analyses of agreement between instruments. As stated by Giavarina (2015), a correlation is not recommended to compare methods since it only indicates the relationship between 2 measures and does not take into consideration the difference between these measures [31]. Presently, BA plots and diagnosis agreement between the 3 versions showed that the 3 versions of the AES present good internal consistency and that the AES-I version present good test-retest reliability, but also revealed that the 3 versions reflect different points of view on apathy. Based on the statistical results, the use of the informant version is more reliable than the patient’s version for test-retest. This is in line with the results of the validation study made originally by Marin et al. in several diseases [15], where the overall validity of the AES-S was less than that of the AES-I.
A limitation of the present study is that the AES-Clinician version was only completed once due to patient’s time constraint and burden. Future studies must assess test-retest reliability as well as the inter-rater reliability of the AES-C. Moreover, further studies should explore the AES-C concurrent validity.
CONCLUSIONS
In all, the 3 versions of the AES showed good internal consistency. The AES informant version showed good test-retest reliability. While further studies are necessary to validate some of the AES-C properties, the present study supports 1) the use of the AES-C as a tool for clinician to assess apathy in DM1 when assessed on one single occasion only (e.g. for descriptive studies of apathy, or when apathy is needed to be assessed as a control factor), and 2) the use of the AES informant version as a reliable instrument to measure apathy in patients with DM1 for either clinical or research purposes.
DECLARATION OF CONFLICTS OF INTEREST
All authors declare to have no competing interest.
Footnotes
ACKNOWLEDGMENTS
This research was financially supported by the Association française contre les myopathies (grant no. GM/NF/2013-0767 # Dossier 16885) and the Marigold Foundation. CG holds a career-grant funding from Fonds de recherche en santé du Québec [grant no. 31011]. BG holds a postdoctoral fellowship from Myotonic dystrophy Foundation/Wyck Foundation.
