Abstract
Remote assessments of individuals with a neurological disease via telemedicine have the potential to reduce some of the burdens associated with clinical care and research participation. We aim to evaluate the feasibility of conducting the Montreal Cognitive Assessment remotely in individuals with movement disorders. A pilot study derived from two telemedicine trials was conducted. In total, 17 individuals with movement disorders (8 with Parkinson disease and 9 with Huntington disease) had Montreal Cognitive Assessment examinations evaluated in-person and remotely via web-based video conferencing to primarily determine feasibility and potential barriers in its remote administration. Administering the Montreal Cognitive Assessment remotely in a sample of movement disorder patients with mild cognitive impairment is feasible, with only minor common complications associated with technology, including delayed sound and corrupted imaging for participants with low connection speeds. The Montreal Cognitive Assessment has the potential to be used in remote assessments of patients and research participants with movement disorders.
Introduction
Individuals with a neurological disease generally require routine doctor visits, consisting of motor and cognitive assessments, which overtime may become burdensome to the individual and their caregivers. Participation in clinical research studies also becomes an added challenge for these individuals, particularly with travel and time as major barriers, subsequently hindering recruitment and retention. At-home assessments via web-based video conferencing may reduce such inconveniences by eliminating travel and wait times, reducing caregiver burden, and providing a comfortable environment applicable to the subject’s daily living, while still having the face-to-face interaction with the clinician. Additionally, the utilization of telemedicine for remote web-based assessments could substantially decrease administrative costs and permit the use of a single centralized rater for important clinical trial outcomes.
Recent studies among individuals with neurodegenerative disease have demonstrated the feasibility of conducting motor assessments remotely1,2 and have been shown to be effective in enhancing enrollment into research studies and trials. 3 These clinical assessments, modified when conducted remotely to only include items that can be evaluated visually, 4 demonstrated excellent reliability and validity when compared to the in-person assessment. Cognitive assessments are also important clinical tools used frequently in this population. The Montreal Cognitive Assessment (MoCA) is a common and simple screening tool for detecting mild cognitive impairment, consisting of visual (visuospatial/executive, naming) and verbal (attention, language, abstraction, delayed recall, orientation) portions. 5 Although the MoCA has been deemed reliable and well validated in individuals with movement disorders such as Parkinson disease (PD)6–9 and Huntington disease (HD),10–12 the feasibility of remote administration has yet to be reported in this population. Therefore, using web-based video conferencing, we evaluated the feasibility and potential validity of assessing the MoCA remotely in patients with PD and HD.
Methods
This was a feasibility study derived from two separate PD 13 and HD 14 telemedicine trials evaluating patient symptoms longitudinally using remote monitoring. Study participants were recruited from the HD Society of America 2012 annual convention in Las Vegas, NV, as well as regionally at the University of Rochester’s movement disorder clinic. Patients who had a clinical diagnosis of PD or HD and had access to a printer and computer with high-speed internet were eligible to participate. The initial baseline study visit was conducted in-person during which participants were consented and asked information on demographics and medical history. Among those with PD, the study’s movement disorder specialists evaluated baseline scores for the Unified Parkinson’s Disease Rating Scale (UPDRS), Parkinson’s disease questionnaire (PDQ-39; a quality of life measure), Hoehn and Yahr, and the MoCA. Among participants with HD, the total functional capacity (TFC) and motor portions of the Unified Huntington’s Disease Rating Scale (UHDRS) were assessed along with the MoCA.
Follow-up MoCA exams were conducted remotely, by the same baseline rater, via web-based video conferencing at 7 months for PD participants and at 3 months for HD participants. These time points were set prior to the initiation of this study as part of the initial PD 13 and HD 14 telemedicine trials. Significant changes in cognition were not anticipated to occur over the two follow-up periods based on recent evidence demonstrating non-significant changes on the total MoCA per year; 15 however, results were stratified by disease group to eliminate potential biases. Administration and scoring of the MoCA was identical to its original instructions (www.mocatest.org) with slight adaptations in administering the visual portion in order to accommodate remote administration. Namely, the visuospatial/executive subsection (including the alternating trail making task, copying the cube, and drawing the clock) as well as the naming subsection (naming of three animals—lion, rhinoceros, and camel—pictured on the scale) were emailed to and printed by the participants immediately before the web-based assessment began. The rater gave verbal instructions for completing these subsections and then asked participants to hold the paper in front of the web-camera, which was captured as a screenshot by the rater for evaluation (see Figure 1). The naming subsection was administered by instructing the participant to name each animal pictured on the form one at a time from left to right. Participants were also asked to either scan and email or mail the hard copy of the visual portion to the rater to ensure accuracy. The remaining subsections in the verbal portion of the exam were conducted orally as they would be in person. The telemedicine trials13,14 in which the data originate from were approved by the University of Rochester’s Institutional Review Board.
Example screenshot taken by the rater of the visual portion of the MoCA.
For the in-person and remote assessments, the mean and standard deviation (SD) of the sum of each subsection, the visual and verbal portions, and the total MoCA score were computed. All analyses were done using SAS version 9.3 (SAS Institute, Inc., Cary, NC, USA).
Psychometric analyses
As a secondary outcome of this study, we conducted a crude assessment of the remote MoCA scale properties. To estimate the reliability and agreement between the in-person MoCA and remote MoCA, the intraclass correlation coefficient (ICC) and 95 per cent confidence intervals (CIs) were calculated. We defined the threshold for reliability as follows: excellent agreement (ICC ≥ 0.75), fair to good agreement (0.40 < ICC < 0.75), and poor agreement (ICC ≤ 0.40). 16 The MIXED procedure was used for ICC calculations. Internal consistency was measured using Cronbach’s alpha for the remote MoCA, although we did not anticipate the method of administration to influence the homogeneity of items in relation to the measure. Values of 0.80 or higher were considered highly consistent, while values between 0.70 and 0.80 were adequate and less than 0.70 were poor. 17
To estimate how well the remote MoCA assessment predicts the previously validated in-person assessment (concurrent validity), remote MoCA scores were correlated with in-person MoCA scores. Pearson’s correlation coefficients (r) and p values were calculated using the CORR procedure. A correlation coefficient greater than or equal to 0.70 was considered excellent and demonstrated very strong concurrent validity, while correlations from 0.40 to 0.69, 0.30 to 0.39, 0.20 to 0.29, and 0.01 to 0.19 demonstrated strong, moderate, weak, and negligible concurrent validity, respectively. 18
Agreement on the direction of individual change from a normal MoCA score (≥26 out of 30) to an abnormal score (<26), or vice versa, was assessed between the in-person MoCA and remote MoCA. Theoretically, a decline in MoCA score from baseline to follow-up meant worsening cognitive function, while an increase in MoCA score meant an improvement in cognition. Analyses were conducted among the full sample and stratified by disease group and performed using the kappa test in PROC FREQ to obtain kappa coefficients and 95 per cent CIs. Per cent agreement was calculated by summing the concordant (diagonal) elements and dividing by the sample size for that group. Values of kappa were interpreted as follows: almost perfect agreement (0.81–0.99), substantial agreement (0.61–0.80), moderate agreement (0.41–0.60), fair agreement (0.21–0.40), slight agreement (0.01–0.20), and less than chance agreement (<0). 19
Results
A total of 17 individuals were recruited for this study. Eight participants had PD and were on average 65.1 years of age (SD = 11.9 years), mostly male (87.5%), with average baseline UPDRS, PDQ-39, and Hoehn and Yahr scores of 46.9 (SD = 12.3), 29.3 (SD = 17.8), and 2.3 (SD = 0.7), respectively. Nine participants had HD and were on average 57.7 years of age (SD = 12.6 years), majority female (55.6%), having average baseline TFC and motor UHDRS scores of 10.9 (SD = 1.7) and 29.9 (SD = 11.4), respectively. The distribution of total and sectional MoCA scores for the in-person and remote assessments is shown in Table 1. Non-parametric analyses for the in-person assessment (median = 25, interquartile range (IQR) = 23–27, mode = 27) and remote assessment (median = 27, IQR = 25–29, mode = 28) confirmed normality of our data. Baseline total MoCA scores did not differ between the two disease groups (p = 0.65). Overall, MoCA scores increased slightly from the baseline in-person assessment to the follow-up remote assessment, more so in the PD sample (+1.63 ± 2.20, p = 0.08) than in the HD sample (+0.67 ± 2.78, p = 0.49).
Distribution of MoCA scores and the psychometric properties when assessed remotely versus in-person in individuals with movement disorders.
MoCA: Montreal Cognitive Assessment; ICC: intraclass correlation coefficient; SD: standard deviation; PD: Parkinson’s disease; HD: Huntington’s disease.
In-person MoCAs were measured at baseline.
Remote MoCAs were conducted via web-based video conferencing at 7 months for PD patients and at 3 months for HD patients.
With regard to the feasibility of administering the MoCA using web-based video conferencing, the raters and study participants experienced very minor complications associated with technology. Specifically, participants with slow internet connection speeds (n = 5) resulted in delayed sound and stuttered imaging. In a few cases, clear screenshots were difficult to obtain due to a lack of adequate lighting and symptoms (i.e. tremors) that made it challenging for participants to steadily hold the paper up to the web-camera. In one case of delayed sound, which occurred in the later portion of the exam, the video assessment was discontinued and the verbal portion was completed over the phone. A few participants initially had issues logging into the video software system, but were quickly resolved. All participants completed every item in the MoCA when administered remotely. Besides the few minor technical difficulties described above, all participants expressed a positive experience with the rater and using the web-based video conferencing tool. The most consistent reasons for their acceptability were (1) the reduced burden on their caregivers and (2) less time commuting and waiting in offices.
The crude psychometric estimates of the MoCA when administered remotely are also presented in Table 1. Examining movement disorder participants as a whole, the total MoCA demonstrated evidence for fair to good reliability (ICC = 0.59), adequate internal consistency (α = 0.74), and strong concurrent validity (r = 0.59). The visual and verbal portions alone performed slightly better in all three psychometric properties than the combined total MoCA, while some subsections (naming, language, abstraction, delayed recall, and orientation) had moderate to poor properties. In the PD sample, the total MoCA was suggested to have moderate predictive value (r = 0.37), but poor internal consistency (α = 0.54) and reliability (ICC = 0.37). The visual portion alone, however, supported adequate agreement and very strong concurrent validity. The verbal portion alone suggested acceptable reliability and validity scores in this sample, but poor consistency. Again noted were some subsections (language, abstraction, delayed recall, and orientation) with moderate to poor properties. In the HD sample, the total MoCA demonstrated potentially adequate reliability (ICC = 0.65) and internal consistency (α = 0.79) and had strong predictive value (r = 0.65). The visual and verbal portions alone performed marginally better in all three properties, while only three subsections (naming, language, and abstraction) had moderate to poor properties.
The direction of individual change (normal vs abnormal total MoCA score) between the in-person and remote assessments in the full sample produced a kappa statistic of 0.53 (95% CI: 0.15, 0.92) and per cent agreement of 76.5, suggesting moderate agreement. Similar results were obtained between disease groups, yielding a kappa statistic of 0.50 (95% CI: −0.02, 1.00) and 75.0 per cent agreement for the PD sample and a kappa statistic of 0.55 (95% CI: 0.00, 1.00) and 77.8 per cent agreement for the HD sample.
Discussion
Based on the evidence from this study, administering the MoCA remotely via web-based video conferencing is feasible, with only minor common complications associated with technology. Given the narrow range of MoCA scores in our sample, feasibility can only be implied in individuals with mild cognitive deficits. Crude validity testing remains inconclusive as compared to the in-person assessment. Overall, psychometric estimates suggested potential predictive value (concurrent validity) but not reliability of total MoCA scores remotely, while internal consistency performed better in the HD sample than the PD sample. The lower crude psychometric property values in the PD sample may partially be explained by the longer follow-up time, and thus more variation, between the in-person and remote assessments. In addition, performance on the verbal portion seems to contribute mostly to the lack of consistency between the two assessments. However, since the verbal portion was administered identically using both methods, it is slightly less relevant than the visual portion, which performed similarly and strongly in both samples. In light of this, high per cent agreements were seen for both disease groups, suggesting the type of movement disorder to be of little relevance when evaluating the MoCA remotely. The moderate to poor properties of the scale subsections were likely due to the limited range of response values and must be interpreted with caution.
To our knowledge, no other studies exist that evaluate remote administration of the MoCA that we can effectively compare our results to. Similarly, we are unaware of any studies that examine remote assessments of cognition in a sample of movement disorder patients. There are studies, however, that validate the remote administration of other cognition tools, namely, the Color-Shape Test and the Mini Mental State Exam (MMSE) in the elderly, 20 and the MMSE in patients with dementia. 21 In the latter study consisting of 20 patients, administration of the MMSE via Telehealth demonstrated strong concurrent validity (r = 0.90) when compared to the in-person assessment. Although our results are not as striking, it raises the question of applicability in the movement disorder population.
While this study has limitations, it is a critical step in the utilization of web-based video conferencing for the assessment of movement disorder outcomes remotely. Although the primary purpose was to determine feasibility, we recognize that the small sample, and thus large variability, limits the value of the psychometric estimates. In addition, somewhat long follow-up periods, more so for the PD sample, may produce unfavorable psychometric results that might have been more consistent if the time between the in-person and remote assessments were significantly shortened. The timing of measures, however, has no influence on the primary aim to assess feasibility. The possibility of practice effects (i.e. improving scores on repeated assessments of the same test) should also be considered as a potential confounding factor. 22 Although complications associated with technology were minor, there is the possibility of glitches in Internet connection, sound or visual problems, lack of access to the necessary technology (e.g. printers, scanners) for patients particularly in non-westernized regions, and limited knowledge of its use. In addition, very little is known about the compliance with telemedicine use in this population and whether movement disorder patients would rather be seen in-person or in the comfort of their own home. Our results could also be a reflection of the setting in which the assessments took place, 23 resulting in potential symptom variability between environments. As most participants in our study had moderate disease severity and a generally high level of cognition (25.06 ± 2.61), this remote approach should be tested in a larger sample of patients with advanced disease progression and wider range of cognitive function who are burdened with travel constraints due to progressive disability.
The assessment of clinical outcomes remotely via web-based video conferencing can increase access to clinical trials for individuals who would otherwise find travel to study sites burdensome and therefore may improve recruitment for future trials. Remote web-based assessments will also obviate the need for frequent in-person assessments and hence improve retention. Consequently, both patients and clinicians would benefit economically by eliminating travel time and expenses as well as increasing the number of patients seen by the doctor. Similar services for research trials are already being provided by the company MedAvante (www.medavante.com), which conducts various central nervous system assessments using high-quality video conferencing or telephone by trained centralized raters. Confirming the reliability and validity of the MoCA administered remotely in a larger randomized trial may lay the foundation for its use in remote assessments of patients and research participants with neurodegenerative diseases.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The original telemedicine trials were funded by Google, Excellus/Blue Cross, and Lundbeck, Inc.