Clinical Significance in Dementia Research

Abstract

Clinical research traditionally relies on measures of statistical significance to assess the strength of evidence while less attention is paid to the practical import of the results. The objective of this study was to provide a critical overview of the current approaches to measuring clinical significance in dementia research and to provide suggestions for future research. A systematic search was conducted of Medline and Embase for original, English-language, peer-reviewed articles published before July 2012. A total of 18 articles met the inclusion criteria, of which 13 used multiple approaches to measure clinical significance. In all, 5 articles used expert opinion as anchors; 4 also used distribution-based approaches. In all, 8 articles used Goal Attainment Scaling; 7 of these also relied on clinician-based impressions of change. Another 3 articles used only clinical global impressions of change, 1 article used changes in symptomatology, and another used the value from literature.

Introduction

Clinical research relies on measures of statistical significance to assess the strength of evidence. In contrast, relatively less attention is paid to the clinical significance and practical import of the results. The term minimal clinically important difference (MCID) was first described by Jaeschke and colleagues in 1989 as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient's management.” ¹ The definition has taken varied constructs since then but the common denominator remains that of the smallest numerical difference which has been defined in some way as being clinically important. ^2,3 The change itself may be toward harm or benefit and is usually variable between outcomes of interest and patient populations.

Change serves as an important index of disease progression in dementia. However, the assessment and quantification of change can often be quite challenging for researchers and clinicians alike as dementia manifests as a variety of symptoms that present, stay the same, reoccur, and/or subside over the course of the illness. ^{4 –6} As such, measurements of MCID are doubly important in providing researchers and physicians with a frame of reference to evaluate clinically relevant progress or deterioration of their patients. According to an annual report released by the Alzheimer’s Association, 5.4 million Americans of all ages were estimated to have Alzheimer’s disease (AD) in 2012, with a projected rise to 11 to 16 million people by 2050. ⁷ Current initiatives emphasizing patient-centered approaches and expected increases in dementia incidence and prevalence in the coming decades combine to underscore the need to define and use outcomes with clinical significance.

Unfortunately, little consensus on optimal approaches and best practices exists at present. Outcome scales like the Alzheimer’s Disease Assessment Scale–Cognitive subscale (ADAS-Cog) ⁸ have long been used in research and clinical trials with a 4-point change at 6 months taken to be clinically meaningful. ⁹ However, the ADAS-Cog has been shown to have little correlation with other clinical measures and the 4-point change to have little inherent meaning for the purposes of patient assessment and/or physician decision making. ^{10 –12} Moreover, there is incredible variability in the terminology and definitions of MCID used in the literature. Although some only have a nuanced difference in semantics and are identical operationally, others are completely different approaches altogether. ³ For instance, Molnar et al outline differences between MCID from minimally detectable difference (MDD) and Goal Attainment Scaling (GAS) among other similar approaches. Like MCID, MDD is the smallest detectable change in the outcome scale but unlike MCID, it is not necessarily a clinically meaningful one. ¹³ The GAS, on the other hand, is a different approach that relies on patient or caregiver assessment of meaningful changes in behaviors identified as being important. ¹⁴ Unlike MCID, neither MDD nor GAS take into account the side effects and costs associated with the intervention and as such do not mirror the cost–benefit considerations undertaken by the patients, caregivers, and physicians. ¹³ King presents a very concise account of some of the definitions and terminologies in common use and their evolution over time. ³

The purpose of this study was to conduct a systematic search of the existing dementia literature for measures of clinical significance, outline their strengths and limitations, and provide recommendations for future research.

Methods

An electronic literature search of Medline (Pubmed) and Embase was conducted for articles published before July 1, 2012. The following free-text search was carried out using different possible permutations of MCID:

((“clinically meaningful change”) OR (“clinically important difference”) OR (“minimally important difference”) OR (“minimally detectable difference”) OR (“minimal important difference”) OR (“clinically important change”) OR (“clinically significant difference”) OR (“clinically relevant change”) OR (“minimal clinical difference”) OR (“Goal Attainment Scaling”) OR (“MCRC”) OR (“MCID”) OR (“MCSD”))

AND

((“cognitive impairment”) OR (“cognitive decline”) OR (“dementia”) OR (“Alzheimer’s”) OR (“memory decline”) OR (“memory impairment”) OR (“mental impairment”)).

The systematic review was restricted to studies published in English-language, peer-reviewed journals. Figure 1 details the literature search and study selection process. The initial combined search of Medline (Pubmed) and Embase retrieved 75 published articles including 28 duplicates. After removal of the duplicates, the remaining 48 studies were assessed for (1) a diagnosis of dementia in at least some of the participants, (2) a measure of clinical significance in study design or analysis, and (3) original research (ie, no editorials, reviews, comments).

Figure 1.

Study selection process.

Results

Eighteen studies met the inclusion criteria for the systematic review. Most studies (13 of 18; 72%) used multiple approaches to measure clinical significance. In all, 5 articles used expert opinion as anchors of which 4 used distribution-based approaches (such as effect size and standardized response means) as well. Eight articles used GAS of which most also used clinical global impressions of change. In all, 3 other articles only used clinical global impressions of change, 1 article used changes in symptomatology, and another used the value of MCID from literature.

Anchor-Based Methods

Anchor-based methods employ the relationship between the instrument of interest and any number of external indicators to give meaning to the degree of change. ¹ As such, these external criteria (or anchors) must have an appreciable association, both conceptually and statistically, with the instrument measure. Second, they must be interpretable as well as able to identify, to the greatest precision possible, individuals who have changed to a small but meaningful degree. ¹⁵ There are many ways in which anchor-based methods may be employed and can be broadly categorized as cross-sectional approaches and longitudinal approaches. In cross-sectional approaches, the instrument of interest is compared between groups that differ in some criteria, such as disease severity, and the difference is taken to establish the MCID. Ideally, the groups should be similar in every regard except for the differentiating criteria.

None of the studies retrieved in our systematic search employ a cross-sectional approach as defined earlier but several used expert opinion as anchors. Burback et al surveyed 161 geriatricians and neurologists about the “the smallest changes in the Folstein Mini-Mental State Examination (MMSE) scores that are compatible with a noticeable change in the patient's overall condition.” ^{(p 535)} They then used the mean of the responses (3.72) to interpret the clinical significance of the results of published randomized controlled trials (RCTs) assessing the efficacy of tacrine in the treatment of AD. ¹⁶ On the other hand, Carpenter et al used a 1-point change in the Minimum Data Set (MDS) Activities of Daily Living Scale to denote a clinically meaningful change in the functional abilities and cost of care when examining change in physical function in nursing home residents with moderate to severe dementia. ¹⁷ The MCID was determined during the original development of the MDS-Resident Assessment Instrument.

Anchors derived from expert opinions (via conferences, surveys, committees, or other means) have several limitations. They are highly prone to error or bias ¹⁸ and do not directly incorporate patient or caregiver perspectives. Moreover, as Burback et al ¹⁶ point out, the MCID varies with the severity of dementia. As the question in their survey did not take that into account, the mean of the responses is unlikely to have been an accurate representation of the MCID. Studies employing this method should specify the degree of illness in the survey but the obvious difficulty in taking the gradient of severity into account poses a serious limitation to the use of expert opinion alone in arriving at an estimate of the MCID and its generalizability to other studies.

Longitudinal approaches to anchor-based methods typically involve comparing the instrument of interest in a group that has changed in some meaningful way, such as disease severity. One commonly employed approach is to use global assessment scales. These are cumulative measures that take into account changes in cognitive, behavioral, and psychological functions in conjunction with caregiver reports to arrive at an overall numerical assessment of improvement or deterioration. Our systematic search retrieved 7 articles that used global assessment scales ^{10,11,19 –23} of which 5 used Clinician’s Interview-Based Impression of Change Plus Caregiver Input (CIBIC-Plus) secondary to GAS or ADS-Cog ^{10,11,19,21,22} for comparison and/or correlation. The CIBIC-Plus ²⁴ is a widely used metric in drug trials and measures overall global change relative to baseline on a 7-point Likert-type scale ranging from 1 (markedly improved) to 7 (markedly worse). There is also systemic evaluation of patient’s cognition, behavior, and function based on patient and caregiver interviews. The CIBIC-plus represents the clinician’s perspective of change, and while patient or partner-administered global ratings are available and have been described, ²⁵ they are not feasible in patient populations having mild or more severe AD. Global assessment scales are particularly useful in cases where changes in different domains are not individually significant but altogether amount to a readily observable difference. Conversely, it is difficult to relate the global assessment of change back to the individual domains and determine their MCID.

The ability of the participants to make decisions is particularly important for GAS. The GAS is a global outcome measure in which problem areas for each individual patient are identified and goals for improvement are set by the patient, caregiver, and/or the clinician. The goals can be medical, behavioral, or psychological in nature. Following some intervention, each patient is assessed at a previously defined follow-up time for improvement or deterioration. The goals are scored at a “0” while somewhat better outcomes and much better outcomes are scored at a +1 and a + 2, respectively. Similarly, somewhat worse outcomes and much worse outcomes are scored at a −1 and a −2, respectively. A standardized formula is used to sum achievement of weighted goals and come to a measure of overall goal attainment. ¹⁴ Since the goals represent the needs of the patient, GAS scores represent meaningful differences in the patients’ functional abilities. However, individuals with cognitive impairment often have a limited understanding of their own capabilities and as such may be unable to set goals for themselves. In these cases, caregiver- and/or clinician-assigned goals are a good substitute. Additionally, GAS is concerned with significant, observable changes, not necessarily minimally significant changes, and as such may lead to larger measures than the MCID.

Our systematic review found 8 articles that used GAS in dementia research. Gordon et al assessed feasibility of GAS in long-term care of nursing home patients. Because 77% of the patients had dementia, the goals were set by 2 geriatricians in consultation with the nursing staff instead of the patients themselves. ²⁶ Rockwood et al and Asp et al used a modified GAS approach by setting 2 groups of goals—one by the physicians and nurses and another by the patients and caregivers with the help of the field researchers. ^{10,11,19,21,22} They noticed that patients and caregivers were more likely to set goals related to behavior, leisure, and social interaction while physicians were more likely to set goals in domains that are testable, such as memory or cognition. This shows that patient collaboration invites a perspective that is important to the patients and not necessarily accounted for by their physicians.

Tractenberg et al describe an approach, not very dissimilar from the GAS, in which changes in behaviors of patients with AD were assessed by their physicians over a period of time. They used 37 items from the Behavior Rating Scale for Dementia and recorded clinically significant changes such as cessation, emergence, improvement, and intensification. An overall assessment of improvement or deterioration was then made by simply counting the number of behaviors that cease or show improvement as opposed to behaviors that emerge or show deterioration. ²⁷

Symptomology or accomplishments of goals are generally more applicable to patients and participants having later stages of dementia when observable behaviors and symptoms start manifesting themselves. Additionally, symptomology does not take into account the fact that many behaviors or symptoms in dementia are associated. So it is possible for 2 patients to show the same change overall even when one may have a certain number of changes in associated behaviors and the other has the same number of changes in unassociated behaviors even though clinically they are not the same and present a very different clinical picture.

In general, longitudinal approaches to anchor-based methods more directly measure change and are as such better suited for MCID measurements compared to cross-sectional approaches. However, none of the methods retrieved in our search took the patients’ perspectives into account or the risk–benefit analysis necessary for MCID measurements. Although patient or caregiver input is highly desired in measures of clinical significance, it is often affected by recall bias, expectations, or comparison to counterparts and can therefore result in smaller assessments of change. ²⁸ Patients may also be unable to report their condition due to a lack of insight, impaired memory, or simply an inability to communicate. These conditions are particularly relevant in moderate and late stages of AD and in this setting an exception has to be made in the original definition posited by Jaeschke and colleagues. ¹ Similarly, caregivers are affected by many of the same biases and their reports may additionally be affected by their emotional state and their relationship with the patient.

Distribution-Based Methods

Several studies in our systematic review used distribution-based approaches in addition to expert opinions. Distribution-based methods examine the relationship between the change in the instrument of interest and a measure of variability. This variation may be in the sample (effect size), change in outcome measures (standardized response mean), or the measurement precision of the instrument (standard error of the mean [SEM]). The effect size is the change in outcome measures for every pretest standard deviation (SD). ^29,30 It is therefore dependent on the initial sample distribution with more heterogeneous cohorts producing a smaller effect size for any given outcome change. In contrast, standardized response mean is the change in outcome measures for every SD of that change. ³¹ It is therefore dependent on the variability in the changes with more variability in otherwise comparable changes resulting in a smaller standardized response mean. Finally, SEM is the change in outcome measures for every standard error of measurement. ³² This method relies on the assumption that measurement error is constant across the range of changes in outcome measures. But in practical application, the SEM is smaller at the extremes of the changes than it is in the middle.

Howard et al assessed the functional and cognitive effects of donepezil and memantine in community-living patients with Alzheimer's disease using MCIDs they had published a year earlier. They sought expert opinion from psychiatrists and geriatricians through e-mail discussions prior to a meeting and during face-to-face discussions. They were unable to reach agreement on a single value for the standardized MMSE (sMMSE) that would serve as the MCID but decided that it was between 1.0 point and 2.0 scale points. They also reviewed the SDs for both the baseline and the change from baseline scores on the sMMSE, Bristol Activities of Daily Living Scale (BADLS), and Neuropsychiatric Inventory (NPI). An SD of 0.4 in the sMMSE gave a value of 1.4 on the outcome measure which met the opinion-based range. They then used 0.4 SD of the change in score from baseline to give MCIDs of 3.5 points for the BADLS and 8.0 points for the NPI. ^33,34

The process whereby multiple values from different approaches are used to converge to a single value or a narrower range of values is known as triangulation. Anchor-based and distribution-based methods, while limited when used alone, can provide very useful estimates of MCID when used in conjunction with each other. ³⁵ Howard et al used expert opinions to guide their selection of MCID for sMMSE but only extrapolated from their distribution estimate to arrive at the MCIDs for BADLS and NPI. The authors do point out that the MCIDs were very close to those chosen for the AD2000 study by expert opinion. ³⁶ As discussed earlier, expert opinions lack the perspective of the patients and the consideration of risks and benefits that they undertake. As such, even with triangulation, measures of MCID do not fully conform to the original definition put forth by Jaeschke et al.

Van Iersel et al had a geriatrician, a neurologist, and a physiotherapist assess video recordings of elderly patients for clinically relevant change in gait, defined as change in the expected risk of falling, after 2 weeks of multidisciplinary treatment. They then carried out several measures of responsiveness at both group and individual levels. ³⁷ Schrag et al also used half baseline SD and SEM in addition to clinical assessments of significant change to establish the MCID for the ADAS-Cog for patients with AD. ¹²

Distribution-based approaches are purely statistical methods that allow us to express the change in a standardized metric, which in turn makes them easier to execute compared to anchor-based approaches and provides a means for comparison across different tests and samples. ¹⁵ They are independent of the sample size and therefore not bound by the limitations of methods based on statistical significance. However, distribution-based approaches by themselves do not provide information regarding the clinical relevance of the observed change. Interpretation of the resulting distribution-based indices is neither intuitive nor well defined. Researchers often use certain rules of thumb for some methods like the effect size and standardized response mean, where a magnitude of 0.2 is considered a small difference, a magnitude of 0.5 is considered a medium difference, and a magnitude of 0.8 is considered a large difference. ²⁹ But even when these methods allow us to establish change as probably being clinically significant and meaningful to patients, there is little information as to whether the change is truly minimal. There is evidence in the literature that an effect size of 0.50 tends to equal the MCID arrived at by anchor-based determinations but that is not always the case. ¹⁵ As a result, distribution-based methods must always be qualified and given meaning with anchor-based estimates.

Discussion

Neely et al demonstrate that even small numerical differences in outcome measures can become statistically significant, given a large enough sample size. ³⁸ Therefore, the MCID and the sample size calculations are particularly important in evaluating and interpreting the study results. ³⁹ However, very few studies place due import to the clinical and practical significance of their results. An extensive review of quality of life (QOL) reporting standards in 82 RCTs using the European Organization for Research and Treatment of Cancer QOL questionnaire core 30 (EORTC QLQ-C30) found that clinical significance was only addressed in 38% of the articles. This percentage was only marginally higher (44%) for the 70% of articles that met the criteria for high-quality reporting. Most studies relied on statistical significance or lack of it to interpret QOL outcomes, while articles that addressed clinical significance relied on simple anchors like a >10-point change. ⁴⁰ The RCTs in dementia and Alzheimer’s research do not fare any better. A 2009 study of 4 systematic reviews on the comparative efficacy of antidepressants, corticosteroids, Alzheimer’s drugs, and targeted immune modulators found that 42% of trials did not provide enough information to determine whether they were powered to assess MCID. ⁴¹ Another recent systematic review reported that 54% of the reviewed dementia drug RCTs did not mention clinical significance in their reports, and none of the studies employed measures of clinical significance that incorporated patient or caregiver considerations of risk and benefit of the drug’s use. ¹³

Measures of clinical significance have a particularly important role to play in dementia as change serves as an important index of disease progression. However, the assessment and quantification of change can often be quite challenging as dementia manifests as a variety of symptoms that present, stay the same, reoccur, and/or subside over the course of the illness. We have attempted to outline several approaches of determining and applying MCID measurements in dementia research. Their use remains relatively rare in published dementia literature and little consensus on best practices exists at present. Additional research is needed to define optimal approaches for measurements of clinical significance in dementia research.