Abstract
Background:
Clinical diagnosis of cognitive disorders depends on accurate reporting of medical history, yet little is known about the reliability and the validity of such reports, particularly in older patients with and without cognitive impairment.
Methods:
In 2 studies, we examined the reliability and the validity of reported histories of select medical events in adults with and without cognitive impairment from a large national cohort.
Results:
Information from subjects (N1 = 3664), obtained from 2 time points, 6 to 12 months apart, was consistent across most medical events, regardless of the diagnostic group (range = 97.6%-100% agreement; Cohen κ range = 0.712-0.945), with few exceptions. Validity analyses (N2 = 382) revealed that 3 of 5 medical events assessed showed substantial agreement between self-report information and clinician diagnosis.
Conclusions:
These data represent some of the first to demonstrate the reliability and the validity of reported select medical events in older adults with and without cognitive impairment.
Introduction
Obtaining medical history information often relies on self-report using interview and/or questionnaire methods. In older patients with suspected or known cognitive impairment, the accuracy of such information may be called into question. This is a particularly important issue, as the likelihood of cognitive impairment and medical diseases both increase dramatically with age.1,2 In addition to various types of dementia, memory impairment can also result from neurological and cardiac events, such as traumatic brain injury (TBI),3–6 stroke, 7 seizures, 8 heart failure, 9 cardiac arrest, 10 and atrial fibrillation. 11 Thus, it is important in any clinical evaluation to obtain an accurate report of a patient’s medical history, as this can be essential to identify the nature and course of cognitive impairment, as well as providing insight into a patient’s prognosis and possible treatment recommendations.
Despite the importance of accurate medical history reporting, little is known about its reliability and validity, particularly among older and/or cognitively impaired individuals. Research by Okura et al 12 investigated the agreement of patients’ self-reported medical conditions compared with that found in medical records within a large cognitively intact, diverse sample of community-dwelling adults aged 45 years and older (N = 2037). The authors found self-reports of congestive heart failure (κ = 0.46) were considerably less consistent than other vascular and neurological events (κs = 0.75-0.80). In another investigation of 2380 community-dwelling elderly patients, Kriegsman et al 13 found that such patients provided fairly accurate reports for many chronic diseases documented in their medical records, such as cardiac disease and diabetes (κs = 0.69 and 0.85, respectively), yet provided inaccurate reports of some other diseases such as atherosclerosis and arthritis (κs = 0.38 and 0.31, respectively). Wilmoth et al 14 demonstrated that subjects aged 50 years and older in a large national database reliably self-reported TBIs across 2 time points and that subjects’ level of cognitive impairment did not predict reporting consistency.
Taken together, these findings suggest that some reports of medical events may be unreliable, even in cognitively intact individuals, and very little is known about medical history reporting among older individuals with varying degrees of cognitive impairment. Although prior studies have focused mainly on medical history reporting in healthy adult samples, little research has been conducted on (a) the stability of reporting across time and (b) the concordance of individual and provider/informant reports in patients with cognitive impairment. The purpose of this investigation was to examine the reliability and the validity of various reported medical history events in individuals with normal cognition (NC), mild cognitive impairment (MCI), and Alzheimer disease (AD) using a large national database.
Study 1
Methods
Data were obtained from the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) form versions 1 and 2, which has compiled sociodemographic and clinical information from subjects volunteering for research at all NIA-funded Alzheimer Disease Centers (ADCs) in the United States since September 2005. 15 Written informed consents were obtained from participants at each ADC and approved by the ADC’s Institutional Review Board (IRB). Research using the NACC database was approved by the University of Washington (UW) IRB (36178). The University of Washington IRB does not require tracking of external IRB numbers because NACC only distributes data sets that are Health Insurance Portability and Accountability Act limited, at a minimum. The maintenance of the NACC is covered by the UW IRB and is limited to the Coordinating Center itself. The individual ADCs that collect the human subjects data have their own approvals that they maintain.
Data received from NACC are either completely deidentified or contain limited data elements that would not allow the researcher to readily identify individuals in the data set. The maintenance of the (potentially) identifiable data in the NACC is covered by the UW IRB approval, and the secondary use of deidentified data would likely not require an IRB approval from external institutions as it is not identifiable to researchers (ie, it does not fit the definition of “human subjects research” per regulations). The NACC database is funded by NIA/National Institutes of Health grant U01 AG016976.
This analysis used data from 33 different ADCs. Sociodemographic and clinical data were obtained for initial and follow-up visits completed between September 2005 and June 2015. Subjects were included if they were aged 50 years or older, completed an initial and follow-up visit within 6 to 12 months, and were classified as being an NC or having a clinical diagnosis of MCI or possible/probable AD at both time points. Clinical diagnoses were made by experienced clinicians at each UDS visit using standard Petersen diagnostic criteria for MCI 16 and National Institute of Neurological and Communicative Disorders and Stroke (NINCDS)/Alzheimer’s Disease and Related Disorders Association (ADRDA) criteria for AD. 17
As part of the standard UDS, subjects and accompanying informants are asked at each ADC visit whether they have experienced different neurological and cardiac events, among other medical history questions. Each medical event question was recorded as having never occurred (“absent”), having occurred within the past year or still requiring active management that is consistent with available information (“recent/active”), or having occurred more than a year prior to the visit but was resolved with no current treatment underway (“remote/inactive”). As our intent was to focus on remote reporting of select medical history events, and because events occurring during the follow-up period would adversely impact test-retest reliability, individuals reporting “recent/active” events or those having follow-up visits after 12 months were excluded. Subjects with missing data were also excluded.
We chose to focus on neurological and cardiac events because of their potential impact on cognitive functioning. The neurological events examined included stroke, transient ischemia attack, seizure, and TBI with either <5 minutes loss of consciousness (LOC) or ⩾5 minutes LOC. Cardiac events included were heart attack/cardiac arrest, atrial fibrillation, angioplasty/endarterectomy/stent implantation, cardiac bypass procedure, pacemaker implantation, and congestive heart failure. Other medical diagnoses such as hypertension and diabetes were excluded due to the ongoing nature of medical treatment and medication adherence, which may confound remote history reporting.
Analyses
Descriptive and frequency analyses were conducted for age, sex, education, and race (see Table 1). Chi-square and 1-way analyses of variance (ANOVAs) examined group differences in demographic factors with the significance level set at .05. To characterize the overall functional status of the sample, the Clinical Dementia Rating Scale Sum of Boxes (CDR-SOB) scores for each participant were included and averaged across diagnostic groups.
Demographic characteristics and dementia rating for study 1 normal cognition, MCI, and AD subjects.
Abbreviation: MCI, mild cognitive impairment; AD, Alzheimer disease; CDR-SOB, Clinical Dementia Rating Scale Sum of Boxes.
Analysis of variance; bPearson χ2.
Self-report reliability was examined using Cohen κ for each neurological and cardiac event. Within each cognitive group, inconsistencies in reporting were defined as participants who endorsed the medical event at time 1 but not time 2 or vice versa (ie, yes/no or no/yes). Confidence intervals (95.00% CIs) are provided for each κ coefficient, with the lower CI providing the reader with an estimate of the lowest estimate that might be found with similar samples. Kappa coefficients greater than or equal to 0.60, which also contain a lower bound value of the 95% CI greater than or equal to 0.60, were deemed to have met criteria for substantial agreement. 18 Analyses were conducted using IBM SPSS Statistics Package version 22.
Results
Our initial query from the NACC UDS yielded data from 31 792 participants. After applying the exclusion criteria, there were 1765 subjects identified as NC, 627 with MCI, and 1272 with AD included in the analyses (see Table 2). Diagnostic groups differed significantly by sex and race, χ2 = 72.31 and 21.94, respectively (P < .001), and they also differed with respect to level of education and age, F(2, 3600) = 69.05 and F(2, 3611) = 17.51, Ps < .001. A Tukey post hoc test revealed that the NC group was significantly younger than the MCI and AD groups, but the MCI and AD groups did not differ significantly for age (P > .05). Consistent with expectations, participants with AD displayed the greatest functional deficits as reflected by CDR sum of boxes ratings, while still considered relatively mild for AD, and the MCI group displayed greater functional deficits than the NC group (meanCDR-SOB = 6.2, 1.3, and 0.1, respectively).
Demographic characteristics and dementia rating for study 2 normal cognition, MCI, and AD Subjects.
Abbreviation: MCI, mild cognitive impairment; AD, Alzheimer disease; CDR-SOB, Clinical Dementia Rating Scale Sum of Boxes.
Analysis of variance; bPearson χ2.
Data were examined across the initial and follow-up visits to examine consistency in medical history reporting. Cohen κ analyses revealed significant agreement for all medical events for each diagnostic group, with few exceptions, Ps < .001. The κ coefficients and CIs pertaining to each neurological and cardiac event across diagnostic groups, the relative percentage agreement between baseline and follow-up visits, and the number of participants who reported remote events at both time points 1 and 2 for each event can be found in Tables 3 and 4.
Test-retest reliability of neurological event reporting.
Abbreviation: MCI, mild cognitive impairment; AD, Alzheimer disease; TIA, transient ischemic attack; TBI, traumatic brain injury; LOC, loss of consciousness; CI, confidence interval.
All Ps < .001.
Test-retest reliability of cardiac event reporting.
Abbreviation: MCI, mild cognitive impairment; AD, Alzheimer disease; CI, confidence interval.
All Ps < .001.
All κ coefficient CIs met criteria for substantial agreement (lower bound κ CI ⩾ 0.60), 18 with 3 exceptions: the seizure condition for both the NC and the AD groups, and the congestive heart failure condition for the NC group. Generally, a more robust agreement was seen within cardiac events overall compared with neurological events. Among neurological events, TBI with less than 5 minutes LOC showed the lowest observed percentage agreements across all cognition groups, and TBI with ⩾5 minutes LOC revealed the highest percentage agreement within the AD group. About 60.10% of medical events revealed a higher number of reporters at follow-up than baseline.
Discussion
Older individuals and assisting caregivers appear to be reliable in reporting of most of the remote neurological and cardiac events queried, regardless of the presence or the absence of cognitive impairment. However, there was a trend across all conditions for a greater number of participants having reported a remote medical event at the follow-up visit than baseline. A closer examination of participants’ reports revealed that no participant within any diagnostic group reported the event at baseline but then denied the event at time point 2 (yes/no) for stroke, transient ischemic attack, or any of the 6 cardiac events. For participants who reported such events at baseline, 100% reliability in reporting was observed at follow-up.
Within each diagnostic group, remote history of a seizure(s) and TBI yielded several yes/no and no/yes inconsistencies. The observed contrast in yes/no inconsistencies for the seizure(s) event could be due to the large variability in presentation and subjective experience and reporting of seizure phenomena, which can overlap with other conditions (eg, migraines and psychiatric disorders). Participants may have been more reliable to report seizure(s) if they required hospitalization, chronic medication, and/or if it was subsequently verified by a medical professional.
Regarding TBI, differences in reporting at both time points could be due to a variety of reasons. Determining the length of LOC after injury, which may have occurred years or even decades earlier, can prove difficult, particularly if post-traumatic amnesia was present. In addition, a witness to the injury, which caused the TBI, is often necessary to estimate the duration of LOC. Recent findings within the literature suggest poor validity of self/parent-reported TBIs that have been verified via medical records. Mckinlay et al 19 recently demonstrated in a longitudinal study that even when young adults were carefully cued to recall remote documented instances of TBI, many instances of TBI, including those severe enough to require hospitalization, were not accurately recalled. Interestingly, the agreement among reporting for all TBI events was similar across the 3 groups of normal, MCI, and AD individuals, suggesting that the presence of cognitive impairment does not substantially affect such reporting in the NACC population.
Within the present study, both TBI events revealed relatively strong reliability across diagnostic groups, with κ coefficients ranging from 0.802 to 0.836 for TBI < 5 minutes LOC and 0.857 to 0.924 for TBI ⩾ 5 minuets LOC. However, a larger number of inconsistent responders were found within the TBI < 5 minutes LOC group. These findings are consistent with those by Wilmoth et al. 14 Given the variability of symptom reporting among milder concussions, it should not come as a surprise that there may be greater variability in event reporting than in more serious injuries, such as those requiring hospitalization. It is unclear as to why our agreement rates were much higher, although this may relate to methodological differences in subject characteristics, data collection, and analysis. In addition, in the NACC procedure for medical history reporting, information is obtained from patients presenting for evaluation, sometimes with an accompanying informant. As such, a study partner may have assisted with medical history reporting. Although the presence of accompanying caregivers more closely mimics clinical scenarios in which cognitively impaired patients are evaluated, the inability to know whether or not the informants provided information in individual cases may limit the generalizability of these findings.
Conversely, the implantation of a pacemaker was the only event to demonstrate 100% agreement across each diagnostic group. This is not surprising given the nature of the surgery, as people do not seem to forget this procedure, although the UDS does not offer information regarding when any of these remote events occurred, which could affect reporting reliability. Interestingly, the congestive heart failure and angioplasty conditions yielded stronger agreement rates for those classified as having MCI and AD compared with the NC group. Of note, the observed strength of agreement observed within the congestive heart failure condition differs from the results observed by Okura et al. 12 The reasons for these observed trends are unclear.
Study 2
Methods
In a separate sample, initial visit data were collected for participants who completed form version 3 of the UDS, the most recently updated protocol used within NACC ADCs, up until the June 2017 data freeze. This analysis used data from 23 different ADCs. Form version 3 of the NACC UDS, which was implemented in 2015, introduced a section pertaining to clinician-verified medical conditions while removing the assessment of participant self-reported medical histories at each follow-up visit. As such, to assess the validity of medical history reporting within this population, a separate sample from the one used in study 1 was necessary. Subjects were included if they were aged 50 years or older, completed an initial visit using form version 3 of the UDS, and were classified as having NC, MCI, or AD.
The presence of medical conditions that were assessed under similar conditions were included in analyses as follows: myocardial infarction, atrial fibrillation, angioplasty, congestive heart failure, and stroke. As fewer identical conditions were assessed across both clinician-assessed and patient health history sections on the UDS form version 3, a smaller list of medical events were included in study 2 compared with study 1. Only medical events that similarly assessed individual and clinician report were included in analyses. Chi-square analyses examined the concordance between self-report and clinician report among diagnostic groups (NC, MCI, and AD). Clinician-assessed medical conditions were based on a review of all available information, including new diagnoses made during the current ADC visit, previous medical records, procedures, laboratory tests, and the clinical examination.
Descriptive and frequency analyses were conducted for age, sex, education, and race (see Table 2). Chi-square and 1-way ANOVAs examined group differences in demographic factors with the significance level set at .05. To characterize the overall functional status of the sample, the Clinical Dementia Rating Scale Sum of Boxes (CDR-SOB) scores for each participant were included and averaged across diagnostic groups. Validity was assessed by agreement between self-report and clinician documentation of conditions using Cohen κ. Confidence intervals (95.00%) are provided for each κ coefficient, with the lower CI providing the reader with an estimate of the lowest estimate that might be found with similar samples. Kappa coefficients greater than or equal to 0.60 that also contain a lower bound value of the 95% CI greater than or equal to 0.60 were deemed to have met criteria for substantial agreement. 18
Results
Our second query from the NACC UDS yielded 382 subjects after applying the exclusion criteria. Diagnostic groups differed significantly by sex and race, χ2 = 10.34 and 26.10, respectively (Ps < .05), as well as level of education, F(2, 3600) = 69.05, P < .001, but not age F(2, 379) = 1.10, P = .38. Again, as expected, participants with AD displayed the greatest functional deficits as reflected by CDR sum of boxes scores, and the MCI group displayed higher scores than the NC group (meanCDR-SOB = 5.4, 1.6, and 0.0, respectively).
Cohen κ analyses revealed significant agreement (lower bound κ CI ⩾ 0.60) for the following conditions: myocardial infarction, atrial fibrillation, and congestive heart failure. Stroke (κ CI = –0.10 to 0.94) and angioplasty (κ CI = 0.40-0.91) displayed poor rates of agreement (see Table 5). Diagnostic groups did not differ significantly by the proportion of inconsistent responders, χ2(2, N = 382) = 1.77, P > .05.
Concordance of reported and clinician-assessed medical histories.
Abbreviation: CI, confidence interval.
All Ps < .001.
Discussion
Although significant concordance was observed in many of the medical conditions assessed by clinicians compared with patient report, no clear distinction was observed between the type of events that elicited a high level of agreement (eg, myocardial infarction) compared with events that elicited a lower level of agreement (eg, stroke). Whereas past research has observed differences in chronic disease reporting, 13 our results were limited to select medical events. Investigating the differences in reporting between chronic and acute conditions may provide a fruitful area of future research. Our results regarding myocardial infarction (κ = 0.83) are similar to those observed by Okura et al 12 (κ = 0.80). A possible explanation for the observed results within the stroke condition could be the manner of diagnosis. For example, if clinician(s) observed signs of a remote lacunar infarct on magnetic resonance imaging (MRI), it is possible the patient may not have noticed or reported clinical symptoms, although the MRI was later read as reflecting a stroke, which is not uncommon in clinical practice.
The medical literacy of individuals endorsing medical conditions found in their charts may also have influenced results. For example, patients may deny a previous myocardial infarction but endorse a “heart attack.” The way information is conveyed to patients, including the specific terms used, may influence their understanding of their medical conditions.
Summary
In summary, these findings are some of the first to demonstrate the reliability and the validity of medical event history reporting in older adults with and without cognitive impairment, including those with mild AD. Given the necessity for valid medical history reporting and the extent to which health care providers and researchers rely on self-report information, these findings provide cautionary support for the ongoing use and inclusion of self and assisting caregiver report information within these populations.
Our results identified that for several different neurological and cardiac events, there was a high level of consistency in reporting between baseline and follow-up visits occurring 6 to 12 months later. Similar high rates of reporting agreement were found by Okura et al 12 in their sample of 2037 older Minnesota residents, noting that rates of agreement were highest for life-threatening events with a rapid onset (eg, stroke and myocardial infarction), as well as in chronic disorders which required ongoing care (eg, diabetes). From a psychological standpoint, it makes sense that potentially life-threatening or life-altering events would be remembered more vividly, and ongoing medical conditions would be reported with good consistency, as they were present at both time points. Combining results from both studies highlights the reliability of reporting major medical event information by older individuals/informants. In addition to supporting the results from the previous studies of healthy older samples, our results extend these findings to aging individuals with different levels of cognitive impairment.
More complex and/or heterogeneous conditions appear to be reported less consistently and accurately based on these results. This observation may be due to numerous factors, such as the transient nature of such events, less severe nature, and/or the presence of overlapping symptoms with other conditions. Although TBI and other events yielded generally strong reliability, agreement between self-report and medical records could not be assessed due to the limitations of the available database. Along these lines, Mckinlay et al 19 found large inconsistencies regarding agreement between self-reported and medical record reports of TBI in a sample of 1265 individuals age 25 years and younger. Of note, most of the TBIs sustained in the study by Mckinlay were considered “mild,” which likely contributed to lower rates of agreement, as definitions and diagnoses of mild TBI vary across settings. Within our study, we were limited to specifiers within the NACC database such as estimated duration of LOC (ie, < or >5 minutes), which reveal little information regarding the severity of the injury. Furthermore, no information was available regarding subjects’ age at the time of TBI or whether repeated injuries were sustained. As such, it may be important to take into account the nature and the severity of the injury/event when gathering self-report information. Clinicians would be well advised to use a variety of collateral information to guide treatment decisions in cases involving such events. These findings also point to the need for continued research within these populations, as obtaining accurate clinical history is not only important for understanding the progression and treatment of cognitive impairment, but for treating the patient in general.
These studies have several limitations that may limit generalizability. First, all participants were seen at ADCs and thus are involved in volunteer research programs. As such, these individuals may not be representative of the general population and may have varying levels of medical literacy, and replication in community-based samples is needed. As previously noted, many ADC participants present with accompanying informants to complete the NACC UDS questionnaires and interview questions. Although this scenario more mimics the clinical setting wherein cognitively impaired patients often present with an informant or caregiver, we were limited to the acquired data from NACC, which do not indicate whether patient and/or caregiver provided the medical history information or offered confirmation of its veracity. It may be that healthy participants and those with MCI would tend to rely less on informants for medical history information/validation than those with dementia, although this remains a question that goes beyond the available data. Future studies may wish to examine the reliability of self-report data separately from patients and caregivers to provide additional measures of reporting reliability based on the source of information. Nevertheless, despite this methodological limitation, it is worth noting that similar high rates of reporting reliability were seen across these large samples of healthy aging subjects and those with MCI and dementia who were systematically and thoroughly evaluated and diagnosed using standard procedures across multiple sites nationally. Furthermore, in both studies, the samples were primarily white and highly educated. Last, our AD samples were only mildly impaired (meanCDR-SOB1 = 6.2, SD = 3.93; meanCDR-SOB2 = 5.4, SD = 3.8), as CDR-SOB scores ranging from 4.5 to 9.0 are considered “mild.” 20 Thus, our findings may not be reflective of patients with more severe cognitive impairment, as medical history reporting may become less reliable with greater deficits, which merits further investigation. Last, our findings reflect the reliability and the validity of reporting only certain neurological and cardiac events/conditions, which may differ from reporting of other common diseases such as hypertension and diabetes.
Conclusions
Two studies examined the consistency of reported histories of select medical events in older adults with and without cognitive impairment from a large national cohort. Results demonstrated that retrospective recall of neurological and cardiac events appears to be reliable, even in those with cognitive impairment due to MCI and mild AD. Results also yielded substantial agreement between select clinician and patient-reported medical event histories. These data represent some of the first to document the reliability of reported medical histories in older adults with and without cognitive impairment.
Footnotes
Funding:
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Funding for this study was provided in part by the Texas Institute for Brain Injury and Repair at the University of Texas (UT) Southwestern Medical Center. Additional support came from the UT Southwestern Alzheimer’s Disease Center’s National Institute of Aging (NIA) grant, P3012300-19, and the Texas Alzheimer’s Research and Care Consortium. The National Alzheimer’s Coordinating Center (NACC) database is funded by NIA/National Institutes of Health grant U01 AG016976.The NACC data are contributed by the NIA-funded Alzheimer Disease Centers: P30 AG019610 (Principal Investigator [PI] Eric Reiman, MD), P30 AG013846 (PI Neil Kowall, MD), P50 AG008702 (PI Scott Small, MD), P50 AG025688 (PI Allan Levey, MD, PhD), P50 AG047266 (PI Todd Golde, MD, PhD), P30 AG010133 (PI Andrew Saykin, PsyD), P50 AG005146 (PI Marilyn Albert, PhD), P50 AG005134 (PI Bradley Hyman, MD, PhD), P50 AG016574 (PI Ronald Petersen, MD, PhD), P50 AG005138 (PI Mary Sano, PhD), P30 AG008051 (PI Thomas Wisniewski, MD), P30 AG013854 (PI M. Marsel Mesulam, MD), P30 AG008017 (PI Jeffrey Kaye, MD), P30 AG010161 (PI David Bennett, MD), P50 AG047366 (PI Victor Henderson, MD, MS), P30 AG010129 (PI Charles DeCarli, MD), P50 AG016573 (PI Frank LaFerla, PhD), P50 AG005131 (PI James Brewer, MD, PhD), P50 AG023501 (PI Bruce Miller, MD), P30 AG035982 (PI Russell Swerdlow, MD), P30 AG028383 (PI Linda Van Eldik, PhD), P30 AG053760 (PI Henry Paulson, MD, PhD), P30 AG010124 (PI John Trojanowski, MD, PhD), P50 AG005133 (PI Oscar Lopez, MD), P50 AG005142 (PI Helena Chui, MD), P30 AG012300 (PI Roger Rosenberg, MD), P30 AG049638 (PI Suzanne Craft, PhD), P50 AG005136 (PI Thomas Grabowski, MD), P50 AG033514 (PI Sanjay Asthana, MD, FRCP), P50 AG005681 (PI John Morris, MD), P50 AG047270 (PI Stephen Strittmatter, MD, PhD).
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
All authors were involved in the design, literature review, data acquisition, analysis, and interpretation for both studies involved in this manuscript. Additionally, all authors participated in drafting and revising the manuscript. Final approval was obtained from each author prior to journal submission.
