Abstract
The gold standard for health information is the health record. Hospitalization and outpatient diagnoses provide health systems with data on which to project health costs and plan programmatic changes. Although health record information may be reliable and perceived as accurate, it may not include population-specific information and may exclude care provided outside a specific health care facility. Sole reliance on medical record information may lead to underutilization of health care services and inadequate assessment of population health status. In this study, we analyzed agreement, without assuming a gold standard, between self-reported and recorded chronic conditions in an American Indian/Alaska Native cohort. Self-reported health history was collected from 3821 adult participants of the Alaska EARTH study during 2004-2006. Participant medical records were electronically accessed and reviewed. Self-reported chronic conditions were underreported in relation to the medical record and both information sources reported the absence more reliably than the presence of conditions (across conditions, median positive predictive value = 64%, median negative predictive value = 94%). Agreement was affected by age, gender, and education. Differences between participant- and provider-based prevalence of chronic conditions demonstrate why health care administrators and policy makers should not rely exclusively on medical record–based administrative data for a comprehensive evaluation of population health.
Assessment of population health status and utilization of services informs decisions about health care policies and programs. Populations are diverse and health status and utilization of health care services vary by geographic location and demographic composition. Reports aggregating large population groups, such as American Indian and Alaska Native (AIAN) people, with multiple subgroups spread over wide geographic distances, may overgeneralize and, in the case of smaller subgroups, such as Alaska Native people, entirely mask pertinent health outcomes or health care needs.1,2 Assessment of health outcomes at the local, subregional, or state level provides health care providers and decision makers more reliable data.
One method of collecting health outcome information is through self-reported health surveys, such as the Behavioral Risk Factor Surveillance System or the National Health and Nutrition Examination Survey. Reliability of self-reported health conditions are influenced by factors, including age, gender, and education. Reliability of self-report has only been tested in limited health conditions among few population sectors. 3
Another source of health information is administrative data abstracted from the health record. Medical record data, however, may be fragmented because of the use of multiple facilities. 4 Information may be lost causing abstracted data to be inconsistent with self-reported health surveys. Assessment of health status, individually and at the population level improves through corroboration of sources. Increasing agreement requires understanding how and why sources diverge.
Cohort studies allow examination of agreement between self-report and medical record. However, agreement studies are not routinely reported; those reported often validate self-report accuracy using the medical record as the gold standard and/or employ differing statistical methods.3-11 Rather than validating accuracy of self-reported information against the medical record, this study measured the agreement between sources in an AIAN cohort receiving health care through the Alaska Tribal Health System without assumption of a gold standard and employed complete statistical measurements that fostered comparison.
Methods
Study Setting and Participants
The Alaska Education and Research Toward Health (EARTH) study is part of a multisite cohort conducted in 3 US regions: Northern Plains, the Southwest, and Alaska. 12 In the Alaska EARTH study, participants in 3 regions were enrolled during 2004-2006. At the time of this study, medical records in the 3 participating Tribal health organizations were maintained electronically through the Resource and Patient Management System (RPMS).The Alaska EARTH study, including access to RPMS health records was approved by the Alaska Area Institutional Review Board and the participating Tribal health corporations.
Residents of 25 rural communities and 1 urban center, who self-identified as AIAN ethnicity, were invited to participate. Participants responding to local radio and newspaper announcements and flyers posted in public locations were at least 18 years old, not pregnant, and not currently receiving chemotherapy. Signed informed consent for interviews and medical record reviews was obtained and a unique study identification code (ID) was assigned.
Study questionnaire
Demographic data included age, sex, and education. Health history information was obtained using questions prefaced with “Did a doctor or health care provider ever tell you that you had . . . ?” for each of the terms found in Table 1. Response choices were “yes,” “no,” “skip,” “refuse,” or “don’t know.” Prior reports based on EARTH survey data shed light on chronic disease prevalence among Alaska Native people.13-17
Agreement of Self-Report With Medical Record for 16 Chronic Conditions. a
Abbreviations: PPV, positive predictive value; NPV, negative predictive value; COPD, chronic obstructive pulmonary disease; fractures ≥18 years, fractures occurring after 18 years of age.
Sensitivity, specificity, PPV and NPV presented with medical record as the referent.
Medical Record Review
Medical record reviews were conducted by trained and experienced research nurses. Regional health records were electronically accessed and visually inspected for the chronic conditions studied using the International Classification of Diseases, Ninth Edition codes. All codes were confirmed by review of available electronic health record information and narrative reports.
Abstracted data were manually entered into EARTH study software. Abstractors were blinded from all participant self-reported health history information. Fifty participants (<1.3%) did not have an RPMS medical record. A majority of these were younger than 30 years, suggesting no regional provision of care for a chronic condition. Percentage prevalence of chronic conditions was calculated for self-reported surveys and medical record abstracted diagnoses. Medical record abstract data were matched with the self-reported responses and demographic data for final analysis.
Statistical Methods
All statistical analyses were performed using SAS version 9.2 (SAS Institute Inc, Cary, NC). A “skip,” “refuse,” or “don’t know” response for a self-reported condition was coded a “no” response. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated.
Self-reported data were initially tested against medical record data as the reference. Sensitivity and specificity of self-report measured against the medical record is equivalent to the PPV and NPV (respectively) of the medical record to self-report, as shown in Table 2. Because of this reciprocal relationship, conducting all 4 measurements with a single referent provided all values for both referents, thereby enhancing cross-study comparison.
Equivalents When Gold Standard Is Reversed.
Abbreviations: PPV, positive predictive value; NPV, negative predictive value.
Kappa values were calculated to differentiate between true agreement and agreement produced by chance. Kappa values were interpreted as follows κ ≤ 0.40, fair-to-poor agreement; κ = 0.41 to 0.60, moderate agreement; κ = 0.61 to 0.80, substantial agreement; and κ = 0.81 to 1.00, excellent agreement. 18 Results include κ values for each condition before and after stratification by age, gender, and education. Chi-square tests were performed for sensitivity, specificity, and predictive values within strata to determine significant differences at P < .05.
Results
Demographic and health history data were collected from 3821 Alaska EARTH study participants. Missing (skip, refuse, or don’t know) responses for medical conditions ranged from n = 46 (1.2%) for stroke to n = 238 (6.2%) for high cholesterol. Fractures after age 18 years (28%) and hypertension (25%) were the most prevalent self-reported conditions. For all conditions except cancer, the prevalence of chronic conditions obtained by self-report was lower than prevalence obtained through medical record review.
Self-report was more specific than sensitive (Table 1). Specificity was 90% or greater in all conditions except fractures after age 18 years (80%). Table 2 illustrates sensitivity and specificity of self-report with the medical record as reference.
The PPV of self-report ranged from 27% to 80%. Self-reported diabetes (80%) and thyroid disease (79%) were the conditions most frequently confirmed by the medical record. Least frequently confirmed were chronic obstructive pulmonary disease (COPD; 27%) and kidney disease (31%). Overall, self-report more accurately predicted the absence rather than the presence of a condition in the medical record, with NPVs ranging between 84% (fractures ≥age 18 years) and 99% (cancer and kidney disease).
In all but cataracts, COPD, glaucoma, heart disease, high cholesterol, and liver disease, stratification by gender resulted in significant (P < .05) differences in at least one measurement of agreement (Table 3). Asthma, gallbladder disease, and thyroid disease demonstrated significant differences in 3 or more measurements.
Patterns of Agreement for All Conditions by Participant Characteristics of Sex, Age, and Education Level. a
Abbreviations: Sens, sensitivity; Spec, specificity; PPV, positive predictive value; NPV, negative predictive value; COPD, chronic obstructive pulmonary disease; Fractures ≥18 years = fractures after age 18 years.
Sensitivity, specificity, PPV, and NPV are reported as percentages.
P < .05; **P < .001.
Education level produced significant differences in at least one measurement of agreement for arthritis, asthma, cancer, COPD, fractures after age 18 years, gallbladder disease, and liver and thyroid diseases (Table 3). Only fractures after age 18 years demonstrated significant differences in all four measurements of agreement.
Several consistent patterns of agreement emerged when data were stratified by age group (Table 3). In all conditions, sensitivity of self-report remained lower than specificity. NPV was inversely associated with age; the youngest age group was significantly higher than the oldest age group. For all conditions except COPD, specificity also decreased with age; differences between youngest and oldest age groups were statistically significant except with hypertension and kidney disease.
Substantial agreement by κ (0.6-0.8) was observed for cancer, diabetes, hypertension, and gallbladder disease. Moderate agreement (κ = 0.4-0.6) was observed for arthritis, asthma, cataract, fractures after age 18, high cholesterol, stroke, and thyroid disease. Liver, kidney, and heart diseases, glaucoma, and COPD demonstrated poor-to-fair agreement (κ < 0.4). Kappa values demonstrated differences in agreement levels within all three strata, age, gender, and education, although without consistency (Table 3).
Specificity of self-report to the medical record was higher than sensitivity. Based on the inverse relationship between measurements of agreement when the reference is shifted from medical record to self-report, the NPV of medical record information to self-report was consistently higher than the PPV. While age appeared to effect agreement more frequently than gender or education level, age groups with higher agreement varied by condition.
Discussion
Administrative data analyzed by health care systems inform programmatic decisions. This information source has limitations, including a focus on individual and not population needs and the exclusion of diagnoses and treatment provided outside the monitored health system. This study measured the prevalence of chronic conditions per self-report and medical record review and tested the strength of agreement between these sources in a cohort of AIAN people with access to primarily one health care system. The aim was measuring agreement and identifying potential sources of disagreement between these sources. Unlike similar studies, it did not assume the medical record as the gold standard.
In this cohort, the prevalence of all chronic conditions except cancer was lower per self-report than medical record documentation. The observed underreporting was supported by the lower sensitivity than specificity of self-reported conditions as compared with the medical record. However, both data sources were better able to detect the absence rather than presence of a condition.
Agreement between the 2 sources also varied by age, gender, and education. Greater agreement produced by severe events may account for the higher agreement in diagnoses of cancer in this cohort. 9 The presence of a multidisciplinary statewide diabetes program may have increased diabetes awareness and also contributed to the higher κ score for diabetes. 19 Conditions that are symptom based may also be more reliably self-reported than documented in the medical record. 20
Although NPV is expected to decline with increasing disease prevalence, consistently lower NPV of self-report with advancing age is a concern in populations with rising life expectancies. 21 Overall, low PPV in all age groups for COPD and decreasing PPV with advanced age for diabetes and thyroid disease may signify limitations in medical record documentation. Variations in strength of agreement were noted with gender and education, though this was not true for all conditions.
The strengths of this study are that self-reported and medical record data were matched by participant and 5 measurements of agreement were calculated to enable comparison with either source as a referent and allow us to identify deficiencies in the medical record as well as self-report and examine effects of age, gender, and education on reporting.
There are several limitations in this study. Only 3 population characteristics were examined in relation to agreement. Characteristics such as spoken language, cultural differences, and urban versus rural residence may influence agreement and should be explored. Study participants were voluntarily enrolled, which may limit generalization of the findings; however, the cohort was determined representative of the 3 regions by age and gender. 12 Only 3 age groups and 2 education levels were used. A redistribution of group levels may narrow discordance.
Coding errors, incomplete information, and missing health records may limit reliability of medical record data. The electronic health record systems date only to January 1, 1986 and some diagnoses may have been missed by abstractors. Evaluation of chronic conditions constrains this limitation, as chronic conditions are likely documented multiple times and the Alaska Tribal Health System provides all levels of care throughout the life span.
Findings affirm that participant-perceived prevalence of chronic conditions is lower than prevalence obtained through administrative data based on medical records. Health status reports based solely on self-report underestimate chronic conditions in AIAN people; this supports previous findings that reliance solely on self-reported survey, such as Behavioral Risk Factor Surveillance System and the National Health and Nutrition Examination Survey, may result in loss of health information. Conversely, the medical record cannot be considered a gold standard when evaluating population health as it may lack data available through self-report. Health literacy of the population may also contribute to the lack of agreement between the health record and self-report. Research is needed to explore communication of health information between health care providers and individuals. In the interim, health care providers and policy makers must be aware of the limitations of morbidity reports based solely on a single source and support local and regional efforts to align data to identify the burden of chronic conditions prevalent in the population.
Footnotes
Acknowledgements
We wish to thank our Alaska Native EARTH participants for their voluntary participation; the Alaska Native Tribal Health Consortium, Southcentral Foundation, Southeast Alaska Regional Health Consortium, and the Yukon Kuskokwim Health Corporation for their support and contributions; and Ms Gretchen Day for her very helpful editorial comments.
Authors’ Note
The contents of this article are solely the responsibility of the authors and do not necessarily represent the official view of the National Cancer Institute.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Alaska EARTH study was funded by the National Cancer Institute grants CA88958 and CA96095.
