Restricted accessResearch articleFirst published online 2025
Assessing Cognitive Impairment in the Health and Retirement Study Harmonized Cognitive Assessment Protocol Project: Comparing a Diagnostic Algorithm With a Diagnostic Consensus Panel
Accurate classification of cognitive impairment in population studies is challenging.
Objective
To compare the performance of a diagnostic algorithm with a clinical consensus panel.
Sample
In 2016, the Health and Retirement Study (HRS) implemented the Harmonized Cognitive Assessment Protocol Project (HRS-HCAP) to streamline cognitive assessments for select HRS participants.
Methods
The Manly–Jones HCAP diagnostic classification was used to classify cognitive status as normal, mild cognitive impairment (MCI), or dementia. For this analysis, a consensus panel of five clinicians reviewed 50 cases with high diagnostic uncertainty, each reviewing 30 cases, blinded to the algorithm’s classifications.
Analysis
Diagnostic concordance was assessed using unweighted and weighted Cohen’s kappa (κ).
Results
Unweighted concordance was 70% (35/50), with discordance mostly among MCI cases. Weighted concordance was 84%. Unweighted κ was 0.56 (95% CI 0.30–0.81) and weighted κ was 0.75 (95% CI 0.49–0.91), indicating moderate to substantial agreement between the two methods.
AlbertM. S.DeKoskyS. T.DicksonD.DuboisB.FeldmanH. H.FoxN. C.GamstA.HoltzmanD. M.JagustW. J.PetersenR. C.SnyderP. J.CarrilloM. C.ThiesB.PhelpsC. H. (2011). The diagnosis of mild cognitive impairment due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s and Dementia: The Journal of the Alzheimer’s Association, 7(3), 270–279. https://doi.org/10.1016/j.jalz.2011.03.008
2.
BassilD. T.FarrellM. T.WeermanA.GuoM.WagnerR. G.BrickmanA. M.GlymourM. M.LangaK. M.ManlyJ. J.TippingB.ButlerI.TollmanS.BerkmanL. F. (2023). Feasibility of an online consensus approach for the diagnosis of cognitive impairment and dementia in rural South Africa. Alzheimer’s and Dementia, 15(2), e12420. https://doi.org/10.1002/dad2.12420
3.
BhakthaN.LechnerC. M. (2021). To score or not to score? A simulation study on the performance of test scores, plausible values, and SEM, in regression with socio-emotional skill or personality scales as predictors. Frontiers in Psychology, 12, 679481. https://doi.org/10.3389/fpsyg.2021.679481
4.
CollinsG. S.ReitsmaJ. B.AltmanD. G.MoonsK. G. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. BMC Medicine, 13(1), 1. https://doi.org/10.1186/s12916-014-0241-z
5.
GabelM. J.FosterN. L.HeidebrinkJ. L.HigdonR.AizensteinH. J.ArnoldS. E.BarbasN. R.BoeveB. F.BurkeJ. R.ClarkC. M.DekoskyS. T.FarlowM. R.JagustW. J.KawasC. H.KoeppeR. A.LeverenzJ. B.LiptonA. M.PeskindE. R.TurnerR. S.ZamriniE. Y. (2010). Validation of consensus panel diagnosis in dementia. Archives of Neurology, 67(12), 1506–1512. https://doi.org/10.1001/archneurol.2010.301
6.
GianattasioK. Z.WuQ.GlymourM. M.PowerM. C. (2019). Comparison of methods for algorithmic classification of dementia status in the Health and retirement study. Epidemiology, 30(2), 291–302. https://doi.org/10.1097/EDE.0000000000000945
7.
GravesL. V.EdmondsE. C.ThomasK. R.WeigandA. J.CooperS.BondiM. W. (2020). Evidence for the utility of actuarial neuropsychological criteria across the continuum of normal aging, mild cognitive impairment, and dementia. Journal of Alzheimer's Disease, 78(1), 371–386. https://doi.org/10.3233/JAD-200778
8.
JinH.ChienS.MeijerE.KhobragadeP.LeeJ. (2021). Learning from clinical consensus diagnosis in India to facilitate automatic classification of dementia: Machine learning study. JMIR Mental Health, 8(5), e27113. https://doi.org/10.2196/27113
9.
JonesR. N.ManlyJ. J.LangaK. M.RyanL. H.LevineD. A.McCammonR.WeirD. (2024). Factor structure of the harmonized cognitive assessment protocol neuropsychological battery in the Health and retirement study. Journal of the International Neuropsychological Society, 30(1), 47–55. https://doi.org/10.1017/S135561772300019X
10.
JormA. F. (2004). The informant Questionnaire on cognitive decline in the elderly (IQCODE): A review. International Psychogeriatrics, 16(3), 275–293. https://doi.org/10.1017/s1041610204000390
11.
KhobragadeP.NicholsE.MeijerE.VargheseM.BanerjeeJ.DeyA. B.LeeJ.GrossA. L.GanguliM. (2024). Performance of the informant Questionnaire on cognitive decline for the elderly (IQCODE) in a nationally representative study in India: The LASI-DAD study. International Psychogeriatrics, 36(3), 177–187. https://doi.org/10.1017/S1041610222000606
12.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
13.
LangaK. M.PlassmanB. L.WallaceR. B.HerzogA. R.HeeringaS. G.OfstedalM. B.BurkeJ. R.FisherG. G.FultzN. H.HurdM. D.PotterG. G.RodgersW. L.SteffensD. C.WeirD. R.WillisR. J. (2005). The aging, demographics, and memory study: Study design and methods. Neuroepidemiology, 25(4), 181–191. https://doi.org/10.1159/000087448
14.
LangaK. M.RyanL. H.McCammonR. J.JonesR. N.ManlyJ. J.LevineD. A.SonnegaA.FarronM.WeirD. R. (2020). The Health and retirement study harmonized cognitive assessment protocol Project: Study design and methods. Neuroepidemiology, 54(1), 64–74. https://doi.org/10.1159/000503004
15.
LeeJ.GanguliM.WeermanA.ChienS.LeeD. Y.VargheseM.DeyA. B. (2020). Online clinical consensus diagnosis of dementia: Development and validation. Journal of the American Geriatrics Society, 68(S3), S54–S59. https://doi.org/10.1111/jgs.16736
16.
ManlyJ. J.JonesR. N.LangaK. M.RyanL. H.LevineD. A.McCammonR.HeeringaS. G.WeirD. (2022). Estimating the prevalence of dementia and mild cognitive impairment in the US: The 2016 Health and retirement study harmonized cognitive assessment protocol Project. JAMA Neurology, 79(12), 1242–1249. https://doi.org/10.1001/jamaneurol.2022.3543
17.
McKhannG. M.KnopmanD. S.ChertkowH.HymanB. T.JackC. R.KawasC. H.KlunkW. E.KoroshetzW. J.ManlyJ. J.MayeuxR.MohsR. C.MorrisJ. C.RossorM. N.ScheltensP.CarrilloM. C.ThiesB.WeintraubS.PhelpsC. H. (2011). The diagnosis of dementia due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s and Dementia: The Journal of the Alzheimer’s Association, 7(3), 263–269. https://doi.org/10.1016/j.jalz.2011.03.005
18.
MorrisJ. C. (1993). The clinical dementia rating (CDR): Current version and scoring rules. Neurology, 43(11), 2412–2414. https://doi.org/10.1212/wnl.43.11.2412-a
19.
PlassmanB. L.LangaK. M.FisherG. G.HeeringaS. G.WeirD. R.OfstedalM. B.BurkeJ. R.HurdM. D.PotterG. G.RodgersW. L.SteffensD. C.McArdleJ. J.WillisR. J.WallaceR. B. (2008). Prevalence of cognitive impairment without dementia in the United States. Annals of Internal Medicine, 148(6), 427–434. https://doi.org/10.7326/0003-4819-148-6-200803180-00005
20.
PlassmanB. L.LangaK. M.FisherG. G.HeeringaS. G.WeirD. R.OfstedalM. B.BurkeJ. R.HurdM. D.PotterG. G.RodgersW. L.SteffensD. C.WillisR. J.WallaceR. B. (2007). Prevalence of dementia in the United States: The aging, demographics, and memory study. Neuroepidemiology, 29(1–2), 125–132. https://doi.org/10.1159/000109998
21.
WeirD. R.WallaceR. B.LangaK. M.PlassmanB. L.WilsonR. S.BennettD. A.DuaraR.LoewensteinD.GanguliM.SanoM. (2011). Reducing case ascertainment costs in U.S. population studies of Alzheimer’s disease, dementia, and cognitive impairment-Part 1. Alzheimer's & Dementia: The Journal of the Alzheimer's Association, 7(1), 94–109. https://doi.org/10.1016/j.jalz.2010.11.004
