Restricted accessResearch articleFirst published online 2025-2
Development and Validation of Prediction Models for Incident Reversible Cognitive Frailty Based on Social-Ecological Predictors Using Generalized Linear Mixed Model and Machine Learning Algorithms: A Prospective Cohort Study
This study aimed to develop and validate prediction models for incident reversible cognitive frailty (RCF) based on social-ecological predictors. Older adults aged ≥60 years from China Health and Retirement Longitudinal Study (CHARLS) 2011–2013 survey were included as training set (n = 1230). The generalized linear mixed model (GLMM), eXtreme Gradient Boosting, support vector machine, random forest, and Binary Mixed Model forest were used to develop prediction models. All models were evaluated internally with 5-fold cross-validation and evaluated externally via CHARLS 2013–2015 survey (n = 1631). Only GLMM showed good discrimination (AUC = 0.765, 95% CI = 0.736, 0.795) in training set, and all models showed fair discrimination (AUC = 0.578–0.667, 95% CI = 0.545, 0.725) in internal and external validation. All models showed acceptable calibration, overall prediction performance, and clinical usefulness in training and validation sets. Older adults were divided into three groups using risk score based on GLMM, which could assist healthcare providers to predict incident RCF, facilitating early identification of high-risk population.
AbeT.SeinoS.NofujiY.TomineY.NishiM.HataT.ShinkaiS.KitamuraA. (2021). Development of risk prediction models for incident frailty and their performance evaluation. Preventive Medicine, 153, Article 106768. https://doi.org/10.1016/j.ypmed.2021.106768
2.
AguilaE.López-OrtegaM.Gutiérrez RobledoL. M. (2018). Non-contributory pension programs and frailty of older adults: Evidence from Mexico. PLoS One, 13(11), Article e0206792. https://doi.org/10.1371/journal.pone.0206792
3.
AyturS. A.CarlinoS.BernardF.WestK.DobrzyckiV.MalikR. (2022). Social-ecological theory, substance misuse, adverse childhood experiences, and adolescent suicidal ideation: Applications for community-academic partnerships. Journal of Community Psychology, 50(1), 265–284. https://doi.org/10.1002/jcop.22560
4.
CabitzaF.CampagnerA.SoaresF.García de Guadiana-RomualdoL.ChallaF.SulejmaniA.SeghezziM.CarobeneA. (2021). The importance of being external. methodological insights for the external validation of machine learning models in medicine. Computer Methods and Programs in Biomedicine, 208(11), Article 106288. https://doi.org/10.1016/j.cmpb.2021.106288
5.
ChangZ.ChenY.ZhaoY.FuJ.LiuY.TangS.HanY.FanZ. (2022). Association of sunshine duration with acute myocardial infarction hospital admissions in Beijing, China: A time-series analysis within-summer. The Science of the Total Environment, 828(21), Article 154528. https://doi.org/10.1016/j.scitotenv.2022.154528
6.
ChenC.ParkJ.WuC.XueQ.AgogoG.HanL.HoogendijkE. O.LiuZ.WuZ. (2020). Cognitive frailty in relation to adverse health outcomes independent of multimorbidity: Results from the China health and retirement longitudinal study. Aging (Albany NY), 12(22), 23129–23145. https://doi.org/10.18632/aging.104078
7.
ChristianL. M.GlaserR.PorterK.MalarkeyW. B.BeversdorfD.Kiecolt-GlaserJ. K. (2011). Poorer self-rated health is associated with elevated inflammatory markers among older adults. Psychoneuroendocrinology, 36(10), 1495–1504. https://doi.org/10.1016/j.psyneuen.2011.04.003
8.
ChristodoulouE.MaJ.CollinsG. S.SteyerbergE. W.VerbakelJ. Y.Van CalsterB. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology, 110, 12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004
9.
CohenJ. (1977). Statistical power analysis for the behavioral sciences (2nd ed.). Academic Press.
10.
CollinsG. S.ReitsmaJ. B.AltmanD. G.MoonsK. G. (2015). Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. BMJ, 350(2), g7594. https://doi.org/10.1136/bmj.g7594
11.
DavagdorjK.PhamV. H.Theera-UmponN.RyuK. H. (2020). XGBoost-Based framework for smoking-induced noncommunicable disease prediction. International Journal of Environmental Research and Public Health, 17(18), 6513. https://doi.org/10.3390/ijerph17186513
12.
FokkemaM.Edbrooke-ChildsJ.WolpertM. (2021). Generalized linear mixed-model (GLMM) trees: A flexible decision-tree method for multilevel and longitudinal data. Psychotherapy Research: Journal of the Society for Psychotherapy Research, 31(3), 313–325. https://doi.org/10.1080/10503307.2020.1785037
13.
GrangerE.WilliamsG.Di NardoF.HarrisonA.VermaA. (2017). The relationship between physical activity and self-rated health status in European adolescents: Results of the EURO-URHIS 2 survey. The European Journal of Public Health, 27(suppl_2), 107–111. https://doi.org/10.1093/eurpub/ckw177
14.
GrollA.TutzG. (2014). Variable selection for generalized linear mixed models by L1-penalized estimation. Statistics and Computing, 24(2). https://doi.org/10.1007/s11222-012-9359-z
HanleyJ. A. (2001). A heuristic approach to the formulas for population attributable fraction. Journal of Epidemiology & Community Health, 55(7), 508–514. https://doi.org/10.1136/jech.55.7.508
17.
HosmerD. W. L. S. (2013). Applied logistic regression. John Wiley & Sons.
18.
HuangY. Q.LiangC. H.HeL.TianJ.LiangC. S.ChenX.MaZ. L.LiuZ. Y. (2016). Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, 34(18), 2157–2164. https://doi.org/10.1200/JCO.2015.65.9128
19.
KerrK. F.BrownM. D.ZhuK.JanesH. (2016). Assessing the clinical impact of risk prediction models with decision curves: Guidance for correct interpretation and appropriate use. Journal of Clinical Oncology: Official Journal of the American Society of Clinical Oncology, 34(21), 2534–2540. https://doi.org/10.1200/JCO.2015.65.5654
20.
KivipeltoM.NganduT.LaatikainenT.WinbladB.SoininenH.TuomilehtoJ. (2006). Risk score for the prediction of dementia risk in 20 years among middle aged people: A longitudinal, population-based study. The Lancet Neurology, 5(9), 735–741. https://doi.org/10.1016/S1474-4422(06)70537-3
21.
LiuQ.YangL.ShiZ.YuJ.SiH.JinY.BianY.LiY.JiL.QiaoX.WangW.LiuH.ZhangM.WangC. (2023). Development and validation of a preliminary clinical support system for measuring the probability of incident 2-year (pre)frailty among community-dwelling older adults: A prospective cohort study. International Journal of Medical Informatics, 177(1382), Article 105138. https://doi.org/10.1016/j.ijmedinf.2023.105138
22.
LyuQ.GuanC. X.KongL. N.ZhuJ. L. (2023). Prevalence and risk factors of cognitive frailty in community-dwelling older adults with diabetes: A systematic review and meta-analysis. Diabetic Medicine: A Journal of the British Diabetic Association, 40(1), Article e14935. https://doi.org/10.1111/dme.14935
23.
MolinuevoJ. L.RabinL. A.AmariglioR.BuckleyR.DuboisB.EllisK. A.EwersM.HampelH.KlöppelS.RamiL.ReisbergB.SaykinA. J.SikkesS.SmartC. M.SnitzB. E.SperlingR.van der FlierW. M.WagnerM.JessenF.Subjective Cognitive Decline Initiative SCD-I Working Group. (2017). Implementation of subjective cognitive decline criteria in research studies. Alzheimer's and Dementia: The Journal of the Alzheimer's Association, 13(3), 296–311. https://doi.org/10.1016/j.jalz.2016.09.012
24.
OnozukaD.HagiharaA. (2017). Within-summer variation in out-of-hospital cardiac arrest due to extremely long sunshine duration. International Journal of Cardiology, 231, 120–124. https://doi.org/10.1016/j.ijcard.2016.12.179
25.
PanX.LuoY.ZhaoD.ZhangL. (2022). Associations among drinking water quality, dyslipidemia, and cognitive function for older adults in China: Evidence from CHARLS. BMC Geriatrics, 22(1), 683. https://doi.org/10.1186/s12877-022-03375-y
26.
ParkC.MishraR.SharafkhanehA.BryantM. S.NguyenC.TorresI.NaikA. D.NajafiB. (2021). Digital biomarker representing frailty phenotypes: The use of machine learning and sensor-based sit-to-stand test. Sensors, 21(9), 3258. https://doi.org/10.3390/s21093258
27.
PencinaM. J.D'AgostinoR. B. Sr.D'AgostinoR. B.Jr.VasanR. S. (2008). Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statistics in Medicine, 27(2), 157–172. https://doi.org/10.1002/sim.2929
28.
PengL. N.HsiaoF. Y.LeeW. J.HuangS. T.ChenL. K. (2020). Comparisons between hypothesis- and data-driven approaches for multimorbidity frailty index: A machine learning approach. Journal of Medical Internet Research, 22(6), Article e16213. https://doi.org/10.2196/16213
29.
PepeM. S.FanJ.FengZ.GerdsT.HildenJ. (2015). The net reclassification index (NRI): A misleading measure of prediction improvement even with independent test data sets. Statistics in biosciences, 7(2), 282–295. https://doi.org/10.1007/s12561-014-9118-0
PorellF.DuganE.SilversteinN. M.LeeC. M. (2022). A new measure of community health: Findings from the healthy aging data reports. The Gerontologist, 62(9), e481–e492. https://doi.org/10.1093/geront/gnab118
32.
RuanQ.XiaoF.GongK.ZhangW.ZhangM.RuanJ.ZhangX.ChenQ.YuZ. (2020). Prevalence of cognitive frailty phenotypes and associated factors in a community-dwelling elderly population. The Journal of Nutrition, Health & Aging, 24(2), 172–180. https://doi.org/10.1007/s12603-019-1286-7
33.
RuanQ.YuZ.ChenM.BaoZ.LiJ.HeW. (2015). Cognitive frailty, a novel target for the prevention of elderly dependency. Ageing Research Reviews, 20(1), 1–10. https://doi.org/10.1016/j.arr.2014.12.004
34.
SiH.JinY.QiaoX.TianX.LiuX.WangC. (2021). Predictive performance of 7 frailty instruments for short-term disability, falls and hospitalization among Chinese community-dwelling older adults: A prospective cohort study. International Journal of Nursing Studies, 117(1), Article 103875. https://doi.org/10.1016/j.ijnurstu.2021.103875
35.
SpeiserJ. L.WolfB. J.ChungD.KarvellasC. J.KochD. G.DurkalskiV. L. (2019). BiMM forest: A random forest method for modeling clustered and longitudinal binary outcomes. Chemometrics and Intelligent Laboratory Systems: An International Journal Sponsored by the Chemometrics Society, 185(4), 122–134. https://doi.org/10.1016/j.chemolab.2019.01.002
36.
SteyerbergE. W.VickersA. J.CookN. R.GerdsT.GonenM.ObuchowskiN.PencinaM. J.KattanM. W. (2010). Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 21(1), 128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
37.
SuD.ZhangX.HeK.ChenY. (2021). Use of machine learning approach to predict depression in the elderly in China: A longitudinal study. Journal of Affective Disorders, 282, 289–298. https://doi.org/10.1016/j.jad.2020.12.160
38.
VeroneseN.ManiscalcoL.MatrangaD.LaccaG.DominguezL. J.BarbagalloM. (2023). Association between pollution and frailty in older people: A cross-sectional analysis of the UK biobank. Journal of the American Medical Directors Association, 24(4), 475–481.e3. https://doi.org/10.1016/j.jamda.2022.12.027
39.
WangW.SiH.YuR.QiaoX.JinY.JiL.LiuQ.BianY.YuJ.WangC. (2021). Effects of reversible cognitive frailty on disability, quality of life, depression, and hospitalization: A prospective cohort study. Aging & Mental Health, 26(10), 2031–2038. https://doi.org/10.1080/13607863.2021.2011835
40.
WangZ.ChengY.SeabergE. C.BeckerJ. T. (2020). Quantifying diagnostic accuracy improvement of new biomarkers for competing risk outcomes. Biostatistics, 23(2), 666–682. https://doi.org/10.1093/biostatistics/kxaa048
41.
WuC.SmitE.XueQ. L.OddenM. C. (2017). Prevalence and correlates of frailty among community-dwelling Chinese older adults: The China health and retirement longitudinal study. The Journals of Gerontology. Series A, Biological Sciences and Medical Sciences, 73(1), 102–108. https://doi.org/10.1093/gerona/glx098
42.
YanJ. (2016). Multiple imputation issues and guidance for practice. Chongqing University Press.
43.
YangL.LiuQ.ZhaoQ.ZhuX.WangL. (2020). Machine learning is a valid method for predicting prehospital delay after acute ischemic stroke. Brain and Behavior, 10(10), Article e01794. https://doi.org/10.1002/brb3.1794
ZhangH.ChenD.ShaoJ.ZouP.CuiN.TangL.WangX.WangD.YeZ. (2021). External validation of the prognostic prediction model for 4-year risk of metabolic syndrome in adults: A retrospective cohort study. Diabetes, Metabolic Syndrome and Obesity: Targets and Therapy, 14, 3027–3034. https://doi.org/10.2147/DMSO.S316950
46.
ZhangT.RenY.ShenP.JiangS.YangY.WangY.LiZ.YangY. (2022). Prevalence and associated risk factors of cognitive frailty: A systematic review and meta-analysis. Frontiers in Aging Neuroscience, 13, Article 755926. https://doi.org/10.3389/fnagi.2021.755926
47.
ZhangZ.JinP.FengM.YangJ.HuangJ.ChenL.XuP.SunJ.HuC.HongY. (2022). Causal inference with marginal structural modeling for longitudinal data in laparoscopic surgery: A technical note. Laparoscopic, Endoscopic and Robotic Surgery, 5(4), 146–152. https://doi.org/10.1016/j.lers.2022.10.002
48.
ZhaoY.HuY.SmithJ. P.StraussJ.YangG. (2014). Cohort profile: The China health and retirement longitudinal study (CHARLS). International Journal of Epidemiology, 43(1), 61–68. https://doi.org/10.1093/ije/dys203
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
