Abstract
Background:
People with multiple sclerosis (PwMS) have an elevated risk of macrovascular disease that may be underestimated by vascular risk calculators (VRCs) validated in the general population.
Objectives:
This study evaluated, recalibrated and externally validated five commonly used VRCs for PwMS using population-based data from England, 1987–2023.
Methods:
PwMS and matched controls were identified from Clinical Practice and Research Datalink (CPRD) GOLD (calibration) and CPRD Aurum (validation). Exposure variables included multiple sclerosis (MS) status and risk factors as defined in Atherosclerotic Cardiovascular Disease, Framingham Risk Score (FRS), FRS-BMI (body mass index), QRESEARCH risk estimator version 3 score and Systematic Coronary Risk Evaluation version 2 score (SCORE2). Model performance was assessed using Somers’ D statistic, area under the receiver operating characteristic (ROC) curve and Hosmer–Lemeshow chi-square test. VRCs with ROC < 0.70 in PwMS were recalibrated using Cox regression, incorporating MS status. Ten-fold cross-validation was used to estimate Somers’ D.
Results:
Calibration: 9411 PwMS and 57,805 controls; validation: 45,934 PwMS and 278,452 controls. Discrimination declined with standard thresholds (e.g. SCORE2 sensitivity in PwMS, 30.0%). Only FRS-BMI retained all significant predictors and was successfully recalibrated, improving discrimination (Somers’ D = 0.815 vs. 0.792; Δ = 0.023) and showing good calibration. External validation showed modest gain (Somers’ D = 0.716; Δ = 0.003).
Conclusion:
These findings underscore the limitations of general-population VRCs in PwMS and support the development of MS-specific vascular risk models.
Introduction
People with multiple sclerosis (PwMS) experience a heightened risk of cardiovascular and cerebrovascular disease, 1 which is not entirely accounted for by traditional vascular risk factors such as hypertension, hyperlipidaemia, diabetes and smoking.1,2 Other possible contributing factors may include accelerated biological ageing, systemic inflammation, endothelial dysfunction, prothrombotic and autonomic mechanisms, cerebral hypoperfusion, genetic liability, treatment exposure and comorbidity.3 –5 These risk factors contribute to premature mortality or accelerated disability progression. 4
However, despite their importance, vascular comorbidities are underdiagnosed and undertreated in PwMS. 6
Clinical guidelines recommend the use of vascular risk calculators (VRCs) to stratify individuals based on their 10-year risk of vascular events, enabling targeted preventive interventions. Commonly used tools, including the Framingham Risk Score (FRS), BMI-based Framingham Risk Score (FRS-BMI),7,8 QRESEARCH risk estimator version 3 score (QRISK3), 9 Systematic Coronary Risk Evaluation version 2 score (SCORE2) 10 and the Atherosclerotic Cardiovascular Disease (ASCVD) score, 11 were validated in general populations and may not accurately estimate vascular risk in PwMS. In rheumatoid arthritis, another chronic inflammatory disease, VRCs underestimate cardiovascular risk in low- and medium-risk categories, while overestimating it in high-risk groups. 12 These findings suggest that systemic inflammation and disease-specific characteristics influence vascular risk in ways that are not accounted for by standard VRCs.
Furthermore, age is a key determinant of vascular risk scores.7 –11 Multiple sclerosis (MS) is frequently diagnosed at a mean age of 30 years globally; 13 so PwMS are systematically attributed low absolute risk, even when other clinical risk factors are present. This structural bias means that younger PwMS with subclinical or emerging vascular comorbidities will be excluded from early interventions. Therefore, recalibration of existing VRCs is needed for PwMS, as was also demonstrated with the WHO’s Fracture Risk Assessment Tool (FRAX®). 14
We aimed to (1) evaluate the performance of five commonly used VRCs: FRS, FRS-BMI, QRISK3, SCORE2 and ASCVD in PwMS, compared to people without MS; (2) recalibrate each VRC using population-level data and (3) externally validate those recalibrated VRCs that demonstrated improved/good performance using population-level data from an external data set of PwMS.
Methods
We conducted a population-based study of PwMS and matched controls registered with general practices in England, from 1987 to 2023. The study used data from routinely collected electronic health records from two Clinical Practice and Research Datalink (CPRD) databases. Ethics approval was granted by the Independent Scientific Advisory Committee of the CPRD (protocol numbers: 22_002520 and 18_279R).
Data source
We used two of the largest UK electronic health record databases: CPRD GOLD (1986–2018) and CPRD Aurum (1990–2023), which hold anonymized, routinely collected longitudinal primary care records.15,16 They are representative in terms of socio-demographic characteristics.15 –18 The calibration cohort was drawn from CPRD GOLD, which includes data from general practices using the Vision® software system. The validation cohort was drawn from CPRD Aurum. For both data sources, we limited the analysis to English practices 15 as primary care data are linked to multiple sources, including Hospital Episode Statistics (HES), Office for National Statistics (ONS) mortality and deprivation data, using National Health Service numbers.
Study population
MS cases were identified using a previously described algorithm (supplemental methods). 1 The index date was considered the date of the first MS diagnosis. 1 Inclusion criteria were the same as for prior work (see supplemental methods). 1 To enhance sample size, we included up to two time points per participant: data from the index year and, when available, data collected at age 50 (±5 years). This age range was selected given that vascular risk assessments are commonly performed around age 50 years in England. 19
To assess the discriminative performance of the VRCs, PwMS were randomly matched to as many as six individuals without MS based on general practice, sex and age. Controls were required to have up-to-standard clinical data available during the study period (CPRD Gold only) and to have no recorded diagnosis of MS or any other demyelinating condition (e.g. optic neuritis, transverse myelitis, acute disseminated encephalomyelitis or central nervous system demyelination not otherwise classified); this reduced the likelihood of including individuals who might later develop MS. Multiple controls were matched to each PwMS to decrease variance. We assigned the controls the index date of their matched MS case.
As VRCs estimate the 10-year risk of cardiovascular disease (CVD) or CVD-related mortality, we excluded individuals with a history of these outcomes before the index date, tailoring the inclusion criteria to match the specific definitions of CVD and applicable age ranges used by each calculator (see Supplemental Table 1).
Study variables
All variables considered for this study included age; sex; ethnicity; index of multiple deprivation; 20 BMI; smoking status; systolic blood pressure (SBP) and a measure of SBP variability; total cholesterol and total cholesterol/high-density lipoprotein (HDL) ratio; diagnosis of the following chronic diseases: diabetes, chronic renal disease, migraine, lupus and severe mental illness; treatment with antihypertensive medication, corticosteroid and atypical antipsychotics; and family history of macrovascular disease. Outcomes included the 10-year risk of CVD and CVD-mortality. As part of the CVD definition (see Supplemental Table 1 for details), cerebrovascular disease was identified from CPRD, HES and ONS records using validated, primary care records, stroke-specific ICD-10 codes in hospital data and cause-of-death codes where applicable, an approach that improves completeness and accuracy of stroke capture in English routine data.21 –23 Supplemental Table 1 summarizes the variables included in each VRC.
Statistical analysis
PwMS with missing data on smoking at index were classified as non-smokers if there was no prior indication of smoking history. 24 Those missing data on ethnicity were classified as White. 1 For BMI, SBP, total cholesterol and HDL, if data were missing at the index year, we considered records in the following and previous 5 years. 1 Consistent with previous studies using CPRD data in MS research, 1 we employed the multiple imputation method to handle missing data for time-varying variables, including BMI, SBP, total cholesterol and HDL. After checking the missing data percentage for these variables and multiple imputation assumptions, we employed multiple imputation by chained equation (10 copies) and combined results using Rubin’s rules for BMI and SBP (calibration cohort: 49.9% for blood pressure and 50% for BMI; validation cohort: 24.9% for blood pressure and 25.7% for BMI). 25 Given the high percentage of missing data, multiple imputation was not possible for cholesterol data. The regression model for the multiple imputation included these independent variables: sex, age, ethnicity, smoking status in the index year, number of comorbidities (considering those included in FRS, FRS-BMI, SCORE2, ASCVD and QRISK3)8,26,27 in the index year, treatment with lipid-lowering, oral anti-diabetic, anti-platelet, anti-coagulant and antihypertensive therapies in the index year, number of visits in the previous year, region and index of multiple deprivation (quintiles). 20
The recalibration and validation process comprised several steps. First, we assessed the performance of the original risk equation employed for the VRCs in the calibration cohort by calculating the receiver operating characteristic (ROC) curve area using the Youden index approximation, considering the continuous score and the binary variable defined by the published cut-point for intervention (binary classifier) for each VRC. Second, for ROC < 0.70 for PwMS, we recalibrated each VRC, which involves adjusting baseline hazard functions and population-specific means to better reflect the target group, an approach applied elsewhere. 14 Specifically, we employed Cox regression models including the terms of the original equations as independent variables and added MS status (binary variable) as an additional term. 28 We limited model extension to the inclusion of an MS indicator to preserve real-world applicability since our recalibration was designed for implementation using routinely collected data, also in primary care settings where MS-specific measures such as phenotype, disability status, inflammatory activity and detailed disease-modifying therapy (DMT) exposures are not consistently available. Transformations applied to continuous variables in the original VRCs were preserved during the recalibration and incorporated into the Cox model accordingly. Although we refer to this process as ‘recalibration’, it more accurately reflects a model revision approach. Specifically, we extended the original VRC by including MS status as an additional predictor; this constitutes a hybrid between recalibration and derivation. 29 Third, the recalibration process was conducted only for risk equations in which all terms remained statistically significant (p < 0.05) in the Cox model to ensure the reliability and interpretability of the recalibrated coefficients. Including non-significant predictors would have resulted in unstable or non-reproducible point estimates, potentially introducing bias or reducing generalizability.29,30 As previously suggested, we evaluated the calibration of the derived risk algorithm in our internal data set by stratifying the cohort by deciles of risk score and employing a Hosmer–Lemeshow test. 8 Finally, k-fold (10-fold) cross-validation was employed to calculate the Somers’ D statistics. 31 For VRCs that were successfully recalibrated, an interaction term between MS and other terms was also tested. The recalibrated VRCs were tested externally using the validation and calculating the Somers’ D statistics; 31 for completeness, the ROC curve for the externally validated score was additionally reported in the Supplemental Material.
Results
Baseline characteristics of the calibration cohort
Sample size varied across sub-cohorts. The FRS-BMI sub-cohort included nearly the full calibration sample (9411 PwMS and 57,805 controls), reflecting minimal exclusions beyond those with prior macrovascular disease or age outliers. The SCORE2-based sub-cohort was the smallest cohort (761 PwMS and 5252 controls) (see Supplemental Figure 1). At index date, among PwMS, the mean age ranged from 46.0 to 52.7 years depending on the VRC sub-cohort. Smoking prevalence was higher among PwMS compared with controls across all subgroups. Mean BMI, SBP and lipid profiles were broadly comparable between groups, although PwMS showed marginally lower SBP (Table 1).
Descriptive statistics of the calibration cohorts.
ASCVD = Atherosclerotic Cardiovascular Disease score; QRISK3 = QRESEARCH risk estimator version 3 score; SCORE2 = Systematic Coronary Risk Evaluation version 2 score; PwMS = people with Multiple Sclerosis; SBP = systolic blood pressure; HDL = high-density lipoprotein.
Discrimination performance of the VRCs
VRCs performed well when considering the continuous score and the area under the ROC curve (AUROC, Figure 1). Overall, several VRCs demonstrated moderate-to-good discrimination in the MS and control groups. For example, the ASCVD, FRS-BMI and SCORE2 achieved AUROC > 0.70 in PwMS and controls. QRISK3 and Framingham (cholesterol-based) models showed more modest discrimination, with an AUROC of 0.57–0.66, with slightly lower performance among PwMS than controls.
ROC areas for the vascular risk calculators in each derivation sub-cohort.
However, performance was poor when standard clinical cut-points were applied (Table 2). Sensitivity was poor across all models in PwMS, whereas specificity tended to be higher. For instance, at the ASCVD 7.5% threshold, sensitivity and specificity in PwMS were 58.2% and 70.4%, respectively (AUROC approximation: 0.64). QRISK3 showed a similar pattern, with 60.9% sensitivity and 64.8% specificity (AUROC approximation: 0.63). SCORE2 performed poorly at its recommended threshold, with a sensitivity of 30.0% and specificity of 34.0% in PwMS (AUROC approximation: 0.32).
Summary statistics for the cut-point for intervention for each vascular risk calculator.
ASCVD = Atherosclerotic Cardiovascular Disease score; QRISK3 = QRESEARCH risk estimator version 3 score; SCORE2 = Systematic Coronary Risk Evaluation version 2 score; PwMS = people with Multiple Sclerosis; ROC = Receiver Operating Characteristic; PPV = positive predictive value; NPV = negative predictive value.
Recalibration of vascular risk calculators
Of the five VRCs tested, only the FRS-BMI equation retained all statistically significant predictors (p < 0.05) in the Cox regression and proceeded to recalibration (see Supplemental Table 2).
For the cross-validated recalibrated FRS-BMI model with the original predictors and MS status as an additional covariate (model A), the Somers’ D coefficient was 0.815 (95% CI = 0.801–0.829) (Table 3). The model (B) that incorporated interaction terms between MS and all predictors performed similarly (Somers’ D coefficient: 0.817 (95% CI = 0.803–0.831)) to model A (p = 0.054). When comparing the two newly derived models with the FRS-BMI equation, both models showed significant gain in discriminative performance (Model A: Δ Somers’ D = 0.023, 95% CI = 0.017–0.030; Model B: Δ Somers’ D = 0.026, 95% CI = 0.018–0.033). Calibration, assessed via Hosmer–Lemeshow-type chi-square statistics, showed excellent fit (Model A: χ2 = 74,006; Model B: χ2 = 77,226). Although Models A and B demonstrated comparable performance regarding discrimination and calibration, we selected Model A for parsimony.
Cox model for the recalibration of the Framingham Score with BMI.
The model A did not consider interaction terms for Multiple Sclerosis (MS), while the model B considered interaction terms for MS. BMI = body mass index; SBP = systolic blood pressure; ln = natural log.
For the selected model, an optimal cut-point for intervention was 2.0% (sensitivity: 69.7%, specificity: 73.1%, correctly classified: 73.1%). Applying this cut-point to the PwMS in the calibration cohort, 38.8% of this population would be considered high risk.
Baseline characteristics of the validation cohort
The validation cohort comprised 324,386 individuals, including 45,934 who were PwMS; 69.2% were females (Table 4). Average age at index year was 44 years across all groups, while the proportion of the population that was non-White population ranged from 1% to 2%. Smoking was consistently higher in PwMS (23.9% in females with MS), while BMI was consistently higher in controls as compared with PwMS (controls: 26.1, SD: 4.7; PwMS: 25.8, SD: 4.8). However, diabetes prevalence was consistently higher in PwMS than in controls, although it never exceeded 3% (3% in males with MS).
Descriptive statistics of the validation cohort for the recalibrated Framingham BMI.
PwMS = people with Multiple Sclerosis.
External validation of the recalibrated FRS-BMI risk equation
In the external validation cohort, the recalibrated FRS-BMI score showed a good discriminating ability (Somers’ D = 0.716, 95% CI = 0.710–0.716). When comparing the newly derived score with the original FRS-BMI equation, the score showed a marginal gain in discriminative performance in the validation data set as well (Δ Somers’ D = 0.003, 95% CI = 0.002–0.003; Supplemental Figure 2).
Discussion
We comprehensively evaluated and recalibrated widely used VRCs for PwMS, using a large, population-based data set. While several VRCs (including ASCVD, SCORE2 and FRS-BMI) demonstrated moderate-to-good discrimination when used as continuous scores, their performance was substantially reduced at clinically relevant decision thresholds. Importantly, only the FRS-BMI score retained statistical validity across all terms when applied to the MS population and was recalibrated.
After model extension with the addition of MS status, the revised FRS-BMI model demonstrated improved discrimination (Δ Somers’ D = 0.023–0.026) and retained good calibration. The principle of parsimony supported the selection of this model (without interaction terms between MS and other terms), enhancing interpretability and potential for clinical implementation without compromising predictive accuracy. However, external validation revealed only a modest performance gain over the original model (Δ Somers’ D = 0.003), indicating that the clinical utility of the recalibrated model remains limited. These findings show that the risk factors included in existing models are insufficient, and de novo derivation of an MS-specific vascular risk algorithm is needed to achieve clinically meaningful advances in vascular risk stratification. Future studies should evaluate whether the recalibrated model improves treatment decisions in practice, for example, using decision curve analysis across clinically relevant thresholds and reclassification measures centred on guideline-based thresholds such as those for statin therapy. 32 We note that recent studies suggest that biological age and frailty may be important indices in predicting cardiovascular events, both observations that may be relevant to the MS population. 33
A key strength of our study lies in the large, population-based nature of both calibration and validation cohorts, encompassing more than 300,000 individuals, with consistent MS case definitions and follow-up through linked electronic health record data. The methodological rigour applied in calibration enhances internal validity and reproducibility. Unlike previous studies of model recalibration (e.g. in osteoporosis), our approach explicitly considered model extension, which has been recognized as a best practice when significant shifts in underlying risk are expected.29,30 The long study window spanned substantial evolution in MS diagnosis and management, and in cardiovascular prevention;34,35 we did not model these temporal changes explicitly as this was outside our scope, although prior population-based work showed consistent excess macrovascular risk after adjustment across eras. 1
Nevertheless, several limitations warrant discussion. First, although our recalibrated model showed statistical improvement, the incremental gain in clinical discrimination was small. This may reflect the mismatch between the assumptions embedded in general-population models and the pathophysiological profile of PwMS, including systemic inflammation, neurovascular interactions and differential medication use, which likely influences vascular risk in ways not captured by traditional predictors. Second, high and selective missingness of lipid measurements precluded reliable imputation, likely penalizing cholesterol-based equations over BMI-based ones; this limitation may have contributed to the apparent superiority of FRS-BMI, partly reflecting data availability rather than true model fitness. Also, some DMTs used in MS can affect lipid profiles, 36 highlighting the importance of lipid capture in future data sets. Third, our approach did not explore non-linear interactions. Fourth, including a binary variable for the MS status does not capture relevant MS-specific characteristics such as disease duration, phenotype or treatment exposure, but reflects the intended use in primary care settings as well, where such information is often unavailable. Fifth, by design, we constrained predictors to those in the original tools plus MS status. Therefore, other variables, including alcohol consumption, that can relate to white matter changes and cerebrovascular risk, 37 were not included. Similarly, there may be some residual confounding from variables such as physical activity. These may be considered in a future MS-specific calculator. Furthermore, neuroimaging data were not available, and acute demyelinating lesions in MS can rarely mimic ischemic stroke on diffusion-weighted imaging.38,39 However, in large data sets using linked hospital and mortality data, any resulting misclassification is likely limited and may be bidirectional, tending to bias estimates towards the null.21,40 Finally, although the inclusion of MS status and its interactions improves statistical fit, it does not address potential mediators (e.g. corticosteroid use and frailty status), which may drive excess vascular risk. Future studies should aim to incorporate such MS-specific factors directly into prediction algorithms.
These findings have tangible implications for vascular risk management in MS. First, even a modestly improved score, such as the recalibrated FRS-BMI, could serve as an interim tool for better identifying at-risk individuals pending the development of new calculators. Importantly, current international guidelines for cardiovascular prevention do not include MS as a high-risk condition. 11 Given the consistent evidence of increased vascular risk in PwMS, especially at younger ages and in women, our results support reclassification of MS as a vascular risk-enhancing condition, analogous to rheumatoid arthritis. Second, disease-specific guidelines should consider incorporating vascular risk scoring tools adapted for PwMS, particularly at MS diagnosis. Risk scores could guide targeted prevention, including lipid-lowering therapy, blood pressure control and smoking cessation – interventions which are underutilized in this population despite their proven benefit. 6
Last, researchers and policymakers should prioritize prospective derivation of MS-specific vascular risk models using multi-country electronic health records or registry data sets, potentially incorporating widely available biomarkers or neuroimaging metrics to capture vascular contributions unique to MS pathogenesis.
Conclusion
In summary, we evaluated five established VRCs in a large MS cohort, finding modest clinical improvement after statistical recalibration in the FRS-BMI. The modest gain in predictive performance following model extension suggests that recalibrating general-population models is insufficient for addressing the vascular risk burden in PwMS. Development of new, MS-specific VRCs appears necessary to enable personalized, preventive vascular care in this population.
Supplemental Material
sj-docx-1-msj-10.1177_13524585261444243 – Supplemental material for Recalibration and external validation of vascular risk calculators for multiple sclerosis: A population-based study using CPRD data in England, 1987–2023
Supplemental material, sj-docx-1-msj-10.1177_13524585261444243 for Recalibration and external validation of vascular risk calculators for multiple sclerosis: A population-based study using CPRD data in England, 1987–2023 by Raffaele Palladino, Ruth Ann Marrie, Giuseppina Affinito, Azeem Majeed, Stephen Allan Schaffer and Jeremy Chataway in Multiple Sclerosis Journal
Footnotes
Author contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Raffaele Palladino. The first draft of the manuscript was written by Raffaele Palladino, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Declaration of conflicting interests
The authors declared the following potential conflicts of interest with respect to the research, authorship and/or publication of this article: In the last 3 years, R.P. has received support from the UK MS Society and has taken part in advisory boards/consultancy for MSD and Sanofi. R.A.M. receives research funding from CIHR, MS Canada, National Multiple Sclerosis Society, CMSC, the Arthritis Society, MMSF, Brain Canada, CHRIM, Public Health Agency of Canada and the US Department of Defense, and is a co-investigator on studies receiving funding from Biogen Idec and Roche Canada. She holds the Multiple Sclerosis Clinical Research Chair (Dalhousie University). A.M. is supported by the NIHR Applied Research Collaboration NW London. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. In the last 3 years, J.C. has received support from the Health Technology Assessment (HTA) Programme (National Institute for Health and Care Research, NIHR), the UK MS Society, the US National MS Society and the Rosetrees Trust. He is supported in part by the National Institute for Health and Care Research, University College London Hospitals (UCLH) Biomedical Research Centre, London, UK. He has been a local principal investigator for a trial in MS funded by MS Canada. A local principal investigator for commercial trials funded by Ionis and Roche, and has taken part in advisory boards/consultancy for Biogen, Contineum Therapeutics, FSD Pharma, InnoCare, Pheno Therapeutics and Roche.
Funding
The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was partially funded by the UK MS Society Grant 146.
Ethics considerations
The study used data from routinely collected electronic health records from two Clinical Practice and Research Datalink (CPRD) databases, the CPRD Gold and CPRD Aurum. The Independent Scientific Advisory Committee of the CPRD (protocol number: 18_279R) granted ethics approval.
Consent to participate
The data were anonymized; therefore, informed consent was not required.
Data availability
The data used in this study were obtained under licence from the CPRD and were accessed via a secure server environment per the data provider’s policy. The data cannot be shared publicly. Investigators with a valid research proposal and necessary ethics/regulatory approvals may apply for access to similar data through the CPRD, subject to their standard access and licensing terms.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
