Sage Journals: Discover world-class research

Abstract

Introduction

Bleeding is the main safety concern of treatment with antiplatelet drugs. We aimed to refine prediction of major bleeding on antiplatelet treatment after a transient ischaemic attack (TIA) or stroke by assessing the added value of new predictors to the existing S₂TOP-BLEED score.

Patients and methods

We used Cox regression analysis to study the association between candidate predictors and major bleeding among 2072 patients with a transient ischaemic attack or ischaemic stroke included in a population-based study (Oxford Vascular Study – OXVASC). An updated model was proposed and validated in 1094 patients with a myocardial infarction included in OXVASC. Models were compared with c-statistics, calibration plots, and net reclassification improvement.

Results

Independent predictors for major bleeding on top of S₂TOP-BLEED variables were peptic ulcer (hazard ratio (HR): 1.72; 1.04–2.86), cancer (HR: 2.40; 1.57–3.68), anaemia (HR: 1.55; 0.99–2.44) and renal failure (HR: 2.20; 1.57–4.28). Addition of those variables improved discrimination from 0.69 (0.64–0.73) to 0.73 (0.69–0.78) in the TIA/stroke cohort (p = 0.01). Performance improved particularly for upper gastro-intestinal bleeds (0.70; 0.64–0.75 to 0.77; 0.72–0.82). Net reclassification improved over the entire range of the score (net reclassification improvement: 0.56; 0.36–0.76). In the validation cohort, discriminatory performance improved from 0.68 (0.62–0.74) to 0.70 (0.64–0.76).

Discussion and Conclusion

Peptic ulcer, cancer, anaemia and renal failure improve predictive performance of the S₂TOP-BLEED score for major bleeding after stroke. Future external validation studies will be required to confirm the value of the STOP-BLEED+ score in transient ischaemic attack/stroke patients.

Keywords

Stroke major bleeding antiplatelet treatment prediction

Introduction

Treatment with antiplatelet drugs is indicated following a transient ischaemic attack (TIA) or non-cardioembolic ischaemic stroke.¹ Bleeding is the main safety concern of antiplatelet drugs, with an average one-year risk of 1%–1.5%.^2,3 Accurate prediction of bleeding risk for an individual patient would allow physicians to weigh benefits and risks of antiplatelet treatment.

Clinical prediction scores, such as the S₂TOP-BLEED score, have been developed to improve risk stratification for major bleeding and are mainly based on patient characteristics.⁴ However, performance of available scores is modest with c-statistics generally below 0.70,^4–6 and discrimination might be improved by extending models with other known risk factors for bleeding. The presence of a peptic ulcer is a frequent cause of upper gastro-intestinal (GI) bleeding and increases risk approximately three-fold.^7,8 Also, cancer has been associated with risk of bleeding, which is further enhanced by antithrombotic treatment.^9,10 Furthermore, several laboratory characteristics have been shown to increase risk of major bleeding, including renal failure, liver failure and anaemia.^11–13

The S₂TOP-BLEED score was derived from individual patient data from trials.⁴ As patients with the highest risk of bleeding are often excluded from trials studying antiplatelet therapy, the predictive value of factors that served as exclusion criteria could not be assessed in the development of the score. Extending the model with other risk factors for major bleeding might improve predictive performance. We aimed to investigate whether we can refine prediction of major bleeding on antiplatelet treatment after a TIA or ischaemic stroke by extending the previously developed S₂TOP-BLEED score (S₂TOP-BLEED+) in a population-based cohort of patients without exclusions.

Methods

Study population

The Oxford Vascular Study (OXVASC) is an ongoing population-based cohort study of all acute vascular events in a population of 92,728 individuals registered with 100 general practitioners in 9 general practices in Oxfordshire, UK.¹⁴ Overlapping methods of hot and cold pursuit are used to identify all vascular events in the study population. In the present study, we analysed patients who experienced a TIA, ischaemic stroke, or myocardial infarction (either a first event or recurrent event) between 2002 and 2012, and who were prescribed antiplatelet treatment. These included both patients who were on premorbid antiplatelet drugs as well as patients who started antiplatelet drugs after the index event. The use of antiplatelet drugs was self-reported by patients and cross-referenced with primary care and medical records.

During an initial visit, information on baseline characteristics and risk factors was collected. Patients were followed-up at 30 days, 6 months, 1 year, 5 and 10 years by a study nurse or physician. At each follow-up visit, recurrent ischaemic events and bleeding events that required medical attention were recorded. Bleeding events were also identified by daily searches of all hospital admissions, by review of administrative diagnostic codes from hospital and primary care records, and by searches of blood transfusion records. Only bleeds that required medical attention or were fatal prior to medical attention were included. Bleeds secondary to trauma, surgery or haematological malignancy were excluded. Haemorrhagic transformation of the index event was not included as a major intracranial bleed. However, if the patient presented with a recurrent event and no distinction could be made between a primary intracerebral haemorrhage and a recurrent ischaemic stroke with haemorrhagic transformation, the event was included as a major bleed (n = 6). The severity of bleeds was classified according to the CURE criteria.¹⁵ Major bleeds were bleeds that were substantially disabling with persistent sequelae, intraocular bleeds leading to significant loss of vision or bleeds requiring transfusion of two or more units of blood. Bleeding events that required medical attention but did not fulfil the criteria of major bleeding were recorded as significant non-major bleeds.

Candidate predictors that could potentially improve prediction of bleeding were selected based on the literature.¹⁶ Their inclusion was dependent on availability in OXVASC. Predictors of interest that could be studied were history of peptic ulcer^7,8 (based on face-to-face patient interview plus cross-referencing with primary care and medical records), history of cancer^9,10 (based on face-to-face patient interview plus cross-referencing with primary care and medical records). History of cancer included any type of cancer, i.e. GI, non-GI, haematological malignancies and skin cancer, and included both active cancer and cancer in remission, renal failure (any documented history of chronic renal failure – glomerular filtration rate of <30 ml/minute/1.73 m² estimated using the modification of diet in renal disease (MDRD) equation), liver failure and anaemia^11–13 (haemoglobin <12 g/dL (7.5 mmol/L) for women and <13 g/dL (8.1 mmol/L) for men, as assessed at baseline).

OXVASC has been approved by the local ethics committee. Patients gave written informed consent or assent was obtained from relatives for patients who were unable to provide consent.

Statistical analysis

Data on body mass index (BMI) were missing for 403 patients (13%) in both cohorts combined, and for these patients, the mean BMI was imputed. Twenty-two patients (0.7%) had missing values on variables other than BMI, and these patients were excluded from the analyses. We performed Cox regression analysis to study the association between candidate predictors and major bleeding, while including the linear predictor of the original S₂TOP-BLEED score as variable in the model. This method for model updating is referred to as recalibration plus extension.¹⁷ It has shown to provide well-calibrated results for new patients and is recommended when the data set for updating is relatively small compared with the development cohort¹⁷ (for the current study: 2072 patients in the update cohort vs. 43,112 patients in the derivation cohort), as it limits the number of coefficients that is re-estimated. We constructed a Cox regression model containing the candidate predictors and the original S₂TOP-BLEED score as variable and applied least absolute shrinkage and selection operator (lasso) regression with repeated cross validation to select the optimal model.^18,19 Compared with traditional stepwise selection, lasso is less likely to produce over-fitted models, particularly when the number of events per variable is small.²⁰ The baseline hazard and mean linear predictor were re-estimated after addition of new predictors.

We calculated three-year risk of major bleeding according to both the original and the updated model and classified patients into risk categories of low (<5%), intermediate (5%–10%) and high risk (>10%). We assigned weights to the additional variables by dividing the regression coefficients by the smallest coefficient of the original S₂TOP-BLEED model and rounded them to the nearest integer.

Performance of the proposed updated model was assessed with discrimination, calibration and reclassification measures. Discrimination reflects the ability of the model to distinguish between someone with and without a major bleed and was assessed with the c-statistic. The c-statistic was calculated separately for intracranial bleedings, non-major bleedings and upper GI bleeding, the most frequent type of antiplatelet-related bleeding, which may partly be prevented by treatment with gastro-protective drugs. Furthermore, we separately assessed discriminatory ability for the first three months, during which dual antiplatelet therapy may be considered. Calibration addresses the correspondence between the observed and predicted risk of major bleeding and was investigated with calibration plots. We calculated the net reclassification improvement (NRI) to assess change in risk stratification with the updated model.²¹ The NRI quantifies the percentage of correct movement across risk categories for patients with and without a major bleed. Correct movement is defined as upward classification for patients with an event, and downward classification for patients without an event. Net reclassification was assessed while taking survival time into account.²² Last, we assessed the improvement without applying risk categories, by means of the continuous NRI and the integrated discrimination improvement.²¹ Confidence intervals were obtained with bootstrapping.

We examined the robustness of the updated model by validating the model in MI patients included in OXVASC. We applied the coefficients of the model as determined in the TIA/stroke cohort and calculated the three-year risk of major bleeding for each patient. All patients were assigned zero points for the modified Rankin Scale variable. Again, discrimination, calibration and reclassification were assessed. All analyses were performed with R version 3.3.2. Results are reported in accordance with the TRIPOD statement.²³

Results

Between 2002 and 2012, 2072 patients with a TIA or ischaemic stroke on antiplatelet treatment were included in OXVASC (895 TIA and 1177 ischaemic stroke). Baseline characteristics of patients are presented in Supplementary Table I. During 8302 person-years of follow-up, 117 patients experienced a major bleed (31 fatal bleeds (26%) of which 19 were intracranial and 12 GI (Supplementary Table II)). Three-year risk of major bleeding was 5.7% (5.9% after exclusion of patients on a short course of dual antiplatelet therapy). The validation cohort consisted of 1094 patients with an MI included in OXVASC during the same period, of whom 70 had had a major bleed (three-year risk 7.1%; Supplementary Figure I).

Among patients with a TIA or ischaemic stroke, we identified history of cancer, peptic ulcer, anaemia and renal failure as independent predictors for major bleeding on top of the S₂TOP-BLEED risk factors (Table 1). Addition of those variables to the original S₂TOP-BLEED score led to an increment in c-statistic from 0.69 (0.64–0.73) to 0.73 (0.69–0.78), p = 0.01 (Table 2; for regression equation, see Supplementary Table III). Liver failure was initially selected as predictor in the model, but we decided to exclude it given the very low prevalence (1%) and wide confidence intervals. Three-year predicted risk of major bleeding based on the updated score was in close agreement with the observed risk of major bleeding as estimated with Kaplan–Meier (Supplementary Figure II). The discriminatory performance for upper GI bleeds improved from 0.70 (0.64–0.75) to 0.77 (0.72–0.82) with S₂TOP-BLEED+. The c-statistic for intracranial bleeding was 0.67 (0.60–0.74) and for non-major bleeds was 0.53 (0.47–0.59). The STOP-BLEED+ score showed a c-statistic of 0.71 (0.62–0.80) for prediction of major bleeding during the first three months (Supplementary Table IV). Discriminatory performance of the S₂TOP-BLEED+ score was better for major bleedings than for recurrent ischaemic events (c-statistic 0.57 (0.54–0.61)). The Brier score of the S₂TOP-BLEED+ model was 0.05 (scaled Brier score 15%). Figure 1 shows the distribution of patients with and without events among risk categories based on the original and updated S₂TOP-BLEED scores. Overall, 70% of patients remained in the same risk category. With the updated model, 4.1% of patients with a major bleed (event NRI) and 4.7% of patients without a major bleed (non-event NRI) were correctly reclassified, leading to an overall NRI of 0.09 (−0.04 to 0.22) (Table 2 and Figure 1). Over the entire range of predicted risks, the S₂TOP-BLEED+ score improved reclassification (continuous NRI 0.56; 0.36–0.76, Table 2), mainly driven by correct downward classification of non-events (38%) and a smaller improvement in classification of events (18%) (Supplementary Table V). When patients were divided in quartiles according to their predicted risk, the observed three-year risk of major bleeding ranged from 0.6% to 14.0%, as compared with 0.8% to 11.6% according to the original model (Figure 2). The ratio between risk of recurrent ischaemic events and major bleeding drops from 19:1 in the lowest quartile to 2:1 in the highest quartile of predicted bleeding risk (Supplementary Table VI).

Table 1.

Hazard ratios of candidate predictors for major bleeding, in the presence of original S₂TOP-BLEED variables.

	Multivariable HR (95% CI)^a	Beta coefficients estimated with lasso
Cancer	2.40 (1.57–3.68)	0.847
Peptic ulcer	1.72 (1.04–2.86)	0.511
Anaemia	1.55 (0.99–2.44)	0.429
Renal failure	2.20 (1.57–4.28)	0.766
Liver failure	1.62 (0.40–6.61)	–

^aAdjusted for original S₂TOP-BLEED variables (age, male sex, Asian ethnicity, body mass index, smoking, modified Rankin Scale score ≥3, hypertension, diabetes, prior stroke, type of antiplatelet treatment). HR: hazard ratio; CI: confidence interval.

Table 2.

Performance of S₂TOP-BLEED and S₂TOP-BLEED+ scores in TIA/stroke and MI cohorts.

	C-statistic (95% CI)	Continuous NRI (95% CI)	Categorical NRI (95% CI)	IDI (95% CI)
TIA/stroke cohort
S₂TOP-BLEED	0.69 (0.64–0.73)	Ref	Ref	Ref
S₂TOP-BLEED+	0.73 (0.69–0.78)	0.56 (0.36–0.76)	0.09 (−0.04 to 0.22)	0.019 (0.007–0.03)
MI cohort
S₂TOP-BLEED	0.68 (0.62–0.74)	Ref	Ref	Ref
S₂TOP-BLEED+	0.70 (0.64–0.76)	0.49 (0.21–0.78)	0.14 (−0.03 to 0.31)	0.011 (−0.005 to 0.028)

MI: myocardial infarction; NRI: net reclassification index; CI: confidence interval; TIA: transient ischaemic attack; IDI: integrated discrimination index; Ref: reference.

Figure 1.

Reclassification tables in TIA/stroke cohort. Same risk category: (14.6 + 29.1 + 37.6 + 809.4 + 354.3 + 185.4)/2067 = 70%; Event NRI: ((4.4 + 5.2 + 20.8) − (8.9 + 16.3))/130.4 = 0.041; Non-event NRI: ((265.1 + 106.7) − (54.6 + 43.8 + 182.2))/1936.6 = 0.047; Overall NRI: 0.041 + 0.047 = 0.088. KM: Kaplan–Meier.

Figure 2.

Observed risk of major bleeding across quartiles of predicted risk according to S₂TOP-BLEED and S₂TOP-BLEED+ scores. Q: quartile.

Figure 3.

Predicted three-year risk of major bleeding (%) according to the S₂TOP-BLEED+ score.

Based on the regression coefficients, we assigned points to anaemia (3 points), peptic ulcer (4 points), cancer (6 points) and renal failure (6 points). An updated score chart is presented in Table 3, with predicted risks displayed in Figure 3.

Table 3.

S₂TOP-BLEED and S₂TOP-BLEED+ score.

	S₂TOP-BLEED	S₂TOP-BLEED+
Risk factor	Points	Points
Age
45–54	2	2
55–64	4	4
65–74	6	6
75–84	9	9
85+	12	12
Male sex	2	2
Asian ethnicity	1	1
Current smoking	1	1
Hypertension	1	1
Diabetes	1	1
Prior stroke	1	1
mRS >2	2	2
BMI
<20	2	2
20–25	1	1
Treatment
A, A + D	1	1
A + C	5	5
Peptic ulcer	NA	4
Cancer	NA	6
Anaemia	NA	3
Renal failure	NA	6

A: aspirin; A + D: aspirin + dipyridamole; A + C: aspirin + clopidogrel; BMI: body mass index; mRS: modified Rankin Scale score; NA: not applicable.

The S₂TOP-BLEED+ score as proposed in the TIA/stroke cohort had a c-statistic of 0.70 (0.64–0.76) in the MI cohort, compared with 0.68 (0.62–0.74) for the original S₂TOP-BLEED score (p = 0.39), see Table 2. Calibration was slightly better for the updated model than for the original S₂TOP-BLEED score, although both models underestimated risk of major bleeding (Supplementary Figure III).

Discussion

Prediction of major bleeding can be refined by incorporating cancer, peptic ulcer, anaemia and renal failure in the existing S₂TOP-BLEED score for major bleeding in patients with a TIA or ischaemic stroke on antiplatelet therapy. Compared with the original score, the S₂TOP-BLEED+ score showed higher discriminatory ability and a larger range of predicted probabilities. A slight improvement in discrimination and calibration was also observed when the model was applied to patients with an MI on antiplatelet drugs, supporting the robustness of the model extension.

Although the associations between anaemia, renal failure, peptic ulcer, cancer and major bleeding have been established previously,^8,9,24 the S₂TOP-BLEED+ score is the first to incorporate these characteristics in a model for bleeding among stroke patients. Other scores like REACH⁵ and intracranial-B₂LEED₃S⁶ could not investigate these factors, due to a lack of measurement in the derivation cohorts, and the likelihood that many patients with these characteristics would be excluded from trial populations and selective registries. However, studies in other areas have shown the importance of these variables in prediction of major bleeding, both in patients with atrial fibrillation^25–27 and in patients with acute coronary syndrome.^12,28 The importance of the new variables is also reflected by their relatively large weights in the S₂TOP-BLEED+ score.

The primary goal of updating S₂TOP-BLEED was to improve the discriminatory ability of the model, aiming to better separate patients with and without a major bleed. Although the increment in c-statistic is small, previous studies have shown that the c-statistic is not very sensitive to addition of new predictors, and possible improvements are highly dependent on the strength of the baseline model.²⁹ Relying solely on the c-statistic for the assessment of added value of predictors is therefore not recommended, as it may lead to exclusion of risk factors that do have a relevant impact on risk stratification in clinical practice.³⁰ Reclassification measures such as the NRI have been proposed as alternatives to assess the incremental value of new predictors and aim to assess whether addition of new predictors actually leads to a change in clinical practice. Calculation of both traditional performance measures (c-statistic, calibration plots) and newer reclassification measures is currently recommended to assess added value of new predictors, as these measures provide complementary information.²⁹ Ultimately, the best way to assess robustness of a model extension is by external validation in an independent population.³¹

The absolute risk of major bleeding was higher among patients with an MI in OXVASC, particularly in the early phase. This is likely explained by a higher incidence of procedure-related bleedings and more frequent prescription of dual antiplatelet treatment (66% among MI patients vs. 26% in TIA/stroke patients in the present study). As a consequence, there was some underestimation of bleeding risk in the MI cohort by both models, which was more pronounced for the original score. Although validation in patients with an MI provides some insight in the robustness of the model extension, a validation study among patients with a TIA or ischaemic stroke is still required.

Although prediction of bleeding may be slightly improved with the updated score, a trade off should be made between increasing complexity and improved performance. The updated score may be less suitable for use at the bedside, due to the larger number of predictors and complexity of the weights assigned to each factor. However, given the increasing use of electronic patient records, calculations could be integrated in health records and performed automatically. Moreover, the new variables are likely to be easily available in all patients with a TIA or stroke and do not require additional imaging or biomarker assessment. An important advantage of the updated model is that it will likely provide better predictions for patients who do not fit in a clinical trial profile, but for whom decisions need to be made in clinical practice. A further advantage is the improved prediction of upper-GI bleeding, which is substantially preventable by co-prescription of proton-pump inhibitor (PPI) drugs, such that the score might be used to target the use of PPI drugs or other similar interventions in high-risk patients.

Strengths of our study include the population-based nature of the cohort, thereby representing the entire range of patients with TIA or stroke on antiplatelet drugs and the validation of the updated model in a separate cohort. A limitation of our study is the relatively small number of bleeding events for model updating and subsequent validation. However, we tried to account for this by using state-of-the-art statistical methods. Another limitation is the fact that we could not validate the model in patients with a TIA or ischaemic stroke, for which the model is intended to be used.

In conclusion, we propose the S₂TOP-BLEED+ score as a refinement to the original S₂TOP-BLEED score, aiming to predict bleeding after a TIA or non-cardioembolic ischaemic stroke. Addition of cancer, peptic ulcer, anaemia and renal failure improves discriminatory performance and increases the range of predicted risks. An external validation of the S₂TOP-BLEED+ score among TIA/ischaemic stroke patients is required to confirm its value.

Supplemental Material

ESO898064 Supplemental material - Supplemental material for Refining prediction of major bleeding on antiplatelet treatment after transient ischaemic attack or ischaemic stroke

Supplemental material, ESO898064 Supplemental material for Refining prediction of major bleeding on antiplatelet treatment after transient ischaemic attack or ischaemic stroke by Nina A Hilkens, Linxin Li, Peter M Rothwell, Ale Algra and Jacoba P Greving in European Stroke Journal

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Oxford Vascular Study is funded by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre, Wellcome Trust, Wolfson Foundation and British Heart Foundation. PMR is in receipt of a NIHR Senior Investigator award. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. JPG and NAH are supported by a grant from the Dutch Heart Foundation (grant number 2013T128). JPG is also supported by a VENI grant from the Netherlands Organization for Health Research and Development (ZonMw), grant number 916.11.129.

Ethical approval

OXVASC has been approved by the local ethics committee.

Informed consent

Patients gave written informed consent or assent was obtained from relatives for patients who were unable to provide consent.

Guarantor

JPG.

Contributorship

PMR and JPG conceived the study. LL was involved in data collection. NAH performed statistical analysis and wrote the first draft of the manuscript. All authors interpreted the results, reviewed and edited the manuscript and approved the final version of the manuscript.

Acknowledgements

The authors are grateful to all the staff in the general practices that collaborated in OXVASC: Abingdon Surgery, Stert St, Abingdon; Malthouse Surgery, Abingdon; Marcham Road Family Health Centre, Abingdon; The Health Centre, Berinsfield; Key Medical Practice; Kidlington; 19 Beaumont St, Oxford; East Oxford Health Centre, Oxford; Church Street Practice, Wantage. The authors also acknowledge the use of the facilities of the Acute Vascular Imaging Centre, Oxford.

ORCID iDs

Linxin Li

Peter M Rothwell

Jacoba P Greving

Supplemental material

Supplemental material for this article is available online.

References

Baigent

Blackwell

Collins

, et al. Aspirin in the primary and secondary prevention of vascular disease: collaborative meta-analysis of individual participant data from randomised trials. Lancet 2009; 373: 1849–1860.

Geraghty

Mehta

, et al. Age-specific risks, severity, time course, and outcome of bleeding on long-term antiplatelet treatment after vascular events: a population-based cohort study. Lancet 2017; 39: 490–499.

Alberts

Bhatt

Smith

Jr , et al. Risk factors and outcomes for patients with vascular disease and serious bleeding events. Heart 2011; 97: 1507–1512.

Hilkens

Algra

Diener

, et al. Predicting major bleeding in patients with noncardioembolic stroke on antiplatelets: S2TOP-BLEED. Neurology 2017; 89: 936–943.

Ducrocq

Wallace

Baron

, et al. Risk score to predict serious bleeding in stable outpatients with or at risk of atherothrombosis. Eur Heart J 2010; 31: 1257–1265.

Amarenco

Sissani

Labreuche

, et al. The intracranial-B2LEED3S score and the risk of intracranial hemorrhage in ischemic stroke patients under antiplatelet treatment. Cerebrovasc Dis 2017; 43: 145–151.

Laine

Yang

Chang

, et al. Trends for incidence of hospitalization and death due to GI complications in the united states from 2001 to 2009. Am J Gastroenterol 2012; 107: 1190–1195.

Valkhoff

Sturkenboom

Kuipers

EJ.

Risk factors for gastrointestinal bleeding associated with low-dose aspirin. Best Pract Res Clin Gastroenterol 2012; 26: 125–140.

Sorensen

Hansen

Abildstrom

, et al. Risk of bleeding in patients with acute myocardial infarction treated with different combinations of aspirin, clopidogrel, and vitamin K antagonists in Denmark: a retrospective analysis of nationwide registry data. Lancet 2009; 374: 1967–1974.

10.

Kamphuisen

Beyer-Westendorf

Bleeding complications during anticoagulant treatment in patients with cancer. Thromb Res 2014; 133: S49–S55.

11.

Cuschieri

Drawz

Falck-Ytter

, et al. Risk factors for acute gastrointestinal bleeding following myocardial infarction in veteran patients who are prescribed clopidogrel. J Dig Dis 2014; 15: 195–201.

12.

Barra

Providencia

Caetano

, et al. BLEED-myocardial infarction score: predicting mid-term post-discharge bleeding events. World J Cardiol 2013; 5: 196–206.

13.

Ocak

Rookmaaker

Algra

, et al. Chronic kidney disease and bleeding risk in patients at high cardiovascular risk: a cohort study. J Thromb Haemost 2018; 16: 65–73.

14.

Rothwell

Coull

Silver

, et al. Population-based study of event-rate, incidence, case fatality, and mortality for all acute vascular events in all arterial territories (Oxford vascular study). Lancet 2005; 366: 1773–1783.

15.

Yusuf

Zhao

Mehta

, et al. Effects of clopidogrel in addition to aspirin in patients with acute coronary syndromes without ST-segment elevation. N Engl J Med 2001; 345: 494–502.

16.

Hilkens

Algra

Greving

JP.

Prediction models for intracranial hemorrhage or major bleeding in patients on antiplatelet therapy: a systematic review and external validation study. J Thromb Haemost 2016; 14: 167–174.

17.

Nieboer

Vergouwe

Ankerst

, et al. Improving prediction models with new markers: a comparison of updating strategies. BMC Med Res Methodol 2016; 16: 128.

18.

Tibshirani

The lasso method for variable selection in the cox model. Stat Med 1997; 16: 385–395.

19.

Pavlou

Ambler

Seaman

, et al. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Stat Med 2016; 35: 1159–1177.

20.

Pavlou

Ambler

Seaman

, et al. How to develop a more accurate risk prediction model when there are few events. BMJ 2015; 351: h3868.

21.

Pencina

D’Agostino

Jr , et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statist Med 2008; 27: 157–172.

22.

Steyerberg

Pencina

MJ.

Reclassification calculations for persons with incomplete follow-up. Ann Intern Med 2010; 152: 195–196.

23.

Collins

Reitsma

Altman

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). Ann Intern Med 2015; 162: 735–736.

24.

de Groot

Hagenaars

Smeets

, et al. Primary non-variceal upper gastrointestinal bleeding in NSAID and low-dose aspirin users: development and validation of risk scores for either medication in two large Dutch cohorts. J Gastroenterol 2014; 49: 245–253.

25.

Gage

Yan

Milligan

, et al. Clinical classification schemes for predicting hemorrhage: results from the national registry of atrial fibrillation (NRAF). Am Heart J 2006; 151: 713–719.

26.

Pisters

Lane

Nieuwlaat

, et al. A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro heart survey. Chest 2010; 13: 1093–1100.

27.

Hijazi

Oldgren

Lindback

, et al. The novel biomarker-based ABC (age, biomarkers, clinical history)-bleeding risk score for patients with atrial fibrillation: a derivation and validation study. Lancet 2016; 387: 2302–2311.

28.

Costa

van Klaveren

James

, et al. Derivation and validation of the predicting bleeding complications in patients undergoing stent implantation and subsequent dual antiplatelet therapy (PRECISE-DAPT) score: a pooled analysis of individual-patient datasets from clinical trials. Lancet 2017; 389: 1025–1034.

29.

Pencina

D'Agostino

Pencina

, et al. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol 2012; 176: 473–481.

30.

Cook

NR.

Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007; 115: 928–935.

31.

Steyerberg

Vickers

Cook

, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB