Validation and comparison of two stroke prognostic models for in hospital,30-day and 90-day mortality

Abstract

Introduction

We aimed to validate and compare two clinical prognostic models for mortality which include the National Institutes of Health Stroke Scale (NIHSS); the Age and NIHSS Score (ANS) and case mix model (CMM) of the Sentinel Stroke National Audit Program (SSNAP). The NIHSS on admission was also tested as a prognostic score.

Patients and methods

Prospectively collected data from the SSNAP register for a cohort of patients (ischaemic and haemorrhagic stroke) admitted over 1 year to Gloucestershire Royal Hospital, England were accessed. The ANS and CMM were calculated and tested for in hospital, 30-day and 90-day mortality using calibration plots with Hosmer–Lemeshow tests, receiver operating characteristics curves and other measures of prognostic accuracy.

Results

Of 848 patients, 110 (12.9%) died in hospital, 112 (13.2%) at 30 days and 164 (19.2%) at 90 days. Calibration for all three scores was good, although Hosmer–Lemeshow test p values were <0.05 with the NIHSS alone for in hospital and 30-day deaths, suggesting deviation from good fit. The c-statistics for in hospital, 30-day and 90-day mortality were ANS (0.783, 0.782, 0.779) and CMM (0.783, 0.774, 0.758), respectively. The NIHSS alone showed fair discrimination but performed less well. A NIHSS score ≥6 was associated with significant mortality (p < 0.0001) in comparison to a score <6.

Conclusion

A simple prognostic model containing age and admission NIHSS only, performed as well as a more complex score at predicting in hospital, 30-day and 90-day mortality. Admission NIHSS recording should be encouraged for stroke registries.

Keywords

Stroke prognostic score prognosis mortality

Introduction

Stroke is the third highest cause of death worldwide with a case fatality of 17–30% at 30 days in population based studies.^1,2 Mortality in hospital and at 30 and 90 days are widely used outcomes in research and assessment of the quality of routine care.³ Several prognostic models for mortality have been designed for clinical decision making, to aid the design and analysis of randomised trials and to allow for case mix adjusted comparisons between different cohorts of patients.^4–6 Prognostic models should ideally be simple and based on easily collected variables.^4,5 Some prognostic models have been criticised for lack of external validation and limited generalisability.^4,5 Independent validation, ideally in several cohorts, is considered as an important step in the development of a truly useful model.^5,6

The National Institutes of Health Stroke Scale (NIHSS)⁷ is a component of some prognostic scores including the Get with the Guidelines–Stroke (GWTG-Stroke) risk model,⁸ the Age and NIHSS prognostic score – model 2 (ANS),⁹ Sentinel Stroke National Audit Program (SSNAP) case mix model – model A (CMM)¹⁰ and the THRIVE score¹¹ derived from patients undergoing endovascular treatment. The ANS comprises age and NIHSS score within 6 h of onset and predicts 100 day mortality in ischaemic stroke patients admitted to acute stroke units in Germany.⁹ It has been externally validated in German hospitals and in a dataset from clinical trials.^12,13 It has not been tested in a cohort including primary intracerebral bleeds or for in hospital or 30-day mortality. Case mix models for standardised reporting of 30 day stroke mortality have been derived from data from SSNAP, the national register of acute stroke in England and Wales.¹⁰ The CMM contains age, NIHSS, presence of AF and stroke type (ischaemic or primary haemorrhage).¹⁰ The model has been externally, but not independently, validated using a cohort of stroke patients in London.¹⁰ It has not been validated for 90-day mortality. The ANS and CMM have not been compared for prognostic accuracy on the same dataset before. The NIHSS score alone has also been found to be useful in predicting in hospital deaths, 30-day mortality and poor outcomes at 90 days (either death or nursing home residence).^8,14,15

This study aimed to independently validate the ANS and the CMM for the prediction of in hospital, 30-day and 90-day mortality in a cohort of patients (comprising ischaemic and haemorrhagic strokes) with prospectively collected data. We also wished to compare the calibration, discrimination and other measures of prognostic accuracy of the two scores and NIHSS scores alone to determine their usefulness and identify the simplest model for our dataset.

Patients and methods

Study population and setting

All patients admitted to the stroke units of Gloucestershire Royal Hospital (GRH), Gloucester, England from 25 July 2015 to 25 July 2016 with a confirmed diagnosis of ischaemic stroke or primary intracerebral haemorrhage were included in the study. This 1 year time frame was chosen as it was the period after a major service reorganisation at our centre and we were confident of the completeness of NIHSS scoring in our dataset for this period. This time frame was also chosen to avoid the inclusion of any patients who may have been included in the derivation sample for the SSNAP CMM. Patients with a diagnosis of transient ischaemic attack (TIA) and stroke mimics were excluded.

Stroke was defined as ‘a clinically defined syndrome of rapidly developing symptoms or signs of focal loss of cerebral function with no apparent cause other than that of vascular origin with symptoms lasting more than 24 h or leading to death’.¹⁶ All patients had imaging to aid the diagnosis and confirm the nature of the stroke. The stroke units at GRH consist of a 15 bed acute unit and a 44 bed rehabilitation stroke unit where patients are transferred from the acute unit typically within 72 h to 7 days of admission. The acute unit has facilities for continuous physiological monitoring and enhanced senior presence. The rehabilitation stroke unit is able to treat patients for up to several weeks depending on clinical need. The catchment area for GRH has a population of 560,000 of predominantly Caucasian origin. All stroke services for the county of Gloucestershire are centralised at GRH and includes a 24/7 thrombolysis service. Patients for thrombectomy are transferred to a tertiary centre, currently within office hours.

Prognostic models studied

The two prognostic models studied were the Age and NIHSS score prognostic model II (ANS)⁹ and the case mix model – model A (CMM).¹⁰

The ANS was calculated from age in years and NIHSS score on admission using the intercept and coefficients of the derivation logistic regression model.⁹ Using the intercept −7.040 and regression coefficients 0.049 and 0.155, $ANS = Age * 0.049 + NIHSS * 0.155 - 7.040$ .

The CMM was derived from age (<60, 60–69, 70–79, 80–89, and ≥90 years), NIHSS on admission, presence of atrial fibrillation on admission, and stroke type (ischaemic or primary intracerebral haemorrhage).¹⁰ Using the intercept and regression coefficients of the original regression model, the CMM was calculated as follows

CMM = regressioncoefficientforagecategory [if < 60 = 0, if (60 - 69) = 0.624, if (70 - 79) = 1.033, if (80 - 89) = 1.488, if \geq 90 = 1.781] + [NIHSS * 0.137] + [0.425 ifAFpresent] + [0.870 ifprimary int arcerebralhaemorrhage] - 5.250

Outcome events

The outcome events of interest were all cause death in hospital and all cause death within 30 and 90 days.

Data collection

Data were extracted from the SSNAP dataset¹⁷ for GRH. This clinical audit programme collects a minimum dataset for stroke patients in England, Wales and Northern Ireland. Data are entered prospectively by stroke teams on all patients admitted to hospital with confirmed stroke using a secure web portal with prompts for data completeness and checks for accuracy.¹⁷ Variables include demographic details, stroke phenotype, NIHSS score on admission (assessed by NIHSS certified clinicians as soon as possible after arrival), comorbidities, several processes of care, clinical outcomes including modified Rankin Scores (mRS)¹⁸ and deaths by discharge. To ascertain death at 30 and 90 days (if after hospital discharge), the hospital ‘Infoflex’ database (Chamelion Information Management Services) was accessed. This database records details of all admissions to hospital, discharge summaries and death. The database directly records deaths in hospital and has robust links with general practices, county registration officers and funeral directors for capturing deaths occurring in the community. All patient identifiers were removed from the dataset before analysis. The study was exempt from ethical review and individual patient consent and had authorisation from the Health Research Authority, England.

Statistical analysis

Analysis included the calibration (observed versus predicted event rates for groups of patients) of the two prognostic scores for in hospital, 30-day and 90-day mortality using calibration plots along with the Hosmer–Lemeshow test of goodness of fit.¹⁹ Discrimination (how well they distinguished between patients who did or did not have the outcome event) of the two scores for in hospital, 30-day and 90-day mortality were assessed by receiver operating characteristics (ROC) curves with area under the curve (AUC or c-statistic).¹⁹ The two c-statistics were compared for statistical significance using the non-parametric (two-sided) DeLong test.^20,21 A sample size calculation for a single prognostic test using the R package pROC was performed.^21,22 We determined optimal cut points for the prognostic scores using criteria based on sensitivity and specificity, the Youden Index.²³ Tests of prognostic accuracy including sensitivity and specificity were calculated.²⁴ As the total duration of follow-up was often more than 90 days, Kaplan–Meier plots were drawn for all deaths at the end of follow up stratified by the cut points and statistical significance assessed by the log rank test. Similar methods were used for the NIHSS score alone and the results compared with the two prognostic scores. The statistical software “R” was used for analysis.²⁵

Results

Between 25 July 2015 and 25 July 2016, 848 patients with confirmed stroke were admitted to the stroke units at GRH. There were no missing variables required for score calculation or outcome for all 848 patients. The sample size was well above our calculated minimum sample of 152 patients (with at least 31 outcomes of interest) obtained using a power of 95%, a significance level of 0.05, a conjectured AUC of 0.7 and a proportion of outcomes (deaths) to living patients of 20%.^21,22 The baseline characteristics of the cohort are shown in Table 1. Median onset to hospital arrival time was 7.2 h, interquartile range (IQR) 1.7–13.9 h. Median length of stay in hospital for all patients, alive or dead at discharge, was 8 days (IQR; 3–23 days). Follow up was for a median period of 223 days from stroke onset (IQR, 117–337 days). The proportion of patients dying in hospital was 104/809 (12.9%). At 30 days, 112 of 809 (13.8%) were dead and at 90 days 164 of 809, that is 20.3% had died (online Supplementary Figure 1).

Table 1.

Baseline characteristics of the cohort (n = 848).

Age – median (IQR)	78 (68–85)
Sex (males), n (%)	431 (50.8)
Type of stroke
Ischaemic, n (%)	729 (86.0)
Intracerebral haemorrhage, n (%)	119 (14.0)
Stroke onset to hospital arrival (hours) – median (IQR)	7.2 (1.7–13.9)
Number of ischaemic strokes thrombolysed, n (%)	79 (10.8%)
Pre-stroke mRS – median (IQR)	0 (0–2)
NIHSS on admission – median (IQR)	4 (2–9)
Atrial fibrillation, n (%)	227 (26.8)
Hypertension, n (%)	434 (51.2)
Diabetes, n (%)	171 (20.2)
Previous stroke or TIA, n (%)	215 (25.4)
Cardiac failure, n (%)	76 (9.0)

IQR: interquartile range; mRS: modified Rankin Scale; NIHSS: National Institutes of Health Stroke Scale; TIA: transient ischaemic attack.

Figure 1 shows calibration plots (predicted vs. observed deaths) for the ANS, CMM and NIHSS scores for in hospital, 30-day and 90-day mortality. Inspection of the plots show broadly similar calibration for the ANS and CMM scores with non-significant p values derived from the Hosmer–Lemeshow test (suggesting good calibration). The NIHSS score performed less well (p < 0.05) for in hospital and 30-day deaths, suggesting deviation from good fit for these time periods. The p value for the NIHSS score for deaths at 90 days, however, was non-significant. The p values for death in hospital were; ANS (p = 0.1454), CMM (p = 0.1552) and NIHSS (p = 0.0034). 30-day mortality; ANS (p = 0.27), CMM (p = 0.267) and NIHSS (p = 0.0055). 90-day mortality; ANS (p = 0.4842), CMM (p = 0.5488) and NIHSS (p = 0.186).

Figure 1.

Calibration plots for the ANS, CMM and NIHSS scores for in hospital, 30-day and 90-day mortality.

ROC curves and the AUC (c-statistics) with 95% CI are shown for the three scores in Figure 2. The best c-statistics were obtained for the ANS, 0.78 for in hospital, 30-day and 90-day deaths although the confidence intervals overlapped for the three scores. Significance tests for the difference in c-statistics using the DeLong²⁰ method were conducted and the results shown in Table 2. These results suggest that there was a consistently significant difference between the ANS and NIHSS score for in hospital, 30-day and 90-day mortality and a significant difference between the CMM and NIHSS only for 90-day deaths. As the ANS has not been validated in intracerebral haemorrhage before, the performance of the three scores was tested separately in infarcts and haemorrhages (online Supplementary Table 1). Although the number of haemorrhages in our sample was small (n = 119), there appeared to be no major difference in the c-statistics of the ANS and CMM while the c-statistic of the NIHSS was lower (0.698) for patients with haemorrhages dying in hospital.

Figure 2.

Receiver operating characteristics curves for the ANS, CMM and NIHSS scores for in hospital, 30-day and 90-day mortality.

Table 2.

Comparison of c-statistics and significance test results for the three scores.

Prognostic score comparison	In hospital mortality – c-statistics and p values^a	30-day mortality – c-statistics and p values^a	90-day mortality – c-statistics and p values^a
ANS vs. CMM	0.783 vs. 0.783 (p = 0.7099)	0.782 vs. 0.774 (p = 0.628)	0.779 vs. 0.758 (p = 0.0925)
ANS vs. NIHSS	0.783 vs. 0.744 (p = 0.02304)	0.782 vs. 0.747 (p = 0.0187)	0.779 vs. 0.725 (p = 0.001)
CMM vs. NIHSS	0.783 vs. 0.744 (p = 0.1231)	0.774 vs. 0.747 (p = 0.1313)	0.758 vs. 0.725 (p = 0.0286)

ANS: age and NIHSS score; CMM: case mix model; NIHSS: National Institutes of Health Stroke Scale.

p Values by the DeLong method.

Optimal cut points for the three scores for our dataset were derived using the Youden Index method.²³ The cut points were −2.321 for the ANS, −2.94 for the CMM and 6 for the NIHSS. These cut points were used to calculate the actual numbers of true positive and negative predictions made using the three scores (Table 3) along with other measures of prognostic accuracy such as the sensitivities and specificities. The best sensitivity was 82.7% for the ANS in predicting in hospital mortality and the NIHSS alone demonstrated a sensitivity of 80.9% for in hospital mortality. Specificities, however, were much lower in the range of about 60–70%.

Table 3.

Table of prognostic significance tests using the ANS, CMM and NIHSS scores.

	ANS	CMM	NIHSS
Deaths in hospital
True positive	91	84	89
False positive	278	213	314
True negative	460	525	425
False negative	19	26	21
Sensitivity	82.7% (74.3–89.3%)	76.4% (67.3–83.9%)	80.9% (72.3–87.8%)
Specificity	62.3% (58.7–65.8%)	71.1% (67.7–74.4%)	57.5% (53.8–61.1%)
Positive predictive value	24.7% (22.0–36.2%)	28.3% (25.1–39.0%)	22.1% (19.6–32.4%)
Negative predictive value	96.0% (93.6–96.6%)	95.3% (92.8–96.0%)	95.3% (92.6–95.9%)
Positive likelihood ratio	2.2 (1.9–2.5)	2.6 (2.2–3.1)	1.9 (1.7–2.2)
Negative likelihood ratio	0.3 (0.1–0.4)	0.3(0.2–0.5)	0.3 (0.2–0.5)
30-day mortality
True positive	91	88	89
False positive	278	245	314
True negative	458	491	422
False negative	21	24	23
Sensitivity	81.3% (72.8–88.0%)	78.6% (69.8–85.8%)	79.5% (70.8–86.5%)
Specificity	62.4% (58.6–66.0%)	66.7% (63.2–70.1%)	57.3% (53.7–61.0%)
Positive predictive value	24.7% (22.0–35.7%)	26.4% (23.5–37.1%)	22.1% (19.6–32.0%)
Negative predictive value	95.6% (93.1–96.2%)	95.3% (92.8–96.0%)	94.8% (92.0–95.5%)
Positive likelihood ratio	2.2 (2.0–2.4)	2.4 (2.1–2.7)	1.9 (1.6–2.1)
Negative likelihood ratio	0.3 (0.2–0.4)	0.3 (0.2–0.5)	0.4 (0.3–0.5)
90-day mortality
True positive	127	119	112
False positive	242	214	238
True negative	443	471	447
False negative	36	44	51
Sensitivity	77.9% (70.8–84.0%)	73.0% (65.5–79.7%)	68.7% (61.0-75.7%)
Specificity	64.7% (61.0–68.3%)	68.8% (65.1–72.2%)	65.3% (61.6–68.8%)
Positive predictive value	34.4% (31.0–43.9%)	35.7% (32.1–44.6%)	32.0% (28.6–40.1%)
Negative predictive value	92.5% (89.4–93.5%)	91.5% (88.3–92.7%)	89.8% (86.2–91.2%)
Positive likelihood ratio	2.2 (1.9–2.5)	2.3 (2.0–2.7)	1.9 (1.7–2.3)
Negative likelihood ratio	0.3 (0.2–0.4)	0.4 (0.3–0.5)	0.5 (0.4–0.6)

ANS: age and NIHSS score; CMM: case mix model; NIHSS: National Institutes of Health Stroke Scale. Confidence intervals (95%) are shown where available.

For all patients discharged alive, follow-up was for more than 90 days. Kaplan–Meier curves were obtained for all deaths by the end of the study stratified by the cut points obtained for the ANS, CMM and NIHSS score as high risk and low risk (online Supplementary Figures 2–4). All the scores demonstrated highly significant p values (<0.0001) by the log rank test. A NIHSS score of ≥6 was associated with a significant higher mortality in comparison to scores <6.

Discussion

In this study, we have independently validated two clinical prognostic scores incorporating the NIHSS, the ANS and CMM scores and compared their performance on the same cohort of patients. The results of our study suggest that both scores had fair discriminatory performance (AUC between 0.7 and 0.8) but the simplest risk score (ANS) based on just two variables, age and NIHSS on admission, performed as well as the CMM which contains additional factors. In fact, the c-statistics of the ANS were marginally, but not significantly, better than the c-statistic of the CMM for in hospital, 30-day as well as 90-day mortality. The NIHSS score on admission alone provided AUCs which would be considered to be fair, although significantly lower than the ANS score.

Both the scores were used slightly differently in this study than originally intended; the ANS was not designed to predict in hospital or 30-day mortality and had never been intended for cohorts containing haemorrhagic strokes.⁹ It was designed in acute stroke units in Germany and its external validity is uncertain.^9,13 The NIHSS scores had been collected within 6 h of onset in the original ANS derivation study,⁹ but were calculated much later in our study as the median time to admission in our cohort was 7.2 h. The derivation cohort of the CMM had contained both ischaemic and haemorrhagic strokes but the score had been designed and validated only for 30-day mortality.¹⁰

Several prognostic scores have been designed to predict stroke mortality and functional outcomes. Most include age and some measure of stroke severity. Examples include the GTWG-Stroke score,⁸ iScore,²⁶ the six simple variable score,²⁷ SPAN-100,²⁸ ASTRAL,²⁹ THRIVE,¹¹ DRAGON³⁰ and TPI³¹ in addition to the two scores validated in this study. Many include vascular risk factors and co-morbid conditions while others have added imaging to the scores.^11,26,30 The most useful prognostic models are usually simple and derived from routinely collected clinical data. Recording of the NIHSS on admission is already mandatory for patients undergoing thrombolysis and mechanical thrombectomy. Many registries, including SSNAP, encourage the routine collection of the NIHSS for all stroke patients.¹⁷

The c-statistic of the ANS for 90-day mortality in our study (0.789) compares favourably with the value of 0.706 in an earlier validation study using a clinical trial cohort.¹³ This was in spite of our different setting, inclusion of patients with intracerebral haemorrhage and later NIHSS scores ascertainment. There are no other validation studies that have studied in hospital and 30-day mortality with this score but our c-statistics for the ANS, 0.783 and 0.782 clearly suggest fair discrimination for this score. The CMM c-statistic of 0.774 for 30-day mortality compares with values of 0.864 and 0.866 for internal and external validation in the original derivation study.¹⁰ The original CMM training dataset was derived from hospitals in England and Wales which are broadly similar to our centre.¹⁰ Their external validation dataset was drawn from a population based registry in London.¹⁰ The CMM has not been tested for in hospital or 90-day mortality before and our c-statistics of 0.783 and 0.758, respectively again suggest fair discrimination.

The in hospital mortality c-statistic of the NIHSS score alone was 0.744 in this study and was 0.83 in the original GWTG-stroke risk model study.⁸ Our c-statistic of the NIHSS alone for 30-day mortality was 0.747 compares the c-statistic of 0.82 in data from Medicare beneficiaries.¹⁴ The NIHSS optimal cut point of 6 obtained in our study is in keeping cut points of NIHSS severity that discriminate between low and significantly higher mortality derived in previous studies such as the dataset from Medicare beneficiaries¹⁴ and the GWTG-Stroke cohort where patients who survived had a median NIHSS of 4.⁸ The c-statistics obtained in our validation as well as the sensitivities and specificities of the two scores suggest that they may not be accurate enough to predict individual patient outcomes but should be very suitable for the purpose of case mix adjustments in trials and comparison of outcomes for different service providers.

The strengths of this study were a reflection of the SSNAP data collection method; all data were prospectively collected via a secure web portal within inbuilt prompts for validation and completion. Consecutive patients admitted to our centre were entered into the database with no selection bias. Diagnosis was confirmed by experienced stroke physicians with the help of imaging and did not rely on coding. The NIHSS scores were ascertained by NIHSS certified clinicians as soon as possible after admission. There were no missing NIHSS scores or other variables used to calculate the two prognostic scores. No patients were lost to follow up and outcome data were complete. We are confident that we were able to ascertain all deaths in hospital (which are collected prospectively) and out of hospital because of the robust links of our ‘Infoflex’ database with various sources of death ascertainment.

The weaknesses of this study include retrospective calculation of the risk scores and ascertainment of 30-day and 90-day mortality. The sample size was more than adequate for ROC curves analysis for a single prognostic score. However, for the comparison of two or more scores with minor difference in c-statistics a larger sample would almost certainly be needed. Our sample size of nearly 850 patients admitted over 1 year is typical of larger hospitals in the UK and suggests that any of the two scores could be used for case mix comparisons for outcomes. Our median age of 78 suggests an older population in keeping with the demographics of our region. The median NIHSS of 4 in our sample is consistent with data from SSNAP for other centres in England and Wales and our 30-day mortality of 13.8% compares with a crude mortality of 13.6% for all hospitals in England and Wales in 2015–2016.¹⁷ Our centre is slightly unusual in having a large number of rehabilitation stroke beds on site where patients are transferred from the acute stroke unit and can spend several weeks. Many stroke services transfer patients from acute stroke units to external rehabilitation units. Given these caveats, we feel that our results are generalisable to most other stroke centres in similar healthcare settings.

Conclusion

A simple prognostic score comprising age and NIHSS score on admission performed just as well as a more complex score in predicting in hospital, 30-day and 90-day mortality in a cohort of ischaemic and haemorrhagic strokes. In keeping with previous data, the NIHSS score alone showed fair predictive value and a NIHSS ≥6 was associated with significantly higher mortality. Routine recording of NIHSS for all stroke patients should be considered for all stroke registries.

Footnotes

Declaration of Conflicting Interests

The author(s) declared following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DD is the principal investigator of a Bayer sponsored randomised controlled trial and Boehringer Ingelheim sponsored AF registry. The others have no conflicts of interest to declare.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: SSNAP is funded by the Healthcare Quality Information Partnership (HQIP) on behalf of NHS England. There was no specific funding for this study.

Ethical approval

This study was exempt from ethical approval and individual consent. The Health Research Authority, England approved the study (IRAS ID 216501).

Guarantor

DD.

Contributorship

DD conceived the study and DD, AC and EB reviewed the literature. Data were accessed by DD, AC and EB. DD was involved in the research proposal, gaining Health Research Authority approval and data analysis. DD wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

References

Thorvaldsen

Asplund

Kuulasmaa

et al.

Stroke incidence, case fatality, and mortality in the WHO MONICA project. World Health Organization Monitoring Trends and Determinants in Cardiovascular Disease. Stroke 1995; 26: 361–367.

Feigin

Lawes

Bennett

et al.

Worldwide stroke incidence and early case fatality reported in 56 population-based studies: a systematic review. Lancet Neurol 2009; 8: 355–369.

Katzan

Spertus

Bettger

et al.

Risk adjustment of ischaemic stroke outcomes for comparing hospital performance a statement for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2014; 45: 918–944.

Counsell

Dennis

. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 2001; 12: 159–170.

Steyerberg

Moons

KGM

van der Windt

et al.

Prognosis research strategy (PROGRESS) series 3: prognostic models. PLoS Med 2013; 10: e1001381.

Moons

KGM

Royston

Vergouwe

et al.

Research methods & reporting. Prognosis and prognostic research: what, why, and how?

BMJ 2009; 338: 1317–1320.

Brott

Adams

Olinger

et al.

Measurements of acute cerebral infarction: a clinical examination scale. Stroke 1989; 20: 864–870.

Smith

Shobha

Dai

et al.

Risk score for in-hospital ischaemic stroke mortality derived and validated within the get with the guidelines – stroke program clinical perspective. Circulation 2010; 122: 1496–1504.

Weimar

König

Kraywinkel

et al.

Age and National Institutes of Health Stroke Scale Score within 6 hours after onset are accurate predictors of outcome after cerebral ischemia: development and external validation of prognostic models. Stroke 2004; 35: 158–162.

10.

Bray

Campbell

Cloud

et al.

Derivation and external validation of a case mix model for the standardized reporting of 30-day stroke mortality rates. Stroke 2014; 45: 3374–3380.

11.

Flint

Cullen

Faigeles

. Predicting long-term outcome after endovascular stroke treatment: the totaled health risks in vascular events score. Am J Neuroradiol 2010; 31: 1192–1196.

12.

German Stroke Study Collaboration GSS. Predicting outcome after acute ischemic stroke: an external validation of prognostic models. Neurology 2004; 62: 581–585.

13.

König

Ziegler

Bluhmki

et al.

Predicting long-term outcome after acute ischaemic stroke: a simple index works in patients from controlled clinical trials. Stroke 2008; 39: 1821–1826.

14.

Fonarow

Saver

Smith

et al.

Relationship of National Institutes of Health Stroke Scale to 30-day mortality in medicare beneficiaries with acute ischemic stroke. J Am Heart Assoc 2012; 1: 42–50.

15.

Muir

Weir

Murray

et al.

Comparison of neurological scales and scoring systems for acute stroke prognosis. Stroke 1996; 27: 1817–1820.

16.

Warlow

. Epidemiology of stroke. Lancet 1998; 352: S1–S4.

17.

Sentinel Stroke National Audit Programme. London: Royal College of Physicians. http://www.rcplondon.ac.uk/projects/sentinel-stroke-nationalaudit (accessed 20 February 2017).

18.

Rankin

. Cerebral vascular accidents in patients over the age of 60. II. Prognosis. Scott Med J 1957; 2: 200–215.

19.

Altman

Vergouwe

Royston

et al.

Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: 1432–1435.

20.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

21.

Robin

Turck

Hainard

et al.

pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011; 12: 77.

22.

Obuchowski

Lieber

Wians

. ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clin Chem 2004; 50: 1118–1125.

23.

Lopex-Raton

Rodriguez-Alvarez

Cadarso-Suárez

et al.

OptimalCutpoints: an R package for selecting optimal cutpoints in diagnostic tests. J Stat Softw 2014; 61: 1–36.

24.

Eusebi

. Diagnostic accuracy measures. Cerebrovasc Dis 2013; 36: 267–272.

25.

R Development Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2014, http://www.r-project.org/ (accessed 4 November 2014).

26.

Saposnik

Kapral

Liu

et al.

IScore: a risk score to predict death early after hospitalization for an acute ischemic stroke. Circulation 2011; 123: 739–749.

27.

Counsell

Dennis

McDowall

et al.

Predicting outcome after acute and subacute stroke: development and validation of new prognostic models. Stroke 2002; 33: 1041–1047.

28.

Saposnik

Guzik

Reeves

et al.

Stroke Prognostication using Age and NIH Stroke Scale: SPAN-100. Neurology 2013; 80: 21–28.

29.

Ntaios

Faouzi

Ferrari

et al.

An integer-based score to predict functional outcome in acute ischemic stroke: the ASTRAL score. Neurology 2012; 78: 1916–1922.

30.

Strbian

Meretoja

Ahlhelm

et al.

Predicting outcome of IV thrombolysis-treated ischemic stroke patients: the DRAGON score. Neurology 2012; 78: 427–432.

31.

Kent

Selker

Ruthazer

et al.

The stroke–thrombolytic predictive instrument. Stroke 2006; 37: 2957–2962.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.23 MB