Background. Comparing prediction models using reclassification within subgroups at intermediate risk is often of clinical interest. Objective. To demonstrate a method for obtaining an unbiased estimate for the Net Reclassification Improvement (NRI) evaluated only on a subset, the clinical NRI. Study Design and Setting. We derived the expected value of the clinical NRI under the null hypothesis using the same principles as the overall NRI. We then conducted a simulation study based on a logistic model with a known predictor and a potential predictor, varying the effects of the known and potential predictors to test the performance of our bias-corrected clinical NRI measure. Finally, data from the Women’s Health Study, a prospective cohort of 24 171 female health professionals, were used as an example of the proposed method. Results. Our bias-corrected estimate is shown to have a mean of zero in the null case under a range of simulated parameters and, unlike the naïve estimate, to be unbiased. We also provide 2 methods for obtaining a variance estimate, both with reasonable type 1 errors. Conclusion. Our proposed method is an improvement over currently used methods of calculating the clinical NRI and is recommended to reduce overly optimistic results.
CookNR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928–35.
2.
HlatkyMAGreenlandPArnettDK. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009;119(17):2408–16.
3.
PencinaMJD’AgostinoRBSD’AgostinoRBJVasanRS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2007;27(2):157–72.
4.
Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) Final report. Circulation. 2002;106(25):3143.
5.
CookNR. Comments on “Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond” by PencinaM. J.., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27(2):191–5.
6.
GreenlandS. The need for reorientation toward cost-effective prediction: comments on “Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond” by PencinaM. J.., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27(2):199–206.
7.
BrautbarABallantyneCMLawsonK. Impact of adding a single allele in the 9p21 locus to traditional risk factors on reclassification of coronary heart disease risk and implications for lipid-modifying therapy in the Atherosclerosis Risk in Communities study. 2009;2(3):279–85.
8.
ChamblessLECummiskeyCPCuiG. Several methods to assess improvement in risk prediction models: extension to survival analysis. Stat Med. 2011;30(1):22–38.
9.
NambiVChamblessLFolsomAR. Carotid intima-media thickness and presence or absence of plaque improves prediction of coronary heart disease risk: the ARIC (Atherosclerosis Risk in Communities) study. J Am Coll Cardiol. 2010;55(15):1600–7.
10.
MealiffeMEStokowskiRPRheesBKPrenticeRLPettingerMHindsDA. Assessment of clinical validity of a breast cancer risk model combining genetic and clinical information. J Natl Cancer Inst. 2010;102(21):1618–27.
11.
MelanderONewton-ChehCAlmgrenP. Novel and conventional biomarkers for prediction of incident cardiovascular events in the community. JAMA. 2009;302(1):49–57.
12.
RipattiSTikkanenEOrho-MelanderM. A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses. Lancet. 2010;376(9750):1393–400.
13.
CookNRPaynterNP. Performance of reclassification statistics in comparing risk prediction models. Biometr J. 2011;53(2):237–58.
14.
McNemarQ. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12(2):153–7.
15.
KrampeAKuhntS. Bowker’s test for symmetry and modifications within the algebraic framework. Comput Stat Data Anal. 2007;51(9):4124–42.
16.
PencinaMJD’AgostinoRBSteyerbergEW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30(1):11–21.
17.
RexrodeKMLeeICookNRHennekensCHBuringJE. Baseline characteristics of participants in the Women’s Health Study. J Women Health Gender Based Med. 2000;9(1):19–27.
18.
RidkerPMCookNRLeeIM. A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women. N Engl J Med. 2005;352(13):1293–304.
19.
RidkerPMBuringJERifaiNCookNR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds risk score. JAMA. 2007;297(6):611–9.
20.
PaukerSGKassirerJP. The threshold approach to clinical decision making. N Engl J Med. 1980;302(20):1109–17.
21.
PepeMSKerrKFLongtonGWangZ. Testing for improvement in prediction model performance. UW Biostatistics Working Paper Series. 2012; Working Paper 379.
22.
HarrellFELeeKLMarkDB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–87.
23.
WhittemoreAS. Evaluating health risk models. Stat Med. 2010;29(23):2438–52.
