The Effect of Disease-prevalence Adjustments on the Accuracy of a Logistic Prediction Model

Abstract

The accuracy of a logistic prediction model is degraded when it is transported to pop ulations with outcome prevalences different from that of the population used to derive the model. The resultant errors can have major clinical implications. Accordingly, the authors developed a logistic prediction model with respect to the noninvasive diagnosis of coronary disease based on 1,824 patients who underwent exercise testing and coronary angiography, varied the prevalence of disease in various "test" populations by random sampling of the original "derivation" population, and determined the accu racy of the logistic prediction model before and after the application of a mathematical algorithm designed to adjust only for these differences in prevalence. The accuracy of each prediction model was quantified in terms of receiver operating characteristic (ROC) curve area (discrimination) and chi-square goodness-of-fit (calibration). As the prevalence of the test population diverged from the prevalence of the derivation pop ulation, discrimination improved (ROC-curve areas increased from 0.82 ± 0.02 to 0.87 ± 0.03; p < 0.05), and calibration deteriorated (chi-square goodness-of-fit statistics increased from 9 to 154; p < 0.05). Following adjustment of the logistic intercept for differences in prevalence, discrimination was unchanged and calibration improved (maximum chi-square goodness-of-fit fell from 154 to 16). When the adjusted algorithm was applied to three geographically remote populations with prevalences that differed from that of the derivation population, calibration improved 87%, while discrimination fell by 1 %. Thus, prevalence differences produce statistically significant and potentially clinically important errors in the accuracy of logistic prediction models. These errors can potentially be mitigated by use of a relatively simple mathematical correction al gorithm. Key words: logistic prediction model; accuracy; prevalence adjustments; model transportability. (Med Decis Making 1996;16:133-142)

Get full access to this article

View all access options for this article.

References

Wasson JH , Sox HC , Neff RK , Goldman L.

Clinical prediction rules: applications and methodologic standards

N Engl J Med. 1985;313:793-9.

O'Conner GT , Plume SK , Olmstead EM, et al.

Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery

Circulation. 1992;85:2110-8.

Pollock SG , Abbott RD , Boucher CA Beller GA , Kaul S.

Independent and incremental prognostic value of tests performed in hierarchical order to evaluate patients with suspected coronary artery disease

Circulation. 1992;85:237-48.

Mark DB , Lam LC , Lee KL, et al.

Identification of patients with coronary disease at high risk for loss of employment

Circulation. 1992;86:1485-94.

Morise AP , Detrano R. , Bobbio M. , Diamond GA

The development and validation of a logistic regression derived algorithm for estimating the incremental probability of coronary artery disease before and after exercise testing

J Am Coll Cardiol. 1992;20:1187-96.

Pryor DB , Shaw L. , McCants CB, et al.

Value of the history and physical in identifying patients at increased risk for coronary artery disease

Ann Intern Med. 1993;118:81-90.

Pryor DB , Lee KL

Methods for the analysis and assessment of clinical databases: the clinician's perspective

Stat Med. 1991;10:617-28.

Diamond GA

Future imperfect: the limitations of clinical prediction models and the limits of clinical prediction

J Am Coll Cardiol. 1989;14:12A-22A.

Poses RM , Cebul RD , Collins M. , Fager SS

The importance of disease prevalence in transporting clinical prediction rules

Ann Intern Med. 1986;105:586-91.

10.

Wijensinha A. , Begg CB , Funkenstein HH , McNeil BJ

Methodology for the differential diagnosis of a complex data set: a case study using data from routine CT examinations

Med Decis Making. 1993;3:133-54.

11.

Morise AP , Diamond GA , Detrano R. , Bobbio M.

The incremental value of exercise electrocardiography and thallium testing in men and women concerning both the presence and extent of coronary artery disease

Am Heart J. 1995 (in press).

12.

Hanley JA , McNeil BJ

The meaning and use of the area under a receiver operating characteristic (ROC) curve

Diagnost Radiol. 1982;143:29-36.

13.

Hanley JA , McNeil BJ

A method of comparing the areas under receiver operating characteristic curves derived from the same cases

Radiology. 1983 ;148:839-43.

14.

Armitage P. , Berry G. Statistical Methods in Medical Research Oxford, U.K.: Blackwell Scientific Publications, 1987:112-5.

15.

Meyer SL Data Analysis for Scientists and Engineers New York: John Wiley & Sons, 1975 :368-9.

16.

Diamond GA , Denton TA

Risky business: prospective applicability of models (letter)

Circulation. 1993;87:1054-5.

17.

Begg CB , Greenes RA

Assessment of diagnostic tests when disease verification is subject to selection bias

Biometrics. 1983; 39:207-15.

18.

Diamond GA

Affirmative actions: can the discriminant accuracy of a test be determined in the face of selection bias?

Med Decis Making. 1991;11:48-56.

19.

Heckerling PS , Conant RC , Tape TG , Wigton RS

Reproducibility of predictor variables from a validated clinical rule

Med Decis Making. 1992;12:280-5.

20.

McCullagh P. , Nelder JA Generalized Linear Models New York: Chapman and Hall, 1989:206.