Improving risk prediction is not easy

We have come a long way. Decades have passed since the first identification of blood pressure and cholesterol as risk factors for cardiovascular disease (CVD) and the prediction of absolute total cardiovascular risk is now based on a combination of well-defined atherothrombotic risk factors. Investigators from the Framingham study were the first to combine the key classic risk factors in a single risk score to estimate the absolute CVD risk over a 10-year period. ¹ Several alternative risk scores have been developed since then, which are almost all based on larger cohorts than the Framingham study. Although different in many aspects from the original Framingham risk score, modern risk scores almost universally use the same set of classic CVD risk factors of age, sex, current smoking, (systolic) blood pressure and cholesterol. Although the accuracy of these modern risk scores is reasonable, substantial risk heterogeneity occurs within the risk categories. ² A considerable number of events occur in those patients estimated as at low risk and many patients estimated as at high risk may receive intensive treatment in vain. The main problems, however, occur in the so-called intermediate risk categories. Because of its large size, many middle-aged adults fall into this category and many CVD events will thus eventually occur in these people. However, the aggregate group’s risk leaves clinicians and guideline writers in doubt as to whether this risk is high enough to justify (multi-)drug regimens in all people in this group. Hence, the call for the refinement of risk estimation, particularly for those people at intermediate risk levels, is about as old as the first risk scores and persists today.

Intuitively, three types of added risk information could improve risk stratification in addition to the existing risk scores. The first is refinement of existing information on classic risk factors, such as improved blood pressure measurements, more detailed information on smoking intensity, or refined parameters of cholesterol metabolism. The second type of useful additional information could be a strong, novel atherothrombotic risk factor unrelated to the classic risk factors. The third option is to include the measures of vascular damage that precede overt clinical CVD. Such measures would add information on individual susceptibility to the classic risk factors by demonstrating that the process of atherosclerosis has actually caused some degree of damage, which may ultimately progress to a CVD event.

Improving risk estimation by adding new risk information is, however, not a walk in the epidemiological park. Basic multiple regression analyses identifying the added information as determinants of CVD outcomes independent from classic risk factors by no means guarantees that a risk score can actually be improved by adding this same information. Also, the magnitude of independent relative risks associated with novel risk factors greatly overestimates their average impact on the calculated risks. ³ In the last 15 years or so, biostatisticians have been working hard to provide sophisticated statistical techniques to assess whether risk scores can be improved by added information. A first step towards a basic understanding these techniques is to acknowledge the principal difference between diagnosing existing disease versus the prediction of future disease. ⁴ This difference essentially precludes the use of diagnostic parameters such as sensitivity, specificity and discrimination (or C-statistics) for disease prediction. Instead, parameters such as calibration (indicating how well the predicted risks match the actual observed risks) and reclassification (indicating how many patients correctly move from one risk category to another after adding information to the risk score) need to be used.

With the use of such contemporary biostatistical standards, few, if any, of the so-called modern risk factors have managed to qualify as serious candidates to improve risk scores ⁵ and most risk scores used to date still rely on simple demographic data, blood pressure, a diagnosis of diabetes mellitus, smoking and lipid profiles. A particularly troubling phenomenon impeding progress in risk stratification science, apart from the use of inappropriate statistical techniques, is publication bias. It has been estimated that ‘positive papers’ in this field may have an up to 25 times larger likelihood of getting published than ‘negative papers’. ⁶

Elsewhere in this issue, Osawa et al. ⁷ present data from the Multi-Ethnic Study of Atherosclerosis study. They aim to demonstrate the added value of the coronary artery calcium (CAC) score and the carotic intima-media thickness (cIMT) for the prediction of stroke/transient ischaemic attack (TIA) and the potential benefit from treatment with statins. None of the state-of-the-art statistical techniques described here were used. Instead, Osawa et al. ⁷ used a straightforward approach by looking at the observed stroke/TIA risks in the strata of CAC scores and cIMT. They also compared the risks predicted by risk score with the observed risk after further sub-stratification of the CAC and cIMT categories. Assuming the risk reduction in stroke/TIA to be the same as in the ASCOTT lipid-lowering trial, ⁸ they then calculated the numbers-needed-to-treat for statin therapy.

Osawa et al. ⁷ are to be commended for providing a strategy that could be applied relatively easily in clinical practice. Intuitively, it makes sense to first calculate the estimated risk using an established risk score and to then perform one or two tests for preclinical vascular damage to help inform treatment decisions. The tests they used are not difficult to perform and have both been formally studied previously in terms of their ability to improve risk stratification. In these previous studies, the CAC score in particular had a strong evidence base in terms of added value. ⁹ The same is, however, much less clear for cIMT. ¹⁰ Publication bias severely distorts published work on the cIMT.⁶ Even in the MESA study itself, cIMT was a weak predictor. ¹¹

Several aspects of the study limit its applicability to clinical practice. First, clinicians are generally not interested in preventing only stroke/TIA, but instead consider preventive interventions in the context of the total CVD risk, pertaining to all clinically relevant events, particularly including coronary artery disease. Second, clinicians want to set rational priorities between several preventive interventions. Osawa et al. ⁷ only focus on treatment with statins, but clinicians will also need to consider antihypertensive drugs, glucose-lowering drugs and antithrombotic drugs.

Apart from the design-related limitations, a few methodological issues deserve comment. First, the cIMT and CAC score were di- and trichotomized, respectively, and such categorizations almost universally decrease the prognostic information that can be obtained from such inherently continuous measures. Second, the sCAC and IMT cut-off values were not age dependent, whereas age is a prime determinant of both.

What would an ideal future risk stratification strategy look like? This is a hard question. Personally, I think an improved risk score for all adults should be one in which the current classical risk factors are complemented by information that is simple and cheap to obtain. A standardized detailed family history, for example, or more refined information on smoking history or other lifestyle factors, have hardly been studied in terms of their potential to improve current CVD risk scores. The Reynolds risk score incorporates family history and a simple extra blood test (high-sensitivity C-reactive protein), but received relatively little attention after its first publication in terms of further validation studies or widespread support for implementation. ¹²

If we could improve current risk score by adding such simple ‘low-brow’ information, a more sophisticated test of preclinical vascular damage may still be of value in some people – for example, those with intermediate CVD risk levels. At this time, the CAC score seems to have the best résumé of such vascular damage indicators. However, even limiting its application to those patients at intermediate risk may prove a huge and costly operation. Ever since the first risk score was published, we have learned a lot, but are yet to make substantial practical progress.