Some methodological points to consider when performing systematic reviews in comparative effectiveness research

Abstract

Background The purpose of comparative effectiveness research (CER) is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels. There is an ongoing discussion as to what types of evidence are appropriate to inform CER and how best to interpret various forms of evidence.

Purpose The purpose of this article is primarily to highlight several interesting methodological issues in the conduct of CER reviews.

Methods We describe several key challenges related to randomized trials, with a particular focus on noninferiority studies, which include active comparators used to assess ‘assay sensitivity’ (defined below), and on the use of randomized studies to perform indirect comparisons between therapies. We touch briefly on the use of observational studies in CER, particularly because of the importance of observational studies in assessing infrequently occurring harms.

Results We argue that studies that may be perceived as unsuitable to address some CER questions may well be appropriate to address others. As an example, noninferiority studies (assuming they include an appropriate comparator at an appropriate dose), are sometimes discounted or excluded from consideration because of concerns that the sponsor’s incentive is to conduct a study that is biased toward showing no difference between the treatment groups. If the purpose of a systematic review of CER is to show superiority with respect to a purported benefit, including studies that may be biased toward equality of treatments would tend to underestimate the proposed benefit, that is, the bias works against the sponsor.

Limitations This is not a comprehensive review of all methodological issues related to CER, and we recognize that there may be dissenting opinions regarding some of the points we raise.

Conclusions In considering the use of systematic reviews, we believe it is sound advice to perform the head-to-head comparison when possible, in the relevant populations, using endpoints relevant to patients, caregivers, physicians, or payers. These endpoints should include patient-reported outcomes (e.g., symptoms), when relevant. Indirect comparisons and mixed treatment meta-analyses may be useful for simultaneously comparing multiple treatments, provided the assumptions underlying such analyses are plausible.