Efficacy and effectiveness

The COVID-19 Pfizer-BioNTech vaccine has been described as being up to 95% effective in preventing COVID-19, ¹ but some would describe this as an estimate of its efficacy.

In plain English, ‘efficacy’ and ‘effectiveness’ are synonyms but when used in a medical context, a distinction is often made between the efficacy of an intervention and its effectiveness, the former interpreted under ‘ideal’ conditions and the latter under ‘real world’ conditions. However, the distinction relies on having to define ‘ideal’ and ‘real world’ conditions if, indeed, a satisfactory definition can be made for either. The terms ‘ideal’ and ‘real world’ are not helpful due to their lack of clarity.

The results of a randomised clinical trial are often regarded as reflecting the true effect of intervention. But this is rarely the case because, with the exception of randomised cross-over trials, it is rare for 100% of people in the intervention group to both accept the intervention and fully adhere to it, as well as no one having the intervention in the control group. Although an intention-to-treat analysis (results based on the treatment assigned not on the treatment received) avoids selection bias, it will necessarily underestimate efficacy because not everyone adheres to their assigned treatment. If the intervention is long-term treatment, it becomes more likely that people will fail to continue to take the treatment for the duration of the trial. The intervention may also take a few years to reach a full effect; as a consequence, the overall result from a trial of, say, five years duration will underestimate the full long-term effect.

There is merit in regarding efficacy and effectiveness as synonyms, hereafter referred to as effectiveness, and defined as the performance of an intervention in achieving a specified outcome assuming 100% of people accept and adhere to the intervention compared to the outcome in people who do not have the intervention. The derivation of effectiveness needs to be explicit. To this end, the comparison of the intention-to-treat result with the on-treatment result is often helpful. If the differences between the two results can be explained by random non-adherence to the assigned treatments (which requires judgement that can be assisted by knowing the reasons for non-adherence), a valid estimate of the full effect free from selection bias can be made from the on-treatment result. For example, stating that breast cancer screening is 15% effective in reducing the risk of dying of breast cancer when half of women are not screened is of limited value and even misleading. It is better to say that it is 30% effective and that in a given population only 50% of women accepted the offer of screening. Women deciding on whether to undergo breast cancer screening need to know the 30% estimate, not the 15% estimate. Effectiveness as defined above tends to be pushed to the side by policy makers and regulators who often focus on the intention-to-treat results.

In practice, it is best to estimate effectiveness and separately estimate the proportions of people who accept a given intervention and adhere to it to obtain an indication of the population impact. This has been adopted in antenatal screening. For example, an antenatal screening method for Down’s syndrome may have a 90% detection rate (sensitivity) for a 2% false-positive rate if all pregnant women have the screening test. If only 50% have the test, the screening performance (the measure of effectiveness) is still a detection rate of 90% for a false-positive rate of 2%, but among all pregnant women, only 45% of affected pregnancies are identified and 1% will have a false-positive result. Estimates of uptake and estimates of effectiveness (screening performance) are kept separate and not conflated. In antenatal screening, effectiveness has always been reported in this way without any problem. Evaluation of other interventions could and should be done in a similar fashion. For example, the COVID-19 Pfizer-BioNTech vaccine is 95% effective, but it remains to be seen what the uptake will be in practice. Estimates of effectiveness, derived from trials, can be used with estimates of vaccine uptake, derived from vaccination programmes, to determine the population impact.

In summary, it is suggested that (i) efficacy and effectiveness are synonyms when used in a medical context, as is the case when used generally, (ii) effectiveness should be estimated on the basis of full (i.e. 100%) acceptance and adherence to the intervention in question and (iii) separate estimates are made for acceptance and adherence to the intervention in question to determine the population impact of the intervention. The separation of (ii) and (iii) avoids any apparent discrepancy in effectiveness from different studies when no discrepancy exists.