Beta errors in anaesthesia randomised controlled trials in which no statistical significance is found: Is there an elephant in the room?

Abstract

Most anaesthetists will be familiar with the concept of β errors in randomised controlled trials (RCTs). However, many may not appreciate the range of uncertainty that these errors impose when it comes to interpreting a trial’s results if no statistical significance is found. In this editorial we present an analogy of ‘an elephant in the room’ in an attempt to better depict the source and range of this uncertainty. We also use an example to demonstrate how overlooking this uncertainty might lead to more definitive inferences than a trial’s results may warrant.

Let us say you have plausible reasons to suspect there is an elephant in a room, but you do not have access to the room to check. You can only look through the windows from the outside. There are five windows, each providing a view of a different 20% of the room above the height of the windowsills (Figure 1(a)). One of the windows is shuttered. There are two possible scenarios when you look through the windows. One is that you see an elephant, or a part of an elephant, through one of the unshuttered windows (Figure 1(a)). You can then be reasonably sure there is an elephant in the room, and that it is at least as tall as the windowsills, although there is a small possibility that what you are seeing is not an elephant, but something that looks like an elephant (i.e. a false positive).

Figure 1.

(a) You see part of an elephant through one of the unshuttered windows. You can then be reasonably sure there is an elephant in the room, although there is a small possibility that it is not an elephant, but only looks like an elephant (i.e. a false positive). (b) You do not see any part of an elephant through any of the unshuttered windows. What inference can you make if you have plausible reasons for suspecting there is an elephant in the room? Perhaps there is an elephant behind the shuttered window, or a small elephant shorter than the windowsill height?

The other scenario is that you do not see an elephant, or any part of an elephant, through the unshuttered windows (Figure 1(b)). So what should you conclude in this scenario about whether or not there is an elephant in the room? Remember you have plausible reasons for suspecting there is an elephant in the room. Clearly you would have to be circumspect. Although there is no evidence that there is an elephant in the room, you would have to acknowledge that there is a 20% probability of missing an elephant in the room taller than the windowsills if such an elephant were there. You would also have to acknowledge that you can make little inference about whether or not there is an elephant in the room shorter than the windowsill height (Figure 1(b)), because it would be physically impossible for you to see such an elephant.

Assessing the primary outcome of an RCT presents similar conundrums. If we observe a statistically significant result (e.g. P ≤ 0.05 or the 95% confidence intervals (CIs) do not include zero (or 1.0 for a ratio)), we can be reasonably certain there is a true effect (difference or association), although there is a small probability (= the α error, typically 0.05) that this is a false positive. However, if we do not obtain a statistically significant result (e.g. P > 0.05), how should we interpret the negative finding? Are there statistical equivalents of shuttered windows and windowsill heights that we should consider?

RCTs do not have unlimited power to obtain a statistically significant result, even if there is a true effect.^1

–5 Unlimited power would require an unlimited sample size, which would be impossible. Instead, the power of most RCTs is in the range of 80%–90%.^1

–5 Power is the probability of obtaining a statistically significant result, given that there is a true effect at least as large as a pre-specified effect size.^1

–5 Using our analogy, if the elephant in the room represents ‘a statistically significant result’, the proportion of unshuttered windows represents the power, the proportion of shuttered windows represents the β error (power = 1 – β), and the windowsill height represents the ‘pre-specified effect size’. Clearly the power and the pre-specified effect size will have a major influence on how we interpret the results, particularly if we do not find a statistically significant effect.

Statistical power, which is a mathematical construct, is a function of the pre-specified β error, the pre-specified α error (e.g. 0.05), the pre-specified effect size, the anticipated variability of the effect in the population, and the sample size. In practical terms, the first four variables are used to calculate the requisite sample size for any requisite power.^1

–5 This requisite sample size is the minimum required to obtain a statistically significant result for a true effect greater than or equal to the pre-specified effect, with that level of power. It follows that it is not possible with this sample size to obtain a statistically significant result for a true effect less than this pre-specified value with the same level of power: the sample size would be too small. This means the probability of a true effect less than this pre-specified value is unknown. In other words, no firm inference can be made on the probability of a true effect less than the pre-specified effect size used in a power/sample size calculation, in the same way as no firm inference can be made on whether there is an elephant in the room shorter than the windowsill height in the earlier example (Figure 1(b)).

The term ‘elephant in the room’ has another connotation, as a metaphor for ‘an obvious major problem or issue that people avoid discussing or acknowledging’.⁶ We contend that there is ‘an elephant in the room’ for most RCTs in which no statistical significance is found for the primary outcome, in the form of the range of uncertainty related to β errors and pre-specified effect sizes. While the 95% CIs for the primary outcome are typically presented in an RCT’s results section, and the possibility of a type II error is often mentioned in the discussion, it is our impression that the range of uncertainty accompanying the non-significant finding is rarely explicitly acknowledged. For example, in a sample of high quality RCTs in which no statistical significance was found for a primary outcome that included postoperative mortality, published between January 2019 and December 2020,^{7

–16} nine out of 10 had β errors (or estimated β errors) of 10%–20% (Table 1).^{7

–10,12

–17} (see Supplementary material online for search strategy). Furthermore, the pre-specified minimum effect size, on which the sample sizes were based, ranged from an absolute risk reduction (ARR) of 0.8%–6.1% in the four trials reporting mortality,^7,
8,
13,
14 and an ARR of 1.5%–15.5% in the remaining trials that reported a composite of death or major complications (Table 1). Yet five of the ten trials presented conclusions of ‘no’ difference, effect or evidence,^{8
–10,
13,
14} with the remainder reporting either ‘no significant’ or ‘no statistically significant’ difference or effect (Table 1). Only three of the ten trials included any mention of uncertainty in their conclusions,^12,
14,
15 and none of the conclusions drew attention to the range of uncertainty related to the β error or the pre-specified effect size on which the sample size was based.

Table 1.

Randomised controlled trials reporting no statistical significance for mortality as a primary outcome, either alone or as a composite, published between January 2019 and December 2020.

Authors, journal, year	Conclusion	Power	Pre-specified effect size in sample size calculation	Observed effect size (95% CIs) for primary outcome
Landoni et al., N Eng J Med, 2019⁷	… among patients undergoing elective, isolated CABG, an intraoperative anaesthetic regimen that included volatile anaesthetics did not result in significantly fewer deaths at 30 days or one year than a regimen of total intravenous anaesthesia	90%	ARR3% to 2% (=1%)RRR33%	One year mortalityRR 0.94 (0.69 to 1.29) 30-day mortalityRR 1.11 (0.70 to 1.76)
Sieber et al., Br J Anaesth, 2019⁸	… the results from this analysis show that there is no difference in mortality or return to ambulation one year after hip fracture surgery in elderly patients receiving heavier and lighter intraoperative sedation	80%	ARR31.5% to 16% (=15.5%)RRR49%	Intention-to-treat analysis showed no significant difference (0%)HR 0.85 (0.44 to 1.97) after accounting for age and MMSE scores
Myles et al.^a, J Thorac Cardiovasc Surg, 2019⁹	In patients undergoing coronary artery surgery, preoperative aspirin did not reduce death or severe disability, or thrombotic events through to one year after surgery	90%	ARR10% to 7% (=3%)RRR30%	RR 1.17 (0.76 to 1.81)
Myles et al.^a, J Thorac Cardiovasc Surg, 2019¹⁰	In this trial of patients having coronary artery surgery, tranexamic acid did not reduce death or severe disability through to one year after surgery	90%	ARR10% to 7% (=3%)RRR30%	RR 0.85 (0.64 to 1.13)
Futier et al., JAMA, 2019¹¹	Among patients at risk of postoperative kidney injury undergoing major abdominal surgery, use of HES for volume replacement therapy compared with 0.9% saline resulted in no significant difference in a composite outcome of death or major postoperative complications within 14 days after surgery	95%	ARR25% to 15% (=10%)RRR40%	RR 1.10 (0.91 to 1.34)
Lomivorotov et al., JAMA, 2020¹²	Among infants younger than 12 months undergoing cardiac surgery with CPB, intraoperative administration of dexamethasone, compared with placebo, did not significantly reduce major complications and mortality at 30 days. However, the study may have been underpowered to detect a clinically important difference.	80%	ARR40% to 25% (=15%)RRR37.5%	HR 0.82 (0.60 to 1.10)
Short et al., Lancet, 2019¹³	… we found no evidence that mortality and serious complications of anaesthesia and surgery were influenced by targeting a BIS of either 50 or 35	80%	ARR10% to 8% (=2%)RRR20%	HR 0.88 (0.73 to 1.07)
Sessler et al., Anesthesiology 2019¹⁴	Real-time alerts to triple-low events did not reduce 90-day mortality, although there were fewer responses to the alerts than expected	80%	ARR2.9% to 2.1% (=0.8%)RRR28%	HR 1.14 (0.96 to 1.35)
Lamontagne et al., JAMA, 2020¹⁵	Among patients ≥65 years receiving vasopressors for vasodilatory hypotension, permissive hypotension compared with usual care did not result in a statistically significant reduction in mortality at 90 days. However, the CIs around the point estimate for the primary outcome should be considered when interpreting the clinical importance of the study.	90%	ARR35% to 28.9% (=6.1%)RRR17%	RR 0.93 (0.85 to 1.03) When adjusted for pre-specified baseline variables, the OR for 90-day mortality was 0.82 (0.68 to 0.98)
Sessler et al., Anesthesiology, 2020¹⁶	Neither perioperative aspirin nor clonidine have significant long-term effects on mortality or non-fatal MI after noncardiac surgery	84%^b	ARR^b6.1% to 4.6% (=1.5%)RRR^b25%	HR aspirin 1.00 (0.89 to 1.12) HR clonidine 1.07 (0.96 to 1.20)

^aThese two articles provide data on different outcomes of the same trial.

^bBased on the POISE-2 Trial power/sample size calculation.¹⁷

CI: confidence interval; ARR: absolute risk reduction; RRR: relative risk reduction; RR: relative risk; HR: hazard ratio; OR: odds ratio; MMSE: Mini-Mental State Examination; HES: hydroxyethyl starch; CPB: cardiopulmonary bypass; CABG: coronary artery bypass graft; BIS: bispectral index; MI: myocardial infarction.

Not drawing attention to uncertainty is understandable if the range of uncertainty is clinically irrelevant. Clinical relevance is also a judgement that may differ depending on the clinical context. However, can a possible reduction or increase of up to eight deaths per 1000 procedures be judged clinically irrelevant, let alone up to 61 deaths per 1000 procedures? In other words, can we dismiss as ineffective, interventions about which we can make no firm inference on whether they prevent up to eight deaths per 1000 procedures (ARR up to 0.8%), or perhaps even up to 61 deaths per 1000 procedures (ARR up to 6.1%)? Alternatively, can we accept as safe, interventions about which we can make no firm inference on whether they are associated with up to an additional eight deaths per 1000 procedures, or perhaps even up to an additional 61 deaths per 1000 procedures? The same considerations would apply to the composite of death or major complications.

Considering relative rather than absolute risks makes this uncertainty even more stark. For example, can we dismiss as ineffective, or accept as safe, interventions about which we can make no firm inference on whether they reduce or increase the relative risk of death by 17%–33% (or the relative risk of a composite of death or major complication by 25%–49%) (Table 1)? While this uncertainty does not mean that relative risks in this range exist, it does mean that the RCT cannot inform us on whether or not they exist. Remember that the authors of these RCTs had plausible reasons for suspecting a true difference, otherwise they would not have gone to the effort and expense of the RCT. Remember too, that the findings of the RCTs will be considered ‘level 1 evidence’.¹⁸

Perhaps RCTs can escape these conundrums by focusing on point estimates and 95% CIs rather than P values and power. Unfortunately, this does not reduce uncertainty. CIs do not provide information on either the β error or the pre-specified effect size on which the power/sample size is based. Moreover, CIs widen as sample sizes decrease, thereby increasing the range of possible true effect sizes, some of which could be considered clinically relevant (even if the 95% CIs for relative risk include 1.0).^1

–5 For example, in most of the RCTs cited, the 95% CIs included a relative risk (or hazard ratio) of at least 10%,^{7

–10,12,13,15,16} with several including relative risks of 15%–30% or more (Table 1). Another option is to avoid classifying a trial’s results into significant and non-significant and instead focus on the point estimates of the effect size observed.^19,
20 However, this approach is rarely taken by authors, and was not taken by the majority of authors in our sample, most of whom appeared to base their conclusions on the absence of statistical significance, despite in many cases observing relative risk point estimates greater than 10%.

While we have used the example of recent anaesthesia RCTs on postoperative mortality and major complications that found no statistical significance, our comments apply in principle to all anaesthesia RCTs that find no statistical significance, and indeed all such RCTs. This does not imply criticism of the RCTs. In particular, it does not imply criticism of the ten RCTs cited, which are exemplary scientific achievements. Our comments are, rather, an appeal for a change in the method of couching non-significant results, given that even the highest quality RCTs have practical limitations in relation to cost, time and other resources. These limit their sample size, which in turn limits their ability to reduce their β error (and 95% CI width) and, more importantly, their ability to choose an estimated effect size for their power/sample size calculation that includes all clinically relevant effect sizes. This introduces a range of uncertainty that cannot be avoided. We recommend that authors of RCTs in which no statistical significance is found make this range of uncertainty explicit in their conclusions, and that this should be required by journal editors. If not, we recommend that readers consider unconditional negative findings (those based on non-significance alone) as a red flag, to prompt them to carefully scrutinize the β error and the range of effect sizes on which the RCT is uncertain, before dismissing interventions as ineffective, or accepting them as safe. Our comments support those of Sidebotham et al., who recently counselled, on the basis of Bayesian modelling, that ‘it is plausible that many studied interventions have clinically important effects that are missed’.²¹

We hasten to add that none of our comments about overlooking inherent uncertainty related to power in RCTs are novel.^22
–24 To quote an accomplished group of statisticians in 2016, ‘Misinterpretation or abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant’.²⁴ Nevertheless, we hope that our ‘elephant in the room’ analogy will assist readers to better understand this inherent uncertainty, and the statistical principles on which it is based. We also acknowledge that potential misinterpretation of a trial’s results is not limited to trials in which no statistical significance is found.^21,
25,
26 Conclusions of clinical superiority related to statistically significant results also require scrutiny, and should be interpreted with attention to the clinical relevance of the observed effect,^26,
27 and in the light of prior findings.^28

–31

Footnotes

Author contribution(s)

Neville M Gibbs: Conceptualization; Data curation; Formal analysis; Methodology; Supervision; Writing – original draft.

William M Weightman: Conceptualization; Formal analysis; Writing – review & editing.

ORCID iD

Neville M Gibbs

Supplemental material

Supplemental material for this article is available online.

References

Katz

MH.

Study design and statistical analysis: A practical guide for clinicians. Cambridge: Cambridge University Press, 2006, pp.127–140.

Myles

Gin

Statistical methods for anaesthesia and intensive care. Edinburgh: Butterworth Heinemann, 2000, pp. 24–28, 122–134.

Kirkwood

Sterne

JAC.

Essential medical statistics 2nd ed. Oxford: Blackwell Publishing Company, 2003, pp.58–79, pp.413–428.

Bland

Introduction to medical statistics 4th ed. Oxford: Oxford University Press: 2015, 115–130.

Altman

Machin

Bryant

, et al. Statistics with confidence 2nd ed. Bristol: BMJ Books, 2000, pp.139–152.

‘Elephant in the room’. Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/elephant%20in%20the%20room (accessed October 2021).

Landoni

Lomivorotov

Nigro Neto

, et al. Volatile anesthetics versus total intravenous anesthesia for cardiac surgery. N Engl J Med 2019; 380: 1214–1225. doi:10.1056/NEJMoa1816476

Sieber

Neufeld

Gottschalk

, et al. Depth of sedation as an interventional target to reduce postoperative delirium: Mortality and functional outcomes of the Strategy to Reduce the Incidence of Postoperative Delirium in Elderly Patients randomised clinical trial. Br J Anaesth 2019; 122: 480–489. doi:10.1016/j.bja.2018.12.021

Myles

Smith

Kasza

, et al. Aspirin in coronary artery surgery: 1-year results of the Aspirin and Tranexamic Acid for Coronary Artery Surgery trial. J Thorac Cardiovasc Surg 2019; 157: 633–640. doi:10.1016/j.jtcvs.2018.08.114

10.

Myles

Smith

Kasza

, et al. Tranexamic acid in coronary artery surgery: One-year results of the Aspirin and Tranexamic Acid for Coronary Artery Surgery (ATACAS) trial. J Thorac Cardiovasc Surg 2019; 157: 644–652. doi:10.1016/j.jtcvs.2018.09.113.

11.

Futier

Garot

Godet

, et al. Effect of hydroxyethyl starch vs saline for volume replacement therapy on death or postoperative complications among high-risk patients undergoing major abdominal surgery. JAMA 2020; 323: 225–236. doi:10.1001/jama.2019.20833

12.

Lomivorotov

Kornilov

Boboshko

, et al. Effect of intraoperative dexamethasone on major complications and mortality among infants undergoing cardiac surgery. JAMA 2020; 323: 2485–2492. doi:10.1001/jama.2020.8133

13.

Short

Campbell

Frampton

, et al. Anaesthetic depth and complications after major surgery: An international, randomised controlled trial Lancet 2019; 394: 1907–1914. doi:10.1016/S0140-6736(19)32315-3

14.

Sessler

Turan

Stapelfeldt

, et al. Triple-low alerts do not reduce mortality. A real-time randomized trial. Anesthesiology 2019; 130: 72–82. doi:10.1097/ALN.0000000000002480

15.

Lamontagne

Richards-Belle

Thomas

, et al. Effect of reduced exposure to vasopressors on 90-day mortality in older critically ill patients with vasodilatory hypotension. JAMA 2020; 323: 938–949. doi:10.1001/jama.2020.0930

16.

Sessler

Conen

Leslie

, et al. One-year results of a factorial randomized trial of aspirin versus placebo and clonidine versus placebo in patients having noncardiac surgery. Anesthesiology 2020; 132: 692–701. doi:10.1097/ALN.0000000000003158

17.

Devereaux

Mrkobrada

Sessler

, et al. Aspirin in patients undergoing noncardiac surgery. N Engl J Med 2014; 370: 1494–1503. doi:10.1056/NEJMoa1401105

18.

Oxford Centre for Evidence-based Medicine: Levels of evidence (March 2009), https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009 (accessed October 2021).

19.

Sullivan

Feinn

Using effect size—or why the p value is not enough. J Grad Med Educ 2012; 4: 279–281. doi:10.4300/JGME-D-12-00156.1

20.

Wasserstein

Schirm

Lazar

NA.

Moving to a world beyond “p < 0.05”. Am Statistician 2019; 73(Suppl. 1): 1–19. doi:10.1080/0003.305.2019.1583913

21.

Sidebotham

Popovic

Lumley

A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care. Br J Anaesth 2021; 127: 487–494. doi:10.1016/j.bja.2021.06.026

22.

Yentis

SM.

The struggle for power in anaesthetic studies. Anaesthesia 1996; 51: 413–414. doi:10.1111/j.1365-2044.1996.tb07781.x

23.

Gibbs

Weightman

WM.

Errors in the interpretation of ‘no statistically significant difference’. Anaesth Intensive Care 2013; 41: 151–153. doi:10.1177/0310057X1304100203

24.

Greenland

Senn

Rothman

, et al. Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. Eur J Epidemiol 2016; 31: 337–350. doi:10.1007/s10654-016-0149-3

25.

Gibbs

Weightman WM. An audit of the statistical validity of conclusions of clinical superiority in anaesthesia journals. Anaesth Intensive Care 2014; 42: 599–607. doi:10.1177/0310057X1404200509

26.

Gibbs

Weightman

WM.

Beyond effect size: Consideration of the minimum effect size of interest in anaesthesia trials. Anesth Analg 2012; 114: 471–475. doi:10.1213/ANE.0b013e31823d2ab7

27.

Gibbs

Weightman

WM.

A forcing strategy to improve the evaluation of clinical superiority in anaesthesia trials. Br J Anaesth 2016; 117: 281–283. doi:10.1093/bja/aew255

28.

McCulloch

TJ.

Bayesian statistics: how to quantify uncertainty. Anaesth Intensive Care 2011; 39: 1001–1003. doi:10.1177/0310057X1103900603

29.

Sidebotham

Are most randomised trials in anaesthesia and critical care wrong? An analysis using Bayes’ theorem. Anaesthesia 2020; 1386–1393. doi:1111/anae.15029

30.

Charlesworth

Pandit

JA.

Negative outcomes in critical care trials applying the wrong statistics – or asking the wrong questions. Anaesthesia 2020; 1284–1288. doi:1111/anae.15028

31.

Frost

Alexandrou

Schulz

, et al. Interpreting the results of clinical trials, embracing uncertainty: A Bayesian approach. Acta Anaesthesiol Scand 2021; 65: 146–150. doi:10.1111/aas.13725

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB