Continuously updated network meta-analysis and statistical monitoring for timely decision-making

Abstract

Pairwise and network meta-analysis (NMA) are traditionally used retrospectively to assess existing evidence. However, the current evidence often undergoes several updates as new studies become available. In each update recommendations about the conclusiveness of the evidence and the need of future studies need to be made. In the context of prospective meta-analysis future studies are planned as part of the accumulation of the evidence. In this setting, multiple testing issues need to be taken into account when the meta-analysis results are interpreted. We extend ideas of sequential monitoring of meta-analysis to provide a methodological framework for updating NMAs. Based on the z-score for each network estimate (the ratio of effect size to its standard error) and the respective information gained after each study enters NMA we construct efficacy and futility stopping boundaries. A NMA treatment effect is considered conclusive when it crosses an appended stopping boundary. The methods are illustrated using a recently published NMA where we show that evidence about a particular comparison can become conclusive via indirect evidence even if no further trials address this comparison.

1 Introduction

In 1898, George Gould, the first president of the Association of Medical Librarians, presented his vision regarding the optimal use of existing evidence. He was looking forward to a situation where “a puzzled worker in any part of the civilized world shall in an hour be able to gain a knowledge pertaining to a subject of the experience of every other man in the world”. ¹ Highlighting the increasing information overload and the pivotal role of systematic reviews in health care, ² Mike Clarke updated Gould’s vision in 2004, hoping for a system in which decision makers “would be able, in 15 minutes, to obtain up-to-date, reliable evidence of the effects of interventions they might choose, based on all the relevant research.” ³

In essence Gould and Clarke call for cumulative (network) meta-analyses of randomized trials of health care interventions.^4–7 Ideally, cumulative meta-analyses are prospectively planned: investigators establish a collaboration before the design of their trials is finalized, so that study procedures, interventions and outcomes can be harmonized and analyses can be done as soon as the results become available.^6,7 Prospectively planned meta-analyses have the potential to reduce bias because key decisions on inclusion criteria, outcome definition and other procedures are made a priori. ⁷ Several prospective meta-analyses have been conducted in recent years, for example in cardiology^8,9 or oncology.^10,11

However, the vast majority of meta-analyses are not prospectively planned. Reviewers tend to update their meta-analysis when relevant studies are published but have no direct influence over the planning of future studies. Nevertheless, after each update they need to characterize the evidence (for a particular treatment comparison and outcome) as conclusive or not, decide whether future updates of the evidence are needed and recommend the realization of further studies or not. The Cochrane Collaboration has a policy about when a systematic review should be updated. ¹² Updating meta-analysis (either because it is prospectively planned or because its result would be used to form a decision about conclusiveness) involves multiple tests as evidence accumulates and effect sizes are recalculated at each step, resulting in an inflated type I error.^13–15 Sequential methods for standard pairwise meta-analysis have been developed to account for multiple testing and adjust nominal significance.^5,16–19

For many conditions, several treatment options exist and data on their comparative effectiveness are of primary interest to clinicians. At present comparisons of one treatment with no treatment, or with placebo, continue to dominate clinical research, and head-to-head comparisons remain uncommon. Network meta-analysis (NMA) addresses this situation. Under the condition that studies are similar with respect to the variables that might modify the treatment effects, NMA can synthesize evidence from trials that form a network of interventions in a single analysis. Summary estimates of comparative effectiveness for all treatment options are thus obtained, including treatments that have not been compared directly in head-to-head comparisons.^20,21 In line with recent calls for comparative evidence at the time of market authorization,^22,23 Naci and O’Connor suggested the use of prospective, cumulative NMA in the regulatory setting. ¹⁹ Evidence on relative effects of treatments can become conclusive even if there are no new trials that directly compare them because of new studies contributing indirect evidence.

In this article we extend ideas of sequential monitoring of trials to provide methods for updating NMAs. We argue that sequential methods are relevant in any setting where a decision is to be made based on the results of an updated meta-analysis; when future studies are to be planned based on existing meta-analytic results (prospective meta-analysis) or when decisions are made about the necessity of future updates. We introduce cumulative NMA, discuss ways to adjust for multiple testing and recommend graphical representations of the sequential NMA process. We then discuss how important outputs of NMA can be monitored when updating a NMA.

2 Illustrative example: Coronary revascularization in diabetic patients

To illustrate the methodology we use a recently published NMA evaluating the optimal revascularization technique in diabetic patients. ²⁴ The primary outcome examined is a composite of all-cause mortality, non-fatal myocardial infarction and stroke measured using odds ratio (OR). Authors combined 15 studies examining the effectiveness of three interventions; percutaneous coronary intervention with bare mental stent (BMS) or drug eluting stents (DES) and coronary artery bypass grafting (CABG). For illustration purposes we consider that NMA has been undertaken sequentially; each study included in the data as soon as it is published and the systematic reviewers have to decide, after each update, whether future updates of the NMA are necessary to provide a conclusive answer. This particular NMA was chosen because it examines few treatments and includes a substantial number of studies to ensure that methods will be easily exemplified and the sequential process will be conveniently presented. Throughout we assume that comparability between trial populations and characteristics that may act as potential effect modifiers is justified, so that the synthesis of the planned trials in a NMA model is sensible. The data set comprises 12 two-arm studies and three three-arm studies. NMA suggests that the best treatment is CABG which is significantly better than BMS (OR 0.59; 95% confidence interval 0.44 to 0.78) and marginally better than DES (OR 0.73; 95% confidence interval 0.54 to 0.98). Studies were published between 2007 and 2013 and it would be interesting to see whether significance is sustained after correcting for multiple testing and if yes, at which point in time the accumulated evidence was conclusive. Note that when updating NMA the comparison of ‘BMS versus CABG’ can become statistically significant even when ‘BMS versus DES’ studies are published via indirect comparison.

In order to undertake a sequential analysis, one needs to specify type I and type II errors, as well as the alternative hypothesis. The specification of the effect size to be detected is of crucial importance as the alternative hypothesis should express a clinically important effect reflecting the perspectives, needs and preferences of different individuals.^15,25–28 However, determination of an effect that reflects patient perceptions is very challenging when the primary outcome is a composite endpoint. ²⁴ For illustrative reasons, in the remainder of the paper we will use arbitrarily (yet clinically plausible) log ORs for the three comparisons to be ${\delta }_{\text{BMSvsCABG}}=0.28$ , ${\delta }_{\text{BMSvsDES}}=0.10$ , and ${\delta }_{\text{DESvsCABG}}=0.18$ ; these correspond to ORs of 1.32, 1.11 and 1.20, respectively. In clinical applications however, we recommend the consideration of a variety of alternative hypotheses taking into account patient preferences that may be driven by discomfort, inconvenience and risk of adverse events. ²⁸ Note that in a NMA context, the alternative treatment effects need to be consistent (e.g. here ${\delta }_{\text{BMSvsDES}}={\delta }_{\text{BMSvsCABG}}-{\delta }_{\text{DESvsCABG}}$ ). Particular attention is needed when more than three treatments are examined; alternative effect sizes should be determined for all comparisons in the network and consistency between them needs to be satisfied. Clinicians who suggest values for the alternative effect sizes are often asked to guess absolute effects for the various treatments. Consequently, the assumption of consistency would be satisfied in practice.

3 Methods

3.1 Cumulative NMA

Consider a network of n trials forming a set $\mathcal{P}=\{X,Y,Z,\dots \}$ of T competing interventions for a healthcare condition. We assume that the evidence base is updated sequentially; each trial indexed with $i=1,\dots ,n$ enters the analysis when its results become available. After the inclusion of each study, pairwise and NMA models are updated and cumulative treatment effects are derived. We assume that the number and timing of interim analyses are not known at the start of NMA and that updates take place after the publication of any new study that meets the inclusion criteria. The method can be generalized to NMAs that are updated after more than one studies are included.

Let ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ be the vector of all cumulative direct relative effects for each treatment comparison after the inclusion of trial i. Vector ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ contains the respective cumulative NMA treatment effects, derived from any appropriate statistical NMA model which integrates direct and indirect evidence and accounts for the correlation introduced by multi-arm trials.^29–31 Elements of ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ and ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ are replaced with the addition of each study i to represent the updated treatment estimates. As evidence is accumulated and treatments are added in the evidence base, ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ and ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ may change dimensions to include additional treatment effects with the dimension of ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ being equal to or smaller than the dimension of ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ . In the last step ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ and ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ will contain at most $(\begin{array}{c}T\\ 2\end{array})$ treatment effects and will be denoted as ${\hat{\mu }}^{\mathbf{D}}$ and ${\hat{\mu }}^{\mathbf{N}}$ respectively; note that the dimensions of ${\hat{\mu }}^{\mathbf{D}}$ and ${\hat{\mu }}^{\mathbf{N}}$ will be exactly $(\begin{array}{c}T\\ 2\end{array})$ in a fully connected network. We may focus on each element of ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{D}}$ and ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ or restrict ourselves to a subset of comparisons that are of more interest. Reasons to restrict the set of comparisons of interest may include the establishment of their comparative effectiveness or safety, their association with adverse events or even the withdrawal of certain treatments from the market. Consider for instance the comparison ‘Y versus X’; ${\hat{\mu }}_{i,\mathit{XY}}^D$ and ${\hat{\mu }}_{i,\mathit{XY}}^N$ denote the respective cumulative direct and NMA treatment effects with standard errors $\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^D)$ and $\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N)$ where $(X,Y)\in \mathcal{P}$ and i index the last study introduced. Similarly to cumulative pairwise meta-analysis, a cumulative NMA is a mechanism of displaying the cumulative NMA treatment effects ${\hat{\mu }}_{i,\mathit{XY}}^N$ along with their confidence intervals for $i=1,\dots ,n$ in a table or in a plot. Each ‘Y versus X’ NMA cumulative effect is modified not only when a study comparing the particular set of treatments is performed, but also when indirect evidence that informs the ‘Y versus X’ comparison becomes available. From this point we will focus on ${\hat{\mu }}_{i,\mathit{XY}}^N$ to illustrate the sequential methodology for NMA estimates. The developments equally apply to any element of ${\hat{\mu }}_{\mathbf{i}}^{\mathbf{N}}$ as well as to the direct cumulative estimates.

3.2 Assumptions underlying the updating of NMA

The justification of similarity in effect modifiers is important to ensure the plausibility of the transitivity assumption after each update of the network.^19,20,32 Throughout, we assume that the transitivity assumption is epidemiologically evaluated and deemed reasonable. The consistency assumption is the statistical manifestation of transitivity and lies on the statistical agreement between different sources of evidence. ³³ A statistical test for inconsistency can be monitored as soon as its evaluation is possible, that is when a closed loop (not composed only by multi-arm trials) is formed. Large amounts of inconsistency should prohibit a joint synthesis of the data and explore the differences between the various sources of evidence. However, the power of inconsistency tests might be low even after the inclusion of several studies in NMA.^34,35 In collaborative prospective NMAs inconsistency is likely to be avoided through the efforts of the researchers to ensure the comparability of the studies and maximize the chances of transitivity.

We adopt a random effects NMA model and we assume a network specific heterogeneity variance ${\tau }^2$ . One could re-estimate the heterogeneity variance at each step of the analysis; this process would be associated with poor estimation of heterogeneity while the number of included studies is small.^36,37 To overcome this limitation we choose to inform the unknown heterogeneity parameter by predictive distributions conditional on the type of outcome and treatment comparison based on findings from previous meta-analyses.^38,39 In order to account for uncertainty in the imputations of the heterogeneity parameter, we suggest the use of the 25th, the 50th and the 75th quantiles of the respective predictive distribution of heterogeneity formulated in Turner et al. (for binary outcomes) and Rhodes et al. (when continuous outcomes are assessed).^38,39 Setting a priori an expected value for heterogeneity might be more appropriate in the setting of prospective meta-analysis as studies are prospectively designed and their inclusion criteria are similar. Alternative strategies for heterogeneity, such as the re-estimation of heterogeneity after a sizeable number of included studies, could be applied.

3.3 Z-score and relevant information of cumulative network estimates

The cumulative network estimate ${\hat{\mu }}_{i,\mathit{XY}}^N$ is assumed to approximately follow the normal distribution ${\hat{\mu }}_{i,\mathit{XY}}^N\hspace{1em}\sim N({\mu }_{i,\mathit{XY}}^N,({\sigma }_{i,\mathit{XY}}^N{)}^2)$ ; we assume variances $({\sigma }_{i,\mathit{XY}}^N{)}^2$ to be known and equal to the sampling variances, denoted as $\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N{)}^2$ . The null hypothesis $H_0:{\mu }_{i,\mathit{XY}}^N=0$ is tested using the statistic

$$Z_{i,\mathit{XY}}^N=\frac{{\hat{\mu }}_{i,\mathit{XY}}^N}{\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N)}\hspace{1em}\sim N(0,1)$$

which we refer to as z-score. It is rejected if $|Z_{i,\mathit{XY}}^N|\ge z_{1-\frac{a}{2}}$ for a two-sided test where the value $z_{1-\frac{a}{2}}$ is the $1-\frac{a}{2}$ quantile of the $N(0,1)$ distribution.

Several approaches have been suggested for measuring information in pairwise meta-analysis; we adopt an approach which is directly related to the precision of the meta-analytic estimates and consequently to the amount of evidence accumulated.^5,15,40,41 According to that approach the information contained within each comparison in the network can be measured as

$$I_{i,\mathit{XY}}^N=\frac{1}{\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N)}$$

We will conventionally refer to $I_{i,\mathit{XY}}^N$ as the ‘amount of information’. Plotting the z-score versus the amount of information at each update i provides a visualization of the accumulation of evidence for the network estimate ‘Y versus X’.

3.4 Construction of efficacy stopping boundaries

Several methods have been proposed to control type I error in clinical trials when multiple looks at the data are taken through the construction of stopping boundaries for deciding whether or not to reject the null hypothesis. These methods include the Haybittle-Peto method, the Pocock boundaries and the O’Brien-Flemming monotone decreasing boundaries. ⁴² Application of different stopping boundaries can lead to different conclusions regarding early stopping of a clinical trial in an interim analysis. It has been suggested that the O’Brien-Flemming method is more close to the behavior of data monitoring committees who require a great beneficial effect to stop a trial at an early stage. ⁴³ An important problem associated with standard sequential methods is the necessity to define the number of interim analyses at the beginning and the requirement of equally spaced interim analyses. These problems are handled by the introduction of alpha spending functions which extend group sequential designs to allow flexibility in the number and timing of interim analyses. ⁴⁴ An alpha spending function $a(t)$ describes the rate at which the total significance level is spent at each intermediate testing; information fraction t indicates the proportion of the information that has been accumulated.

Appending efficacy boundaries to the $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane can lead to a stopping framework when updating NMA. We adopt the continuous alpha spending function which resembles the O’Brien-Flemming boundaries, defined as

$$a(t_{i,\mathit{XY}}^N)=2(1-\Phi (z_{1-\frac{a}{2}}/z_{1-\frac{a}{2}}\sqrt{t_{i,\mathit{XY}}^N}\sqrt{t_{i,\mathit{XY}}^N}))$$

where Φ represents the cumulative standard normal distribution.^42,44 The parameter $t_{i,\mathit{XY}}^N\in (0,1]$ indicates the position in the analysis regarding the accumulated information and is calculated as $t_{i,\mathit{XY}}^N=\frac{I_{i,\mathit{XY}}^N}{I_{\text{max},\mathit{XY}}^N}$ .

As the total amount of information that will be employed is unknown, the specification of $I_{\text{max},\mathit{XY}}^N$ needs to rest on assumptions. In order to specify the respective quantity in a sequential framework for pairwise meta-analysis, Wetterslev et al. assume that studies are approximating one big trial and follow conventional calculations made in sequential analysis of individual trials. ⁴¹ Higgins et al. use values obtained from the O’Brien and Flemming design for specific values of the alternative effect size, type I and type II errors. ¹⁵ We specify $I_{\text{max},\mathit{XY}}^N$ following conventional power calculations, imposing consistency between alternative effect sizes, ${\delta }_{\mathit{XY}}$ , for all comparisons involved in the network and taking into account the multiplicity induced by multiple comparisons. Specifically, $I_{\text{max},\mathit{XY}}^N$ is derived as the information that would be needed in an adequately powered multi-arm trial. As $\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N)$ involves the estimation of heterogeneity, $I_{i,\mathit{XY}}^N$ and $t_{i,\mathit{XY}}^N$ are also affected by the heterogeneity value. In particular, larger heterogeneity values are associated with smaller $I_{i,\mathit{XY}}^N$ and $t_{i,\mathit{XY}}^N$ with the respective meta-analytic estimates occupying places which are further to the left in the $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane. Details on derivation of $I_{\text{max},\mathit{XY}}^N$ can be found in Appendix A1.

The alpha spending function is used to allocate a portion of the total a to each $Z_{i,\mathit{XY}}^N$ . The efficacy boundaries $E_{i,\mathit{XY}}^N$ are the quantiles corresponding to $a(t_{i,\mathit{XY}}^N)/2$ . If $Z_{i,\mathit{XY}}^N$ crosses the boundaries ( $-E_{i,\mathit{XY}}^N,E_{i,\mathit{XY}}^N)$ the meta-analysis has reached a conclusive answer for the ‘Y versus X’ comparison. Note that even when a NMA effect estimate is deemed conclusive, indirect evidence may continue to feed into that particular comparison if the rest of the comparisons in the network do not contain sufficient evidence to infer about their conclusiveness. Similarly, a NMA effect estimate is updated and might reach conclusiveness even in the absence of trials addressing that particular comparison because of indirect evidence. A counter-intuitive situation can occur when a conclusive result becomes inconclusive in the next update. This could be the result of an important increase in heterogeneity or inconsistency leading to less precise effect estimates when data is synthesized using the random effects approach. In such situations, formal exploration and interpretation of sources of variability is required before inclusion of the new evidence in warranted updating of the network.

Figure 1 panel a presents the $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane of a fictional example where nine studies are synthesized sequentially and conclusiveness is achieved after eight studies for $a=0.05$ . The $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane along with the derived efficacy stopping boundaries can equivalently be presented by repeated confidence intervals on the estimates of the summary effects as ${\hat{\mu }}_{i,\mathit{XY}}^N\pm E_{i,\mathit{XY}}^N\mathit{se}({\hat{\mu }}_{i,\mathit{XY}}^N)$ .^44,45 The repeated confidence interval would include 0 when $|Z_{i,\mathit{XY}}^N|\le E_{i,\mathit{XY}}^N$ . This particular representation provides the same information regarding stopping decisions while it offers the advantage of displaying the NMA stopping framework in a forest plot along with the effect estimates ¹⁵ as shown in Figure 1 panel b.

Figure 1.

Panel a: Hypothetical stopping framework for efficacy and futility. Futility here means that Y will not be shown better than X by more than 0.5 effect size. Panel b: Hypothetical forest plot with repeated confidence intervals (dotted lines).

3.5 Construction of futility stopping boundaries

Future updates of NMA can be considered unnecessary when there are early signs of efficacy or because it is considered unlikely that the relative superiority of a treatment will be shown in subsequent steps of analysis. Such decisions in clinical trials are known as stopping for futility. ⁴⁶ Roughly, there are four major methods used to stop further experiments for futility: conditional power, predictive power—which is the analogue of conditional power in Bayesian analysis—, construction of triangular regions—also known as sequential probability ratio tests—, and beta spending functions.^46–48 We choose to transfer the later method for stopping for futility in NMA because of its analogy to the alpha spending functions and its convenient visualization along with the efficacy boundaries in the $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane. Note that the use of conditional power in NMA has been considered elsewhere. ⁴⁹

We adopt a method described by Lachin to determine futility boundaries. ⁵⁰ Without loss of generality we assume that positive values of ${\hat{\mu }}_{i,\mathit{XY}}^N$ represent a relative advantage of treatment Y. We consider that a study is futile if it cannot show that Y is better than X with an effect size of at least $f_{\mathit{XY}}$ . The treatment effect parameter $f_{\mathit{XY}}$ is an additive measure and should be defined so that it represents a clinically significant advantage of Y over X. Then, a decision to stop for futility can be specified if the upper confidence limit of the interim effect estimate ${\hat{\mu }}_{i,\mathit{XY}}^N+z_{1-\frac{a}{2}}/z_{1-\frac{a}{2}}{I}_{i,\mathit{XY}}^N{I}_{i,\mathit{XY}}^N$ does not exceed the pre-defined value $f_{\mathit{XY}}$ . It turns out that the futility confidence limit is equivalent to the determination of a futility stopping boundary for the interim $Z_{i,\mathit{XY}}^N$ value. Then, the futility stopping rule for a relative advantage of Y over X can be expressed as

$$Z_{i,\mathit{XY}}^N<f_{\mathit{XY}}{I}_{i,\mathit{XY}}^N-z_{1-\frac{a}{2}}$$

and we define the futility boundaries for Y over X on the $(Z_{i,\mathit{XY}}^N,I_{i,\mathit{XY}}^N)$ plane as $F_{i,\mathit{XY}}^N=f_{\mathit{XY}}{I}_{i,\mathit{XY}}^N-z_{1-\frac{a}{2}}$ . Note that while $f_{\mathit{XY}}$ is constant throughout the analysis, $F_{i,\mathit{XY}}^N$ depends on the amount of information accumulated at the ith update.

The value $f_{\mathit{XY}}$ could be set equal to ${\delta }_{\mathit{XY}}$ employed in power calculations. Values of clinical significance $f_{\mathit{XY}}$ should be chosen so as to satisfy consistency; that is for a triangular network that includes treatments X, Y and Z, we need to specify the respective values for only two out of the three treatment comparisons. If we specify $f_{\mathit{XY}}$ and $f_{\mathit{XZ}}$ it turns out that the value of clinical significance for the comparison ‘Z versus Y’ is $f_{\mathit{XZ}}-f_{\mathit{XY}}$ . In the hypothetical example illustrated in Figure 1 panel a we present the futility boundary for the case that we expect Y being better than X with an effect size of at least 0.5 which results to a decision of stopping for futility after the inclusion of the third study.

It has been shown that under the alternative hypothesis $f_{\mathit{XY}}\ne 0$ , stopping for futility inflates type II error. ⁵¹ A common solution to this limitation is the delay in making inferences regarding stopping for futility in the updating procedure, for instance appending futility boundaries only after at least half of the total planned information has been accumulated (that is at $t_{i,\mathit{XY}}^N\ge 0.5$ ). ⁵⁰ The vertical line in Figure 1 panel a indicates this point in the analysis which is termed ‘half-information futility assessment’.

3.6 Other network characteristics to be monitored

Monitoring changes in the conclusions from NMA should be accompanied by an evaluation in changes in the inconsistency and heterogeneity (if re-estimated at each update) so as to put results into context. Investigators planning NMA should make sure that the inclusion criteria of the studies ensure their comparability and maximize the chances of transitivity and that the distribution of effect modifiers is comparable across treatment comparisons. However, even after careful planning, there is always the possibility of inconsistency in the assembled data.^20,52 Thus, we consider that in each update of NMA an estimation of inconsistency is included; here, we consider the cumulative performance of the loop specific approach. ⁵³ Taking into account the low power of tests for inconsistency, we do not recommend adjusting for multiple testing.^34,35 Any signs of inconsistency in interim stages should be explored and the inclusion of new evidence should be carefully reconsidered.

Monitoring changes in the treatment ranking might also be useful in particular in large networks where many treatments are compared. Probabilities for each treatment being at each possible rank can be obtained and the surface under the cumulative ranking probabilities (SUCRAs) and their equivalent P-scores or mean ranks can be illustrated in graphs.^54,55 As these measures are based on the estimated summary effects at each update, their uncertainty should be expressed by the repeated confidence intervals while P-scores could be based on the adjusted p values.

4 Application

We apply our methodology to the network of trials for coronary revascularization in diabetic patients. ²⁴ Arm level data for the 15 studies along with the year of publication and the respective ORs are given in Appendix Table 1. For a ‘non-pharmacological versus any’ intervention comparison type and a semi-objective outcome a log-normal distribution for heterogeneity ${\tau }^2\hspace{1em}\sim \mathit{LN}(\begin{array}{cc}-2.89,& \hfill 1.{91}^2\hfill \end{array})$ has been recommended corresponding to 25th, 50th and 75th quantiles $\tau =0.12$ , $\tau =0.24$ , and $\tau =0.45$ respectively (Appendix Figure 4). ³⁹ We adopt a significance level of $\alpha =0.05$ and a type II error $b=0.1$ . Using the alternative effect sizes described in section 2 we estimate the maximum information needed to detect them as $I_{\text{max},\text{BMSvsCABG}}^{\text{N}}=13.24$ , $I_{\text{max},\text{BMSvsDES}}^N=38.56$ , and $I_{\text{max},\text{DESvsCABG}}^N=20.16$ . To derive futility boundaries we assumed $f_{\mathit{XY}}$ values equal to the alternative effect sizes; that is we consider it is futile to continue undertaking trials if we cannot show that CABG is better than DES and BMS with log ORs $0.18$ and $0.28$ , respectively. From consistency it follows that $f_{\text{BMSvsDES}}=0.10$ (in favor of DES).

4.1 Description of the accumulation of evidence

When evidence is updated regularly, researchers perform both pairwise and NMA and evaluate the criteria of stopping early for efficacy or futility for both procedures. Appendix Figure 5 shows the cumulative pairwise and NMA effect estimates along with their confidence and predictive intervals after the inclusion of each study.

Figure 2 shows the stopping framework for the three evaluated comparisons in the network assuming a heterogeneity standard deviation equal to the median of the predictive distribution, $\tau =0.24$ .

While inference regarding the comparison ‘BMS versus CABG’ is inconclusive using evidence only from the four trials providing direct evidence this is not the case for the accumulated evidence from NMA. More specifically, the 13th study was conducted in December of 2012 and examined the relative effectiveness of DES compared to CABG. This study informs the comparison ‘BMS versus CABG’ indirectly leading to a conclusion that further research is not needed for that particular comparison. Note that the comparison ‘BMS versus CABG’ would have become marginally significant after the inclusion of the 12th study in an unadjusted cumulative NMA as the respective ‘z-score’ lies on the dotted boundary which represents conventional stopping. The inclusion of nearly half of the included studies rendered the ‘DES versus CABG’ comparison statistically significant in favor of DES in an unadjusted cumulative NMA; adjusting for multiple testing though, both ‘DES versus BMS’ and ‘DES versus CABG’ comparisons remain inconclusive using either pairwise meta-analysis or NMA (Figure 2).

Figure 2.

Stopping framework for efficacy (solid lines) and futility (dashed lines) for the network of coronary revascularization in diabetic patients. Maximum information is not displayed in the graphs as it is everywhere larger than 10. Heterogeneity standard deviation is assumed to be equal to the median of the predictive distribution, $\mathbf{\tau }=0.24$ . Black circles indicate that the latest update comes from a study with direct evidence; blue circles indicate that the latest update comes from indirect evidence and red circles indicate that the latest update comes from a three-arm trial (both direct and indirect evidence). Stopping for efficacy is taking place if observations are outside the efficacy boundaries. The arrow on the Y-axis indicates the side of the futility boundary that suggests stopping. Conventional significance thresholds are represented with dotted lines.

For all three comparisons, the accumulated data do not cross the futility boundaries so no decision over stopping for futility is being made throughout the updating process of NMA or pairwise meta-analysis.

During the updating process, the inclusion of the 13th study would lead investigators to reach conclusive results about the relative effectiveness of one out of the three evaluated comparisons indicating that CABG is better than BMS. After the inclusion of 15 studies, DES would appear to have an insignificant advantage over BMS and CABG a non-statistical significant benefit over DES. As $I_{\text{max},\text{BMSvsDES}}^N$ and $I_{\text{max},\text{DESvsCABG}}^N$ would not have been reached, studies would continue to be performed (if the particular comparisons were still of interest).

The information regarding stopping for efficacy using results from pairwise meta-analysis or NMA given in Figure 2 can also be visualized in the form of repeated confidence intervals (Appendix Figure 6).

Assuming a 25th and 75th quantile of the predictive distribution for heterogeneity instead of the median in our calculations does not markedly change the conclusions of the stopping framework (Appendix Figure 7 and Appendix Figure 8). The influence of the 13th study continues to be pronounced in the stopping decisions. In general, greater values of heterogeneity render the repeated confidence intervals larger and consequently stopping for efficacy is less likely to occur.

Appendix Figure 9 shows the cumulative estimates of the inconsistency factor for the loop ‘DES-BMS-CABG’. It suggests that the initial inconsistency factor of 1.34 (on a logOR scale) in 2007 was decreased to 0.84 in 2009 and finally to a relatively small inconsistency factor of 0.26 in 2013. The confidence intervals become smaller as more studies are included and, although the method is underpowered, initial concerns that the network might be inconsistent are challenged.

We calculate the SUCRAs of the three treatments in each interim analysis allowing for uncertainty expressed by the repeated confidence intervals. Cumulative estimation of SUCRAs is illustrated in Figure 3. Repeated SUCRAs are relatively close to each other in the first years of the sequential NMA while their distinction is growing as evidence is accumulated.

Figure 3.

Accumulated SUCRAs for the network of coronary revascularization in diabetic patients. Heterogeneity standard deviation is assumed to be equal to the median of the predictive distribution, $\mathbf{\tau }=0.24$ . SUCRA: surface under the cumulative ranking probabilities.

It is important to note that the assumptions feeding into the analysis (values for heterogeneity, type I and type II error, alternative effect sizes) may not be universally acceptable to all health-care professionals and patients. Thus, results from a sequential NMA should be interpreted in the light of such decisions. Moreover, firm recommendations on the need of further studies should take into account that new studies might be useful for the examination of a secondary outcome; indeed Tu et al. point out that although CABG seems to be better than BMS and DES in terms of the primary outcome, it is associated with an increased risk of stroke and might not be preferred for patients at high risk of such an event. ²⁴ In that case, it is even more important to avoid undertaking further trials that involve CABG because its superiority has been established and further experimentation might be deemed unethical. Instead, indirect evidence, e.g. by planning more ‘BMS versus DES’ studies, should be sought for all comparisons of interest. In general, clinical judgment considering several outcomes that might be of interest to patients is necessary to evaluate which intervention is appropriate to which patient group.

5 Discussion

We suggest formal statistical monitoring when decisions need to be made every time a ΝMA is updated. The outlined method is adopted from the respective methodologies developed for clinical trials and pairwise meta-analyses. We consider two situations in which our methodology can be appropriate; in both situations the analyses of studies are performed as their results become available. The first one is the prospective design of a NMA at the time of market entry of a new drug as suggested by Naci and O’Connor. ¹⁹ In contrast to the current practice that drug approval often relies on the evaluation of each drug in placebo-controlled trials, such a procedure would feed regulatory agencies with the optimal level of evidence regarding the comparative efficacy and safety of the new drug. The establishment of designing prospective NMAs in the regulatory setting may be challenged by the potential reluctance of manufacturers to compare their treatments with all competing alternatives, which might lead to selective inclusion of pieces of evidence. Moreover, efforts to reduce the cost of performing a series of trials might lead to postponing the design of prospective NMA until a competing company has collected enough relevant evidence. Informing policy decision-making by health technologies assessments could also include an evaluation of the sufficiency of included evidence using methods described in this paper.

The second context that our method can be used in is the regular update of systematic reviews that contain multiple treatments when new trials become available. Application of the statistical monitoring is of particular interest to organizations that produce and maintain systematic reviews such as the Cochrane Collaboration. As the main aim of the Cochrane Collaboration is to provide the best available and most up-to-date evidence, authors not only prepare systematic reviews but are also committed into updating them. This commitment aims to minimize the risk of the reviews to become out-of-date and potentially misleading. Frequent updates of systematic reviews, however, can result in an inflated type I error, in a similar manner as in a genuinely prospective NMA.

Appending a stopping rule to the meta-analysis context has received considerable criticism.^15,56 In particular, expressed concerns highlight the lack of direct control over the process of collecting and synthesizing studies in the sense that the meta-analyst is not in a position to decide whether more trials are to be conducted or not. We consider the formal statistical monitoring to be relevant for situations in which a researcher can have control—or at least provide recommendations—over future updates of the meta-analysis.

Our methods are similar to those proposed by Whitehead and Higgins et al. for pairwise meta-analyses, extended to the case where multiple treatments are competing.^5,15 Whitehead has developed a sequential method for meta-analysis using the triangular test in a series of concurrent clinical trials and Higgins et al. focused on the restricted procedure of Whitehead, equivalent to an O’Brien and Flemming boundary.^5,15 Wetterslev et al. have developed an alternative sequential method for pairwise meta-analysis ⁴¹ ; they have also created software (www.ctu.dk/tsa) which has been largely applied in practice and they argue that their methodology should be adopted by Cochrane authors. ²⁵ Their approach has technical and conceptual similarities and dissimilarities with that proposed by Higgins et al. ^15,25 For instance, Wetterslev et al. adjust the required sample size by a factor that depends on the estimated heterogeneity. As estimation of heterogeneity is difficult at the beginning of the sequential process, the estimate at the final update is employed. This is something that can be feasible only on retrospective cumulative meta-analysis. Higgins et al. explores several ways to handle heterogeneity in a sequential random-effects meta-analysis approach including incorporating a prior distribution for the between-studies variance parameter. They argue in the discussion that “further empirical research is needed to characterize the degree of heterogeneity that can be anticipated in a meta-analysis with particular clinical and methodological features, so that realistic informative prior distributions can be formulated.” ¹⁵ As such empirical research has been conducted since then,^38,39 we here employ informative priors for the heterogeneity variance.

Whitehead suggests that the sequential procedure in meta-analysis may be more justifiable for safety outcomes while Higgins et al. propose the area of adverse effects of pharmacological interventions as a potential application of sequential methods.^5,15 Whether the proposed methods work well in the context of rare events is an issue that remains to be investigated. It has been argued that when a major adverse event is rare it might be inappropriate to control over inflated type I error as even a small signal could be sufficient for the meta-analysis to ‘stop’. ⁵ In any case, the practice of accumulating evidence in a formal way becomes even more imperative in the context of rare events.

6 Concluding remarks

The evolvement of technology can decisively contribute to the realization of living systematic reviews—the high quality, up-to-date online summaries, updated as new research become available—by providing semi-automation to the production process. The inclusion of all available treatment options in such ‘real-time’ syntheses has been termed as “live cumulative network meta-analysis” and can further facilitate informed research prioritization and decision-making. ⁵⁷ Development, refinement, and evaluation of appropriate statistical methodology as well as guidance over the optimal update of systematic reviews can aid the attempt of living systematic reviews and live cumulative NMA to bridge the gap between research evidence and health care practice. ⁴

Methodology described in this paper should ideally be viewed as part of a holistic framework for strengthening existing evidence by judging when evidence summaries provide conclusive answers,^28,58 planning new studies when needed^27,49,58,59 and subsequently updating meta-analysis to include the—assumed justified—future studies. While methodological developments regarding parts of this process have appeared in the literature, they are rarely used in practice. In order to shift the paradigm to evidence-based research planning, methodology needs to be refined and summarized in a comprehensive global framework while its properties need to be evaluated in real world examples. The development of user-friendly software routines along with educational material could also contribute to the usefulness and applicability of the methodology.