The impact of studies with no events in both arms on meta-analysis of rare events: A simulation study using generalized linear mixed model

Abstract

Objective

The common approach to meta-analysis with double-zero studies is to remove such studies. Our previous work has confirmed that exclusion of these studies may impact the results. In this study, we undertook extensive simulations to investigate how the results of meta-analyses would be impacted in relation to the proportion of such studies.

Methods

Two standard generalized linear mixed models (GLMMs) were employed for the meta-analysis. The statistical properties of the two GLMMs were first examined in terms of percentage bias, mean squared error, and coverage. We then repeated all the meta-analyses after excluding double-zero studies. Direction of estimated effects and p-values for including against excluding double-zero studies were compared in nine ascending groups classified by the proportion of double-zero studies within a meta-analysis.

Results

Based on 50,000 simulated meta-analyses, the two GLMMs almost achieved unbiased estimation and reasonable coverage in most of the situations. When excluding double-zero studies, 0.00%–4.47% of the meta-analyses changed the direction of effect size, and 0.61%–8.78% changed direction of the significance of p-value. When the proportion of double-zero studies increased in a meta-analysis, the probability of the effect size changed the direction increased; when the proportion was about 40%–60%, it has the largest impact on the change of p-values.

Conclusion

Double-zero studies can impact the results of meta-analysis and excluding them may be problematic. The impact of such studies on meta-analysis varies by the proportion of such studies within a meta-analysis.

Keywords

Meta-analysis rare events generalized linear mixed model statistical properties simulation study

Introduction

Meta-analysis is a crucial statistical tool to synthesize information of similar studies and is commonly employed in medicine, education, sociology, psychology and related areas.^1–4 Standard meta-analytic methods based on asymptotic theory with a large sample assumption usually achieve unbiased estimation.⁵ However, in the case of dichotomous outcomes, especially for rare outcomes (e.g., rare adverse events) or for outcomes where zero events were reported, the assumption no longer holds and may result in biased or even invalid estimation.⁶

Studies with zero events are generally due to low incident rate, small sample size, or limited follow-up period. Dealing with studies of no events in both arms has been a tough problem because the relative estimates, such as odds ratio (OR), and the variance are hard to be defined.^7–11 A common approach for studies with no events in both arms is to apply a continuity correction of 0.5, while this method has been criticized for a number of drawbacks.¹² The more common approach is to remove such studies by treating them as non-informative.¹³ However, some researchers argued that these studies may contain usable information for statistical inference and omitting them in analysis may lead to selection bias.^14,15

Methodologists therefore developed several models to handle this issue. For example, Kuss et al. and Chu et al. proposed the beta-binomial model and their simulation verified the performance of this model.^16,17 Liu et al. used the p-function model (within the exact framework¹⁸ based on Fisher’s mid-p correction to combine studies with no events.¹⁹ Ren et al. compared the performance of standard methods (e.g., Peto, continuity correction), exact methods, and Bayesian methods, and recommended using the exact and Bayesian methods when there were studies with no events.²⁰ An alternative option was the generalized linear mixed model (GLMM) described by Stijnen et al.²¹ and Simmonds and Higgins.²² This model is a one-stage framework and allows studies with no events in both arms to “borrow strength” from studies with events without adding a priori assumption or post hoc correction.^21–24 Jackson et al. discussed six generalized linear mixed models for rare events and suggested that GLMMs performed better in the statistical inference than standard two-stage random-effects model.²⁵

Our previous work, by fitting a random-intercept GLMM model, has demonstrated that studies with no events contain important information and excluding them would have an impact on the results of meta-analysis.²⁶ We observed that the impact of studies with no events in both arms on meta-analysis may differ by the number of studies with no events in both arms within a meta-analysis. It is unclear whether the impact still exist with different number of studies with no events in both arms within a meta-analysis. In this article, we employed a large-scale simulation study to explore the impact of such studies on meta-analysis in terms of the proportion of them within a meta-analysis. For simplicity, we will refer “studies with no events in both arms” to as studies with no events in the following context.

Methods

Data generation

Our simulation is designed and reported according to the guidance from Pateras K et al., Burton A et al., and Morris TP et al.^27–29 The protocol of current simulation has been documented in our previous work.²⁶ We generated grouped data (2 × 2 table) in the simulation. The following fixed or random parameters were designed for current simulation: sample size of each trial (random), sample size ratio of two arms (random), number of studies within a meta-analysis (random), event risk in the control arm (fixed), true effect size (fixed), between-study variance (fixed). The true effect size and between-study variance were set to form different scenarios (see below). In order to generate a dataset that was aligned to the real-world settings, we performed a simulation with the parameter setting based on systematic reviews published from January 2003 to May 2018 in the Cochrane Database of Systematic Reviews (CDSR).^30,31 From the CDSR, we identified 442 meta-analyses (with data from 3652 trials) which contained studies with no events (supplementary file 1) could be used for analysis.²⁶

The sample size information (the above 442 meta-analyses) was fitted into 71 commonly used distributions and the log-normal distribution fitted well in both the experimental and the control arms. Considering the potential correlations on the sample size of the two arms, we further analyzed the sample size ratio of them and utilized the ratio and the log-normal distribution of the control arm to get the sample size of the experimental arm. To be simple, here we took the uniform distribution of the first to third quartiles (0.84–2.04) of the sample size ratio (simply referred to as ratio henceforth) from the empirical data. Let $n_{1}$ and $n_{2}$ be the sample sizes of experimental and control arms; then, log $n_{2}$ ∼ N (3.3537, 0.9992), $n_{1}$ = exp (log $n_{2}$ )*ratio, and ratio ∼ uniform (0.84, 2.04).

The mean risk of the control arm from the 3652 trials was 0.07; we however set it as 0.01 to improve the possibility for generating studies with zero events. This would allow us to get sufficient numbers of meta-analyses that contain studies with no events. The risk of the experimental arm was determined by effect size (i.e., OR) and the between-study variance (tau-square). To cover more scenarios, five monotonic increased values with equally-spaced ranges from small to large for effect size (1, 2, 3, 4, 5) and tau (0.2, 0.4, 0.6, 0.8, 1.0) were considered. Table 1 summarizes the simulation settings.

Table 1.

Simulation parameter setup.

Parameter	Assigned values
Incidence rate of the control group (pc)	0.01
Number of patients in control group ( $n_{2}$ )	mean(log) = 3.3537, sd(log) = 0.9992
Sample size ratio (ratio)	Uniform (0.84, 2.04)
Number of patients in experimental group ( $n_{1}$ )	$n_{1}$ = exp (log $n_{2}$ )*ratio
Effect sizes (OR)	1, 2, 3, 4, 5
Between-study variance ( $τ^{2}$ )	$τ$ =(0.2, 0.4, 0.6, 0.8, 1.0)
Number of studies included in each meta-analysis (m)	Uniform (4, 10)

In the CDSR data, the study numbers in each meta-analysis ranged from 2 to 78, with the first to third quartiles as 4 and 10, respectively (Table S1, supplementary file 2). The study numbers was then randomly sampled from 4, 5, 6, 7, 8, 9 and 10. This setting also facilitated the calculation of the proportion (0%–100%) of studies with no events in each meta-analysis. We generated 50,000 iterations for each scenario; among them, we removed those failed to contain studies with no events. We removed those meta-analyses with all studies as no events. In the GLMM model for meta-analysis, if the studies with the dependent variable was constant, we removed it owing to the basic assumption of likelihood-based methods for the parameter estimation.^32,33

Data analysis

The GLMMs were employed to combine data based on the link function “logit” under the “binominal” family. Here we considered two standard GLMMs for meta-analysis, i.e., the random slope GLMM and the random intercept and slope GLMM.^25,34 Suppose there are i studies with j intervention status (1 for experimental and 0 for control). The random slope model employs random treatment effects $θ_{i} \sim N (θ, τ^{2})$ with fixed study effects ( $γ_{i}$ ) based on the hierarchical logistic regression within a frequentist framework: $l o g i t (π_{i j}) = γ_{i} + {j θ}_{i}$ . Here $θ$ is log OR, $τ$ is the between-study variance due to varies treatment effect, and $π$ is the event risk. We call it the random slope GLMM since the constant term is generally excluded in the model.^25,34 The random intercept and slope model employ both random study effects $γ_{i} \sim N (γ, σ^{2})$ and random treatment effects $θ_{i} \sim N (θ, τ^{2})$ . The later accounts for the variance of study effects ( $σ^{2}$ ) across studies and is expected to be more conservative.

We first examined the statistical properties of GLMMs for combining studies with no events based on current parameter settings. The percentage bias, mean squared error (MSE), and coverage were utilized as indicators for the statistical properties of point and interval estimation. The model fitting was also compared based on Akaike information criterion (AIC) and Bayesian information criterion (BIC) statistics.³⁵ We further repeated the meta-analyses after excluding studies of no events to investigate the impact of studies of no events for meta-analysis. The direction of the estimated ORs and p-values for including and excluding studies with no events were subsequently compared. The amount of between-study variance, namely, interclass correlation (ICC), was also compared; while this is only possible for the random intercept and slope model due to the model assumption.³⁶ Paired meta-analyses that changed the direction of ORs and p-values were summarized. The threshold of the direction for OR was set as one and for p-value as 0.05 separately. For example, if an OR changed from >1 to <1 (or from <1 to >1), or a p-value changed from >0.05 to <0.05 (or from <0.05 to >0.05), then we considered that the direction was changed.

To investigate the potential impact of the number of studies with no events within each meta-analysis on the estimation, we stratified the whole samples into 9 ascending groups by the proportion of the number of studies with no events in a meta-analysis: 0.01%–10%, 10.01%–20%, 20.01%–30%, 30.01%–40%, 40.01%–50%, 50.01%–60%, 60.01%–70%, 70.01%–80%, and 80.01%–90%. The GLMMs were implemented by the meglm command in STATA/SE 14.0 (Stata, College Station, TX). The command for the random slope GLMM is: meglm r study group || study: group, nocons family(binomial n) intme(lap). Also, the command for random intercept and slope GLMM is: meglm r study group || study: group, family(binomial n) intme(lap). Where r is the number of cases and group is the intervention status. We used the Laplacian method for the estimation of the log likelihood; it is the default method for crossed random-effect model in Stata (version 14.0/SE, Stata, College Station, TX). The simulation code is presented in the supplementary file.

Results

Statistical properties of GLMMs

Figure 1 presents the percentage bias, MSE, and coverage of the two GLMMs under different scenarios. Our simulation suggested that the two GLMMs achieved almost unbiased estimation on the effect size in most of the situations when the between-study variance was not substantial (tau<0.8). We observed some extents of underestimation on effect size when true OR = 1. Even when the between-study variance was large (tau = 0.8 or 1.0), the percentage bias remained in a relative low level (1.89%–35.2%). The MSE ranged from 0.78 to 1.23 for random slope GLMM and 1.03 to 1.32 for the random intercept and slope GLMM. A larger between-study variance or smaller effect size tended to lead to a larger MSE. Both GLMMs’ coverages were over the nominal level of 0.95 in all of the scenarios. The random intercept and slope GLMM showed lightly higher bias, MSE, and coverage than the random slope GLMM. In terms of the model fitting, the random slope GLMM model had slightly lower AIC and BIC than the random intercept and slope model in all of the scenarios (Table S2, supplementary file 2).

Figure 1.

The statistical properties in terms of percentage bias, MSE, and coverage of the two GLMMs (Model 1: random slope GLMM. Model 2: random intercept and slope GLMM).

The impact of studies with no events for meta-analysis

Effect size

Figure 2 presents the impact of studies with no events on the effect size for meta-analysis. In total, when excluding studies with no events, there were 0.0% (OR = 5, tau = 0.2) to 4.47% (OR = 1, tau = 1) of the meta-analyses with ORs changed the direction based on the random slope GLMM. A smaller proportion (0.0%–2.47%) were observed in the random intercept and slope GLMM. The proportion varied in terms of the magnitude of effect size and between-study variance: those meta-analyses with larger effect size or smaller between-study variance tended to be more insensitive of the impact of studies with no events. There was almost no impact on the direction of OR when the true effect was large (e.g., OR = 4 or 5).

Figure 2.

The impact of excluding studies with no events in both arms on the direction of effect size and significance of p-value.

P-value

In total, when excluding studies with no events, there were as many as 8.78% of the meta-analyses (OR = 2, tau = 1) with the p-value changed the direction of significance based on the random slope GLMM and 7.33% based on the random intercept and slope GLMM. Again, those meta-analyses with smaller between-study variance tended to be more insensitive of the impact of studies with no events. Interestingly, we observed that an OR = 2 or three had the largest impact on the direction of p-value in both GLMMs (Figure 2).

Between-study variance

Table S3 (supplementary file 2) presents the comparison of the between-study variance, namely, ICC, of including versus excluding studies with no events. The results suggested that excluding studies with no events tend to slightly enlarge the between-study variance in all of the scenarios. However, we also observed that the ICC almost kept unchanged for the scenarios with different tau-squared. This implies that the between-study variance was seriously underestimated for meta-analysis of rare events.

Number of studies with no events in a meta-analysis and the impact on the results

We stratified meta-analyses into 9 groups by the proportion (0.01%–90%) of the number of studies with no events. Table S4 (supplementary file 2) presents the number of meta-analyses in each group. Although we set a large-scale simulation, we noticed in some situations that there were insufficient sample sizes (less than 100) for the analysis where the proportion was high and the effect size was large (e.g., 80.01%–90%). The results of these scenarios should be treated with caution.

Our results suggested that, for effect size, when more studies with no events involved in a meta-analysis, the impact of excluding them would be larger (Table 2, Table S5). In detail, when there was an increase in the proportion of studies with no events in a meta-analysis, there was a higher probability of the effect size (OR) changed the direction in all 25 scenarios. This was not the case for the direction of p-value on the significance (Table 3, Table S6). For p-value, we observed that in most of the situations, when the number of studies with no events accounted for 50% (centered around 40%–60%) in each meta-analysis, they tend to have the largest impact on the direction of p-value; however, we also observed that when OR = 2, which had largest impact by studies with no events, as the increase of the proportion of such studies, the impact of excluding them tend to be larger.

Table 2.

The proportion of OR changed the direction for including against excluding studies with no events in the 9 groups with different proportions of such studies (OR = 1 and 2).

Changes on the direction according to the 9 groups, OR = 1
Model 1	Total	Group 1	Group 2	Group 3	Group 4	Group 5	Group 6	Group 7	Group 8	Group 9	Model 2	Total	Group 1	Group 2	Group 3	Group 4	Group 5	Group 6	Group 7	Group 8	Group 9
Tau = 0.2	1.89%	0.00%	0.70%	1.20%	1.80%	1.96%	2.51%	2.30%	2.32%	3.59%	Tau = 0.2	1.78%	0.00%	0.76%	1.18%	1.57%	1.91%	2.33%	2.19%	2.19%	3.43%
Tau = 0.4	1.93%	0.11%	0.83%	1.31%	1.92%	2.11%	2.26%	2.30%	2.45%	3.01%	Tau = 0.4	1.77%	0.00%	0.86%	1.12%	1.56%	1.88%	2.30%	2.10%	2.51%	2.84%
Tau = 0.6	1.83%	0.20%	0.71%	1.13%	1.82%	1.93%	2.24%	2.53%	2.28%	3.21%	Tau = 0.6	1.69%	0.10%	0.66%	1.09%	1.59%	1.73%	2.21%	2.19%	2.35%	3.03%
Tau = 0.8	2.24%	0.00%	0.95%	1.46%	1.91%	2.26%	2.80%	3.43%	4.16%	3.97%	Tau = 0.8	1.75%	0.00%	0.74%	1.05%	1.63%	1.81%	2.33%	2.30%	3.14%	2.72%
Tau = 1	4.47%	0.00%	2.01%	2.86%	3.72%	4.18%	6.71%	8.07%	8.63%	9.67%	Tau = 1	2.47%	0.00%	0.87%	1.55%	1.98%	2.40%	4.11%	4.47%	4.29%	3.59%
Changes on the direction according to the 9 groups, OR = 2
Tau = 0.2	1.02%	0.02%	0.62%	0.81%	1.22%	1.66%	2.74%	3.86%	2.84%	4.08%	Tau = 0.2	0.97%	0.02%	0.61%	0.77%	1.13%	1.60%	2.69%	3.19%	2.84%	6.12%
Tau = 0.4	1.04%	0.00%	0.69%	0.68%	1.31%	1.87%	2.56%	3.39%	2.04%	1.92%	Tau = 0.4	0.95%	0.00%	0.60%	0.68%	1.16%	1.79%	2.26%	2.98%	1.81%	1.92%
Tau = 0.6	1.00%	0.02%	0.59%	0.86%	1.21%	1.48%	2.31%	3.98%	2.80%	6.98%	Tau = 0.6	0.97%	0.02%	0.58%	0.84%	1.12%	1.48%	2.27%	3.73%	2.37%	6.98%
Tau = 0.8	1.20%	0.02%	0.45%	0.86%	1.49%	2.04%	3.15%	4.68%	4.39%	5.45%	Tau = 0.8	0.89%	0.00%	0.35%	0.63%	1.04%	1.57%	2.39%	3.37%	3.51%	1.82%
Tau = 1	2.74%	0.00%	1.04%	2.03%	3.09%	4.92%	5.93%	9.53%	9.63%	13.43%	Tau = 1	1.40%	0.00%	0.54%	1.02%	1.62%	2.64%	3.09%	4.71%	3.48%	2.99%

Note: Model one is random slope GLMM, Model 2 is random intercept and slope GLMM.

Table 3.

The proportion of p-value changed the direction for including against excluding studies with no events in the 9 groups with different proportions of such studies (OR = 1 and 2).

Changes on the direction according to the 9 groups, OR = 1
Model 1	Total	Group 1	Group 2	Group 3	Group 4	Group 5	Group 6	Group 7	Group 8	Group 9	Model 2	Total	Group 1	Group 2	Group 3	Group 4	Group 5	Group 6	Group 7	Group 8	Group 9
Tau = 0.2	0.61%	0.00%	0.30%	0.87%	0.76%	0.83%	0.62%	0.37%	0.03%	0.00%	Tau = 0.2	0.40%	0.00%	0.18%	0.83%	0.49%	0.49%	0.39%	0.21%	0.03%	0.00%
Tau = 0.4	0.64%	0.00%	0.36%	0.82%	0.83%	0.90%	0.60%	0.39%	0.06%	0.00%	Tau = 0.4	0.42%	0.00%	0.24%	0.82%	0.45%	0.52%	0.39%	0.27%	0.03%	0.00%
Tau = 0.6	0.72%	0.00%	0.26%	1.07%	0.94%	0.89%	0.68%	0.47%	0.20%	0.00%	Tau = 0.6	0.52%	0.10%	0.43%	0.78%	0.58%	0.74%	0.41%	0.24%	0.07%	0.00%
Tau = 0.8	1.04%	0.00%	0.46%	1.27%	1.24%	1.29%	1.19%	0.84%	0.19%	0.21%	Tau = 0.8	0.76%	0.09%	0.59%	1.05%	0.93%	0.96%	0.69%	0.47%	0.11%	0.00%
Tau = 1	2.86%	0.23%	2.07%	3.23%	3.51%	3.36%	2.77%	2.49%	0.89%	0.28%	Tau = 1	2.29%	0.08%	1.85%	2.82%	2.77%	2.73%	1.75%	1.75%	1.21%	1.38%
Changes on the direction according to the 9 groups, OR = 2
Tau = 0.2	3.51%	0.12%	2.05%	3.78%	4.44%	5.71%	6.39%	7.98%	6.87%	10.20%	Tau = 0.2	4.22%	0.25%	2.22%	4.07%	5.38%	7.18%	9.02%	9.84%	11.14%	10.20%
Tau = 0.4	3.54%	0.25%	2.03%	3.83%	4.40%	5.43%	7.07%	7.45%	8.84%	5.77%	Tau = 0.4	4.33%	0.23%	2.47%	4.11%	5.37%	6.99%	9.27%	11.11%	13.61%	9.62%
Tau = 0.6	3.75%	0.17%	2.09%	3.74%	4.87%	5.96%	7.61%	7.71%	7.97%	2.33%	Tau = 0.6	4.54%	0.20%	2.73%	4.27%	5.69%	7.24%	9.88%	9.77%	9.91%	4.65%
Tau = 0.8	4.87%	0.16%	2.91%	4.45%	6.11%	7.72%	9.64%	10.45%	10.31%	5.45%	Tau = 0.8	5.44%	0.39%	3.23%	4.92%	6.86%	8.65%	11.04%	10.88%	9.87%	9.09%
Tau = 1	8.78%	0.21%	5.08%	8.30%	11.38%	13.48%	15.00%	16.05%	14.96%	10.45%	Tau = 1	7.33%	0.37%	4.54%	6.67%	9.14%	10.93%	13.72%	14.24%	13.73%	8.96%

As a result, when there was a low proportion of studies with no events in a meta-analysis (e.g., proportion≤10%), the impact of excluding studies with no events on the direction of OR and p-value was small (Tables S5 and S6). In this case, about 0.00%–0.20% meta-analyses would change the direction on OR and 0.00%–0.46% (0.25% in the random slope GLMM) would change the direction on p-value, regardless of the magnitude of effect size and between-study variance.

Discussion

We conducted a simulation study to investigate the impact of studies with no events in both arms for the results of meta-analysis. Our findings verified that studies with no events in meta-analysis have considerable impact on the results that may change the conclusions. Although this impact was generally small on effect size, it was large on p-value. The impact depends on the proportion of the number of such studies within a meta-analysis: a higher proportion has a larger impact on the effect size, while a proportion of about 50% tend to have the largest impact on the p-value. We further observed that for meta-analyses included studies with no events less than 10%, there would have the least impact on the direction of OR and p-value.

Studies with no events in both arms are commonly seen in meta-analyses of safety. Based on our ongoing research, there were 63.45% of the meta-analyses that with safety as exclusive outcome contains zero-events studies, and of which at least 50% contains studies with no events in both arms.³⁷ Such studies bring challenges for both statisticians and systematic review authors on how to deal with them. Currently, the routine approach is to simply remove them from the synthesis which simply treat them as “non-informative”.³⁸ Our study verified that such studies could not be treated as “non-informative” and the routine approach may be inappropriate that it’s time for a change.

In our simulation, when there was no true effect (OR = 1) or the true effect was small (i.e. OR ≤2), excluding studies with no events has considerable impact on the direction of OR. When the true effect was large (OR >3), studies with no events have limited impact on effect size. This can be expected because compared with a small effect that close to 1, it needs more “evidence” to push a large effect toward an opposite direction. However, these findings did not signify that studies with no events could be discarded when there was large effect. This is because excluding them is expected to change the between-study variance as well and further impacts the statistical inference.

We found that in most of the situations when studies with no events account for 50% of the numbers within a meta-analysis, it has the largest impact on the direction of p-value. This is because when the proportion is small, the weight of these studies is too weak to predominate the result; and when the proportion is large, these studies will however push the pooled effect to 1 as the expected OR of balanced studies with no events was 1, which make it hard to change the direction of p-value (when OR≈1, the p-value≈1 as well). Interestingly, we observed that moderate effect sizes (OR = 2 or 3) have the largest impact on the direction of p-value. One possible explanation is that, for meta-analyses with OR = 1 the p-value is large and converges to 1, and for meta-analyses with large effect (OR = 4 or 5) the p-value is small and converges to 0; both are less sensitive to the impact of excluding studies with no events on the direction of p-values. However, it is hard to explain that when OR = 2, the impact on the direction p-value enlarged as the proportion of studies with no events increased. Even so, it should be noted that for meta-analysis of rare events, the value of p-value for the inference is limited because of the extremely low power.^39,40

These findings from this simulation concurs with our previous study.²⁶ where we observed that, based on empirical data modeled with the random intercept GLMM, when excluding studies with no events, for those meta-analyses changed the direction of p-values the proportion of studies with no events within a meta-analysis mainly clustered in 5%–30%. For meta-analyses that changed the direction of OR, the ORs mainly clustered in 0.84–1.04²⁶ The current simulation employed two random effect GLMMs to verify these findings. Based on our previous study and the current one, we found that the random effect GLMMs were less sensitive on the impact of excluding studies with no events for effect size than the random intercept GLMM, while comparable sensitive for p-values. Both these findings reinforce that studies with no events in both arms are not necessarily non-informative and should not be simply excluded from the synthesis.^26,41,42

The current study verified the properties of GLMMs for pooling studies with no events in meta-analysis. One of the predominant advantages for GLMMs for meta-analysis is that they allow the estimation of the effect of studies with no events without requiring a priori assumption or post hoc correction. When the total events in both arms are sufficient (i.e., ≥10 ^33,43), the model would achieve a good estimation. In our previous simulation, we observed a very small percentage bias for GLMMs.²⁶ However, when the total events are small in either of the arms, the performance of GLMMs deteriorates due to the “small sample bias” of likelihood estimation. Researchers have proposed several solutions to this bias, for example, the penalized likelihood.⁴⁴ However, these methods have been verified to be not superior to the standard likelihood estimation.⁴⁵ and the GLMMs would make no sense when the total events in either of the groups were zero. The beta-binomial model may be promising solutions for meta-analyses with insufficient number of total events.^{16,17,38,46,47} Due to the complexity of such meta-analyses, further guidance on how to deal with studies with no events is needed.

Strength and limitations

As far as we know, this is currently the largest simulation study for meta-analysis of rare events. Even so, the results of current simulation study should be explained with caution due to one major limitation, that is, the “events” of change the direction of OR and p-value are rare. Although we used as more iterations as possible to increase the total sample size (50,000 times), the “events” in each subgroup are rare (Table S4). This is especially in those groups with a high proportion of studies with no events, with large effects, and those with large pre-setting heterogeneity. Dividing the total sample into 9 groups facilitate us to explore how studies with no events impact the conclusions, however also brings the risk of large random errors that make the estimation less accurate. Maybe a simulation of 5,000,000 interactions based on high computational speed computer in the future is helpful to verify the current findings.

Conclusions

Based on our simulation, studies with no events contain information and have considerable impact on the results of a meta-analysis; the impact varies by the proportion of numbers of studies with no events. Excluding these studies is inappropriate and may lead to misleading conclusions and should be avoided in the future. The GLMMs could be a potential solution to combine studies with no events in meta-analysis.

Supplemental Material

Supplemental Material - The impact of studies with no events in both arms on meta-analysis of rare events: A simulation study using generalized linear mixed model

Supplemental Material for The impact of studies with no events in both arms on meta-analysis of rare events: A simulation study using generalized linear mixed model by Chang Xu and Lifeng Lin in Research Methods in Medicine & Health Sciences

Footnotes

Acknowledgements

None

Author’s contributions

CX conceived and designed the study, analyzed the data, and drafted the manuscript; CX and LL interpreted the results; LL provided careful comments on the statistical methods; all authors contributed careful edits for the manuscript. All authors approved the final version.

Declarations of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The current study was supported by the National Natural Science Foundation of China (72204003) and three funding bodies from Anhui Medical University (9021783201, 0301001882, and 0301035204). .

Availability of data and material

Available at:

ORCID iDs

Chang Xu

Lifeng Lin

Supplemental Material

Supplemental material for this article is available online.

References

Murad

Montori

. Synthesizing Evidence. JAMA 2013; 309(21): 2217–2218.

Doi

Barendregt

Khan

, et al. Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model. Contemp Clin Trials 2015; 45(Pt A): 130–138.

Gurevitch

Koricheva

Nakagawa

, et al. Meta-analysis and the science of research synthesis. Nature 2018; 555(7695): 175–182.

Shadish

Lecy

. The meta-analytic big bang. Res Synth Methods 2015; 6(3): 246–264.

Hedges

Tipton

Johnson

. Robust variance estimation in meta-regression with dependent effect size estimates. Res Synth Methods 2010; 1(1): 39–65.

Efthimiou

Mavridis

Nikolakopoulou

, et al. A model for meta-analysis of correlated binary outcomes: the case of split-body interventions. Stat Methods Med Res 2019; 28(7): 1998–2014.

Rücker

Schwarzer

Carpenter

, et al. Why add anything to nothing? the arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Stat Med 2009; 28(5): 721–738.

Sweeting

Sutton

Lambert

. What to add to nothing? use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med 2004; 23(9): 1351–1375.

Bradburn

Deeks

Berlin

, et al. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med 2007; 26(1): 53–77.

10.

Shuster

Jones

Salmon

. Fixedvs random effects meta-analysis in rare event studies: The Rosiglitazone link with myocardial infarction and cardiac death. Stat Med 2007; 26(24): 4375–4385.

11.

Mathes

Kuss

. A comparison of methods for meta‐analysis of a small number of studies with binary outcomes. Res Synth Methods 2018; 9(3): 366–381.

12.

Bhaumik

Amatya

Normand

, et al. Meta-analysis of rare binary adverse event data. J Am Stat Assoc 2012; 107(498): 555–567.

13.

Kuss

. Statistical methods for meta-analyses including information from studies without any events-add nothing to nothing and succeed nevertheless. Stat Med 2015; 34(7): 1097–1116.

14.

Xie

Kolassa

Liu

, et al. Does an observed zero-total-event study contain information for inference of odds ratio in meta-analysis. Stat Interface 2018; 11: 327–337.

15.

FinkelsteinFinkelstein

Levin

. Observation. Colo L Rev 2012; 39: 123–123.

16.

Kuss

Hoyer

Solms

. Meta-analysis for diagnostic accuracy studies: a new statistical model using beta-binomial distributions and bivariate copulas. Stat Med 2014; 33(1): 17–30.

17.

Chu

Nie

Chen

, et al. Bivariate random effects models for meta-analysis of comparative studies with binary outcomes: methods for the absolute risk difference and relative risk. Stat Methods Med Res 2012; 21(6): 621–633.

18.

Yang

Liu

Wang

, et al. Meta‐analysis framework for exact inferences with application to the analysis of rare events. Biometrics 2016; 72(4): 1378–1386.

19.

Liu

Xie

. Exact meta-analysis approach for discrete data and its application to 2 × 2 tables with rare events. J Am Stat Assoc 2014; 109(508): 1450–1465.

20.

Ren

Lin

Lian

, et al. Real-world performance of meta-analysis methods for double-zero-event studies with dichotomous outcomes using the cochrane database of systematic reviews. J Am Stat Assoc 2019; 34(6): 960–968.

21.

Stijnen

Hamza

Özdemir

. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat in Med 2010; 29(29): 3046–3067.

22.

Simmonds

Higgins

JPT

. A general framework for the use of logistic regression models in meta-analysis. Stat Methods Med Res 2016; 25: 2858–2877.

23.

Riley

Price

Jackson

, et al. Multivariate meta‐analysis using individual participant data. Res Synth Methods 2015; 6(2): 157–174.

24.

Debray

Moons

Valkenhoef

GetReal Methods Review Group , et al. Get real in individual participant data (IPD) meta‐analysis: a review of the methodology. Res Synth Methods 2015; 6(4): 293–309.

25.

Jackson

Law

Stijnen

, et al. A comparison of seven random-effects models for meta-analyses that estimate the summary odds ratio. Stat in Med 2018; 37(7): 1059–1085.

26.

Lin

, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol 2020; 123: 91–99.

27.

Pateras

Nikolakopoulos

Roes

. Data-generating models of dichotomous outcomes: Heterogeneity in simulation studies for a random-effects meta-analysis. Stat in Med 2018; 37(7): 1115–1124.

28.

Burton

Altman

Royston

, et al. The design of simulation studies in medical statistics. Stat in Med 2006; 25: 4279–4292.

29.

MorrisMorris

White

Crowther

. Using simulation studies to evaluate statistical methods. Stat in Med 2019; 38: 2074–2102.

30.

Lin

Chu

Murad

, et al. Empirical Comparison of Publication Bias Tests in Meta-Analysis. J Gen Intern Med 2018; 33(8): 1260–1267.

31.

Lin

, et al. Performance of between-study heterogeneity measures in the cochrane library. Epidemiology 2018; 29: 821–824.

32.

Cheng

Cai

, et al. The use of one-stage meta-analytic method based on individual participant data for binary adverse events under the rule of three: a simulation study. PeerJ 2019; 7: e6295.

33.

Lin

Chu

, et al. Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data. BMC Med Res Methodol 2020; 20(1): 152.

34.

. Use of generalized linear mixed models for network meta-analysis. Medical Decis Mak 2014; 34: 911–918.

35.

Vrieze

. Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychol Methods 2012; 17(2): 228–243.

36.

Hemming

Hughes

McKenzie

, et al. Extending the I-squared statistic to describe treatment effect heterogeneity in cluster, multi-centre randomized trials and individual patient data meta-analysis. Stat Methods Med Res 2021; 30(2): 376–395.

37.

Zhou

Zorzela

, et al. Utilization of the evidence from studies with no events in meta-analyses of adverse events: an empirical investigation. BMC Medicine 2021; 19(1): 141.

38.

Furuya‐Kanamori

Lin

. Synthesis of evidence from zero‐events studies: A comparison of one‐stage framework methods. Res Synth Methods 2022; 13: 176–189. DOI: 10.1002/jrsm.1521

39.

Jackson

Turner

. Power analysis for random-effects meta-analysis. Res Synth Methods 2017; 8(3): 290–302.

40.

Jia

Lin

Kwong

JSW

, et al. Many meta-analyses of rare events in the Cochrane Database of Systematic Reviews were underpowered. J Clin Epidemiol 2021; 131: 113–122.

41.

Furuya-Kanamori

Zorzela

, et al. A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies. J Clin Epidemiol 2021; 135: 70–78.

42.

Xiao

Lin

Hodges

, et al. Double-zero-event studies matter: a re-evaluation of physical distancing, face masks, and eye protection for preventing person-to-person transmission of COVID-19 and its policy impact. J Clin Epidemiol 2021; 133: 158–160.

43.

Wynants

Bouwmeester

Moons

, et al. A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data. J Clin Epidemiol 2015; 68(12): 1406–1414.

44.

Benedetti

Platt

Atherton

. Generalized linear mixed models for binary data: are matching results from penalized quasi-likelihood and numerical integration less biased? PLoS One 2014; 9(1): e84601.

45.

Thomas

Platt

Benedetti

. A comparison of analytic approaches for individual patient data meta-analyses with binary outcomes. BMC Med Res Methodol 2017; 17(1): 28.

46.

Bakbergenuly

Kulinskaya

. Beta-binomial model for meta-analysis of odds ratios. Stat in Med 2017; 36(11): 1715–1734.

47.

Mathes

Kuss

. Beta-binomial models for meta-analysis with binary outcomes: variations, extensions, and additional insights from econometrics. Stat Methods Med Res 2021; 2(2): 82–89.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB