Meta-analysis Framework as a Form of Original Research

Abstract

This article explains that systematic reviews with meta-analysis (SRMAs) resemble original research in many ways, and should therefore be considered as such in journal listings, for academic appointments and promotions, and for consideration for awards. SRMAs are best conceptualized as a specialized form of original research.

Original research or an original article (OA), where a researcher begins with an a priori research question, conceptualizes the study protocol to answer this question, collects and analyzes data, interprets the findings, and finally, prepares and publishes a manuscript, is traditionally considered the most essential and respected form of research activity. Indeed, this is reflected in academic requirements for career advancement and obtaining tenured positions; candidates are typically required to show their publication output, and OAs are prioritized in these requirements, ¹ though other forms of research are now also permitted by a recent amendment. ²

Discussion

The publications mandate has led to a burgeoning output over the last few decades, creating challenges for academicians who wish to stay up to date with the growing research landscape. Review articles provide one solution to this issue; they summarize the existing literature and aim to provide a balanced view of the findings, with all assertions supported by relevant and adequate citations. Review articles save time for busy readers who might otherwise spend several hours combing through literature search results to find pertinent articles and to read and assimilate their findings. However, there is no guarantee that all relevant articles will be cited in a narrative review article, which presents no systematic effort to search, screen, and shortlist articles of interest. Thus, the subjective selection of articles in a narrative review may introduce bias in the conclusions of the review. ³

This gap is addressed by systematic reviews. These reviews use search strategies and protocols that are formulated and documented a priori and executed across multiple electronic databases, resulting in a systematic process for identifying articles, extracting data, assessing risk of bias, and drawing conclusions. Systematic review conclusions are deemed more reliable and comprehensive than narrative review conclusions because they emerge from a rigorous, transparent, and reproducible method for synthesizing research evidence. Indeed, following publications by Cochrane and Chalmers, ⁴ and the advent of the Cochrane Collaboration (https://www.cochrane.org/), systematic reviews have gained widespread popularity since the turn of the century.

Systematic reviews, however, have limitations. For instance, consider a clinician who wishes to know whether adjunctive treatment with N-acetyl cysteine (NAC) is efficacious in improving negative symptoms in schizophrenia and, if yes, what the pooled effect size is. Imagine that in three randomized controlled trials (RCTs), NAC is significantly superior to placebo, and in five RCTs it is no better than placebo; how would a systematic review answer the clinical question? Using one’s judgment results in subjective conclusions. Counting studies (e.g., five minus three) is no better because all studies are treated as being equal. A simple mathematical averaging of findings across these trials is not a correct approach to estimate the pooled effect size, because the contribution of each RCT to the pooled estimate should be based on factors such as study sample size and the precision of the findings, in addition to the actual findings. The clinical question can therefore be answered only if the systematic review includes a meta-analysis (MA).

What is a Meta-analysis?

The term MA, coined by Glass in 1976, ⁵ refers to a mathematical procedure wherein effect size measures from individual studies are statistically combined and summarized to obtain a pooled estimate. The procedure “weights” studies; thus, larger studies with more precise estimates contribute more to the overall result. ⁶ The MA is usually nested within a systematic review and performed if the studies selected through the systematic review are judged suitable for statistical aggregation of individual study estimates. Readers will now understand that not all systematic reviews may have an MA component. Therefore, there can be two types of systematic reviews: Those with and those without an accompanying MA. Readers may note that the focus of this article is the systematic review with MA (SRMA); our arguments do not apply to systematic reviews without an accompanying MA.

Is SRMA Original Research?

One way to gauge how the scientific community views an SRMA vis-à-vis OA is to examine how scientific journals publish SRMAs: As OA or in a separate section, such as under reviews. On the one hand, most of the high-ranking journals, such as the Journal of the American Medical Association (JAMA), ⁷ and JAMA network journals such as JAMA Cardiology, ⁸ and JAMA Psychiatry, ⁹ The Lancet, ¹⁰ and its associated group of journals, ¹¹ The New England Journal of Medicine,^12,13 The British Medical Journal (BMJ), ¹⁴ and the BMJ group of journals, ¹⁵ publish SRMAs as original research supporting the abovementioned arguments. This is reflected in their editorial policies as well.^16–18

On the other hand, a few high-ranked journals, ¹⁹ such as Annals of Internal Medicine, ²⁰ and more specifically to psychiatry, Acta Psychiatrica Scandinavica, ²¹ Molecular Psychiatry, ²² Journal of Psychopharmacology, ²³ Indian Journal of Psychiatry, ²⁴ and the Indian Journal of Psychological Medicine, ²⁵ publish SRMAs as reviews, and not OAs .

These differences point to a division among publishers and call for a critical analysis of the similarities and differences between SRMAs and OAs for two reasons. First, because some scientific journals do not publish SRMA articles as OA, researchers may be discouraged from pursuing MA investigations, as these may not count toward the number of “original” articles they need to meet their selection and promotion criteria. This could have negative implications for science, practice, and policy, as SRMAs are important for guideline makers, policymakers, practitioners, and funding agencies to make evidence-based decisions. One of the earliest and most impactful examples of an SRMA leading to a change in practice is the cumulative MA of treatment trials in acute myocardial infarction (AMI) by Lau and colleagues. ²⁶ It showed clear benefits of thrombolytic therapy and catalyzed the adoption of thrombolysis as the standard of care in AMI. SRMAs also help identify knowledge gaps, which can then be addressed through new research.

Second, the debate on whether an SRMA constitutes original research may undermine the value of a type of research that, when conducted rigorously, can clarify what works, what does not, and the extent to which the available evidence can be trusted. This is particularly important in disciplines such as psychiatry, where disorders and treatment response are heterogeneous and driven by multiple genetic and environmental factors. ²⁷

As an additional note, in OAs, research subjects may include humans, animals, bacteria, tissue cultures, or other organisms. In SRMAs, research subjects are research articles with quantitative data. In other regards, however, OAs and SRMAs are structurally similar. These are discussed in the next section.

SRMAs Are like OAs

SRMAs Require A Priori Registration in a Review Registry

It is now widely accepted that any clinical trial must be prospectively registered in a clinical trials registry. Non-registered trials are difficult to publish; editors are often unwilling to accept them. Registration of a trial is meant to promote transparency, reduce protocol deviations and resultant bias, prevent duplication of effort, and promote replicability. For the same reasons, SRMA researchers are encouraged to prospectively register their systematic review protocols with registries such as the International Prospective Register of Systematic Reviews (https://www.crd.york.ac.uk/prospero/). The prospective nature of the registration is highlighted because the site requires a self-declaration that data extraction has not yet begun at the time of registration. This is similar to the declaration asked for when registering a clinical trial on a registry, stating that the first patient will be recruited only after the trial registration is complete.

SRMA Outcomes Are Framed A Priori

As with an OA, an SRMA starts with an a priori hypothesis, tests it, and provides an evidence-based answer. For example, an SRMA of psychedelic-assisted therapies (PAT) can test the hypothesis that PAT is superior to control therapies for reducing depression symptom severity (primary outcome), and for improving response and remission rates (secondary outcomes). Furthermore, when several hypotheses are tested, statistical correction for Type 1 errors may be applied, as in OAs. ²⁸ As a side note, SRMA articles do not always state their hypotheses explicitly; this is also true for OAs.

SRMA Study Methods Are Outlined A Priori

This distinguishes a narrative review from a systematic review and links SRMAs with OAs. The use of a common, reproducible, and specific methodology to limit bias and random error is why SRMAs carried out by the Cochrane Collaboration are widely accepted as the highest level of evidence in evidence-based health care. As with OAs, SRMAs typically have a well-structured methods section, formulated a priori. The methodology in SRMA papers includes subsections for different components, such as search strategy, study selection criteria (inclusion and exclusion), data extraction, risk of bias appraisal, and a statistical analysis plan outlining the methods and models used to synthesize data, with details of subgroup and sensitivity analyses. ²⁹ As with OAs, the design, execution, and analysis of SRMAs is a team effort with elements that require training, expertise, judgment, and time. As with OAs, some meta-analyses may take years to formulate, execute, and complete.

SRMAs Present Original Results

SRMAs are not reviews that qualitatively interpret studies in the field. As with OAs, SRMAs focus on data acquisition and analysis. The result of an MA is typically a pooled estimate with a confidence interval, obtained by statistical aggregation of effect sizes from individual studies. The pooled estimate thus obtained represents an original finding that has not been reported in prior literature, akin to the findings from an OA. Readers may note that, unlike a systematic review, which merely summarizes the findings of included studies, an SRMA goes beyond summarizing the original findings; it involves a new synthesis using statistical methods to combine data from included studies. This leads to new findings that can be interpreted to support a more generalizable understanding of the phenomenon being examined. This is an effective counter to the argument that SRMAs lack originality or fail to generate novel findings comparable to those of primary research.

As an additional note, the findings of an MA are more generalizable than those of individual studies because the included studies would have drawn participants from a broader range of populations and settings; this is an advantage of SRMAs over OAs. As a final note, similar to OA, the statistical techniques employed in an SRMA are constantly evolving. Currently, complex and unique analytical methods are used in several types of advanced meta-analyses, such as individual participant data MA, network MA, and Bayesian approaches to MA. Last but not least, SRMAs are verifiable and replicable. SRMAs are not opinions; they are based on verifiable and replicable processes, as OAs are.

SRMAs Versus OAs

The two differ in some regards. However, the differences often pointed out are not as substantive as might be believed. We consider these below.

The Units of Analysis Differ

The unit of analysis in an OA is usually a single human, animal, bacterium, or other organism. In an SRMA, the unit of analysis is an eligible study. This, by itself, does not imply that SRMAs must be viewed differently from OAs. In fact, there are several types of OA where the unit of analysis differs. For instance, in a couple therapy study, the unit of analysis is a couple or dyad. The household is the analytic unit in research examining household-level interventions. In a cluster RCT, the unit of analysis may be a hospital, ward, school, or an entire village. All of these are still OAs; we do not label them any differently merely because the unit of analysis differs.

The Nature of Data Differs

An OA collects new, primary data from participants or patients, whereas an SRMA uses existing, published data from eligible studies. The data collection in OA may be prospective or retrospective, whereas an SRMA collects data retrospectively from public sources.

The Data May Be Biased and Subject to Other Limitations

The sources of bias in an original study may include factors such as selection bias (sample is not adequately representative of the target population), information bias (such as measurement bias, when the measured value is systematically different from the “true” value), and attrition bias (participants who drop out systematically differ from those who remain in the study). ³⁰ Sources of bias unique to SRMAs include between-study heterogeneity, publication bias, and selective reporting. ³¹

However, there are similarities in the sources of bias as well. For instance, selection bias may influence the results of an SRMA if the search did not identify all available data on a topic. Similarly, the results of an SRMA may be impacted by measurement bias if the (primary) researchers did not define study variables consistently or accurately.

The findings of SRMAs and OAs differ with respect to the generalizability vs. specificity of the evidence. Original research findings are context-specific about matters such as sample selection and study setting. In contrast, the SRMA results are more generalizable because the studies included exhibit diverse characteristics. ³²

Finally, recognizing meta-analyses as original research can positively impact academic credit, funding, and research training. It will allow authors to receive recognition and credit similar to those of empirical researchers publishing primary research. More specifically, it may boost career advancement prospects for early-career researchers who may lack access to sufficient patient data, research funding, or high-end laboratory equipment required for impactful primary research but can still generate high-impact publications through rigorous evidence synthesis. It will also augment research training by advancing evidence appraisal skills, which are essential for both practitioners and academicians.

As a final note of caution, the increasing use of artificial intelligence (AI) tools, while improving efficiency, may simplify the SRMA process to the point of compromising methodological rigor and leading to questionable validity of the conclusions drawn. Readers may note that this concern extends to primary research as well, where AI may be used in areas such as drug discovery, medical imaging, early diagnosis of disorders, and data analysis, interpretation, and manuscript preparation. When done without adequate oversight, this may introduce bias into findings and errors into conclusions.

Conclusions

SRMAs are not narrative reviews. SRMAs focus on data acquisition and analysis. SRMAs are similar to OAs in many respects, including prior registration, prior formulation of primary and secondary outcomes, prior statement of study methods, prior outline of the analysis plan, need for a trained and skilled team of investigators, need for systematic data extraction and appropriate analysis, and presentation of original findings. As with OAs, SRMAs are intended to be replicable. SRMAs are best conceptualized as specialized research designs within the rubric of original research. Based on our arguments, we encourage academic journals and institutions to consider SRMAs as original research for academic publications and promotions, and for consideration of awards.