Levels of Evidence in Rhinology and Skull Base Surgery Research

Abstract

Objective

The purpose of this study was to evaluate the quality of evidence of rhinology and rhinologic skull base surgery (RSBS) research and its evolution over the past decade.

Study Design

Review article.

Setting

We reviewed articles from 2007 to 2019 in 4 leading peer-reviewed otolaryngology journals and 3 rhinology-specific journals.

Methods

The articles were reviewed and levels of evidence were assigned using the Oxford Centre for Evidence-Based Medicine 2011 guidelines. High quality was defined as level of evidence 1 or 2.

Results

In total, 1835 articles were reviewed in this study spanning a 13-year period. Overall, the absolute number of RSBS publications increased significantly 22.6% per year, from 108 articles in 2007 to 481 in 2019 (P < .001; 95% CI, 7.9-37.2). In 2007, only 13 articles, or 15%, were high quality, and this grew to 146 articles, or 39%, in 2019. A 14.0% per year exponential increase in the number of high-quality publications was found to be statistically significant (P < .001; 95% CI, 7.2, 20.7). Overall, high-quality publications represented just 25.8% of RSBS articles overall. There was no significant difference in quality between rhinology-specific journals and general otolaryngology journals (χ² = 3.1, P = .077).

Conclusion

The number of overall publications and of high-quality RSBS publications has significantly increased over the past decade. However, the proportion of high-quality studies continues to represent a minority of total RSBS research.

Every 7 years, the volume of medical literature and research published doubles with approximately 20,000 medical papers published each week.^1,2 In theory, physicians can incorporate the latest data to improve their practice, but the sheer volume of research presents a practical problem. Clinicians must develop strategies to classify the information by relevance and strength of evidence. This screening process is imperative as it is this high-level evidence combined with clinician knowledge and patient preference that will ultimately translate to meaningful patient care. ³

Practicing evidence-based medicine (EBM) allows physicians to filter through the available published literature and use the strongest evidence to make appropriate decisions for individual patient care. ³ Although a greater number of scientific articles are being published, there is concern that the quality of the research remains poor. Within the field of general otolaryngology, there has been an explosion in the number and quality of published research. ⁴ However, within both rhinology and skull base surgery (RSBS) literature, the quality of data has not yet been fully explored.

The goal of the study is to identify research trends within RSBS. We will ascertain the growth within this subspecialty and rate the quality, including level of evidence, of the articles being published over the past decade. We hypothesize that there is an increasing number of journal articles in RSBS being published each year and that the overall quality of the publications is improving.

Methods

This study was reviewed by the institutional review board (IRB) of the Faculty of Medicine at McGill University. It is exempt from ethics review as all information was publicly available.

Study Design

We identified the 4 highest circulating peer-reviewed otolaryngology journals, as per Wasserman et al, ⁵ and 3 rhinology-focused journals as per the Web of Science published impact factors in 2015. These journals included the Annals of Otology, Rhinology, and Laryngology (AORL); Archives of Otolaryngology–Head & Neck Surgery (now retitled as JAMA Otolaryngology–Head and Neck Surgery); The Laryngoscope; and Otolaryngology–Head and Neck Surgery, as well as International Forum of Allergy and Rhinology (IFAR), American Journal of Rhinology and Allergy (AJRA), and Rhinology. Each journal was reviewed from 2007 to 2019. IFAR was established only in 2011 and contributed articles to our data set from this point to the end of the study. All studies relevant to rhinology and to skull base surgery were included ( Figure 1 ). Using the Oxford Centre for Evidence-Based Medicine 2011 Levels of Evidence (OCEBM), each article was classified by its level of evidence to determine the quality of the publication. ⁶

Figure 1.

Systematic review literature review process.

Search Strategy

The search terms used were derived from the American Rhinologic Society, Rhinology and AJRA online manuscript submission forum, to determine relevance to rhinology. Key words are included in Supplementary Appendix 1 (in the online version of the article). The article’s title, abstract, and key words were screened for relevance by 2 independent reviewers. Articles discussing workshops, symposium reports, regular journal features (“How I Do It,”“Clinical Photographs,”“Case of the Month,”“Resident Pathology Page,”“Imaging Case of the Month,” and “Clinicopathological Consultation”), editorials, letters to the editor, general interest articles (book reviews, historic vignettes, biographies), and brief communications of drugs, devices, or instruments were excluded. Articles were subsequently classified as clinical or nonclinical based on studies involving patient care or if the study focus was then correlated with patient care, patient history, or disease state. Nonclinical articles included those of animal or laboratory studies, postmortem studies that do not meet clinical requirements, questionnaires, and questions of manpower, education, cost-effectiveness, or quality control.

Parameters

Information compiled for all articles included classification of clinical vs nonclinical, publication within a rhinology-specific journal vs otolaryngology journal, and the topic of rhinology vs skull base surgery. For clinical research, each article was given its OCEBM level of evidence score according to study focus and study methodology. Levels of evidence ranged from 1 (highest quality) to 5 (lowest quality). High-quality evidence was defined as levels 1 and 2. The OCEBM grading scheme was applied to this study, as it was originally developed to evaluate clinical questions. It classifies articles by type of foreground question and then type of study methodology. As opposed to other grading schemes, the OCEBM is simple to use and provides a single number for efficient categorization of article quality.

Statistical Analysis

To ensure interrater reliability of OCEBM scoring, authors (J.A.S., G.T., and J.S.S.) reviewed articles independently. J.A.S. and G.T. independently reviewed 120 articles within the data set to further confirm this agreement. Minor discrepancies were discussed and resolved to establish a consensus. The senior author verified all discordant articles after the joint reviewing process. Percentage agreement and Cohen’s κ coefficient were calculated to measure interrater agreement. Means and 95% CIs were used to describe descriptive statistics. Chi-square testing was used to compare frequency data among the different categories. Data were analyzed using R software (version 3.0; R Foundation for Statistical Computing) using libraries GLM and DescTools.

Results

In total, 11,175 articles from 7 journals were reviewed for this study. After exclusion criteria, 1835 articles were included in our study ( Figure 1 ). Percent agreement among authors was 84%, and Cohen’s κ coefficient was 0.64, representing substantial agreement.

Quantity

A total of 1835 RSBS publications were included from 2007 to 2019. In 2007, 108 RSBS articles were reviewed, and this increased to 481 in 2019. The number of RSBS articles overall increased by 22.6% per year, or 56.5 new articles per year, and is statistically significant (P < .001; 95% CI, 7.9-37.2; Figure 2 ).

Figure 2.

Growth of articles and annual breakdown: exponential increase of total number of articles over study period. A 22.6% increase in overall number of articles published per year. Table with the breakdown of journal contributions of articles.

Quality

Overall, 25.8% of articles reviewed during the study period (2007-2019) were found to be high quality. In 2007, only 15.1% of articles were of high quality, whereas 39.3% of articles were high quality at the conclusion of the study period. There was a significant exponential increase in the proportion of high-quality publications at approximately 14.0% per year (95% CI, 7.2-20.7; P < .001; Figure 3 ).

Figure 3.

Proportion of high-quality articles: high-quality publication growth. Proportion of high-quality articles increased from 15.1% to 39.3% (blue). Absolute number increased from 13 to 146 (red). In total, 25.8% are high quality overall.

Trends

Only 7.9% of articles initially reviewed from general otolaryngology journals were included in our study, while 48.6% of publications within rhinology-specific journals were included. After the screening process, high-quality publications represented 25.8% of total reviewed clinical articles. In total, 30.1% of articles from general otolaryngology journals were high quality and 34.5% of publications in rhinology-specific journals were high quality. There was no significant difference in the proportion of high-quality articles in rhinology-specific journals (χ² = 3.1, P = .077; Figure 4 ).

Figure 4.

Rhinology vs skull base: high-quality articles within rhinology (red) and within general otolaryngology journals (blue), no significant difference between journal types (χ² = 3.1, P = .077).

There have been a greater number of clinical publications vs nonclinical articles between 2007 and 2019. In total, there were 1443 clinical and 392 nonclinical studies reviewed. The number of clinical articles overall increased by 22.2% per year and is statistically significant (P < .001; 95% CI, 8.85%-35.5%).

Within the clinical studies, 1248 articles were focused on rhinologic diseases and 195 were focused on skull base pathologies ( Figure 5 ). Of the rhinology subset, 432 (34.6%) were high-quality articles, and within the skull base articles, 40 (20.5%) were high quality. A significantly higher proportion of high-quality papers have been published about rhinologic diseases as compared to skull base surgery topics (χ² = 15.2, P < .001).

Figure 5.

Rhinology vs skull base article growth: rhinology (red) vs skull base surgery (blue). In total, 1248 articles were focused on rhinologic diseases and 195 were focused on skull base pathologies. Total number of clinical articles (green).

Discussion

This study aimed to assess the progression of research within the field of RSBS by evaluating the quantity of published literature and the quality of these studies. The quantity of RSBS publications has increased over the past decade. While the proportion of these publications deemed to be high quality still makes up a minority of the research overall, representing 25.8%, there was statistically significant exponential growth observed from 2007 to 2019.

It can be noted that once screened for relevance to RSBS, there are a greater number of high-quality articles within rhinology-specific journals as compared to general otolaryngology journals. As well, there is greater emphasis on clinically focused research compared to nonclinical publications.

As a benchmark for quality of evidence research trends in the field of rhinology, our findings highlight that we are on par with previous studies. Results from Wasserman et al ⁵ were very similar in their determination that 31% of general otolaryngology research was deemed high quality. However, it should be noted that their study reviewed articles from 1993 to 2003, and in the 13-year time span that this study covered, there is no change in the overall quality. The improvement in research quality within RSBS is nonetheless an improvement compared to the findings of similar studies conducted in other otolaryngology subspecialties. Xu et al ⁷ evaluated quality of evidence research trends within the field of facial plastic surgery using the OCEBM grading scheme as well. An improvement over the 10-year study period was observed (3.8% in 1999 compared to 8.3% in 2008), but these figures are considerably lower than those observed in the RSBS literature.

There are several limitations of this study that are worth mentioning. Research performed by Chang et al ⁸ determined that a bias exists that is inherent to evaluating the level of evidence of a publication as it is heavily based on study type. ⁹ Within the OCEBM framework, randomized controlled trials are high quality, whereas case series, case report, review, or expert opinion publications are of lower quality. In surgical subspecialties, there are often barriers to the implementation of randomized trials, making them less frequently performed. In 2000, it was reported that only 3% to 9% of clinical study designs are randomized trials. ¹⁰ Within the field of otolaryngology, randomized controlled trials only represented 3.3% of articles from 2011 to 2013. ¹¹ McCulloch et al ¹² explored the underlying reasons for the lack of randomized controlled trials in the surgical literature, and explanations that were cited included urgent situations, surgical learning curves, and patient hesitancy, among others. In an emergency, obtaining informed consent is difficult, and in these fast-paced and stressful environments, it may be impossible to randomize the techniques due to surgeon availability and skill, as well as the priority of life-sustaining measures. Surgeons also improve their surgical technique over an individual learning curve, which can bias studies that randomize patients between a familiar and a new procedure. Patients may hesitate to participate in trials as they do not want chance to decide their treatment or if the potential adverse effects may be irreversible. Some suggestions from their study included continuous quality control evaluation to determine appropriateness of randomized controlled trials, measurement and tracking of learning curves and variations of techniques, and a nonrandomized initial phase of trials to determine study feasibility, study design, and patient cooperation. ¹²

An additional limitation of our study includes potentially overlooking high-quality RSBS research, which may have been published in high-quality nonotolaryngology journals that were excluded in our publication search. For example, the Journal of Allergy and Clinical Immunology is a common journal to which rhinologists submit and publish high-quality studies. As the journal has an impact factor of 12.485 in 2015, this is a significantly stronger journal than the highest impact factor journal that we included (Archives of Otolaryngology–Head & Neck Surgery, which had an impact factor of 3.502).

Skull base surgery represented a minority of publications reviewed in this study. This is likely reflective of the journal inclusion criteria, as a skull base surgery–specific journal was not included. The number of high-quality skull base surgery studies was also very low, possibly because of the limited number of randomized controlled trials in the field. ¹³

Furthermore, while the OCEBM grading scheme uses study question type and study design, it does not monitor methodology specifically. Research may therefore appear to be of higher quality despite methodologic limitations such as improper randomization, deficient blinding, or inadequate follow-up, all criteria not assessed within the OCEBM framework. ⁵

Conclusion

Our review of the RSBS literature over the past 13 years indicates that there has been growth in quantity and quality of published literature. Furthermore, while high-quality publications still comprise a minority of the published RSBS literature, there was a period of statistically significant growth. This review provides an opportunity to appraise the subspecialty of rhinology to understand the field’s current state and trajectory.

Supplemental Material

sj-docx-1-oto-10.1177_0194599820987131 – Supplemental material for Levels of Evidence in Rhinology and Skull Base Surgery Research

Supplemental material, sj-docx-1-oto-10.1177_0194599820987131 for Levels of Evidence in Rhinology and Skull Base Surgery Research by Jennifer A. Silver, Marco Mascarella, George Tali, Rickul Varshney, Marc A. Tewfik, Bobby A. Tajudeen and Joseph S. Schwartz in Otolaryngology–Head and Neck Surgery