Abstract
Objective
Citation skew is a phenomenon that refers to the unequal citation distribution of articles in a journal. The objective of this study was to establish whether citation skew exists in Otolaryngology—Head and Neck Surgery (OHNS) journals and to elucidate whether journal impact factor (JIF) was an accurate indicator of citation rate of individual articles.
Methods
Journals in the field of OHNS were identified using Journal Citation Reports. After extraction of the number of citations in 2020 for all primary research articles and review articles published in 2018 and 2019, a detailed citation analysis was performed to determine citation distribution. The main outcome of this study was to establish whether citation skew exists within OHNS literature and whether JIF was an accurate prediction of individual article citation rate.
Results
Thirty-one OHNS journals were identified. Citation skew was prevalent across OHNS literature with 65% of publications achieving citation rates below the JIF. Furthermore, 48% of publications gathered either zero or one citation. The mean and median citations for review articles, 3.66 and 2, respectively, were higher than the mean and median number of citations for primary research articles, 1 and 2.35, respectively (P < .001). A statistically significant correlation was found between citation rate and JIF (r = 0.394, P = 0.028).
Conclusions
The current results demonstrate a citation skew among OHNS journals, which is in keeping with findings from other surgical subspecialties. The majority of publications did not achieve citation rates equal to the JIF. Thus, the JIF should not be used to measure the quality of individual articles. Otolaryngologists should assess the quality of research through the use of other metrics, such as the evaluation of sound scientific methodology, and the relevance of the articles.
Key Points
Question: Does the journal impact factor (JIF) predict individual article citation rate in Otolaryngology––Head and Neck Surgery (OHNS) journals?
Findings: Citation skew was prevalent across OHNS literature with 65% of publications achieving citation rates below the JIF. A statistically significant correlation was found between citation rate and JIF (r = 0.394, P = 0.028).
Meaning: The JIF does not predict individual article citation rate and should not be used to assess quality of individual articles.
Introduction
Journal impact factor (JIF) is a bibliometric tool first coined by Eugene Garfield in 1955 historically used to aid librarians in ranking journals of different scientific disciplines. 1 -3 JIF is now universally considered as a surrogate indicator of journal quality. JIF is calculated as the number of citations a journal receives for its articles divided by the number of citable items in that journal published in the two preceding years. 2,4,5 This means that a publication in a journal with an impact factor of 10 would yield 10 citations per year, on average, over the first two years after publication. However, only a small number of articles will achieve a high number of citations and the majority of articles will not reach the expected number of citations––this phenomenon is known as citation skew. 6 In general, citation counts are considered to represent the article’s and journal’s impact, as the more the article is cited the higher the journal’s influence and impact. For this reason, it is important to establish whether citation skew exists in Otolaryngology––Head and Neck Surgery (OHNS) journals.
Within other surgical specialties, citation skew has been noted in Orthopedic Surgery and Plastic Surgery journals. 7,8 From 2003 to 2015, the impact factor of OHNS journals had increased by 51.3% and the number of articles that were cited over 100 times had increased to over 1000% between 1999 and 2009. 9,10 Therefore, we intend to study whether citation skew exists in OHNS journals and elucidate whether JIF is an accurate indicator of citation rate of individual articles.
Methods
Journal Citation Reports (JCR) 2020 was used to identify OHNS journals (Clarivate Analytics). General otolaryngology journals, as well as ones with a subspecialty focus, were included in the analysis. The number of citations in 2020 for all primary research articles and review articles published in 2019 and 2018 were extracted from the JCR. Publications included in the analysis were primary research articles and review articles only. Citation distribution was evaluated, and subgroup analyses were conducted for individual OHNS journals.
To identify the percentage of publications achieving JIF, the number of citations of each publication was compared to the JIF of the journal in which the article was published. 8 The number of publications with no citation was divided by the total number of publications to determine the percentage of publications that did not receive any citations during the initial 2-year period after publication. The interquartile range (IQR) of each journal, and the entire dataset, was obtained by sorting the publications by the number of citations they achieved and extracting the 25th percentile, 75th percentile, and median. The mean number of citations was then calculated and compared using t-test analyses. In addition, the median number of citations for review articles and primary research articles were compared using a Kruskal–Wallis H test. The percentage of publications that contributed to the total percent of citations for all publications and for the top three journals with the highest impact factors (JAMA Otolaryngology—Head & Neck Surgery, International Forum of Allergy & Rhinology, and Rhinology) was calculated using the Lorenz curve. Pearson’s correlation coefficient was calculated between the number of citations per journal and the JIF. All statistical analysis was performed using Python programming language (Python’s SciPy library).
Results
Otolaryngology—Head and Neck Surgery journals and their 2020 impact factors.
A significant proportion of publications analyzed had either zero citations (25%) or one citation (23%) during the two-year period, demonstrating a right-skew citation distribution. The journal with the lowest percentage of publications with zero citations was JAMA Otolaryngology—Head & Neck Surgery at 9%. Figure 1 illustrates the citation distribution of the analyzed OHNS journals. The mean and median citations for review articles, 3.66 and 2, respectively, were higher than the mean and median number of citations for primary research articles, 1 and 2.35, respectively (P < .001). Thus, reviews achieved higher number of citations than primary research articles on average (P < .001). Citation distributions for the top 10 journals are illustrated in Figure 2, demonstrating right-skewed distributions for each individual journal. Citation distribution of all Otolaryngology—Head and Neck Surgery journals. *red line represents mean impact factor in 2020. Citation distribution of top 10 Otolaryngology—Head and Neck Surgery journals. *red line represents mean impact factor in 2020.
Only 34% of OHNS publications achieved citation rates higher than their journal’s JIF as noted in Table 1. This data demonstrates that most OHNS publications reached citation rates less than their JIF. Across all OHNS journals analyzed, only 15% of publications were responsible for 50% of citations overall. Furthermore, 50% of the publications accounted for 89% of citations, as demonstrated by the Lorenz curve in Figure 3. A similar trend was observed among three journals with the highest impact factors––in JAMA Otolaryngology—Head & Neck Surgery, only 16% of publications were responsible for 50% of citations, and 50% of publications accounted for 83% of citations; in International Forum of Allergy & Rhinology, only 12% of publications contributed to 50% of citations and 50% of publications accounted for 85% of citations; similarly, in Rhinology, only 16% of publications contributed to 50% of citations and 50% of publications accounted for 85% of citations. A statistically significant correlation was found between citation rate and JIF (r = 0.394, P = 0.028). Lorenz curve for all included articles. Lorenz curve for all included articles. This displays that 15% of publications contributed to 50% of citations as noted by the green line, and 50% of publications accounted for 89% of citations as noted by the red line.
Discussion
The current study assessed and analyzed the validity of JIF in OHNS journals. Our results demonstrate a citation skew among OHNS journals. This study revealed that 65% of publications in the studied journals had garnered citations below the JIF. Our data also demonstrated that only 15% of publications were responsible for 50% of the journal’s citations. In addition, 50% of publications accounted for the majority of citations (89%). This strongly indicates the presence of citation skew within OHNS journals. According to these findings, JIF is not an accurate bibliometric tool to reflect on the quality of individual articles in a journal.
Our study results are consistent with findings from precious studies investigating the presence of citation skew in other scientific journals. The presence of citation skew has also been noted in Plastic Surgery journals, with 40% of publications gathering zero citations and 25% having one citation. They also concluded that the majority of publications had achieved citation counts below the JIF. 8 Additionally, orthopedic surgery journals have also been noted to demonstrate citation skew with 85% of publications achieving citations below the JIF. Furthermore, they found that the median number of citations was zero for all included journals except for seven journals. 7
To date, the JIF is still considered the gold standard tool to judge the quality of journals and individual articles. With the advancements in academic research and bibliometric methods, a number of limitations in JIF have been identified. One limitation is that the JIF is heavily influenced by the presence of a small number of highly cited articles skewing the JIF––as thoroughly demonstrated in the present study. Moreover, this provides pressures for journals to encourage publications that achieve high citation rates which consequently artificially inflate the JIF. 11 Another limitation is that JIF uses the mean to measure journal citation distribution, and this introduces the possibility of outliers affecting the distribution. 4,12 -14 Consequently, the median should be used as there is less outlier effects on this measure. 15,16 The persistent use of JIF as a representation of a journal’s quality and impact has encouraged exploitative practices such as self-citation, and selective publication of review articles and studies with negative findings. 4,12,13,17,18 In addition, JIF cannot be used to compare journals from different fields due to different publication criteria; therefore, it has little value in inter-disciplinary comparison. Given these limitations, there is a growing movement in the scientific community to find an alternative to the JIF. 5 One alternative is the Eigenfactor score (ES), which takes into consideration the previous five years of publications and excludes self-citations. It also weighs the ES of the citing journal for incoming citations. 11,19 For these reasons, ES is considered to be an accurate tool in appraising a journal’s impact. 19 Another notable alternative is the h-index that was originally utilized as an author-level metric but has also recently gained a positive reputation as a journal metric for its application in determination of a journal’s impact and productivity. It is the number of papers (h) published in a journal that have been cited atleast h times. Therefore, rather than reflecting only a few of the most highly cited publications in a journal, the h-index is more representative of the majority of publications in a given journal.
Despite the flaws pertaining to the validity of JIF, it is continued to be used as one of the most significant tools in research and academia, mainly justified by its simplicity and familiarity in the field. While JIF remains the gold standard bibliometric tool, its use is certainly not adequate to assess the quality of individual articles. 17,18,20 Evaluation of academic research should continue to be based on research quality such as thorough methodology, replicability, and validity. 17,20,21
To our knowledge, this is the first paper to study the presence and effect of citation skew among OHNS journals. The journals included in this analysis are the top ranked journals in the field and the most recent impact factors for these journals were examined. Nevertheless, this study is not without its limitations. The data was obtained from one source only, the JCR 2020 (Clarivate Analytics) and included citations from only the preceding 2 years. Despite these shortcomings, the goal to highlight the most recent JIF and citations for cornerstone OHNS journals was accomplished.
Conclusion
Citation skew exists within OHNS journals, which is in keeping with findings from other surgical subspecialties. The majority of publications did not reach citation rates equal to the JIF. Only 15% of publications were found to be responsible for 50% of citations overall. Furthermore, 50% of publications accounted for 89% of citations. Thus, the JIF should not be the only measure used to assess the quality of individual articles. Otolaryngologists should assess the quality of research through the use of other metrics, including the evaluation of sound scientific methodology and the relevance of articles.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
