Abstract
Introduction
The impact factor (IF) is a journal-level metric developed by the Institute for Scientific Information in the 1970s to help assess a journal’s influence.1,2 Partially due to its simplicity, the IF is often used as an indicator of a journal’s or individual article’s academic rigor and worth, although this notion is not supported by robust evidence.3,4 For example, it has been reported that some universities, funding agencies, and government entities use IF as a basis for career decisions such as hiring, promotion, funding, and awards.5-8 Despite increasing awareness of the limitations of non-intended use of IF, authors may still feel pressured to consider a journal’s impact factor as a main consideration for manuscript submission.9,10
Radiology research is often interdisciplinary in nature. In the context of publication of radiology research, the authors may have the choice of submission to not only radiology journals, but also to a journal in a different field related to its research topic, such as a journal of general medicine, another medical specialty, biomedical engineering, or computer science. Within clinical medicine, it has been demonstrated that journals of different specialties have different ranges of IF, with radiology journals in the middle of the pack. 10 For example, analysis of the 2010 IFs demonstrated that the top radiology journals had lower IFs compared with those of general medicine and many specialties such as oncology, neurology, cardiology, pathology, and gastroenterology. 10 For some radiology authors, this difference in IF may be a strong incentive to prioritize submission to a non-radiology journal with a higher IF, despite the fact that the study would be better suited to an imaging audience. Therefore, the determinants of radiology journals’ IFs in relation to other medical specialties are important to study because such understanding could help guide radiology authors’ submission preference and improve radiology journals’ attractiveness.
The difference in IFs of journals in different medical specialties could be secondary to characteristics of a given specialty such as its size or level of academic activity.10,11 Another possible contributing factor could be different intra-specialty citation patterns. 12 For example, when authors of a radiology manuscript cite more articles published in radiology journals, there would be a corresponding increase of the overall IFs of radiology journals. Thus, we hypothesized that, compared with another medical specialty with higher IFs, articles published in radiology journals demonstrate a lower number of intra-specialty citations per article. To test this hypothesis, we conducted this study to quantify the intra-specialty citation count and rate in articles published in radiology journals, and compare them with another medical specialty with an overall higher IFs, such as gastroenterology/hepatology.
Materials and Methods
No ethics approval was required for this study that examined publicly available bibliometric data. A study protocol was registered on the Open Science Framework (https://osf.io/wtz3x). Protocol deviations include additional analysis of intra-specialty citation count, in addition to the original emphasis on intra-specialty citation rate.
Journal Selection
For this study, the 2020 IF (June 2021 release) and in-category journal rankings were used. For the purpose of comparison, we chose gastroenterology/hepatology journals as a control, which have higher IFs than radiology journals (see
Four radiology journals were selected from 133 journals included in the Web of Science (Clarivate) “RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING” category: Radiology (IF: 11.105, in-category rank: 2/133), European Radiology (IF: 5.315, rank: 22/133), Diagnostic and Interventional Imaging (IF: 4.026, 35/133), and Canadian Association of Radiologists Journal (IF: 2.248, 94/133), because they are general radiology journals that represent a wide range of impact factors and ranking within the journal category.
Four gastroenterology journals were selected from 92 journals included in the Web of Science “GASTROENTEROLOGY & HEPATOLOGY” category: Journal of Hepatology (IF: 25.083, in-category rank: 2/92), Journal of Gastroenterology (IF: 7.527, rank: 17/92), World Journal of Gastroenterology (IF: 5.742, rank: 28/92), Journal of Clinical Gastroenterology (IF: 3.062, rank: 67/92). They were selected because they are general gastroenterology/hepatology journals that had equivalent percentile rankings within the journal category to the rankings of the selected radiology journals.
Literature Search
A literature search was conducted on July 7, 2022 using the Web of Science to identify all articles published in 2021 in the selected four radiology and four gastroenterology/hepatology journals, with the following search string: (SO = (RADIOLOGY) OR SO = (EUROPEAN RADIOLOGY) OR SO = (DIAGNOSTIC “AND” INTERVENTIONAL IMAGING) OR SO = (CANADIAN ASSOCIATION OF RADIOLOGISTS JOURNAL JOURNAL DE L ASSOCIATION CANADIENNE DES RADIOLOGISTES) OR SO = (JOURNAL OF HEPATOLOGY) OR SO = (JOURNAL OF GASTROENTEROLOGY) OR SO = (WORLD JOURNAL OF GASTROENTEROLOGY) OR SO = (JOURNAL OF CLINICAL GASTROENTEROLOGY)) AND PY = 2021.
Data Collection
Full bibliometric data of the returned results were downloaded from the Web of Science to a local Microsoft Excel spreadsheet that included each article’s type, title, publishing journal, cited articles, and reference count. The “(original research) article,” “review,” “editorial,” and “letter” article types were included. The “meeting abstract,” “corrections,” “early access,” and “biographical items” article types were excluded. Articles with incomplete information of cited articles were excluded. A master Microsoft Excel spreadsheet that contained all individual citations was then created by using the Excel “Power Query” function, with the publishing journal of each citation (“cited journal”) subsequently identified by using the Excel “Text to Columns” function. By this step, if no value was returned in the expected column that contained a citation’s source journal title, that citation was deemed “non-recognizable” and was removed from subsequent analysis; this typically occurred due to missing journal information or inclusion of special characters in the citation record. A full list of all cited journals was then generated. For each cited journal, it was counted how many times the journal was cited by all the included articles. Only journals that were cited at least six times were further analyzed. This cut-off was made to identify a core list of cited journals that supplied the vast majority of citations while avoiding low-yield and problematic effort to categorize uncommon journals/citation sources. Cited journals in the core list were then classified into the following categories: “radiology,” “gastroenterology,” “general medicine,” “other medicine,” and “non-medicine.” For journals that were included in the Web of Science “Science Citation Index Expanded” (SCIE) and “Emerging Sources Citation Index” (ESCI) collections, the “radiology” category corresponds to the Web of Science “RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING” category, the “gastroenterology” category to the Web of Science “GASTROENTEROLOGY & HEPATOLOGY” category, the “general medicine” category to the Web of Science “MEDICINE, GENERAL & INTERNAL” category, the “other medicine” category to the Web of Science “CLINICAL MEDICINE” group (containing 59 medical categories) except journals belonging to the previous three categories, and “non-medicine” category to all other Web of Science categories. For journals that were not included by the Web of Science SCIE/ESCI collections, one author (BG) manually classified each journal into the above categories. The word and reference limits of the selected journals were obtained from each journal’s website on August 26th, 2022.
Data Analysis
As described above, only citations from the core list of cited journals that were cited at least six times were included in the data analysis. The number of published articles in 2021 was calculated for each of the four radiology and four gastroenterology/hepatology journals, with sub-group analysis by article types. The total citation count was also calculated for these studied journals, with a break-down by article types. For each studied journal and article type, the citation count for each category of cited journals was further obtained. Intra-specialty citations were defined as citations from journals of the “radiology” category cited by articles published in the four studied radiology journals, and citations from journals of the “gastroenterology/hepatology” category cited by articles published in the four studied gastroenterology/hepatology journals.
Statistical Analysis
Continuous variables were expressed as mean ± standard error of the mean (SEM), or median and interquartile range (IQR). 13 Two-group comparisons of quantitative variables were conducted using Student’s t-test if the data passed the Shapiro–Wilk normality test (otherwise using Mann–Whitney rank sum test if normality test failed). Proportions and percentages were compared using chi-square (χ2) test, and when pairwise comparisons were appropriate, with additional post hoc z-test. All tests were two-tailed. Statistically significant difference was set at P < .05.
Results
Journal Characteristics
For the purpose of comparison, we chose gastroenterology/hepatology journals as a control. An analysis of 2020 IFs of Web of Science SCIE journals demonstrate that gastroenterology/hepatology as a specialty had higher 2020 IFs than radiology journals (radiology journal IFs: median 2.890, IQR 2.119, range .212-14.805, n = 133; gastroenterology/hepatology journal IFs: median 3.933, IQR 3.121, range .660-46.802, n = 92; Mann–Whitney rank sum test P < .001).
Characteristics of the Selected Four Radiology and Four Gastroenterology/Hepatology Journals, Including the 2020 IF, Web of Science in-Category Ranking, and Word/Reference Limits.
*June 2021 release.
Search Results
The literature search returned 4523 articles published in 2021, including 2149 articles published in the selected four radiology journals and 2347 articles published in the selected four gastroenterology/hepatology journals. The 1016 “Meeting Abstract,” 98 “Correction,” 2 “Early Access,” and 10 “Biographic Item” typed articles were excluded. One “review” article was further excluded due to incomplete data. Thus, a total of 3398 articles (including 2063 radiology and 1335 gastroenterology/hepatology articles) were included in data analysis, including 2190 “original research” (labeled as “article” type by the Web of Science), 503 “review,” 399 “editorial,” and 306 “letter” typed articles.
These 3398 articles cited 128526 individual citations (or “references,” used interchangeably in this study; radiology: 61424, gastroenterology/hepatology: 67102). A total of 1276 citations (1276/128526 = 1.0%; radiology: 684/61424 = 1.1%, gastroenterology/hepatology: 592/67102 = .9%) were excluded because they were non-recognizable. Of the 127250 recognizable citations (radiology: 60740; gastroenterology/hepatology: 66510), 8467 journals/citation sources were identified, including 1945 journals cited at least six times that were included in the subsequent categorization and data analysis (1945/8467 = 23.0%). These 1945 journals supplied 120828, or 95.0% (120828/127250) of recognizable citations (radiology: 57209/60740 = 94.2%; gastroenterology/hepatology: 63619/66510 = 95.7%).
Article Characteristics
Articles Published by the Selected Four Radiology and Four Gastroenterology/Hepatology Journals in 2021, by Article Type.
Overall Citation Count
Total Citation Count Per Article of Radiology and Gastroenterology/Hepatology Articles Published in the Selected Journals in 2021, by Article Type.
SEM indicates standard error of the mean.
To examine if the actual citation counts conformed to journals’ posted reference limits, we selectively analyzed citation counts of original research and review articles published in the Radiology journal and the Journal of Hepatology. For the Radiology journal, 41 of the 256 original research articles (41/256 = 16.0%) and 7 of 26 reviews (7/26 = 26.9%) had a citation count above the posted reference limits (35 references for original research articles, and 100 references for reviews). For the Journal of Hepatology, 6 of 220 original research articles (6/220 = 2.7%) and 9 of 42 reviews (9/42 = 21.4%) had a citation count above the posted reference limits (100 references for original research articles, and 150 references for reviews).
Per-Article Intra-Specialty Citation Count
Per-Article Intra-Specialty Citation Count of Radiology and Gastroenterology/Hepatology Articles Published in the Selected Journals in 2021, by Article Type.
SEM indicates standard error of the mean.
Intra-Specialty Citation Rate
For each article, the intra-specialty citation rate refers to the percentage of intra-specialty references out of the total number of references it cites. While the per-article intra-specialty citation count is directly related to the specialty’s IFs, the intra-specialty citation rate may offer insights into authors’ citation practice.
Intra-Specialty Citation Rate of Radiology and Gastroenterology/Hepatology Articles Published in the Selected Journals in 2021, by Article Type.
SEM indicates standard error of the mean.
Most-Cited Journals
The top five most-cited journals by the radiology articles were Radiology (cited 4760 times), European Radiology (3126 times), American Journal of Roentgenology (1792 times), Journal of Magnetic Resonance Imaging (1120 times), and European Journal of Radiology (991 times).
The top five most-cited journals by the gastroenterology/hepatology articles were Journal of Hepatology (cited 3186 times), Hepatology (2721 times), Gastroenterology (2585 times), Gut (1446 times), and New England Journal of Medicine (1146 times).
Discussion
In this study we demonstrated that, compared to gastroenterology/hepatology articles, radiology articles in the four selected journals cited a lower number of total references as well as a lower number of intra-specialty (i.e., radiology) references per article (i.e., intra-specialty citation count). However, no differences in overall intra-specialty citation rates were found between radiology and gastroenterology/hepatology articles. If this finding is generalizable to all radiology journals, it could help explain the overall lower IFs of radiology journals (compared with gastroenterology/hepatology journals in our study). This finding contributes to the understanding of the factors that lead to different IFs of different medical specialties.
Some of the journals that were selected in this study stated word and reference limits for different article types on their websites. Notably, the stated word and reference limits for original research and review articles were generally lower in radiology journals than gastroenterology/hepatology journals (Table 1). Despite inconsistent level of enforcement as we have demonstrated, we suspect these lower word and reference limits may have contributed to the observed lower number of total references in radiology articles. Interestingly, our results also demonstrate that in original research articles and editorials, but not review articles and letters, the studied radiology articles in the four selected journals cited a greater percentage of intra-specialty references out of all references (intra-specialty citation rate) than gastroenterology/hepatology articles. However, the greater intra-specialty citation rates in these article types were offset by the lower average number of total citations, leading to overall lower intra-specialty citation count per radiology article, with the latter directly linked to a specialty’s overall IFs. 14
Additional research will be helpful to further elucidate the implications of our findings. While we investigated the contribution of intra-specialty citations to the overall IF of a specialty, there could be specialty-specific pattern of contributions from outside of the specialty (citations from other medical specialties or outside of medicine). Furthermore, it is unclear why radiology articles demonstrated a lower intra-specialty citation count compared with gastroenterology/hepatology. Potentially this could reflect differences in the level of academic activity, cross-interconnectivity with other specialties, or the nature of reporting/citation in research fields that involve different extent of technical/clinical focus. While we are not advocating changing academic citation/publication practices solely for the purpose of increasing IFs, academic journals may consider loosening their reference limits if such limits were determined primarily on the basis of resource considerations. It is probably more important that the academic community becomes more aware of the potential contribution of intra- and extra-specialty citation counts to the overall IF of a specialty, and consider them as important contributing factors when comparing IFs across medical specialties, or different research fields in general.
Identifying contributing factors to different IFs of different medical specialties is not merely an academic exercise. While the publication of radiology articles in non-radiology journals likely increases the visibility of radiology as a specialty and helps foster interdisciplinary collaborations, the discrepancy in IFs between radiology and other medical specialties could exert an undue influence on radiology authors when considering the submitting journal. 15 Certain authors may consider IFs as the main, if not the only, incentive to prioritize submission to a non-radiology journal with a higher IF, despite the fact that the study would be better suited to an imaging audience. Furthermore, anecdotal experiences appear to indicate that, compared with non-radiology journals, radiology journals tend to have more stringent requirements of imaging-related technical details due to their expertise in imaging, which serves to improve the quality of academic activities in the field. Thus, if the overall lower IFs of radiology journals lead to a loss of article submissions to radiology journals, it may perpetuate a vicious cycle that is unhealthy for the field.
Other than IF, another variable related to a journal’s reputation is its ranking (and quartile) within its Web of Science category. 16 Our results demonstrate that the journals Radiology, European Radiology and American Journal of Roentgenology were the top three cited journals in the studied radiology articles. However, in the Web of Science “RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING” category, these three journals were ranked 2nd, 22nd, and 36th out of 133 journals in 2020. While the discrepancy in the rankings could be partially explained by the possibility of self-citation (because the journals Radiology and European Radiology were selected in this study), another possible explanation is heterogeneous grouping of journals in the same category. Indeed, the “RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING” category includes many journals that are strictly speaking not in radiology fields, such as journals regarding radiation therapy, cardiology, optics and photonics, and medical physics, which may contribute to an unfair lower journal ranking of radiology compared to other specialties.
This study has limitations. First, the journal selection method could contribute to sampling error given that they represent a small proportion of journals in the overall specialty. The narrow selection was necessary so that the amount of data was within a practical limit for data collection and analysis, and the risk of selection bias was mitigated by the wide range of IFs and rankings of the selected radiology journals, as well as the equivalent rankings of the selected gastroenterology/hepatology journals. Second, the selection of gastroenterology/hepatology specialty as comparison may not represent fields of other medical specialties or other non-medical fields. Third, the classification of articles into the original research, review, editorial, and letter types as processed by the Web of Science was a simplification of the more diverse article types offered by the selected journals, 17 although the most reasonable approach appeared to follow the Web of Science conventions as we did in this study. Fourth, due to the large number of complex factors that contribute to a journal’s IF, a causative relationship between citation patterns and journals’ IFs is difficult to identify.
In conclusion, our results demonstrate that radiology articles cite fewer total references and fewer intra-specialty references per article than gastroenterology/hepatology articles in representative journals selected in our study; the overall intra-specialty citation rates were not statistically different. The significantly lower per-article intra-specialty citation counts of radiology articles likely contributed to the overall lower IFs of radiology journals.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
