Sage Journals: Discover world-class research

Abstract

Background:

The purposes of this study were to analyze the trends in Oxford level of evidence (LOE), LOE of most-cited articles, and association between LOE and journal impact factor (IF) and SCImago Journal Rank (SJR) over a 10-year period (2009-2018) in 3 prominent hand surgery journals, specifically HAND, Journal of Hand Surgery (American Volume) (JHS), and Journal of Hand Surgery (European Volume) (JHSE).

Methods:

All articles published from 2009 to 2018 in HAND, JHS, and JHSE were reviewed for assigned or available LOE. Data were pooled and analyzed for trends in LOE; relationship among IF, SJR, and LOE; and citation density.

Results:

A total of 3921 total publications were tabulated from 2009 to 2018, with the majority of studies being level V (1700, 43%) and fewer studies being level I (146, 4%). Over the 10-year study period, there was no significant change in frequency of level I studies for any journal. HAND trended significantly toward higher LOE, JHS trended toward higher LOE, and JHSE trended toward decreased LOE without significance. Among all journals, the annual number of articles and the average LOE were independent significant predictors of IF and SJR. Statistically significant correlations were found between citation density and LOE for JHS and HAND.

Conclusions:

Higher quality evidence is becoming more prevalent in the hand surgery literature over the past 10 years. Annual articles, average LOE, and level I and II and level IV articles were significant predictors of increasing IF and SJR.

Keywords

hand surgery level of evidence impact factor SCImago Journal Rank top 10 most-cited

Introduction

Evidence-based medicine has become the foundation for clinical decision-making.¹ Level of evidence (LOE) was developed in 1989 to characterize the quality of evidence based on study design and was later modernized by Oxford.^2,3 Marking the first introduction of LOE to orthopedic literature, The Journal of Bone and Joint Surgery started publishing LOE with each article in 2003, and there has been an increase in the overall quality and quantity of studies in orthopedic and hand literature since the implementation of LOE assignments.^4-7 Trends in levels of evidence and correlation with publication performance metrics are emerging fields in bibliometric studies. Publication performances can be assessed by a variety of factors and measurements. Journal impact factor (IF), introduced by Garfield⁸ in 1955, was the first to assign a scientometric index to a journal. Impact factor is a function of the annual average number of citations generated by a journal in the last 2 years.⁹ The SCImago Journal Rank (SJR) was developed in 2007 as an updated metric to measure the influence of a journal based on annual average number of citations in the last 3 years and the importance of the journals that generate the citations.^10,11

The purposes of this study were to analyze the trends in LOE, LOE of most-cited articles, and association between LOE and journal IF and SJR over a 10-year period (2009-2018) in 3 prominent hand surgery journals, specifically HAND, Journal of Hand Surgery (American Volume) (JHS), and Journal of Hand Surgery (European Volume) (JHSE). We hypothesized that the hand surgery literature has increased the rate of higher LOE and that there is a correlation between LOE and IF and SJR.

Methods

All articles published from 2009 to 2018 in HAND, JHS, and JHSE were reviewed for LOE with 3921 total studies. These journals were used because they had 10-year IF and SJR data. All 3 of these journals were accessed online (accessed date November 30, 2020).

Level of evidence assigned by the journal was used where available, or the study methods were assessed to determine LOE based on the Oxford system.³ Briefly, level I (highest level of evidence) was defined as randomized controlled trials (RCTs) with >80% of follow-up. Level II was defined as cohort studies or RCTs with <80% of follow-up. Level III was defined as case-control studies. Level IV was defined as case-series and poor-quality cohort and case-control studies. Level V (lowest level of evidence) was defined as expert opinion pieces without critical appraisal. Level of evidence of systematic reviews was assigned based on the lowest LOE of included studies.³

Journal metrics were accessed via Scopus online database for SJR (Elsevier, The Netherlands, accessed May 1, 2021) and Web of Science for impact factor (Clarivate Analytics, Philadelphia, accessed May 1, 2021). Web of Science was also used to analyze the top 10 cited papers annually during the study period (accessed May 10, 2021). Articles with at least 1 citation were included.

Data normality was assessed using the Shapiro-Wilk test. Multiple linear regression analysis was used to assess the relationship among number of articles, median LOE, and frequency of each LOE with IF and SJR. Spearman correlation coefficient (ρ) was also used to assess the relationship between IF, SJR, and LOE. Impact factor is derived from citations from publications in the preceding 2 years, whereas SJR is derived from those in the preceding 3 years. The association between LOE and IF was assessed for publications in the preceding 2 years and for publications in the preceding 3 years for SJR. Median LOE was calculated as 2- and 3-year moving averages to assess correlation with IF and SJR, respectively.^8,10 Level I and II studies were combined for assessing correlation to IF and SJR due to relative paucity of prospective trials. Citation density was defined as the total number of citations divided by the number of years in circulation.¹² Alpha was defined as <0.05. The terms “higher” or “improved” refer to the quality of evidence, rather than the numerical value, with level I studies being of a higher level of evidence than level V.

Results

A total of 3921 total publications were tabulated from 2009 to 2018 by online database search with 805 (20%) from HAND, 2141 (55%) from JHS, and 975 (25%) from JHSE. Median annual publications were 74 (interquartile range [IQR], 19) from HAND, 218 articles (IQR, 40.5) for JHS, and 96 (IQR, 8) from JHSE (Tables 1 and 2; supplementary Figures 1–3).

Table 1.

Annual Articles by Oxford Level of Evidence From HAND, Journal of Hand Surgery (American Volume), and Journal of Hand Surgery (European Volume).

Year	Oxford level of evidence (%)
Year	I	II	III	IV	V
2009	14 (4%)	44 (12%)	57 (15%)	52 (14%)	207 (55%)
2010	11 (3%)	32 (8%)	54 (14%)	72 (19%)	216 (56%)
2011	18 (5%)	38 (10%)	62 (17%)	60 (19%)	186 (51%)
2012	21 (5%)	60 (13%)	83 (18%)	96 (21%)	194 (43%)
2013	16 (4%)	43 (11%)	90 (22%)	86 (21%)	174 (43%)
2014	13 (3%)	61 (16%)	93 (24%)	92 (24%)	125 (33%)
2015	13 (3%)	65 (14%)	112 (23%)	121 (25%)	168 (35%)
2016	7 (2%)	35 (11%)	65 (20%)	86 (27%)	125 (39%)
2017	14 (4%)	46 (13%)	65 (18%)	108 (30%)	123 (35%)
2018	19 (5%)	54 (14%)	53 (13%)	91 (23%)	182 (46%)
Total	146 (4%)	478 (12%)	733 (19%)	864 (22%)	1700 (43%)

Table 2.

Spearman correlation coefficient for frequencies of Oxford level of evidence annually from 2009 to 2018 for HAND, JHS, and JHSE.

Journal		Spearman correlation coefficient for frequency of Oxford level of evidence 2009-2018
Journal		I	II	III	IV	V
HAND	ρ	0.261	0.745	0.830	0.358	–0.891
HAND	P value	.467	.0133	.00294	.310	.000542
JHS	ρ	–0.103	0.915	–0.067	0.261	–0.345
JHS	P value	.777	.000204	.855	.467	.328
JHSE	ρ	–0.127	–0.782	–0.042	0.855	–0.827
JHSE	P value	.726	.00755	.907	.00164	.00318
All	ρ	–0.006	0.515	0.176	0.855	–0.648
All	P value	.987	.128	.627	.00164	.0425

JHS = Journal of Hand Surgery (American Volume); JHSE = Journal of Hand Surgery (European Volume).

For all journals, the majority of studies published were level V (1700, 43%) and fewer studies were level I (146, 4%) (Tables 1 and 2). Over the 10-year study period, there was no significant change in the frequency of level I studies for any journal (supplementary Table 1). The frequency of level II studies increased in JHS and HAND and decreased in JHSE (P < .05). The frequency of level III increased in HAND (P < .05). The frequency of level IV increased in JHSE and overall (P < .05). The frequency of level V decreased in JHSE, HAND, and overall (P < .05).

The median level of evidence of the 3 studied hand journals trended toward higher LOE with statistical significance (P = .0470). Among all journals, the annual number of articles and median LOE were independent significant predictors of IF and SJR (P = .00120 and P = .00204, respectively) (supplementary Tables 2–5). The annual frequency of combined level I and II and level IV articles were independent significant predictors of IF and SJR (P < .001 and P < .001, respectively) (supplementary Tables 4 and 5; supplementary Figure 8). The frequency of level II and V studies was not correlated with IF and SJR. Impact factor improved significantly for JHSE (ρ = 0.697, P = .0251), improved without statistical significance for HAND (ρ = 0.515, P = .128), and decreased without statistical significance for JHS (ρ = −0.667, P = .855). SCImago Journal Rank improved significantly for HAND (ρ = 0.867, P = .00117), improved without statistical significance for JHSE (ρ = 0.624, P = .0537), and decreased without statistical significance for JHS (ρ = −0.503, P = .138).

The top 10 most-cited articles for each journal over the 10-year period from 2009 to 2018 yielded 64 for HAND, 100 articles for JHS, and 100 for JHSE (Table 3). Statistically significant correlations were found between citation density and LOE for JHS and HAND (ρ = −0.390, P < .001 and ρ = −0.280, P = .0251, respectively). There was no significant correlation between citation density and LOE in JHSE (ρ = 0.170, P = .356). The odds ratio (OR) for a level I or II article being among the top 10 cited annually was 3.04 (95% confidence interval [CI] = 1.93-4.77) for JHS. This OR was not significant for JHSE and HAND (1.46, 95% CI = 0.9-2.32; 0.6, 95% CI = 0.27-1.34, respectively).

Table 3.

Citation Analysis by Level of Evidence for the Top 10 Cited Articles Annually for Each Journal.

Journal	Oxford level of evidence
Journal	1	2	3	4	5
HAND
Count	2	5	4	32	21
LOE (% all)	4	13	27	6	50
LOE (% top 10)	3	8	6	50	33
Median citation density (IQR)	9.7 (5.3)	6.5 (5.5)	3.1 (3.8)	2.6 (3.2)	1.8 (2.3)
JHS
Count	12	17	15	50	6
LOE (% all)	3	9	11	30	47
LOE (% top 10)	12	17	15	50	6
Median citation density (IQR)	11.8 (4.2)	10.3 (3.5)	9 (6.0)	7.8 (3.3)	7.5 (0.8)
JHSE
Count	9	20	21	35	15
LOE (% all)	4	18	27	19	32
LOE (% top 10)	9	20	21	35	15
Median Citation Density (IQR)	4.5 (3.3)	5.6 (1.7)	5.3 (4.6)	7.0 (4.7)	5.3 (2.6)

IQR = interquartile range; JHS = Journal of Hand Surgery (American Volume); JHSE = Journal of Hand Surgery (European Volume); LOE = level of evidence.

Discussion

The aim of this study was to analyze trends in LOE, association between LOE and journal IF and SJR, and LOE of most-cited articles in 3 prominent hand journals over a 10-year period, namely, HAND, JHS, and JHSE. Among the 3 journals, overall LOE trended toward higher level of evidence. The annual number of articles and median LOE were significant predictors of IF and SJR. There were significant correlations between citation density and LOE for JHS and HAND. The hand literature is overall increasing LOE with stable or improved IF and SJR among these 3 journals.

Among the 3 journals, overall LOE trended toward higher level of evidence with fewer level V studies (P = .0425) and increased level II studies in JHS (P < .001) and HAND (P = .133), increased level III in HAND (P = .00294), and increased level IV studies in JHSE (P = .00164). These align with an emphasis toward higher quality of research for execution and publication, which has been observed in other orthopedic specialties, specifically trauma, pediatrics, foot and ankle, and spine.^1,13-18 The frequency of level I studies, constituting 2% to 5% of annual publication in these 3 journals, remained stable over the 10-year study period and was consistent with the prior decade in general orthopedics, trauma, hand, and other specialty fields.^4,6,17-21 The paucity of level I studies can be attributed to the difficulty and expense in conducting RCTs in surgical specialties.^17,22-24 Stringent patient inclusion criteria, blinding of surgeons and patients to surgical interventions, and surgical placebo use make RCTs difficult or ethically unjustifiable to perform.²⁵

This study investigated the association between LOE and IF and SJR as scientometric indices for scholarly influence. The annual number of articles is an independent predictor of IF and SJR (P = .00120 and P = .00204, respectively), which is expected as increasing publications yields more opportunities or citations. This has been reported in similar studies across multiple specialties, specifically plastic surgery, orthopedics, and anesthesia.^26-29 In addition, the median LOE and the annual frequency of combined level I and II and level IV articles are independent significant predictors of increased IF and SJR (P < .001). This suggests that the quality of studies reflected in level I and II articles correlate with higher number of citations. Level IV studies constitute landmark case series and literature review articles with high numbers of citations, which have been observed in multiple other specialties including orthopedics and plastics surgery.^26,27,30 Amiri et al²⁵ found that more than 70% of the top 50 most-cited articles in spinal surgery were of level IV evidence.

Finally, a citation density analysis for LOE among the top 10 most-cited publications annually redemonstrated favorable number of citations among level I and II and level IV studies. The JHS and HAND showed statistically significant correlations between citation density and LOE (P < .05). No correlation was found for citation density and LOE for JHSE (ρ = 0.170, P = 0356). Moreover, JHS showed a statistically significant OR for level I or II articles being cited among the top 10 annually (OR: 3.04, 95% CI = 1.93-4.77). These findings are contrary to the multitude of studies on citation analysis across various specialties.^31,32 Studies of the 50 most-cited articles in plastic surgery and spinal surgery found that level IV evidence constituted the bulk of the citation analyses (68% and 71%, respectively).^25,33 Our study likely found differing results due to the more recent nature of our analysis and an increased emphasis on RCTs in hand literature due to their relative infrequency. As the number of citations can be indicative of the overall importance of the article to further research, our results reflect an increasing emphasis on higher quality evidence in hand literature.³⁴

There are limitations to this study. First, we sampled 3 hand surgery journals based on the availability of IF and SJR rankings over the past 10-year period but did not include other journals with plastic surgery and peripheral nerve surgery emphases that overlap in hand surgery topics and readership. Second, there is variation when assigning LOE, which was mitigated by standardized definitions.³⁵ Third, the index for HAND is recorded fairly recently, making the journal difficult to track. Despite these limitations, this analysis evaluated 3921 publications and demonstrated that hand surgery research is following trends of other specialties by increasing the overall quality of evidence. We recommend future research evaluating the quality of studies included in highly cited level IV review articles, as these review articles continue to generate large audiences and inform current practices.

Conclusion

This study found that higher quality evidence is becoming more prevalent in hand surgery literature over the past 10 years. Furthermore, annual articles, median LOE, and level I and II and level IV articles were significant predictors of increasing IF and SJR.

The hand surgery literature has evolved with the emphasis of evidence-based medicine. To provide the highest quality of care that the structure of evidence-based medicine provides, the hand surgery literature needs to continue circulating high-quality research.

Supplemental Material

sj-docx-1-han-10.1177_15589447221093672 – Supplemental material for Ten-Year Trends in Level of Evidence in Hand Surgery

Supplemental material, sj-docx-1-han-10.1177_15589447221093672 for Ten-Year Trends in Level of Evidence in Hand Surgery by Reid E. Tompkins, Michael M. Polmear, Kyle J. Klahs, John P. Scanaliato, Leon J. Nesti and John C. Dunn in HAND

Footnotes

Supplemental material is available in the online version of the article.

Authors’ Note

Investigation was performed at William Beaumont Army Medical Center, El Paso, Texas.

Ethical Approval

This study was approved by our institutional review board.

Statement of Human and Animal Rights

This article does not contain any studies with human or animal subjects.

Statement of Informed Consent

Informed consent was obtained from all individual participants included in the study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Reid E. Tompkins

Michael M. Polmear

Kyle J. Klahs

John C. Dunn

References

Djulbegovic

Guyatt

. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390(10092):415-423.

Cook

Guyatt

Laupacis

, et al. Rules of evidence and clinical recommendations on the use of antithrombotic agents [published correction appears in Chest 1994 Feb;105(2):647]. Chest. 1992;102(4 Suppl.):305S-311S.

Oxford centre for evidence-based medicine: levels of evidence (March 2009). Centre for Evidence-Based Medicine, University of Oxford. www.cebm.ox.ac.uk.https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009

Cunningham

Harmsen

Kweon

, et al. Have levels of evidence improved the quality of orthopaedic research? Clin Orthop Relat Res. 2013;471(11):3679-3686.

Reich

Shaw

Barrett

, et al. Level of evidence trends in the Journal of Bone and Joint Surgery, 1980-2010. Iowa Orthop J. 2014;34:197-203.

Sugrue

Joyce

Sugrue

, et al. Trends in the level of evidence in clinical hand surgery research. Hand (N Y). 2016;11(2):211-215.

Schmitt

Talley

Chang

. New concepts and technologies in reconstructive hand surgery. Clin Plast Surg. 2012;39(4):445-451.

Garfield

. Citation indexes for science. A new dimension in documentation through association of ideas 1955. Int J Epidemiol. 2006;35(5):1123-1128.

Roldan-Valadez

Salazar-Ruiz

Ibarra-Contreras

, et al. Current concepts on bibliometrics: a brief review about impact factor, Eigenfactor score, CiteScore, SCImago Journal Rank, Source-Normalised Impact per Paper, H-index, and alternative metrics. Ir J Med Sci. 2019;188(3):939-951.

10.

González-Pereira

Guerrero-Bote

Moya-Anegón

. A new approach to the metric of journals’ scientific prestige: the SJR indicator. J Informetrics. 2010;4(3):379-391.

11.

Falagas

Kouranos

Arencibia-Jorge

, et al. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J. 2008;22(8):2623-2628.

12.

Jelicic Kadic

Kovacevic

Runjic

, et al. Research methodology used in the 50 most cited articles in the field of pediatrics: types of studies that become citation classics. BMC Med Res Methodol. 2020;20(1):60.

13.

Scheschuk

Mostello

Lombardi

, et al. Levels of evidence in orthopaedic trauma literature. J Orthop Trauma. 2016;30(7):362-366.

14.

Wright

Swiontkowski

Heckman

. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1-3.

15.

Hanzlik

Mahabir

Baynosa

, et al. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91(2):425-428.

16.

Grant

Tjoumakaris

Maltenfort

, et al. Levels of evidence in the clinical sports medicine literature: are we getting better over time? Am J Sports Med. 2014;42(7):1738-1742.

17.

Cashin

Kelley

Douziech

, et al. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31(6):721-725.

18.

Zaidi

Abbassian

Cro

, et al. Levels of evidence in foot and ankle surgery literature: progress from 2000 to 2010? J Bone Joint Surg Am. 2012;94:15.

19.

Rosales

Reboso-Morales

Martin-Hidalgo

, et al. Level of evidence in hand surgery. BMC Res Notes. 2012;5:665.

20.

Park

Gil

Kleiner

, et al. Publication characteristics of studies published in The Spine Journal from 2005 to 2015. Orthop Rev (Pavia). 2019;11(3):7786.

21.

Voleti

Donegan

Baldwin

, et al. Level of evidence of presentations at American Academy of Orthopaedic Surgeons annual meetings. J Bone Joint Surg Am. 2012;94(8):e50.

22.

Sinno

Neel

Lutfy

, et al. Level of evidence in plastic surgery research. Plast Reconstr Surg. 2011;127(2):974-980.

23.

Solomon

McLeod

. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery. 1995;118(3):459-467.

24.

Stirrat

. Ethics and evidence based surgery. J Med Ethics. 2004;30(2):160-165.

25.

Amiri

Kanesalingam

Cro

, et al. Level of evidence of clinical spinal research and its correlation with journal impact factor. Spine J. 2013;13(9):1148-1153.

26.

Obremskey

Pappas

Attallah-Wasif

, et al. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87(12):2632-2638.

27.

Rodrigues

Tedesco

ACB

Nahas

, et al. Journal impact factor versus the evidence level of articles published in plastic surgery journals. Plast Reconstr Surg. 2014;133(6):1502-1507.

28.

Bain

Myles

. Relationship between journal impact factor and levels of evidence in anaesthesia. Anaesth Intensive Care. 2005;33(5):567-570.

29.

Weale

Lear

. Randomised controlled trials and quality of journals. Lancet. 2003;361(9370):1749-1750.

30.

de Barros

MPM

Matsunaga

Tamaoki

MJS

. Relation between impact factor in orthopedic journals and level of evidence. Acta Ortop Bras. 2018;26(4):275-277.

31.

Baltussen

Kindler

. Citation classics in anesthetic journals. Anesth Analg. 2004;98(2):443-451.

32.

Lefaivre

Shadgan

O’Brien

. 100 most cited articles in orthopaedic surgery. Clin Orthop Relat Res. 2011;469(5):1487-1497.

33.

Loonen

MPJ

Hage

Kon

. Plastic Surgery Classics: characteristics of 50 top-cited articles in four Plastic Surgery Journals since 1946. Plast Reconstr Surg. 2008;121(5):320e-327e.

34.

Baldwin

Kovatch

Namdari

, et al. The 50 most cited articles in pediatric orthopedic surgery. J Pediatr Orthop B. 2012;21(5):463-468.

35.

Bhandari

Swiontkowski

Einhorn

, et al. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery. J Bone Joint Surg Am. 2004;86(8):1717-1720.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.42 MB