Sage Journals: Discover world-class research

Abstract

Background:

As the movement toward evidence-based medicine grows and publication rates rise each year, critical analysis of the orthopedic literature has become increasingly important. To aid readers in assessing the scientific quality of published research, Foot & Ankle International (FAI) began assigning levels of evidence to all clinical articles in 2008. The purpose of this study was to analyze trends in the characteristics and levels of evidence of articles published in FAI between 2000 and 2015.

Methods:

All articles published in FAI from the years 2000, 2005, 2010, and 2015 were reviewed and categorized into article type (clinical, basic science, review, or technical tip). Each clinical article was assigned a level of evidence (I-V) and study type (prognostic, therapeutic, economic, or diagnostic). Descriptive information was gathered pertaining to country of origin, author credentials, and funding. Statistical analysis was performed using chi-squared tests to detect any trends in levels of evidence and publication characteristics.

Results:

A total of 647 articles were reviewed. From 2000 to 2015, there was a statistically significant increase in the publication of clinical research articles (70% to 83%; P = .013), while the number of basic science articles decreased (29% to 17%; P = .013). Of the clinical articles, there was a significant increase in therapeutic studies (41% to 58%; P = .003). During the study period, the publication of Level I and II evidence significantly increased (2% to 14%; P = .002). Although Level III and V evidence also increased (65% to 71%, P > .99), this was not found to be statistically significant. Publications originated from a total of 39 countries, with a significant increase in the proportion of international papers (33% to 48%; P = .007) over the study period. The proportion of articles authored by Doctors of Podiatric Medicine (DPMs) during the study period significantly decreased (4% to 2%, P = .035). Finally, the percentage of studies that disclosed the use of outside funding increased during the study period, with reported funding from grants or professional groups rising from 3% to 16% (P < .001) and reported funding from commercial sources rising from 0% to 9% (P = .002).

Conclusion:

The proportion of Level I and II studies published in FAI significantly increased from 2000 to 2015. The publication of clinical research rose, with a majority being therapeutic studies. There was a significant increase in articles published by international authors and a significant decrease in articles published by DPMs. During the same time period, there was a rise in the proportion of articles reporting the use of outside funding, both professional and commercial.

Keywords

level of evidence publication trends clinical research

Introduction

Over the last 2 decades, there has been a growing emphasis on the publication quality of orthopedic research. Toward this effort, in 2003, the Journal of Bone and Joint Surgery (American Volume) (JBJS-A) began assigning levels of evidence for all clinical articles that it published¹⁰ based on the Oxford Centre for Evidence-Based Medicine Classification.¹³ This classification system assigns research articles a level of evidence from I (highest, randomized controlled trials) to V (lowest, expert opinion) and classifies the studies as therapeutic, prognostic, diagnostic, or economic.

Since their introduction, many orthopedic journals have started assigning levels of evidence to the articles they publish, including Foot & Ankle International (FAI), which began this practice in 2008. Interobserver agreement in assigning levels of evidence to orthopedic clinical research has been found to be high.^3,6 Various studies have looked at trends in levels of evidence, whereas others have looked for associations between levels of evidence and Journal Impact Factor.^4,6,11,12 In publication since 1980, FAI is the official journal of the American Orthopaedic Foot & Ankle Society (AOFAS) and is considered the premier foot and ankle publication in the world.

The purpose of this study was to analyze trends in the characteristics and levels of evidence of articles published in FAI between 2000 and 2015. We hypothesized that the percentage of Level I and II evidence would increase over time, particularly after the introduction of levels of evidence in 2008.

Methods

Identification and Selection of Articles

All articles published in FAI during 4 time periods, separated by 5-year intervals, were identified using each issue’s table of contents. The time periods consisted of the calendar years of 2000, 2005, 2010 and 2015. These years were chosen because the publication of levels of evidence in FAI began in 2008, thus permitting an analysis of 2 time periods before (2000 and 2005) and after (2010 and 2015) this change.

The following article types from each of the 4 time periods were included: clinical, basic science, review, and technical tip. Editorials, news, announcements, and letters to the editor were excluded.

Assignment of Levels of Evidence and Collection of Article Characteristics

For each clinical article, one of 3 independent reviewers assigned a level of evidence (I-V) and study type (prognostic, therapeutic, economic, or diagnostic) based on the JBJS-A grading system.¹⁰ Each reviewer was provided with and reviewed the classification system before data collection. Levels of evidence determinations were made independent of assignments made by the journal for the years they were available (2010 and 2015). Assignments were compared among the reviewers to confirm agreement. Any disagreements were discussed with the group until a consensus was reached. Additional descriptive information for each article was gathered regarding country of origin, credentials of all listed authors, and funding.

Quality control checks of each reviewer’s assessment was performed by the lead author by randomly auditing 10 articles from each year. Any discrepancies were settled with group discussion.

Statistical Analysis

The data was stratified by article type. Descriptive analysis was performed on levels of evidence for clinical articles. Chi-squared tests were used to determine if there were statistically significant differences over time in levels of evidence, author characteristics, and funding. The P value for significance was set at .05.

Results

A total of 647 articles were included and reviewed (Figure 1). Of those reviewed, 184 were basic science articles and the remaining 463 were clinical articles (Table 1). On average, there were 1 to 2 review articles every year. Of the clinical articles, therapeutic studies were the most common (307 articles, 47%), followed by prognostic (96 articles, 15%), diagnostic (56 articles, 9%), and lastly economic (8 articles, 1%) (Table 2).

Figure 1.

The number of articles for each studied year increased from 2000 to 2005, but then plateaued until 2015.

Table 1.

Article Type by Year.

Article Type	Year
Article Type	2000,n (%)	2005,n (%)	2010,n (%)	2015,n (%)
Clinical	87 (70.2)	125 (69.8)	140 (80.9)	141 (82.5)
Basic science	36 (29.0)	42 (23.5)	23 (13.3)	29 (17.0)
Technical tip	0 (0)	11 (6.1)	8 (4.6)	1 (0.6)
Review article	1 (0.8)	1 (0.6)	2 (1.2)	0 (0)

Table 2.

Clinical Study Type by Year.

Study Type	Year
Study Type	2000,n (%)	2005,n (%)	2010,n (%)	2015,n (%)
Therapeutic	51 (61.4)	75 (66.4)	81 (63.3)	100 (69.9)
Prognostic	26 (31.3)	25 (22.1)	21 (16.4)	24 (16.8)
Diagnostic	5 (6.0)	12 (10.6)	24 (18.8)	15 (10.5)
Economic	1 (1.2)	1 (0.9)	2 (1.6)	4 (2.8)

From 2000 to 2015, there was a statistically significant increase in the publication of clinical research articles (70% to 83%; P = .013), whereas the number of basic science articles decreased (29% to 17%; P = .013) (Table 3). Of the clinical articles, there was a significant increase in therapeutic studies (41% to 58%; P = .003) (Figure 2). When evaluating levels of evidence, there was a significant increase in high-level (I and II) studies published from 2000 to 2015 (2% to 14%; P = .002) (Table 3). Although Level III and V evidence also increased (65% to 71%, P > .99), this was not found to be statistically significant.

Table 3.

Level of Evidence by Year.

Level of Evidence	Year
Level of Evidence	2000,n (%)	2005,n (%)	2010,n (%)	2015,n (%)
I	2 (2.4)	7 (6.3)	3 (2.4)	8 (5.6)
II	1 (1.2)	8 (7.2)	6 (4.8)	15 (10.5)
III	13 (15.7)	17 (15.3)	28 (22.2)	55 (38.5)
IV	46 (55.4)	59 (53.2)	52 (41.3)	64 (44.8)
V	21 (25.3)	20 (18.0)	37 (29.4)	1 (0.7)

Figure 2.

The proportion of prognostic studies decreased over time.

Medical physicians (MDs and DOs) were the most frequent publishers, with a total of 1618 articles (66%) over the course of the 4-year period, whereas DPMs published only 48 articles (2%). The proportion of articles authored by DPMs decreased from 2000 to 2015 (4% to 2%, P = .035) (Table 4).

Table 4.

Author Credentials by Year.

	Year
Author Credentials*	2000,n (%)	2005,n (%)	2010,n (%)	2015,n (%)
MD/DO	279 (69.4)	492 (74.2)	365 (70.6)	616 (71.4)
DPM	15 (3.7)	10 (1.5)	8 (1.5)	15 (1.7)
Other (not MD, DO, DPM)	108 (26.9)	161 (24.3)	144 (27.9)	232 (26.9)
Total authors	402	663	517	863

* Author credentials included all authors listed on each study.

Publications originated from a total of 39 countries, with the majority of articles from the United States (83%) during the 4-year period. The number of countries represented increased at each successive time period: 21 in 2000, 26 in 2005, 32 in 2010, and 35 in 2015. There was a significant increase in the proportion of international papers (31% to 48%; P = .007) over the study period (Figure 3).

Figure 3.

There was a significant increase in the percentage of studies published by authors from countries other than the United States from 2000 to 2015.

The proportion of articles reporting outside funding increased significantly from both grants (2% to 16%, P < .001) and commercial sources (0% to 9%, P = .002) over the study period.

Discussion

This study investigated the recent trends in study characteristics and levels of evidence of articles published in FAI. From 2000 to 2015, a number of significant findings were identified: an increase in the proportion of higher level evidence studies (Levels I and II), an increase in therapeutic studies, more internationally published studies, fewer DPM published studies, and a rise in the proportion of funded studies.

The current Journal Impact Factor of FAI is 2.341, making it among the highest-impact journals specific to the field of foot and ankle surgery. The impact factor is widely used as a surrogate for the quality of a journal based on the idea that a large number of citations relative to the number of published articles is indicative of a higher quality journal. Considered the premier foot and ankle publication in the world based on its consistently highest Journal Impact Factor,⁸ FAI publishes the largest volume of clinical research on foot and ankle topics.² Because of this, the authors sought to investigate the quality of research that foot and ankle surgeons are using to guide their practice.

The current study showed that there was a significant increase in the number of Level I and II studies from 2000 to 2015, with only 2% in 2000 to 14% in 2015. Interestingly, the greatest increase was seen from 2010 to 2015, after FAI began including level of evidence assignments in their articles in 2008. This improvement in foot and ankle research quality has been suggested in previous studies. In 2003, JBJS-A published a review assessing research quality in 9 orthopedics journals and found a high number of Level IV studies.⁶ Over the 6-month review period, FAI had the highest percentage of Level IV evidence (30 of 40, 75%) among the 9 journals. In 2012, Barske and Baumhauer showed that FAI had improved their published level of evidence, as only 26 of 41 (63%) papers were Level IV.² In a review of 720 articles from 3 foot and ankle surgery journals over a time period from 2000 to 2010, Zaidi et al found a significant increase in Level I and II studies, doubling from 5.2% to 10.3%.¹² The trend toward higher levels of evidence over time is not unique to foot and ankle literature. In 2014, Cvetanovich et al⁴ reported similar findings of increased Level I and II studies as well as international representation in AJSM, currently the orthopedic journal with the highest impact factor. Consistent with prior analyses of the orthopedic literature, we found that therapeutic studies were the most frequently published.^6,11,12

Despite the trend toward higher levels of evidence, the current investigation demonstrated that Level III or V studies remain the most commonly published. This is consistent with previous studies of the orthopedic literature. These studies still have a valuable place in orthopedic research, given the challenges in conducting Level I and II studies, particularly for infrequently occurring conditions and when randomization is neither practical nor ethical.

The low proportion of published studies in FAI by podiatrists is consistent with previous studies. In 2012, Barske and Baumhauer found that physicians, not podiatrists, publish the majority of clinical foot and ankle research.² FAI specifically publishes 1.5 to 5 times the number of clinical research articles than the podiatric journals,² despite podiatrists providing approximately 60% of elective insured foot and ankle surgery.⁹

We observed an increase in the global contributions to FAI over the study period, with a greater percentage of studies published by authors outside the United States. This phenomenon has similarly been found in AJSM. The increase in international authorship in FAI to include 35 countries in 2015, up from 21 countries in 2000, suggests the international growth of foot and ankle research. This also may reflect greater circulation of FAI on an international level and the increased influence of research published by FAI.

To date, few studies have looked at funding in the orthopedic literature. In 2007, Okike et al evaluated the outcomes of studies with disclosed conflicts of interest. They reviewed all of the abstracts from all podium presentations at the annual meetings of the American Academy of Orthopaedic Surgeons (AAOS) from 2001 and 2002. They looked at the likelihood that a study would report a positive outcome and found that 40.8% of the studies reported a possible conflict of interest. They found that studies with conflicts of interest related to royalties, consulting services, and stock options were statistically significantly more likely to publish positive outcomes.⁷ Similarly, Amiri et al conducted a systematic review in 2014 of outside funding in spine surgery. They included 864 papers in their review and found that industry-sponsored research was more likely to have Level IV evidence and report positive outcomes. Level I studies were more likely to report negative outcomes.¹ Finally, in 2003, Ezzet looked at funding in total joint literature. Similar to the previous studies, they found that 50% of the studies had commercial funding. Those with commercial funding were more likely to publish positive results compared with independent research. They also found that those receiving royalties only reported positive outcomes.⁵ These studies, however, all report a higher amount of outside funding than the current study with only 16% and 9% receiving grant/institution and commercial funding, respectively. We did not investigate the relationship between funding and the level of evidence or outcome of the study.

This study should be viewed in light of its limitations. First, this study evaluated trends in publication over a 15-year period, but only 4 years were selected for inclusion, without analysis of articles published in the intervening years. For the purposes of looking at publication trends, it was thought that taking a snapshot every 5 years was sufficient. Furthermore, the study included 2 analyses prior to and 2 after introduction of levels of evidence assignments, as it was hypothesized this would lead to an increase in the quality of research published. A second limitation is that this study looked at only a single journal, which may not reflect the overall trend of foot and ankle nor orthopedic literature. We chose this journal as it is the premier journal associated with the AOFAS. Finally, we did not assess interobserver reliability as it relates to level of evidence determination. This was controlled for by having systematic audits of each reviewer by the lead author. Furthermore, previous authors have found that untrained reviewers can apply the JBJS-A level of evidence grading with acceptable interobserver agreement.^3,6

In conclusion, this study found that since the introduction of levels of evidence in FAI, the quality of studies has increased, with a significant increase in the proportion of Level I and II studies published from 2000 to 2015. The publication of clinical research rose, with a majority being therapeutic studies. There was a significant increase in articles published by international authors and a significant decrease in articles published by DPMs. During the same time period, there was a rise in the proportion of articles reporting the use of outside funding, both professional and commercial.

Supplemental Material

Supplemental Material, FAO-20-0027.R1-ReportedAuthorDisclosures - Trends in Publication and Levels of Evidence in Foot & Ankle International From 2000 to 2015

Supplemental Material, FAO-20-0027.R1-ReportedAuthorDisclosures for Trends in Publication and Levels of Evidence in Foot & Ankle International From 2000 to 2015 by Matthew Griffith, Edward Han, Joshua Hattaway and Jeannie Huh in Foot & Ankle Orthopaedics

Supplemental Material

Supplemental Material, FAO931419-ICMJE - Trends in Publication and Levels of Evidence in Foot & Ankle International From 2000 to 2015

Supplemental Material, FAO931419-ICMJE for Trends in Publication and Levels of Evidence in Foot & Ankle International From 2000 to 2015 by Matthew Griffith, Edward Han, Joshua Hattaway and Jeannie Huh in Foot & Ankle Orthopaedics

Footnotes

Ethics Approval

Ethical approval was not sought for the present study because the study did not include any human or animal subjects.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. ICMJE forms for all authors are available online.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Matthew Griffith, MD,

Jeannie Huh, MD,

References

Amiri

Kanesalingam

Cro

Casey

. Does source of funding and conflict of interest influence the quality and outcome of spine research. Spine J. 2014;14(2):308–14.

Barske

Baumhauer

. Quality of research and level of evidence in foot and ankle publications. Foot Ankle Int. 2012;33(1):1–6.

Bhandari

Swiontkowski

Einhorn

, et al. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery . J Bone Joint Surg Am. 2004;86(8):1717–1720.

Cvetanovich

Fillingham

Harris

Erickson

Verma

Bach

Jr . Publication and level of evidence trends in the American Journal of Sports Medicine from 1996 to 2011. Am J Sports Med. 2015;43(1):220–225.

Ezzet

. The prevalence of corporate funding in adult lower extremity research and its correlation with reported results. J Arthroplasty. 2003;18(7 suppl 1):138–145.

Obremskey

Pappas

Attallah-Wasif

Tornetta

3rd Bhandari

. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87(12):2632–2638.

Okike

Kocher

Mehlman

Bhandari

. Conflict of interest in orthopaedic research. An association between findings and funding in scientific presentations. J Bone Joint Surg Am. 2007;89(3):608–613.

Thordarson

. Editorial: levels of evidence, evidence-based medicine, and Foot & Ankle International . Foot Ankle Int. 2008;29(9):881–882.

Weiner

Steinwachs

Frank

Schwartz

. Elective foot surgery: relative roles of doctors of podiatric medicine and orthopedic surgeons. Am J Public Health. 1987;77(8):987–992.

10.

Wright

Swiontkowski

Heckman

. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1–3.

11.

Wupperman

Davis

Obremskey

. Level of evidence in Spine compared to other orthopedic journals. Spine (Phila Pa 1976). 2007;32(3):388–393.

12.

Zaidi

Abbassian

Cro

, et al.

Levels of evidence in foot and ankle surgery literature: progress from 2000 to 2010?

J Bone Joint Surg Am. 2012;94(15):e1121–10.

13.

Oxford Centre for Evidence-Based Medicine. Levels of evidence. 2013. http://www.cebm.net/index.aspx?o=4590. Accessed October 21, 2019.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.11 MB