Abstract
Background
We reviewed the shoulder arthroplasty (SA) literature to correlate citations, methodological characteristics and quality of most-cited articles in this field. We hypothesized that a greater number of citations would be found for high-quality clinical studies.
Methods
We searched the Web of Knowledge database for the 50 most-cited articles about SA and collected author name, publication year, country of origin, journal, article type, level of evidence (LoE), subject of paper, type of arthroplasty and metrics (number of citations and citation rate). Coleman Methodology Score (CMS) was computed for clinical articles. Statistical analysis of variance and correlation coefficients were used to investigate the relationship between different variables.
Results
Out of the selected 50 studies on SA, 26% were nonclinical. There were 15,393 citations overall (mean 307.8), with a mean 19.5 citations per year (range 48.3-6.7). Thirty or 60% of all articles were LoE IV. All studies were published between 1984 and 2011 in 8 journals. Reverse SA (RSA) was the most common subject (36% of studies). The United States was the country responsible for most contributions (50% of studies). CMS ranged from 81 to 38 (mean 59.6). RSA received the highest number of citations (P < .001), independently from country of origin (P = .137) and LoE (P = .723). CMS correlated with citation rate (r = 0.397; P = .013) and publication year (tau = 0.397; P = .013), but not with LoE (P = .204).
Conclusion
In SA literature, citation rate positively correlates with methodological quality of a study, independently from publication country and LoE. Among most-cited papers, RSA is the most common standalone subject.
Introduction
Shoulder arthroplasty (SA) has become a commonly performed procedure over the last 2 decades, 1 mainly due to the advancements in implant designs and to the increased number of indications. However, there is still an open debate about surgical indications and techniques. 2 This may be a direct consequence of the paucity of high level of evidence (LoE) studies in the SA literature. In 2020, a Cochrane review, focused on SA for osteoarthritis, concluded stating that: “no high-quality randomized trials have been conducted to determine whether shoulder replacement might be more effective than other treatments for osteoarthritis or rotator cuff tear arthropathy of the shoulder.” Moreover, the use of shoulder prosthesis for the treatment of proximal humerus fractures has been questioned by prospective randomized controlled trials comparing conservative and surgical treatment. 3 On a general note, the number of peer-reviewed citations has broadly been considered as a proxy of the intellectual influence of an article, and it has become one of most important bibliometric instruments to assess the performance of academics, 4 institutes and journals. 5 However, it should be emphasized that a few studies have demonstrated that the number of citations is often not correlated to the quality of studies.6,7 This means that there is a risk that opinion leadership from a few influential professionals might guide physicians in their decision-making more than high-level studies. What is more, a “citation bias,” recently defined by Higgins et al 8 as a citation (or noncitation) which is related to the nature and direction of the results rather than the scientific quality of a manuscript, might strongly influence the number of citations of a study with chances that even articles of poor methodology could be highly cited.
To the best of our knowledge, the assessment of the quality of the most cited studies about SA has never been performed so far. In this context, our aim was to review the literature about SA in order to correlate the number of citations, the methodological characteristics, and the quality of most-cited articles in this field. We hypothesized that a greater number of citations would be found for high-quality clinical studies.
Materials and Methods
Study Selection
For this study, the Clarivate Analytics Web of Knowledge database was used to gather data and metrics. The literature search took place in December 2021. Journals from the portal “Journal Citations Report” were sorted by the number of total citations possessed by each journal under the category “Orthopedics.” Each single journal was then screened.
Search Strategy
The final Boolean search phrase was: [(shoulder) AND (prosthesis OR arthroplasty OR replacement) AND (shoulder prosthesis OR shoulder arthroplasty OR shoulder replacement)]. There were no date, language, journal, or country of origin restrictions. This resulted in a total of 15,462 articles. Articles that had SA as the focus of the study after a 2-step screening including title and abstract evaluation were included in the analysis, regardless of preoperative diagnosis or prosthetic type. If a study was unclear or if there was a question as to whether it should be excluded, the full article was obtained and reviewed by 2 independent authors (V.R. and A.G.) to ultimately decide on inclusion or exclusion.
Study Assessment and Data Extraction
The 50 studies with the highest number of citations that met the inclusion criteria were included in the analysis. These 50 studies were reviewed to obtain the following information: Author name, publication year, country of origin, journal name, subject of the article (basic science, therapeutic, prognostic, diagnostic, economic, nonclinical), study type (expert opinion, epidemiology prevalence study, review article, descriptive study, case report, case series, case-control study, cohort study, randomized controlled trial), and the LoE for clinical articles based on the guidelines of first-quartile journals. 9
We included Clinical and Nonclinical studies, such as reviews and metanalysis of literature, basic science studies and expert opinions. Reviews with no systematic approach were classified as “Expert opinion.” Meta-analyses and systematic reviews were classified as “Review.” The type of procedure was recorded as hemiarthroplasty, total SA (TSA), reverse SA (RSA), whether being the only surgical procedure described or combined (Es. TSA, TSA + RSA ecc). The citation density, calculated as the number of citations per year since publication, was computed for each of the 50 studies and recorded. The Coleman Methodology Score (CMS, Figure 1) was calculated for clinical studies to assess methodological quality of such articles included in our analysis: that applied to 37 papers out of the 50.10–46 The authors are not aware of any validated methodological quality score that could be used for nonclinical papers, given the extremely heterogeneous variables and outcomes included in those 13 studies, so they were not considered in quality assessment.
The Coleman Methology Score sheet used to evaluate clinical papers.
Statistical Analysis
The association between article metrics (Citation Density index and CMS) and the country of origin of the study, the LoE and the type of implant was based on the Kruskall–Wallis nonparametric analysis of variance followed by pairwise Mann–Whitney U-test for independent samples, without correction for multiplicity. Correlation between article metrics was based on the Spearman coefficient while the relationship between citation density and year of publication was assessed using the nonparametric Kendall's tau correlation coefficient. A P value of .05 was set to define significance. The statistical analysis was performed using Xsoftware.
Results
Results of the Search
The 50 most-cited articles on SA are shown in Table 1 and were published in 8 journals.
The “Journal of Bone and Joint Surgery—American Volume” published 20 (40%) papers, followed by the “Journal of Shoulder and Elbow Surgery” which published 19 (38%) articles. The “Clinical Orthopaedics and Related Research (CORR)” and the “Journal of Bone and Joint Surgery—British Volume” each featured 4 and 3 papers, respectively, while “Clinical Biomechanics”, the “Journal of the American Academy of Orthopaedic Surgeons,” “Orthopaedics & Traumatology—Surgery & Research” and “Orthopedics” each released 1 article of the list. All papers were published between 1984 to 2011, but only 4 articles were found between 1990 and 2000. Conversely, the decade with the highest number of publications was 2001 to 2010 with 37 papers being published. As for the decade 2010 to 2020, 5 articles were published in the year 2011 and represent the totality of the papers published this range of years. In total, 6 papers were written before the year 2000, in the 2 decades between 1980 and 2000. The authors most frequently involved in these papers were Pascal Boileau (6 times as first author and 3 times as co-author) and Gilles Walch (author or co-author in 8 papers), both from France. The most investigated procedure was RSA by itself (18 studies), TSA (Anatomic) 9 times and hemiarthroplasty alone 4 times. All 3 types of procedures were mentioned in the same article 3 times, Anatomic and Hemiarthroplasty were the subject of analysis together 12 times. We conducted an analysis of the indication for which arthroplasty—whether hemi, total or reverse—had been used as therapeutic mean. The indications for surgery have been depicted in Figure 2.
Indication for arthroplasty in the selected articles and the respective number of papers.
In total, 23 times the operation had been performed as elective surgery. Trauma was the only etiology in 7 papers, while both elective and trauma surgery were considered in 18 articles.
Bibliometric Analysis
The number of citations ranged from 788 to 181 (mean 307.8) for a total of 15,393 citations.
The Citation Density calculated for each article ranged from 48.3 1 to 6.7 citations per year, 26 with a mean of 19.5 citations per year. Articles originated from 8 countries, with the United States of America having the most contributions (25), followed by France (14), Switzerland (4), Netherlands (2), Austria, Canada, England and Germany (1) (Table 2.).
Number of Most Cited Articles in the List Published per Country of Origin.
The mean number of citations for Level I articles (6) was 281.5 (range, 445-186), while it was 302.8 (range, 577-200) for Level II articles (5), 271.3 (range, 306-239) for Level III articles (3) times and 308.9 (range, 788-181) for Level IV articles (30). The only paper with an LoE of V 47 was cited 559 times.
The mean CMS score for clinical studies (37 articles) was 59.60 values, 59ranging from 81 to 38.
A significant difference was observed in Citation Density when stratifying articles with respect to the type of implant (P < .001) (Figure 3.).
Correlation between type of prosthesis object of the paper and citation density showing significancy toward reverse shoulder arthroplasty (RSA)-only papers (P < .001).
In particular, articles describing RSA showed a significantly higher CD than those describing ASA (P < .001) or both Anatomic and Hemi arthroplasty (P = .003). No difference was observed based on the country of origin and the LoE (P .137 and .723, respectively) (Figures 4 and 5).
Correlation between country of origin of the article and citation density is nonsignificant (P = .137).
Correlation between level of evidence (LoE) and citation density is nonsignificant (P = .723).
The CMS did not show any significant association with the LoE (P = .204)(Figure 6).
Correlation between LoE and CMS of the articles in the selection is nonsignificant (P = .204). Abbreviations: LoE, level of evidence; CMS, Coleman Methodology Score.
The Citation Density showed a significant moderate correlation with the CMS (Pearson r = 0.397; P = .013) and with the year of publication (Kendall's tau = 0.397; P = .013) (Figure 7a & b.), indicating that more recent articles tend to be cited more in a shorter amount of time after the publication date.
Correlation between the citation density and the Coleman Methodology Score (CMS) (a) and year of publication (b) of the articles in the selection has been proven significant, suggesting that most recent articles and better methodology present a higher number of citations.
Discussion
Compared with similar topics in other specialties, SA has fewer mean citations in scientific publications. Holzer and Hozler 58 published a list of the 50 highest cited papers on hip and knee arthroplasty with a mean citation number of 550 citations (ranging from 2495 to 347), even though from the date of publication of this paper the sheer number of published articles on the subject greatly increased. 59 For instance, on hip arthroplasty alone Zhang et al 60 found in a similar bibliometric analysis a mean of 312 citations overall (ranging from 994 to 191) in a selection of the 100 most cited papers, recently in 2019.
Noticeably, most papers on the list came from the journal with the highest impact factor to date among the 8 journals involved, the JBJS American Volume (5.284 in 2020), while the second on the list for most papers included, the JSES, with just 1 less article, is not the second-ranked based on impact factor. We think this may be due to the highly specialized themes of this journal. Overall, 39 papers on 50 came from these 2 journals. CORR, second-highest journal based on impact factor to date, came in third for the number of articles: this may be due to the fact that most articles are dated in the years between 2001 and 2010, and at that time this same journal had an Impact factor of max 2.5 points, steadily increasing during the following years.
Also, to correlate the dates of publishing of articles regarding certain prosthesis models may give us an idea of the trends in SA literature: anatomic prosthesis and hemiarthroplasty tend to concentrate in early 2000s, while reverse arthroplasty with and without other types of implants seems to be a main theme in late 2000s.
Moreover, looking into the evidence levels in light of the quality assessment using the CMS can give us interesting informations on the fact that level 4 papers, in addition to being the vast majority, tend to have low mean methodological quality, and with the rising of the evidence level the mean methodological quality tends to rise, even though this data didn’t show statistical significance due to the quantitative difference in favor of LoE IV articles.
Nonetheless, papers on the upper-side of the methodological value-curve also tend to get more citations, but with numerous exceptions: probably because of a widespread tendency to quote “opinion leaders” in shoulder-surgery-themed scientific literature, with a less rigorous look at the actual scientific accuracy of said papers. For instance, when cross-checked with the renown Cochrane library selection on 2 recent reviews and meta-analyses on elective SA 2 and the treatment of proximal humerus fracture, 61 we were able to find only 2 papers29,46 that were included both in our citations-guided list and in the elective arthroplasty review. And, even though included as highly relevant from a clinical perspective, both were judged as heavily biased for several reasons (unclear how well and how long participants were blinded; high risk of bias for physician-evaluated outcomes; incomplete outcome data; unclear risk of bias due to selective reporting; major baseline imbalance for patients included). Lastly, the fact that up to 26% of the articles in the 50 most cited about SA explore nonclinical aspects of the matter at hand, should highlight how some papers defining core concepts keep being cited even in most recent years, despite not putting the spotlight on the clinical outcome.
Limitations
This study has several limitations. The identification of the 50 most cited papers, even if based on well-defined criteria, remains a challenging selection. Therefore, important and influential papers with lower citation frequencies may have been overlooked. In some cases, the value of the contribution to the subject cannot be quantified by the number of citations. The number of citations of articles may be influenced by various factors and does not necessarily reflect the importance of the research performed or published, nor does it directly translate into changes in clinical practice. The methodological assessment by CMS couldn’t be applied to all the papers included, given the nonclinical nature of some of them. The search was conducted on the Thomson ISI Web of Science®, therefore citations of articles from other sources such as textbooks, lectures, or digital media, could not be included.
To conduct a statistical analysis given these nonordinal variables is challenging and different testing may lead to different conclusions.
Conclusions
Statistical evidence demonstrates a positive association between the methodological quality of an article addressing SA and the amount of citations that the study receives over time, regardless of the publication country or the LoE. Intriguingly, many nonclinical studies continue to be cited, whereas the vast majority of clinically influential articles are level IV papers. The high ratio of basic science to clinical data and the low evidence level of those clinical studies indicate the need for additional randomized clinical trials to be conducted in the field of SA research, or should at least suggest that level I and II papers have a difficult time garnering attention in this field.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
