Abstract
Background:
Objective measures of research influence are being increasingly utilized to evaluate and compare academic faculty. However, traditional bibliometrics, such as the Hirsch index and article citation count, are biased by time-dependent factors and are limited by a lack of field normalization. The relative citation ratio (RCR) is a new field- and time-normalized article-level metric developed by the National Institutes of Health (NIH).
Purpose/Hypothesis:
The purpose of this study was to evaluate the RCR among fellowship-trained academic sports medicine surgeons and to analyze physician factors associated with RCR values. We hypothesized that the mean RCR score for fellowship-trained academic sports medicine surgery faculty will fall above the NIH standard.
Study Design:
Cross-sectional study.
Methods:
A retrospective data analysis was performed using the iCite database for all fellowship-trained sports medicine surgery faculty associated with Accreditation Council for Graduate Medical Education–accredited orthopaedic surgery residency programs in December 2021. In eligible faculty, the mean RCR, weighted RCR, and total publication count were compared by sex, career duration, academic rank, and presence of additional degrees. A mean RCR value of 1.0 is the NIH-funded field-normalized standard. The data herein are presented as the median and interquartile range, in addition to the mean and standard deviation, to account for outliers of the mean and weighted RCR scores.
Results:
A total of 624 fellowship-trained sports medicine surgery faculty members from 160 orthopaedic surgery residency programs were included in the analysis. Overall, faculty produced impactful research, with a median RCR of 1.6 (interquartile range, 1.0-2.2) and a median weighted RCR of 19.3 (interquartile range, 5.1-69.3). Advanced academic rank and career longevity were associated with increased weighted RCR and total publication count. All subgroups analyzed had an RCR value >1.0.
Conclusion:
Study findings indicate that fellowship-trained academic sports medicine surgery faculty are highly productive and produce impactful research, as evidenced by the high median RCR value relative to the benchmark NIH RCR value of 1.0.
Scholarly activity remains one of the hallmarks of achievement in academic medicine. 9 Objective measures of scholarly activity, such as research productivity, are being increasingly utilized within academic settings to evaluate new faculty hires and the assessment of grant funding. 14 In addition, scholarly activity is often considered a prerequisite for academic advancements for fellows and faculty in tenure-track positions. 6
Historically, objective assessment of research productivity was based on simple count metrics (eg, number of publications, number of presentations, total grant funding). 11,12 However, a major shortcoming of simple count metrics included a lack of quality assessment. 4 While bibliometric indices, such as journal impact factor and the Hirsch index (h-index), have been widely adopted to evaluate and compare the academic impact among researchers, 3,6 their drawbacks are well documented. For example, journal impact factor obscures major differences in the influence of individual articles and is dependent on research discipline. The h-index is generated by combining the frequency of publication and the frequency of citation into a single metric. 8,11 As such, the h-index may underestimate the research influence of a younger author with a smaller number of impactful publications. Conversely, the h-index may overestimate the research influence of an older author with a larger number of less impactful publications. Furthermore, because the h-index is not field normalized, the citation potential of a publication relies on the size of the academic field in which it is published. 2 This serves to limit cross-specialty comparisons. 2,10,13,14
The US National Institutes of Health (NIH) recently released a new article-level metric, the relative citation ratio (RCR), which aims to improve on the drawbacks of traditional bibliometric indices. 14 The RCR utilizes a co-citation network of an article to normalize its research impact to that of others in the field and compares this performance with a peer comparison group. 7 Specifically, the RCR is calculated as the total number of citations per year of a particular publication divided by the mean number of citations received per year by NIH-funded articles in the same field. 11 This method of dynamic field normalization is unique to RCR and allows for analysis across academic fields. In addition, author-level derivates of the RCR may be used to quantify a researcher’s overall impact and productivity. The mean RCR represents the mean of all article-level RCR scores pertaining to a single researcher. This value eliminates bias from time-dependent variables by removing the impact of total publication count and allows for a fair comparison of research impact among authors at different career stages. Yet, the weighted RCR represents the sum of all article-level RCR scores pertaining to a single researcher and therefore serves as a measure of research productivity. 4
Several recent studies have evaluated the use of RCR across a number of academic specialties 10,11,13,14 and have been able to provide insight into the overall and subgroup productivity within various medical and surgical specialties. Herein, we conducted an RCR analysis among fellowship-trained academic sports medicine surgery faculty across the United States. The objective of this study was to provide benchmark data for RCR scores within the field of academic sports medicine surgery and to identify correlates between these scores and various demographic groups, including sex, career duration, academic rank, and acquisition of a doctor of philosophy (PhD) degree. We hypothesized that the mean RCR score for fellowship-trained academic sports medicine surgery faculty will fall above the NIH standard. The information presented in this study may serve as a more accurate gauge of research impact within the sports medicine community and can be used for individual self- and departmental evaluation as well as cross-specialty comparisons.
Methods
Departmental and Faculty Inclusion Criteria
All fellowship-trained sports medicine surgeons employed as faculty at orthopaedic surgery residency programs accredited by the Accreditation Council for Graduate Medical Education were included in our analysis. Individual departmental websites for each accredited residency program were accessed in December 2021 (https://apps.acgme.org/ads/Public/Programs/Search) to identify all fellowship-trained sports medicine surgeons. Faculty sex, academic degrees, academic rank, and residency start year were obtained using physician profiles on departmental websites or via publicly available outlets. From lowest to highest, academic rank included assistant professor, associate professor, and full professor. Clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified were listed as “other.” Residency start years were obtained to categorize faculty into the following groups: ≤1980, 1981 to 1990, 1991 to 2000, 2001 to 2010 and >2010. Our analysis inclusion criteria included any sports medicine surgeons who completed an orthopaedic surgery residency, completed a sports medicine surgery fellowship, and were listed as faculty on their corresponding department websites.
Bibliometric Analysis
The RCR for an individual publication is described as the total number of citations per year for that publication divided by the mean field-specific citations per year received for all NIH-funded publications in the same field. Thus, a ratio of 1.0 represents the field-normalized, NIH-funded standard. Author-level RCR scores (mean and weighted RCR) are calculated from the aggregate article-level RCR scores for all publications produced by an individual author. The mean RCR is simply the statistical mean of all RCR scores for publications produced by an individual author. The weighted RCR is the sum of all RCR scores for publications produced by an individual author.
Fellowship-trained sports medicine surgery faculty were individually indexed using the NIH iCite database website (https://icite.od.nih.gov). Nonoriginal research articles (ie, editorials, reviews, and meeting abstracts) as defined by the iCite database were excluded. The iCite database currently contains PubMed-listed articles between 1980 and the present. The number of total publications, mean RCR score, and weighted RCR score were collected for each author on January 4, 2022.
Statistical Analysis
The mean and weighted RCR scores were collected from the iCite search output for each fellowship-trained sports medicine surgery faculty member and compared by sex, academic degree, academic rank, and career longevity as defined by residency start date. Student t tests were used for 2-group analyses, whereas the analysis of variance test was used for between-group comparisons of ≥3 subgroups. Statistical significance was achieved at P < .05. The data are presented as means and standard deviations as well as medians and interquartile ranges (IQRs) to account for outliers of the mean and weighted RCR scores.
Results
A total of 624 fellowship-trained academic sports medicine surgery faculty members were included in this study. The majority were male (n = 569; 91.2%) and approximately 1.6% had a PhD (n = 10). Table 1 lists the demographic characteristics of the participants. Overall, RCR scores were high but widely variable, with a median RCR of 1.55 (IQR, 1.02-2.21) (Table 2) and median weighted RCR of 19.1 (IQR, 5.0-69.3) (Table 3). The median number of publications produced per faculty member was 15 (IQR, 4.0-45) (Table 4). An overview of the mean and weighted RCR data for all fellowship-trained academic sports medicine surgery faculty members is depicted as box plots in Figure 1. Tables 2 to 4 show RCR scores, weighted RCR scores, and number of publications, stratified by sex, PhD status, academic ranking, and career duration.
Demographic Characteristics of the Study Participants (N = 624)
a Assistant professor includes clinical assistant professor, instructor, and lecturer.
b Other indicates clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified.
RCR Scores Overall and by Sex, PhD Acquisition, Academic Ranking, and Residency Start Year a
a IQR, interquartile range; RCR, relative citation ratio.
b Assistant professor includes clinical assistant professor, instructor, and lecturer.
c Other indicates clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified.
Weighted RCR Scores Overall and by Sex, PhD Acquisition, Academic Ranking, and Residency Start Year a
a Bold P values indicate statistically significant difference in median values within variables (P < .05). IQR, interquartile range; RCR, relative citation ratio.
b Assistant professor includes clinical assistant professor, instructor, and lecturer.
c Other indicates clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified.
Number of Publications Overall and by Sex, PhD Acquisition, Academic Ranking, and Residency Start Year a
a Bold P values indicate statistically significant difference in median values within variables (P < .05). IQR, interquartile range.
b Assistant professor includes clinical assistant professor, instructor, and lecturer.
c Other indicates clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified.
(A) Mean and (B) weighted relative citation ratio (RCR) data for all fellowship-trained sports medicine surgery faculty at academic institutions. The center line shows the median; box limits indicate the 25th and 75th percentiles as determined by R software; whiskers extend 1.5 times the interquartile range from the 25th and 75th percentiles; and dots represent outliers. N = 624 sample points.
Academic Ranking
The most common academic ranking was assistant professor, with 198 members (31.7%); professors represented 17.0% and associate professors 21.3%. The remaining 30.0% were categorized as other.
There was a positive correlation of median publication count (P < .001) and weighted RCR (P < .001) with academic ranking. Full professors were the most productive subgroup in our study, with a median weighted RCR of 83.54 (IQR, 33.22-220.61) and a median 53 publications (IQR, 18.0-120.0). There was no significant association between mean RCR and academic ranking (P = .21).
Sex
There was a statistically significant difference in median weighted RCR scores between female and male participants (26.55 [IQR, 6.50-68.13] vs 19.15 [IQR 4.98-69.09]; P = .043). However, there was no statistically significant difference between the sexes in median RCR score (P = .53) or median publication count (P = .28).
PhD Degree
Faculty members with a PhD had a median RCR of 1.97 (IQR, 1.48-2.51), and non-PhD faculty had a median RCR of 1.53 (IQR, 0.99-2.21); however, this finding was not statistically significant (P = .14). There was also no statistically significant difference between PhD and non-PhD faculty in median weighted RCR scores (P = .25) or median publication count (P = .18).
Career Longevity
Longer career duration, as defined by residency start date, had a significant impact on median weighted RCR scores (P < .001) and total publication count (P = .001). Participants with a residency start date between 1981 and 1990 had the highest median weighted RCR (45.76; IQR, 9.26-161.07) and publication count (31.00; IQR, 8.00-96.00), while those with the shortest career duration (residency start date >2010) had the lowest median weighted RCR (12.99; IQR, 5.51-40.46) and publication count (14.00; IQR, 4.25-28.00). No significant association between median RCR and career longevity was found (P = .185).
Discussion
In the current study, we found that fellowship-trained academic orthopaedic sports medicine surgery faculty produce highly impactful research, as validated by a high median RCR value (1.55; IQR, 1.02-2.21) in comparison with the NIH standard (1.0). The RCR helps to overcome some of the weaknesses of other productivity indices and has the ability to serve as a more comprehensive assessor of research impact within the sports medicine community.
Of all PubMed-listed publications in the iCite database, the median RCR was 0.37 (range [10th-90th percentile], 0-2.24), whereas the median RCR for all NIH-funded publications was 1.00 (range [10th-90th percentile], 0.38-3.81). In our study, the median RCR was 1.55 (IQR, 1.02-2.21), which suggests that publications of fellowship-trained academic sports medicine surgery faculty are highly influential when compared with PubMed-listed and NIH-funded publications.
Several recent studies have evaluated the use of RCR across a number of academic specialties. 10,11,13,14 Rock et al 14 evaluated the use of the RCR among 1299 radiation oncology physician faculty members from 75 institutions and reported a mean RCR score of 1.32 (range, 0.87-1.94). Similarly, Reddy et al 13 studied the use of RCR within the field of academic neurosurgery among 1687 neurosurgery faculty members from 125 institutions, reporting a mean RCR score of 1.37 (range, 0.93-1.97). These findings were supported in a follow-up study of 358 fellowship-trained academic spine neurosurgeons that found a median RCR of 1.38 (range, 0.94-1.95). 4 Our data suggest that research impact among academic sports medicine surgery faculty may be slightly higher than that of other specialties.
Research productivity is an important factor in professional advancement among academic sport medicine surgeons. 3 In a study of 610 faculty members from the AOSSM, cumulative h-index (1.22; P < .001) was strongly correlated with higher academic rank. 3 Similarly, we found a significant association between weighted RCR scores and advancing academic rank (P < .001). In addition, those with greater career longevity had higher weighted RCR scores (P < .001). These findings come as no surprise, as those with longer career durations and higher academic rank also had significantly greater publication counts (P = .001 and P < .001, respectively). Overall, these trends are consistent with prior studies of the older h-index among academic sports medicine surgeons, 1,3,5 indicating the validity of the weighted RCR as a measure of overall research productivity.
A primary criticism of the h-index originates from its tendency to disadvantage younger authors, who are likely to have published less frequently when compared with those of older age and greater career longevity. 13 It is therefore necessary for newer metrics to reduce the influence of total publication count, which naturally increases with age and career duration. 13 The mean RCR represents the mean of all article-level RCR scores pertaining to a single researcher. 13 Because this metric does not incorporate the total citation count of an article, it is able to quantify research impact without bias from time-dependent factors. Thus, mean RCR allows for fair assessment of overall research impact among those at different career stages. In our study, there was no significant association between median RCR score and career longevity (P = .185) or academic rank (P = .210). These findings suggest that while more experienced faculty members of higher academic rank have greater overall research productivity (ie, significantly higher weighted RCR scores and h-indices), publication-level impact has remained consistent over time regardless of academic position or career longevity. Perhaps the use of RCR as a measure of research influence will be incorporated into academic advancement decisions in the near future.
In our study, sex-specific analysis revealed no significant impact on mean RCR (P = .530
Similar to our findings for academic sports medicine surgeons, other studies have found no significant difference in median RCR among male and female neurosurgeons 13 or radiation oncology physicians. 14 Additionally, no significant difference in cumulative h-index has been identified between male and female AOSSM members. 3 However, it is important to note that female faculty were highly underrepresented among these cohorts, accounting for only 7% of AOSSM members, 9% of neurosurgeons, and 31% of radiation oncology physicians studied. Perhaps calculated research impact will continue to change over time with more equal sex representation among medical disciplines.
No significant differences in median publication count (29.0 vs 14.0; P = .180), median RCR (1.97 vs 1.53; P = .140), or median weighted RCR (63.4 vs 18.5; P = .247) were found among academic sports medicine surgeons with and without a PhD. Although PhD faculty seemed to generate a greater number of impactful studies (ie, total publication count and RCR) when compared with those without a PhD, assertions are difficult to make given the underrepresentation of PhDs within this cohort (1.6%; 10 of 624).
Limitations
Our study is not without limitations. Although we evaluated a large number of fellowship-trained orthopaedic sports medicine surgeons (N = 624), only those affiliated with Accreditation Council for Graduate Medical Education–accredited institutions across the United States were included. Our analysis relied on information provided on department websites, which represents a possible limitation as information on the website may not have been up-to-date or accurate at the period of our data collection. In addition, our inclusion criteria required surgeons in our analysis to be fellowship trained. This represents a possible limitation in data for older surgeons, as fellowships have increased in popularity over the past few decades and may exclude surgeons trained in the past when fellowships were less common. Despite these limitations, we employed our inclusion criteria to best capture orthopaedic sports medicine surgery faculty with equivalent training for our analysis.
Therefore, our data may not be generalizable to the entirety of the sports medicine surgery specialty, as surgeons without academic affiliation were not accounted for. However, research productivity among private orthopaedic sports medicine surgeons has been shown to be less than that of their academic counterparts. 3 Reasons for this difference include a lack of research requirement in the private sector, less resident and fellow influence, and decreasing monetary impetus, as advancing rank and salary are mainly driven by clinical activity. 3 Because benchmark RCR data are intended to represent a baseline standard of specialty-specific research influence for the purpose of academic advancement and departmental evaluations, inclusion of academic-affiliated surgeons was thought to provide the most accurate RCR value. Furthermore, a primary advantage of the RCR is the ability to draw cross-specialty comparisons of research influence. As other studies evaluating the use of RCR within other areas of medicine focused on academic-affiliated faculty, 4,11,13,14 it was prudent to do the same.
As with other bibliometric indicators, the RCR does not account for varying levels of author seniority. Furthermore, the iCite website does not differentiate among researchers with the same name. Potential errors were limited by searching middle initials and reviewing individual publication titles. Additionally, the iCite website included only PubMed articles published between 1980 and the present, which may undervalue the RCR of researchers with publications before 1980.
Conclusion
The RCR serves as a new, more accurate metric of research impact that addresses several drawbacks of traditional bibliometrics. Our study showed that fellowship-trained sports medicine surgery faculty are highly productive and produce impactful research when compared with physicians in other specialties and the general scientific community. This information can be used as a standard to assess the improvement of grant outcomes, promotion, fellowship education, and continued evaluation of research productivity in the sports medicine surgery community.
Footnotes
Final revision submitted May 25, 2022; accepted August 2, 2022.
One or more of the authors has declared the following potential conflict of interest or source of funding: R.P. has received education payments from Arthrex. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.