Abstract
Background:
The h-index helps quantify an individual's academic contributions and is widely used for academic promotion. A new bibliometric, called the Research Interest (RI) score, is now also being used to measure academic productivity.
Purpose:
To (1) determine whether the h-index and the RI score are associated with academic rank and other common metrics considered in academic promotion for sports medicine orthopaedic surgeons in the United States; (2) conduct a secondary analysis to find any correlations between patient satisfaction and h-index, RI score, or academic rank.
Study Design:
Cross-sectional study.
Methods:
A list of 183 orthopaedic surgery residency programs was generated using the American Medical Association's Fellowship and Residency Interactive Database Access System. Academic sports medicine orthopaedic surgeons were identified through institutional and personal websites/profiles. The Scopus database was queried to determine the h-index for each surgeon. RI scores were identified through the ResearchGate website. Patients’ satisfaction scores were obtained with Vitals and HealthGrade.
Results:
In total, 835 academic sports medicine orthopaedic surgeons were identified in the United States. H-indices increased with higher academic rank: 6.5 for instructors/other, 6.8 for assistant professors, 14.4 for associate professors, and 29.9 for professors. RI scores also increased with higher academic rank: 363, 503.3, 850.1, and 2893.6, respectively. However, there was no correlation between the h-index and the RI score (r < .001). The mean h-indices for each region of the country (West, Midwest, South, and Northeast) were found to be 16.27, 15.26, 11.49, and 15.74, respectively. South versus West and South versus Northeast had significant differences. No correlation was found between patients’ satisfaction scores and h-index, RI score, and academic rank.
Conclusion:
The h-index is a valid measure of academic productivity and rank among academic sports medicine surgeons in the United States. Some regional differences exist in the h-index score. Both the h-index and the RI score increased with higher academic rank, but they were not correlated. No correlation was found between patient satisfaction scores and academic productivity indices. The RI score has potential bibliometric use, but further research is warranted on its validity as a productivity measure before its inclusion for academic promotion.
When academic committees consider faculty recruitment or promotion, the candidates are typically evaluated based on contributions to education, clinical service, leadership roles, and research as the 4 major scholarly missions.2,5 As research activities are acknowledged as an important factor for such advancements, 15 there have been many attempts to establish a standardized measure to thoroughly and accurately quantify research productivity. These standardized measures throughout time have included but are not limited to the h-index, g-index, a-index, r-index, f-index, and t-index, each with its shortcomings. 38 Of these standardized measures of research productivity, the h-index has been the most widely studied and used to characterize both research quantity and quality.
The h-index was introduced by Jorge E. Hirsch 20 in 2005 as a means to assess the quality of scholarly contributions and complement straightforward numerical metrics, such as the total number of publications. The h-index mathematically adjusts for both quality and quantity by calculating the number of publications (h) that have attained at least h citations each. For example, an author with an h-index of 10 would have 10 published papers that have been cited at least 10 times each. Likewise, an author who has 20 publications, each of which has attained at least 15 citations per article, will have an h-index of 15. The h-index has been shown not only to be useful in determining research productivity but also effective in predicting future academic productivity. 21 It has been widely accepted and is becoming a common metric used for evaluating faculty members applying for promotions. Although the h-index was initially described for the field of natural sciences, it has also been studied in various medical and surgical specialties, including radiology, anesthesiology, emergency medicine, otolaryngology, neurosurgery, and ophthalmology.8,10,25,33,35,36 These studies found a strong association between h-index and academic rank. Because these matrices for academic promotion can differ from specialty to specialty,12,15,36 it is important to investigate the utility of the h-index within orthopaedic subspecialties.
Other emerging metrics similar to the h-index that have been used to quantify research productivity and quality include the original ResearchGate (RG) score as well as the current Research Interest (RI) score.22,24,40 ResearchGate is a rapidly expanding social networking website aimed at helping researchers identify potential research collaborations and share recent publications. Previously, ResearchGate assigned each researcher an RG score that was calculated based on both the number of publications and the amount of positive feedback about each publication, or the number of times that each publication is read or downloaded within the ResearchGate community. However, this metric was noted to be flawed due to there being unexplainable changes to researchers’ scores, the metric not being nontransparent and irreproducible at times, and the incorporation of a journal's impact factor into evaluating researchers. 24 ResearchGate responded to these claims from individual researchers by agreeing that there needed to be a metric that was more intuitive, transparent, and could not be artificially manipulated. 32 As a result, the RG score was abandoned completely in August of 2022. Subsequently, ResearchGate launched a separate metric to replace the RG score, called the RI score—a weighted score composed of multiple components used to measure the impact of research within the scientific community. 28 This score takes into account how many times a research item's summary was read, if it was read entirely and/or downloaded, if another author recommended it, and if it was cited. When these activities occur, the author of that particular research item gets an increase in their RI score by 0.05, 0.15, 0.25, and 0.50, respectively. 28 Although the RI score has been underutilized to date, we believe that it will serve as a reliable metric of academic productivity and research quality when reviewing an orthopaedic sports medicine surgeon being considered for academic promotion. Citations are an important aspect of scientific impact; nonetheless, citations alone sometimes do not take into account the full impact within that scientific community. With the introduction of the RI score, there is now a metric that also takes into account when an article is reviewed online, downloaded, recommended, as well as cited, and these are all important aspects of impact. Thus, we believe that as the ResearchGate community continues to grow, the RI score will also continue to grow in its respective value and can be used in conjunction with the h-index and other metrics to evaluate orthopaedic sports medicine surgeons when considering them for academic promotion.
To date, there has not been an analysis of how the h-index or the RI score relates to academic rank among sports medicine orthopaedic surgeons in the United States. This study aimed to examine the relationship between h-index, RI score, and academic rank, and assess the validity of each bibliometric as a means of predicting academic rank or promotion within sports medicine orthopaedic surgery. Secondary analysis compared the h-index with other measures of academic productivity, such as the number of Scopus 34 referenced publications, sex, age, region within the United States, number of years in practice, number of years spent at respective current institutions, other leadership positions held, additional sports team positions held, additional degrees obtained, and patient satisfaction—measured by Vitals ratings and HealthGrades ratings. We hypothesized that the h-index and the RI score would positively correlate with each other and with increasing academic rank; thus, both measures could be used in conjunction with each other as indicators for productivity when considering an orthopaedic surgeon for academic promotion.
Methods
Data Collection
The American Medical Association's Fellowship and Residency Interactive Database Access System 16 was used to generate a list of the 183 orthopaedic surgery residency programs in the United States. Listings from each program's institutional website were used to identify faculty members. The inclusion criteria included full-time faculty members who were appointed under the division of sports medicine and/or indicated as interested in sports medicine on the institutional website description. To focus solely on sports medicine orthopaedic surgeons, faculty members with primary care sports medicine training and nonphysician degrees were excluded from the study. A total of 835 academic orthopaedic surgeons from 183 programs in 43 states were included in the analysis.
The department website was used to collect basic information about each faculty member, such as name and academic rank. The academic rank for each orthopaedic surgeon was categorized as instructor/other, assistant professor, associate professor, or professor. The h-index of each faculty member was calculated using the Scopus database on June 20, 2022. The length of career was determined using the year of fellowship training completion as documented on the department website as a surrogate for the beginning of an academic career. Each surgeon was also marked yes or no whether they held additional sports positions, such as the team physician for an athletics department, regardless of whether that was through the high school, collegiate, or professional level. Institutional websites, Doximity profiles, LinkedIn profiles, online curriculum vitae, and personal websites were used to determine the length of time at the current institution based on the indicated year hired listed on respective websites. Information from each website was cross-matched to ensure accuracy. If there was uncertainty regarding the accuracy of information, it was not used. Several publications on Scopus for each faculty member were collected, along with the RI scores that were recorded from the ResearchGate website. As a proxy for patient satisfaction, Vitals 14 ratings/number of reviews and Healthgrades 19 ratings/number of reviews were also recorded for each surgeon. Only ratings that had at least a minimum of 10 reviews were included in this study.
Statistical Analysis
Univariate analysis and multivariable regression analysis were performed to control for independent variables. Pearson correlation coefficients were calculated to assess the relationship between h-index and academic rank, RI score, Vitals rating, and Healthgrades rating. All statistical analyses were performed using Python. The fitted regression model was calculated to predict the degree of the h-index association with other measures of academic productivity. The threshold for statistical significance was set at P < .05.
Regression Analysis
Multivariable linear regression was performed using a backward elimination technique to investigate the relationship between the h-index/the RI score and candidate variables. The initial candidate variables were as follows: assistant professor (yes/no), associate professor (yes/no), professor (yes/no), years since fellowship, PHD (yes/no), years at institution, sex (M/F), age, RI score, Healthgrades, and Vitals score. The final model included associate professor, professor, years since fellowship, PhD, years at institution, and RI score. Individual regression models were built to investigate the correlation between the h-index and the following variables: professor (yes/no), assistant professor (yes/no), associate professor (yes/no), RI score (yes/no), Healthgrades score, and Vitals score.
Results
Summary Statistics
A total of 835 academic orthopaedic surgeons from 183 programs in 43 states were identified and included in the analysis. Of these 835, a total of 378 surgeons are members of the ResearchGate community with identifiable RI scores. In total, there were 33 instructors/others, 370 assistant professors, 234 associate professors, and 198 professors. There were 90 female and 745 male surgeons, with a mean age of 52 ± 10.2 years, and the number of years in practice of 16 ± 10.4 years. The mean number of publications was 55.7 ± 83.2, the mean h-index score was 14.4 ± 15, the mean RI score for the 378 surgeons with RG memberships was 1364.7 ± 2218.1, the mean HealthGrades score for 748 surgeons was 4.2 ± 0.76, and the mean Vitals score for 689 surgeons was 4.2 ± 0.60. These demographic characteristics are summarized in Table 1. The mean h-index was 14.7 ± 2.1 and 11.9 ± 2.1 for male and female surgeons, respectively. Surgeons were categorized based on rank by instructor/other, assistant professor, associate professor, and professor. The mean age was 52.2, 48.3, 50.2, and 59.4 years, and the mean h-indices were 6.5 ± 8.6, 6.8 ± 6.9, 14.4 ± 9.4, and 29.9 ± 19.6, respectively (Figure 1). Out of the 835 orthopaedic surgeons, 283 had h-indices that ranged between 0 and 5. The number of surgeons with higher h-indices declined gradually. For simplicity, orthopaedic surgeons with the highest h-indices were classified into 1 group between the range of 40 and 118, and this amount was found to be 59 (Figure 2). Mean RI scores for each academic rank were 363 ± 615.6, 503.3 ± 805.1 ± 902.2, 850.1 ± 902.2, and 2893.6 ± 3282.4, respectively (Figure 3). H-index scores of surgeons with Research Gate profiles (n = 457) were greater than those without (n = 378) (16, 14.3, and 12.5, respectively). Mean h-indices were also calculated for each region of the country (West, Midwest, South, and Northeast) and were found to be 16.27, 15.26, 11.49, and 15.74, respectively (Figure 4). South versus West and South versus Northeast had significant differences (P = .013 and P = .035, respectively).
Demographic Characteristics of Sports Medicine Orthopaedic Surgeons per Academic Rank a
Data are presented as mean ± SD, unless otherwise indicated. H-index, Hirsch index; RI score, Research Interest score.
The mean h-index per academic title. H-index, Hirsch index.
H-indices of academic sports medicine orthopaedic surgeons in the United States. H-index, Hirsch index.
The mean RI score per academic title. RI, research interest.
The mean h-index per region of the United States. H-index, Hirsch index.
Univariate Analyses
The univariate analysis found that professorship and h-index had the strongest correlation at 0.58 (P < .001), corresponding to a moderate correlation. Assistant professorship had a correlation coefficient of 0.45 (P < .001), which is a low correlation. Associate professorship, RI score, Healthgrades score, and Vitals score had correlation coefficients of <0.01, indicating negligible correlations.
Multivariable Regression Analysis for H-index
After eliminating insignificant variables from the initial model, the following variables were included: associate professor (P < .001), professor (P < .001), years since fellowship (P = .010), PhD (P < .001), and years at current institution (P < .001). The R2 and adjusted R2 for the final model were 0.457 and 0.453, respectively. Multiple model regression coefficients for the h-index and their associated P values are summarized in Table 2.
Multiple Model Regression for H-index a
H-index, Hirsch index.
Multivariable Regression Analysis for RI Score
The following variables were significant for the model: professor (P < .001) and years at current institution (P < .001). The R2 and adjusted R2 for the final model were 0.160 and 0.158, respectively. Multiple model regression coefficients for the RI score and their associated P values are summarized in Table 3.
Multiple Model Regression for RI Score a
H-index, Hirsch index.
Discussion
This study demonstrates that the h-index correlates with increasing academic rank within sports medicine orthopaedic surgery, number of publications, years in practice since graduation from fellowship, whether a surgeon has a PhD degree or not, and years at respective current institutions. However, there was a negligible correlation between h-index and RI score, region, sex, and patient satisfaction, measured via Healthgrades and Vitals review scores. The mean h-index in our sample of 835 orthopaedic sport surgeons was 14.4 ± 15. The mean h-index score increased with increasing academic rank, while the RI score showed no correlation. While other studies evaluated the chairperson as an additional rank, it was not included in this study because sports medicine is a subspecialty within the department of orthopaedics, and the chairperson was often not a sports medicine surgeon. According to Bastian et al, 3 the mean h-index of the overall academic orthopaedic population was 3.6 ± 4.5 for assistant professors, 8.4 ± 6.7 for associate professors, 15.1 ± 11.1 for professors, and 17.8 ± 13.7 for chairs. The authors did not look at the instructor’s mean h-index. This study demonstrates on average higher h-indices per academic rank compared with the aforementioned academic orthopaedic surgeon population, but had comparable results to those of Cvetanovich et al, 9 who specifically looked at sports medicine orthopaedic faculty. The discrepancy of the h-index between the overall academic orthopaedic population versus fellowship-trained sports medicine surgeons may be the result of increased research demand for fellowship and academic faculty positions within the sports medicine department.
The h-index has been widely used throughout different scientific disciplines and has gained acceptance for approximating an individual's calculated research impact across various disciplines. 23 However, each discipline has different measures and thresholds for scholarly impact based on several factors, including size. Therefore, it is necessary to validate the h-index within specific specialties and subspecialties. The h-index has been tested in several fields, including neurosurgery, plastic surgery, radiology, urology, anesthesiology, and surgery.4,25,29,31,39 Lee et al 25 reported a large amount of variation in mean h-indices between general surgery, urology, oncology, cardiology, orthopaedic surgery, and neurosurgery for academic rank and promotion. It is also important to note that there is a difference in h-indices of physicians between different subspecialties of surgery and medicine because the calculation of the h-index is inherently based on the number of scientists in a field and the relevance of publications to the field. 37 In addition, there has been noted to be a large difference in h-indices between academic ranks within certain subspecialties. 17 The large gap in h-indices between associate professors and professors can potentially be due to the inclusion of “honorary” authors, such as professors, in manuscripts. 17 Therefore, it is important to assess the validity of the h-index within each specialty and each subspecialty to determine whether it correlates with academic rank and, thus, can be used as a measure of review when considering academic promotion.
Three previous studies demonstrated significant associations between the h-index and academic rank in the field of orthopaedic surgery. Ence et al 12 reported the largest dataset of the h-index, including 4663 orthopaedic surgeons at 142 academic institutions. The median h-indices were 2, 8, 17, and 13 for assistant professors, associate professors, professors, and chairpersons, respectively. Ence et al also showed that female academic surgeons had a significantly lower h-index than male academic surgeons, which they attributed to the shorter career duration of female orthopaedic surgeons. Bastian et al 3 performed a similar analysis on 2061 academic orthopaedic surgeons from 120 orthopaedic surgery residency programs. They found the mean h-indices of 3.6, 8.4, 15.1, and 17.8 for assistant professors, associate professors, professors, and chairs, respectively. Bastian et al also showed that female assistant professors had a significantly lower mean h-index than male assistant professors. However, they did not find a significant difference in the mean h-index between the male and female surgeons at other academic levels, such as associate professor or professor. They were unable to compare at the chair level since there were no female chairs. This study found that male and female sports medicine orthopaedic surgeons had respective h-indices of 14.7 ± 2.1 and 11.9 ± 2.1; however, the difference was not statistically significant. Similar studies in other disciplines have supported this finding. Martinez et al 27 showed no differences in h-indices between male and female musculoskeletal oncology surgeons. Other studies that have reported initial differences, demonstrated a decrease in the difference after controlling for other variables. Therattil et al 37 showed that male plastic surgeons in academia tended to have higher h-indices than female plastic surgeons; however, this was not a significant difference when taking other variables into account. 29 Furthermore, several studies have indicated that female physicians early in their careers have lower h-indices than their male counterparts; nonetheless, this difference reduces later in their careers.11,26
Since the discontinuation of the RG score in August of 2022, ResearchGate has launched the RI score as a more intuitive and transparent means of measuring one’s research productivity. This bibliometric is a weighted score composed of reads by other ResearchGate community members, downloads, recommendations, and citations. Memisevic 28 found that the RI score was strongly correlated with the number of Google Scholar citations, followed by several Web of Science citations and Scopus citations. However, Memisevic also found that the RI score was not highly correlated with any h-indices using Google Scholar, Scopus, or Web of Science. While this study demonstrated no significant association between RI scores and the Scopus-based h-index, RI scores did increase with increasing academic rank. Also, RI scores correlated with the rank of the professor and the number of years at the current institution. This is likely attributed to the fact that with longer careers and higher academic ranks, there is more interaction throughout time within the ResearchGate community, and other researchers are more likely to read, download, recommend, or cite a researcher's work. While the RI score is indeed a more intuitive and transparent method of measuring research interest within the ResearchGate community compared with the original RG score, it is also not without flaws and is inherently a different kind of metric compared with the h-index, which is why it is not surprising there was no such correlation. Simply put, the RI score is a measure of interest in the scientific community rather than a measure of productivity and quality of one’s research. A particular study could be flawed or controversial and be read and downloaded numerous times, because of this, and not necessarily because it is a quality study. Thus, the RI score has the potential to be used in combination with the h-index and other bibliometrics, but it should be used with caution when used alone when considering a researcher for academic promotion. In addition, no study to date has evaluated correlations between academic productivity and patient satisfaction for academic sports medicine orthopaedic surgeons. This study showed no correlation between h-index and patient satisfaction, measured by HealthGrade and Vitals Review scores.
A geographic influence was only noted when comparing the h-index in the South versus the Northeast and West regions. A study by Cvetanovich et al 9 identified 610 faculty members from 90 AOSSM sports medicine fellowship programs and found that the Northeast had a higher productivity. This study's findings were consistent with the results from Ence et al 12 and Cvetanovich et al, 9 who showed significantly lower mean h-indices for academic orthopaedic surgeons practicing in the South compared with other regions. Multiple factors likely affect regional variation in the h-index, including institutional resources availability, differences in program size, competitiveness of residency/fellowship programs, and cultural factors. These cultural factors include the incorporation of clinical educator tracks at academic institutes, where the focus of an academic appointment is centered on clinical and mentorship responsibilities. This contrasts with tenure-track positions, where research productivity plays heavily in faculty appointment, retainment, and promotion. Of note, surgeons with Research Gate accounts had higher h-indices than those without, which further emphasizes how differences in a surgeon’s practice impact these bibliometrics. Further differences in institutional composition of tenure track and clinical educator roles can impact research productivity assessment and affect regional differences.
A physician’s h-index and other bibliometric parameters, while informative, should not be used as the sole marker of achievement. To date, there is no perfect scientific measure of an individual researcher's academic productivity. One criticism of the h-index is that it does not account for the order of the authors, and therefore, a first, middle, and last author would have similar h-indices for a given paper.1,6 Another criticism is that the h-index can potentially increase by increasing the amount of self-citation 30 ; however, this has been shown to be difficult to successfully accomplish. 13 The h-index also remains dependent on the length of an investigator's career, 18 and our study has shown that it is significantly associated with the number of years in practice. Since the h-index can only increase over a physician's career, this metric may not be as reliable to assess recent or future performance. 7 Last, the h-index ignores outlier studies that have a significant number of citations compared with the rest of a researcher's body of work. A particular researcher, for example, could have an h-index of 20 because 20 of their papers have at least 20 citations. However, 1 of those studies could have been a landmark paper that was cited over 1000 times, and the h-index would not take this into account when calculating one's score. Despite these drawbacks, the h-index continues to be the most widely used and accepted measure of research productivity. Further, Hirsch 21 validated that the h-index has predictive power and is better than other indicators considered, such as total citation count, citations per paper, and total paper count. This study shows that the h-index is a good predictor of academic rank and promotion within sports medicine orthopaedic surgeons in the United States.
Limitations
This study has limitations. Analysis was performed on faculty members who were appointed under the sports medicine department and/or indicated sports medicine as their scope of practice on the institutional website. This inclusion criterion was used to embrace the broader definition of orthopaedic sports medicine surgeons. At the same time, however, it can introduce some degree of heterogeneity to the study population. For example, an attending could have specified their interest in sports medicine despite not being fellowship-trained in orthopaedic sports medicine. It is debatable whether they should be included in our study. Second, the data used for the analysis relied on information from institutional websites, personal websites, and online curriculum vitae at the time of data collection, which may not always be readily updated or entirely accurate. To mitigate this limitation, data were cross-referenced across all platforms available online before being extracted, and the included data had been recently updated on the respective platforms. Third, there were limitations associated with the use of the h-index as a measure of academic productivity, as the h-index is not a dynamic measure and only accumulates throughout a career. Fourth, the author position and the total number of authors for each research item were not considered during the analysis, and therefore, the potential for significant variation in authors’ contributions exists. As a result, this would reflect outcomes that are more generous to the total quantity than to the quality of contribution to each research item from these authors. Fifth, only 378 out of 835 surgeons had profiles on ResearchGate and obtainable information. This is a relatively small size compared with the original number of surgeons identified. ResearchGate is still a relatively new website, and many surgeons do not have a robust online presence, thus limiting the capacity that ResearchGate has in accurately reflecting the current research landscape and scientific community as a whole. However, as social media websites such as these continue to grow and ResearchGate has more mainstream use, this limitation will continue to be minimized. Sixth, metrics of research productivity beyond article publications, such as poster presentations, conference or national presentations, and visiting professorships, were not considered when looking at academic promotion. Seventh, the scope of an individual's academic appointment may not involve a research focus. Institutions often have a different track for appointed faculty, which includes clinical educators whose appointments and promotion are not research based; thus, their overall impact is not accurately characterized by the study. Last, there were potential errors in data extraction from the Scopus database with crediting or discrediting authors with associated research due to surgeons having the same or similar names, a change in surname, or working institution. This limitation was mitigated by cross-checking the author’s full name with the institution and field of research in which they were involved. Some authors on Scopus were listed multiple times, and these additional listings were excluded. Moreover, variations in authors’ choice to be listed by their middle name on institutional and personal websites were accounted for when searching the Scopus database.
Conclusion
This study shows that the h-index is a valid measure of academic productivity within academic sports medicine orthopaedic surgeons in the United States. Although the h-index and the RI score increased with increasing academic rank, there was no significant correlation between the h-index and the RI score when compared. The South, compared with both the Northeast and West regions, had a statistically lower h-index score. No correlation was found between patient satisfaction scores and all of the academic productivity indices (h-index, RI score, and academic rank). While the RI score has the potential to be used as a bibliometric, further evaluation of the validity of the RI score is necessary to consider it as a reliable means of research productivity. As it stands, further measures to assess impact and productivity when evaluating scholarship for promotion are warranted.
Footnotes
Final revision submitted March 3, 2025; accepted March 14, 2025.
The authors have declared that there are no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.