Relationship of Altmetric Attention Score,Twitter Performance,and Dimensions Badge Value With Traditional Metrics in Top-Cited Anterior Cruciate Ligament Research Studies

Abstract

Background:

Alternative citation metrics—such as the Altmetric Attention Score (AAS), number of tweets (TN), and dimensions badge value (DBV)—are emerging as new options for assessing the value of scientific works.

Purpose:

To analyze the AAS of highly cited articles on the anterior cruciate ligament (ACL) and to assess the relationship between alternative and traditional metrics such as journal rankings and article citation performance.

Study Design:

Cross-sectional study.

Methods:

A search was conducted using the Web of Science Core Collection of databases with “anterior cruciate ligament” as the search term. Full-text articles published between 2011 and 2021 were reviewed, and the top 100 cited articles were determined. The articles were analyzed by publication year, study design, research topic, journal impact factor, journal h-index, number of total citations (TC), recent citations (RC) (ie, citations in the latest 2 years), and average citation per year (ACY), as well as AAS, TN, and DBV.

Results:

For articles in the top 100 list, the median TC was 160 (interquartile range [IQR], 117-561) and the median AAS was 24.50 (IQR, 1-730). A higher AAS score was achieved by articles on return to sports and anterolateral ligament (P < .05). The AAS and TN were significantly and positively correlated with the RC (r = 0.459 and P = .001; r = 0.438 and P = .001, respectively) and ACY (r = 0.363 and P = .001; r = 0.393 and P = .001, respectively).

Conclusion:

Alternative metrics were linked to traditional metrics but were not a direct representation of bibliometrics. The AAS was not correlated with TC numbers in the ACL research. Higher AAS and Twitter popularity of an article were related to receiving high-volume RC performance. The DBV represented a stronger correlation with traditional metrics than the AAS.

Keywords

anterior cruciate ligament Altmetric Attention Score dimensions badge value Twitter performance

In search of instantaneous information on the impact of an article, scholars have begun exploring new nonformal online platforms that have emerged in the past few decades.^4,19 These innovative bibliometric approaches, or alternative metrics, provide a more recent interaction with the published articles compared with traditional citation-based metrics such as journal rankings and article citation performance, which require 2 or 3 years to reach their real value to accumulate.⁵ One of these alternative metrics—known as Altmetrics— is a web-based quantitative representation of online attention to a research paper.²⁰ Different databases are analyzed and combined to produce an Altmetric donut—a visual representation (Figure 1)—and the Altmetric Attention Score (AAS)—a numerical value. Altmetrics mainly evaluates the online performance of a study beyond medical databases and may be considered a tool to measure the influence of publications on the community.^11,21

Figure 1.

Demonstration of an Altmetric donut, with an Altmetric Attention Score inside the donut. Each color in the donut corresponds to a different source that is available for a given manuscript. The additive value of each source is also indicated at each line.

Another alternative performance metric for published research is dimensions. This includes many sources that belong to each step in the production and postpublishing processes, including Altmetrics.¹⁸ It takes into account the citation performance of the latest 2 years, also called recent citations (RC). It combines data from publications, data sets, patents, and policy documents to calculate a score called a dimensions badge (Figure 2) and produces an interconnecting dimensions map.⁹ In this sense, dimensions may be considered as an effort to unite the scholarly and nonscholarly performance of scientific research papers.

Figure 2.

Demonstration of a dimensions badge value (DBV). The DBV is given in the geometric figure. The values of total citations, recent citations (in the past 2 calendar years), field citation ratio, and recent citation ratios are also available aside. In the given example, the number of recent citations is 157 and the DBV is 709.

Twitter is a popular social media tool not only for the instantaneous sharing of information but also for diverse scientific activities. AlFaris et al² reported that 35% of medical students use Twitter for vocational training, as an increasing number of medical journals publish their Twitter handle on the internet. The notification rate of research papers on Twitter was reported¹⁴ to be ≥20%. The total number of tweets (TN) and retweets for a study is now considered an early estimator of its impact in terms of Altmetrics.¹⁰

Costas et al⁶ published a study in 2015 on the relationship between alternative metrics and traditional metrics. Batooli et al⁴ reported a strong positive correlation between the number of views of the articles in ResearchGate and Mendeley and the number of citations of the articles in Scopus. Thelwall et al²⁵ reported statistically significant associations between higher metric scores and higher citations for articles with positive Altmetric scores in forums such as Twitter, Facebook wall posts, research highlights, blogs, the mainstream media, and forums (P < .05). Scarlat et al²³ mentioned that high-volume publications also have a high-volume social media impact. Silva et al²⁴ reported that the AAS exerted a stronger relationship with the number of citations than the journal impact factor and open access status in sports sciences journals. Hughes et al¹⁵ demonstrated that journals with higher numbers of retweets had higher AAS and were positively associated with impact factor.

On the other hand, Kunze et al¹⁷ did not find a strong correlation between traditional metrics and the AAS in orthopaedic studies. De Gregori et al⁸ reported that alternative metrics showed a statistically weak correlation with Facebook and TN, with the exception of those for Mendeley. These studies suggest that although alternative metrics have interrelations with citation performance, they may not represent them in the same way and may instead have a potential additive value in general.

The anterior cruciate ligament (ACL) is a popular subject not only for researchers but also for the growing sportive population, promising a high AAS in addition to good citation performance. There is a continuing interest in the improvement of ACL treatment modalities to provide an early return to previous sport levels. A good example of this is the resurgence in anterolateral ligament (ALL) reconstruction techniques as an adjunct to ACL reconstruction, which has been reported to decrease revision rates significantly.^12,13 The purpose of the present study was to analyze the alternative metric scores of highly cited recently published articles on the ACL and to assess the relationship between alternative and traditional metrics. We postulated the following hypotheses:

Alternative modes of performance scores of the scientific works—such as AAS, TN, and dimensions badge value (DBV)—might be linked to each other and, at the very least, to some parameters of the traditional metrics.

Alternative metrics—such as the AAS and DBV—are under the influence of social media. Thus, the research focus of alternative metrics and scholarly metrics may be different considering their different data sources.

Methods

Data Collection and Analytical Tools

The present study was conducted in accordance with the Helsinki Declaration of ethical principles. No ethical approval was required from the local committee of ethics regarding the data extracted from the literature. No personal data were included. On July 31, 2021, we conducted a search for articles published between 2011 and 2021 using the search term “anterior cruciate ligament” in the Web of Science Core Collection—including the Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, Conference Proceedings Citation Index–Science, Conference Proceedings Citation Index–Social Science & Humanities, Book Citation Index–Science, Book Citation Index–Social Sciences & Humanities, and Emerging Sources Citation Index databases.The first 100 full-text articles having the highest citations were selected by the authors. Only the articles under the human health category on the epidemiology etiology, pathophysiology, histology, diagnosis, therapy, and prognosis related to the ACL were selected. Articles with only abstracts were excluded.

The impact factor for each included journal was retrieved from the 2019 Clarivate Journal Citation Reports. All journals were searched in the 2020 Scimago Journal and Country Rank (SJCR) for their h-index and quartile index.

For all articles, we recorded the publication year, author names, publication year study design, research category and topic, level of evidence, impact factor, field citation ratio (FCR), total citation (TC), average citation per year (ACY), RC, DBV, AAS, and TN. The FCR is available for a field with >500 articles that have been published in the past 2 years. It is calculated by the division of the TC value of a given article by the ACY value in the same field. Study types and evidence levels were reviewed using the 7-grade list provided by Ackley et al.¹

The Altmetric it tool on the Altmetric website was used to delineate the AAS and TN values of the articles included (https://www.altmetric.com/products/free-tools/bookmarklet/). Given an article’s digital object identifier number, the AAS of each article was calculated automatically by a weighted average process, evaluating the attention of the article in alternative sources and producing an Altmetric donut, with each color symbolizing a distinct source of attention. The DBV for each article was gathered from the dimensions website (http://www.digital-science.com/product/dimensions).

Mapping knowledge domains is a method for visual representations of bibliometric data. VOSviewer (https://www.vosviewer.com) is an online tool that calculates co-occurrence data and builds visual networks in the form of interconnected hotspots.¹⁶ In this study, visual bibliometric coupling analyses on the country of origin and article keywords of the top 100 cited publications were performed with VOSviewer software Version 1.6.17. A threshold value of 2 (ie, the number of publications in which ≥2 keywords were shared) was used in the keyword co-occurrence analysis maps.

Statistical Analysis

Descriptive statistics for continuous data are shown as mean ± SD or median and range or interquartile range (IQR). Categorical data are shown as percentages. The Shapiro-Wilk test was used to test the distributions of continuous variables for normality. The differences in parameters between ≥3 groups were compared using the Kruskal-Wallis (post hoc: Dunn) or analysis of variance (post hoc: Duncan) tests. Spearman or Pearson correlation coefficients were calculated to detect any linear relationships between the variables of interest, in which a correlation coefficient (r) between 0.8 and 1 was considered very strong, between 0.6 and 0.79 was considered strong, between 0.4 and 0.59 was considered moderate, 0.2 and 0.39 was considered weak, and between 0 and 0.19 as very weak.²² Beta coefficients were estimated by univariate linear regression analysis. All analyses were performed by using SPSS Statistics for Windows Version 23.0 (IBM). P < .05 was considered statistically significant.

Results

A total of 25,001 articles using the term “anterior cruciate ligament” were identified in our Web of Science search. Limiting the search to articles published between 2011 and 2021 resulted in 14,868 papers in the preferred indexes. English language limitation resulted in 14,537 papers. A total of 13,124 articles—11,751 reviews and 1373 original articles—were selected. The articles in the top 100 list are presented in the Supplemental Material, available separately.

The overall characteristics of the top 100 articles are provided in Table 1. The included articles had a mean TC value of 181.34 ± 78.15 and were considered high-impact studies, with a mean level of evidence of 4.31 ± 1.40 (Table 1). The included articles were cited 18,134 times by 6877 papers. Excluding self-citations, the search revealed 17,824 citations in 6805 articles. All articles were published between 2011 and 2017, with 90% of the studies published between 2011 and 2015 (Table 2). Cohort studies (n = 18) and systematic reviews (n = 22) were dominant (Table 2). The mean 2019 impact factor of the journals in which these articles were published was 6.10 ± 4.23.

Table 1

General Characteristics of the Top 100 Articles on ACL Research ^a

	Mean ± SD	Median (Range)
TC	181.34 ± 78.15	160 (117-561)
RC	57.56 ± 39.91	47 (15-207)
ACY	21.46 ± 9.93	18.14 (11-55)
Field citation ratio	59.51 ± 27.16	50 (28-164)
Journal IF	6.10 ± 4.23	5.81 (0.3-30)
Journal h-index	192.23 ± 64.08	221 (31-429)
LoE ^b	4.31 ± 1.40	4 (1-7)
AAS	68.76 ± 125.68	24.50 (1-730)
DBV	224.26 ± 101.21	196.50 (0-709)
TN	78.89 ± 153.22	28.50 (0-957)
Time passed, y	7.77 ± 1.69	8 (4-10)

^a ACL, anterior cruciate ligament; AAS, Altmetric Attention Score; ACY, average citations per year; DBV, dimensions badge value; IF, impact factor; LoE, level of evidence; RC, recent citations; TC, total citations; TN, number of tweets.

^b According to Ackley et al.¹

Table 2

Characteristics of the Top 100 Articles ^a

Characteristic	Value	Characteristic	Value
Publication year		Research category
2011	20	Anatomy	7
2012	21	Epidemiology	9
2013	13	Function	28
2014	20	Injury	43
2015	16	Radiology	4
2016	8	Technique	4
2017	2	Biomechanics	15
Study design		Research topic ^b
Case-control	11	Anterolateral ligament	14
Case series	8	Anatomy	12
Cohort	22	Cartilage	24
Controlled laboratory study	9	Clinical results	24
Cross-sectional	2	Comparison of techniques	26
Epidemiological	14	Contralateral rupture	13
Meta-analysis	2	Instructive	8
Randomized controlled trial	4	Laboratory study	11
Retrospective comparative	4	Meniscus	36
Review	6	Osteoarthritis	6
Systematic review	18	Prevention	4
		Radiology	15
		Rehabilitation	9
		Risk factors	29
		Return to sports	25

^a Values are presented as No. of articles.

^b Articles with >1 topic.

Table 3

Journals That Published Articles in the Top 100 List

Journals	%
American Journal of Sports Medicine	49
Arthroscopy	10
British Journal of Sports Medicine	9
Knee Surgery, Sports Traumatology, Arthroscopy	7
Journal of Bone and Joint Surgery–American Volume	5
Journal of Athletic Training	3
Journal of Orthopaedic & Sports Physical Therapy	3
British Medical Journal	2
Acta Orthopaedica Belgica	1
Bone & Joint Research	1
Clinical Journal of Sport Medicine	1
Cochrane Database of Systematic Reviews	1
Exercise and Sport Sciences Reviews	1
Journal of Anatomy	1
The Knee Journal	1
Medicine and Science in Sports and Exercise	1
Orthopaedic Journal of Sports Medicine	1
Radiology	1
Research In Sports Medicine	1
Sports Medicine	1

According to the publication year, the RC was strongly correlated with the AAS and TN (P < .05). The AAS, TN, and ACY increased significantly with publication time (Figure 3 and Table 4).

Figure 3.

Progression of the mean TC, RC, AAS, and TN by publication year, 2011-2017. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.

Table 4

Comparison of TC, RC, AAS, and TN According to Publication Year, 2011-2017 ^a

Publication Year	TC	RC	AAS	TN
2011	150.5 [140-187]	36 [22-47]	12 [7-24.5]	5.5 [1.5-33.5]
2012	174 [139-197]	41 [32-49]	14 [4.5-21]	9 [3.5-28]
2013	167 [125-207]	52 [35-55]	58 [30-220]	58 [30-126]
2014	150.5 [126.5-219]	54 [36.5-68]	38 [26-65]	44 [16-74]
2015	136 [125.5-193.5]	58 [41.5-63.5]	30.5 [14-46]	43 [18-65]
2016	176.5 [162.5-245]	113.5 [45.5-154.5]	70.5 [5.5-297]	86.5 [6.5-373.5]
2017	145.5 [121-170]	85.5 [76-95]	301 [91-511]	349.5 [16-683]
P	.635	.002	.001	.001

^a Data are reported as median [interquartile range]. Bold P values indicate significant differences within the column (P < .05, Dunn or Duncan test). AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.

The study design and research category were not correlated with any of the traditional or alternative citation metrics (Tables 5 and 6). Regarding research topics, studies on return to sports (RTS) were found to be related to increased TC, AAS, and TN (P < .05), and studies on anatomy and on the ALL had higher AAS and TN compared with others (P < .05), RC was statistically correlated with meniscal and cartilage lesions and return to sport (P < .05) (Table 7).

Table 5

Comparison of TC, RC, AAS, and TN According to Study Design ^a

Study Design	TC	RC	AAS	TN
Case control	154 [146-204]	55 [22-70]	27 [14-58]	42 [6-58]
Cohort	160.5 [133-177]	48 [36-74]	23.5 [14-60]	16.5 [10-74]
Case series	178.5 [125.5-236.5]	41 [25.5-61]	4 [3-15]	5 [2-7]
Controlled laboratory study	139 [120-160]	39 [31-41]	5.5 [2-14]	7.5 [2-18]
Cross-sectional	159.5 [124-195]	58.5 [58-59]	26.5 [21-32]	42 [34-50]
Meta-analysis	194.5 [167-222]	46.5 [37-56]	64 [28-100]	62.5 [37-88]
Randomized controlled trial	205 [193.5-214.5]	47.5 [38.5-54.5]	68 [20.5-249]	89.5 [22.5-292]
Retrospective comparative	168.5 [129-254.5]	44.5 [33.5-70]	21 [6-36]	23.5 [3-41.5]
Review	155.5 [138-210]	50.5 [30-58]	14 [12-225]	19 [14-324]
Systematic review	144 [125-188]	47 [40-79]	22.5 [7-80.5]	21.5 [6.5-95.5]
P	.608	.413	.065	.194

^a Data are reported as median [interquartile range]. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.

Table 6

Comparison of TC, RC, AAS, and TN According to Research Category ^a

Research Category	TC	RC	AAS	TN
Anatomy	190 [118-492]	39 [32-121]	264.5 [5-524]	191.5 [63-77]
Biomechanics	142.5 [119-190]	41.5 [25-51.5]	4 [3-7]	6 [4-9]
Epidemiology	138 [123-207]	53 [38-63]	37 [14-66]	41 [14-66]
Function	163 [126.5-214.5]	58 [43.5-71.5]	27 [15-40]	36 [14-54]
Injury	157.5 [129-188]	45 [33-67]	31.5 [12-65]	37.5 [6-94]
Radiology	148 [121-167]	26 [15-37]	51.5 [3-100]	44 [0-88]
Technique	180 [144.5-221.5]	62.5 [38.5-87.5]	15 [3-91]	7 [4-16]
P	.948	.132	.057	.145

^a Data are reported as median [interquartile range]. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.

Table 7

Comparison of Research Topic According to the Distribution of TC, RC, AAS, and TN ^a

Research Topic	TC	RC	AAS	TN
Anatomy	162.5 [129.5-193.5]	39 [32.5-45.5]	5 [3-7]	6 [2-9]
No anatomy	160 [127-208]	50 [36-69]	28 [12-60]	34 [7-74]
P	.824	.085	.006	.022
ALL	162.5 [121-197]	40 [33-53]	6 [3-15]	6.5 [4-16]
No ALL	160 [127-207]	50 [36-66]	27.5 [12-59]	34.5 [6-74]
P	.739	.419	.036	.046
RTS	176 [148-213]	59 [50-76]	32 [17-91]	50 [17-94]
No RTS	154 [125-197]	42.5 [33-56]	22 [7-51]	18 [5-64]
P	.033	.003	.037	.044
Meniscal lesion	153.5 [127-200.5]	40 [30.5-55.5]	21 [7-37]	16 [4-41]
No meniscal lesion	163 [126-209.5]	52 [39-74]	32 [9-71]	35 [7-88]
P	.464	.005	.053	.144
Cartilage lesion	154.5 [137-205]	35 [30-52.5]	19 [5-37]	16 [5-41]
No cartilage lesion	160.5 [126-205.5]	51 [39-73]	27.5 [9-66]	33.5 [6-74]
P	.958	.003	.157	.18
Osteoarthritis	179.5 [145-216]	52.5 [36-56]	33 [22-65]	39 [11-85]
No osteoarthritis	160 [127-202]	46 [36-67]	23 [7-58]	26 [6-65]
P	0.5	0.769	0.564	0.286

^a Data are reported as median [interquartile range]. Bold P values indicate significant differences within the column (P < .05). AAS, Altmetric Attention Score; ALL, anterolateral ligament; RC, recent citations; RTS, return to sports; TC, total citations; TN, number of tweets.

TC and AAS Analyses

The median values for the TC and AAS were 160 (IQR, 117-561) and 24.50 (IQR, 1-730), respectively. The top study, “Return to Sport Following Anterior Cruciate Ligament Reconstruction Surgery: A Systematic Review and Meta-analysis of the State of Play” was conducted by Ardern et al³ and was published in 2011; it achieved a TC of 561 and an AAS of 96, ranking it 16th in terms of AAS performance. “Evidence-based Clinical Practice Update: Practice Guidelines for Anterior Cruciate Ligament Rehabilitation Based on a Systematic Review and Multidisciplinary Consensus” conducted by van Melick et al²⁶ was published in 2016 and had the highest AAS (n = 730). The AAS was not available for 7 studies in the 100 list (Table 8). Annual cumulative totals of TC demonstrated less growth over the captured time compared with the increasing performance of the AAS and TN after 2015 (Figure 3).

Table 8

TC, TN, AAS, and DBV of the Top 10 Articles According to AAS ^a

Rank	First Author	Journal	TC	TN	AAS	DBV
38	van Melick	British Journal of Sports Medicine	176	957	730	709
2	Claes	Journal of Anatomy	492	377	524	606
90	Nagelli	Sports Medicine	121	683	511	560
13	Kyritsis	British Journal of Sports Medicine	234	509	410	491
21	Frobell	British Medical Journal	213	446	400	0
73	Waldén	British Journal of Sports Medicine	127	504	369	392
23	Hewett	American Journal of Sports Medicine	210	324	225	395
88	Joseph	Journal of Athletic Training	122	126	220	370
6	Paterno	American Journal of Sports Medicine	321	265	188	349
3	Ardern	British Journal of Sports Medicine	432	185	172	324

^a AAS, Altmetric Attention Score; DBV, dimensions badge value; TC, total citations; TN, number of tweets.

Twitter Analysis

The analysis of the TN revealed that 87 articles from the top 100 list were shared on Twitter. Of the top 10 articles with the highest TN, 4 dealt with ACL rerupture, 3 with RTS and rehabilitation, and 1 with ALL (Tables 4 -7).

Journal Analysis

According to the 2020 SJCR, all journals had a Quartile 1 index ranking, indicating that they occupied the top 25% of journals within their subject area. Among the 20 journals in which the articles in the top 100 list were published, The American Journal of Sports Medicine (AJSM) published the most cited articles (n = 49) (see Table 3). In contrast, the first 3 articles in the top 100 list were published in the British-based journals (Supplemental Material).

The articles with the highest TC and AAS were published in the British Journal of Sports Medicine—the journal with the highest impact factor in the top 100 list. There was a British journal predominance, with 6 articles in the list of the first 10 papers according to their AAS in the ACL research (Table 8).

Research Topics

The research category with the most cited and tweeted articles was postoperative knee function and concomitant injuries, whereas the main research topics were RTS and rehabilitation. When listed according to their AAS performance, the main research topics of the top 10 articles were RTS (n = 3) and ACL rerupture (n = 3) (Table 8).

Correlation Analysis

The results of the correlation analysis are displayed in Table 9. The AAS was found to have a strong positive correlation with the TN (r = 0.971; P = .001). While the AAS and TN did not have a significant correlation with the TC, they did show a weak positive correlation with the ACY (r = 0.363 and P = .001; r = 0.393 and P = .001, respectively) and a moderate correlation with the RC (r = 0.459 and P = .001; r = 0.438 and P = .001, respectively). The AAS had a weak negative correlation with the publication year (r = –0.315; P = .002).

Table 9

Results of the Correlation Analysis ^a

	FCR	TN	TC	ACY	RC	DBV	Journal IF	Journal h-index	Year ^b	LoE
AAS
r	0.274 ^c	0.971 ^c	0.184	0.363 ^c	0.459 ^c	0.234 ^d	0.209 ^d	–0.098	–0.315 ^c	–0.099
P	.009	.001	.080	.001	.001	.025	.046	.351	.002	.353
FCR
r	1	0.279 ^c	0.854 ^c	0.891 ^c	0.705 ^c	0.850 ^c	0.272 ^c	0.165	–0.293 ^c	–0.072
P	—	.008	.001	.001	.001	.001	.007	.106	.004	.488
TN
r	—	1	0.179	0.393 ^c	0.438 ^c	0.224 ^d	0.205 ^d	–0.131	–0.370 ^c	–0.121
P	—	—	.088	.001	.001	.032	.050	.212	.001	.254
TC
r	—	—	1	0.781 ^c	0.572 ^c	0.935 ^c	0.157	0.149	0.035	–0.088
P	—	—	—	.001	.001	.001	.118	.138	.730	.386
ACY
r	—	—	—	1	0.776 ^c	0.740 ^c	0.239 ^d	0.071	–0.541 ^c	–0.051
P	—	—	—	—	.001	.001	.017	.484	.001	.616
RC
r	—	—	—	—	1	0.594 ^c	0.216 ^d	–0.040	–0.450 ^c	0.018
P	—	—	—	—	—	.001	.032	.693	.001	.863
DBV
r	—	—	—	—	—	1	0.201 ^d	0.112	0.038	–0.035
P	—	—	—	—	—	—	.045	.265	.704	.727
Journal IF
r	—	—	—	—	—	—	1	0.604 ^c	–0.173	–0.148
P	—	—	—	—	—	—	—	.001	.085	.143
Journal h-index
r	—	—	—	—	—	—	—	1	0.012	–0.242 ^d
P	—	—	—	—	—	—	—	—	.902	.016
Year ^b
r									1	0.011
P										.913

^a The r values were obtained from the Spearman rank correlation or the Pearson correlation (n = 100). AAS, Altmetric Attention Score; ACY, average citation per year; DBV, dimensions badge value; FCR, field citation ratio; IF, impact factor; LoE, level of evidence; RC, recent citation; TC, total citation; TN, number of tweets.

^b Publication year.

^c Correlation is significant at the .01 level.

^d Correlation is significant at the .05 level.

There was a weak but positive correlation between the FCR and TN (r = 0.279; P = .008). However, the FCR was strongly correlated with the TC, ACY, and RC. The FCR showed a weak significant negative correlation with the publication year (P < .05). The TN had a weak positive correlation with the ACY and dimensions. On the other hand, it was negatively correlated with the publication year (P < .05).

The journal h-index and journal impact factor were strongly correlated (r = 0.604; P = .001). The journal impact factor was weakly correlated with the AAS, FCR, TN, RC, ACY, and DBV (P < .05). The journal h-index was correlated significantly but weakly with the level of evidence (r = –0.242; P = .016).

The DBV showed strong correlations with traditional citation metrics such as the TC (r = 0.935; P = .001), FCR (r = 0.850; P = .001), and ACY (r = 0.840; P = .001) but weak correlations with alternative metrics such as the AAS (r = 0.234; P = .025) and TN (r = 0.224; P = .032).

Results of Regression Analysis

There was a moderate positive correlation between the AAS and RC (r = 0.459; P = .001) (Figure 4A). According to univariate linear regression analysis, ∼21% of the variation in the AAS was due to recent citations. A 1-unit increase in recent citations resulted in a 1.54 increase in the AAS.

There was a moderate positive correlation between the RC and TN (r = 0.438; P = .001) (Figure 4B). According to univariate linear regression analysis, ∼19% of the variation in the RC was explained by the TN. A 1-unit increase in the TN resulted in a 0.11 increase in the RC. There was a strong positive correlation between the AAS and TN (r = 0.971; P = .001) (Figure 4C). According to univariate linear regression analysis, ∼94% of the variation in the AAS was explained by the TN. A 1-unit increase in the TN resulted in an 0.8 increase in the AAS.

Figure 4.

Scatterplots of the relationship between (A) the AAS and RC, (B) the RC and TN, and (C) the AAS and TN. There is a varying degree of correlation between the AAS, RC, and TN, and the AAS and TN showed the highest correlation. AAS, Altmetric Attention Score; RC, recent citations; TN, number of tweets.

Visualization Analysis

The overall strength of bibliographic coupling linkages with other countries was calculated and visualized for each of the 23 countries in the top 100 list (Figure 5). The magnitude of the nodes corresponds to their publication number and citation performance. The power of interrelation and collaboration among countries was indicated by the thickness and distance of links between nodes (Figure 5). The biggest spot in the country table was occupied by the United States, which was the most competing country in terms of the number of publications and citations (n = 55 documents), TC received (n = 9659 citations), and international cooperation index (n = 6826 link strength). In terms of the number of publications, TC, and international cooperation index, the United States was followed by Australia ([13] [3153] [2814]), Sweden ([10] [1751] [2586]), Norway ([7] [1751] [2853]), Belgium ([6] [1320] [1826]), Canada ([6] [859] [1024]), England ([4] [677] [1223]), France ([4] [660] [771]), Denmark ([3] [529] [555]), and Austria ([2] [378] [1387]).

Figure 5.

VOSviewer output for countries. Schematic representation of the distribution of papers according to countries allows for visualizing the impact of each country. The United States is in the center and has the highest number and strongest interconnections.

The hotspots of the ACL research were produced by the VOSviewer tool by calculating co-occurrence levels of high-frequency keywords. The keyword co-occurrence limit of ≥2 was met by 53 of the 181 extracted keywords, and the network of keywords produced 6 major clusters, represented by different colors in Figure 6. The most prevalent keywords after “anterior cruciate ligament” were as follows: “ACL reconstruction”; “knee”; “anterolateral ligament”; “return to sport”; “revision”; “epidemiology”; and “biomechanics,” with a total occurrence of 36, 29, 11, 8, 7, 6, 5, 4, respectively.

The visual representation of links between the keywords demonstrated that the anatomic studies concentrated around ALL studies. RTS was considered one of the key elements of knee function. Injury was the most recurring keyword, appearing in 3 different clusters, “ACL injury” in the knee injury cluster, “reinjury” in the allograft cluster, and “injury” in the epidemiological studies and meniscectomy cluster (Figure 6).

Figure 6.

Visualization of VOSviewer keywords co-occurrence. The distribution and interrelation of keywords with ≥2 co-occurrences produced 6 major clusters: knee, knee injury, re-injury, revision, anterolateral ligament, epidemiology, each represented by a different color. ACL, anterior cruciate ligament; KOOS, Knee injury and Osteoarthritis Outcome Score.

Discussion

This manuscript was based on the bibliometric information of the top 100 most cited ACL studies between 2011 and 2021 and evaluated the relationship between traditional metrics—including TC, journal h-index, and impact factor—and alternative metrics. To the best of our knowledge, this is the first study to compare traditional metrics with alternative metrics, including Twitter, Altmetrics, and dimensions.

The most striking finding of this study may be the strong correlation between the RC, AAS, and TN according to the publication year, supporting the first hypothesis of this manuscript. RCs are those that are not taken into consideration in journal index performances. Therefore, the parallelism of the AAS performance with the RC renders the AAS a good indicator of the future value of a manuscript. Moreover, being a component of the AAS, Twitter’s popularity may represent the AAS performance, supporting the idea of sharing publications among scholars on Twitter to augment the TC of a given study. According to another study,⁷ journals with a considerable Twitter performance present a high AAS performance as well.

The journal impact factor is considered to be one of the most trusted bibliometric power indicators.⁵ It is a measure of the total performance of all of the articles in the journal rather than a specific one. In this study, the impact factor was found to be weakly correlated with the citation-based and alternative metrics in ACL studies and strongly correlated only with journal h-index. Moreover, journal h-index was weakly correlated with the level of evidence. In other words, the scientific level of a journal is not enough to attract the attention of the online community and does not guarantee the level of evidence for a highly cited article.

One of the initial hypotheses of this paper—the possible difference in the interest fields of patients and scholars—was evaluated in this study. Although the research category and research topic of the studies were not correlated with the citation metrics in this study, research on RTS was found to have a positive influence on the TC and AAS, whereas the ALL research was found to be correlated with the AAS and TN. Thus, RTS is still one of the most important issues for both researchers and social media users, possibly athletic patients. On the other hand, the correlation of ALL studies with increased AAS may be considered as the tendency of scholars to share those articles on the online platforms. Therefore, the AAS may be considered the bridge where the 2 major populations of interest meet, as it represents both scholar and nonscholar interest fields simultaneously.

The DBV was found to be weakly correlated with the AAS and TN but strongly correlated with traditional metrics in this research. In other words, dimensions may have a different category than Altmetrics by being closer to the citation-based metrics. This is probably because of its stronger connections with research data than Altmetrics, which is only a small component of its database. From this view, dimensions value may be regarded as an alternative metric that preserves the density of scholarly content while not ignoring nonscholar sources of attention.

The results of this study reaffirm those of Zhu et al²⁷ who reported that in the past 20 years, the most influential journal in ACL surgery was the American Journal of Sports Medicine and the most productive country was the United States. According to a visual representation of interconnections between countries, the United States is not only the strongest country in publication numbers but also the one with the strongest international relationships with other countries. On the other hand, although the United Kingdom was ranked sixth in the number of publications, journals from the United Kingdom were also successful, especially in terms of publishing papers with the highest TC and AAS values.

Limitations

This study was based on the bibliometric data of the ACL research in a given time, and the variables are time-dependent and thus may show differences with the progression of time. Twitter, AAS, and dimension values were selected to represent alternative metrics. However, more databases exist to evaluate alternative attention sources that were not included in this study. Furthermore, not all social media platforms are open to data searches, such as those used in China, Russia, and North Korea. Additionally, the selected language of this study is English, and thus, the literature in other languages was not included.

Conclusion

Alternative metrics are a new approach to representing the value of a research paper in a broader sense. Altmetrics is not a direct representation of bibliometrics, and the AAS is not correlated with TC numbers, impact factor, or journal h-index. The TN value is a good estimator of the RC value, suggesting that articles that have a high social media performance may be expected to maintain receipt of high-volume citations. Dimensions are an example of the new perspective in the metric systems through the combination of alternative metrics with traditional metrics. Rather than being a complementary tool to traditional metrics that measure the scholarly impact of a research paper, alternative metrics—including attention to social media such as Twitter—seem to change the rules of the “citation race” with their power to reflect the interest of a greater population. Future metrics will be more powerful and instantaneous with the accumulation of alternative metrics data.

Supplemental Material for this article is available at https://journals.sagepub.com/doi/full/10.1177/23259671231166701

Supplemental Material

Supplemental Material, sj-pdf-1-ojs-10.1177_23259671231166701 - Relationship of Altmetric Attention Score, Twitter Performance, and Dimensions Badge Value With Traditional Metrics in Top-Cited Anterior Cruciate Ligament Research Studies

Supplemental Material, sj-pdf-1-ojs-10.1177_23259671231166701 for Relationship of Altmetric Attention Score, Twitter Performance, and Dimensions Badge Value With Traditional Metrics in Top-Cited Anterior Cruciate Ligament Research Studies by Hasan Basri Sezer, Yoann Bohu, Alexandre Hardy, Nicolas Lefevre and Aysun Sezer in Orthopaedic Journal of Sports Medicine

Footnotes

Final revision submitted January 5, 2023; accepted January 27, 2023.

The authors declared that there are no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.

Ethical approval was not sought for the present study.

References

Ackley

Ladwig

Swan

, et al. Evidence-Based Nursing Care Guidelines-E-Book: Medical-Surgical Interventions. Elsevier Health Sciences; 2007.

AlFaris

Irfan

Ponnamperuma

, et al. The pattern of social media use and its association with academic performance among medical students. Med Teach. 2018;40(1):77–82.

Ardern

Webster

Taylor

, et al. Return to sport following anterior cruciate ligament reconstruction surgery: a systematic review and meta-analysis of the state of play. Br J Sports Med. 2011;45(7):596–606.

Batooli

Ravandi

Bidgoli

. Evaluation of scientific outputs of Kashan University of Medical Sciences in Scopus citation database based on Scopus, ResearchGate, and Mendeley scientometric measures. Electron Physician. 2016;8(2):2048–2056.

Collins

Singh

Ananthasekar

, et al. The correlation between altmetric score and traditional bibliometrics in orthopaedic literature. J Surg Res. 2021;268:705–711.

Costas

Zahedi

Wouters

. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. J Assoc Inf Sci Technol. 2015;66(10):2003–2019.

Dardas

Woodward

Scott

, et al. Measuring the social impact of nursing research: an insight into altmetrics. J Adv Nurs. 2019;75(7):1394–1405.

De Gregori

Scotti

De Silvestri

, et al. Does a research group increase impact on the scientific community or general public discussion? Alternative metric-based evaluation. J Pain Res. 2016;9:391–395.

Duffy

. Dimensions. The Charlest Advis. 2021;23(1):9–12.

10.

Eysenbach

. Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13(4):123.

11.

Goyal

Galetta

Fang

, et al. The impact of social media in orthopaedics. Curr Orthop Pract. 2019;30(5):419–422.

12.

Hamido

Habiba

Marwan

, et al. Anterolateral ligament reconstruction improves the clinical and functional outcomes of anterior cruciate ligament reconstruction in athletes. Knee Surg Sports Traumatol Arthrosc. 2021;29(4):1173–1180.

13.

Hardy

Casabianca

Hardy

, et al. Combined reconstruction of the anterior cruciate ligament associated with anterolateral tenodesis effectively controls the acceleration of the tibia during the pivot shift. Knee Surg Sports Traumatol Arthrosc. 2017;25(4):1117–1124.

14.

Haustein

Costas

Larivière

. Characterizing social media metrics of scholarly papers: the effect of document properties and collaboration patterns. PLoS One. 2015;10(3):120495.

15.

Hughes

Murphy

. The use of Twitter by trauma and orthopaedic surgery journals: Twitter activity, impact factor, and alternative metrics. Cureus. 2017;9(12):1931.

16.

Kambhampati

Vaishya

. Trends in publications on the anterior cruciate ligament over the past 40 years on PubMed. Orthop J Sports Med. 2019;7(7):2325967119856883.

17.

Kunze

Polce

Vadhera

, et al.

What is the predictive ability and academic impact of the altmetrics score and social media attention?

Am J Sports Med. 2020;48(5):1056–1062.

18.

Patthi

Prasad

Gupta

, et al. Altmetrics–-a collated adjunct beyond citations for scholarly impact: a systematic review. J Clin Diagn Res. 2017;11(6):16–20.

19.

Perneger

. Relation between online “hit counts” and subsequent citations: prospective study of research papers in the BMJ. BMJ. 2004;329(7465):546–547.

20.

Polce

Kunze

Farivar

, et al. Orthopaedic randomized controlled trials published in general medical journals are associated with higher altmetric attention scores and social media attention than nonorthopaedic randomized controlled trials. Arthroscopy. 2021;37(4):1261–1270.

21.

Priem

Piwowar

Hemminger

. Altmetrics in the wild: An exploratory study of impact metrics based on social media. Metrics 2011: Association for Information Science and Technology. Presented at: Symposium on Informetric and Scientometric Research (SIG MET); October 12, 2011. New Orleans, USA. https://docs.google.com/presentation/d/1JOBBjzKEDmtE5Z6W4ZYrB0kjuqU2O6Hi-hpwmJtVhJY/edit?pli=1#slide=id.i0

22.

Salkind

NJ.

Statistics for People Who (Think They) Hate Statistics. Sage Publications; 2000.

23.

Scarlat

Mavrogenis

Pećina

, et al. Impact and alternative metrics for medical publishing: our experience with International Orthopaedics. Int Orthop. 2015;39(8):1459–1464.

24.

Silva

DDO

Taborda

Pazzinatto

, et al. The altmetric score has a stronger relationship with article citations than journal impact factor and open access status: a cross-sectional analysis of 4022 sport sciences articles. J Orthop Sports Phys Ther. 2021;51(11):536–541.

25.

Thelwall

Haustein

Larivière

, et al. Do altmetrics work? Twitter and ten other social web services. PLoS One. 2013;8(5):64841.

26.

van Melick

Van Cingel

Brooijmans

, et al. Evidence-based clinical practice update: practice guidelines for anterior cruciate ligament rehabilitation based on a systematic review and multidisciplinary consensus. Br J Sports Med. 2016;50(24):1506–1515.

27.

Zhu

Zhang

Wang

Xie

Wang

. The top 100 highly cited articles on anterior cruciate ligament from 2000 to 2019: a bibliometric and visualized analysis. Orthop Traumatol Surg Res. 2021;107(8):102988.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB