Abstract
Background:
Alternative citation metrics—such as the Altmetric Attention Score (AAS), number of tweets (TN), and dimensions badge value (DBV)—are emerging as new options for assessing the value of scientific works.
Purpose:
To analyze the AAS of highly cited articles on the anterior cruciate ligament (ACL) and to assess the relationship between alternative and traditional metrics such as journal rankings and article citation performance.
Study Design:
Cross-sectional study.
Methods:
A search was conducted using the Web of Science Core Collection of databases with “anterior cruciate ligament” as the search term. Full-text articles published between 2011 and 2021 were reviewed, and the top 100 cited articles were determined. The articles were analyzed by publication year, study design, research topic, journal impact factor, journal h-index, number of total citations (TC), recent citations (RC) (ie, citations in the latest 2 years), and average citation per year (ACY), as well as AAS, TN, and DBV.
Results:
For articles in the top 100 list, the median TC was 160 (interquartile range [IQR], 117-561) and the median AAS was 24.50 (IQR, 1-730). A higher AAS score was achieved by articles on return to sports and anterolateral ligament (P < .05). The AAS and TN were significantly and positively correlated with the RC (r = 0.459 and P = .001; r = 0.438 and P = .001, respectively) and ACY (r = 0.363 and P = .001; r = 0.393 and P = .001, respectively).
Conclusion:
Alternative metrics were linked to traditional metrics but were not a direct representation of bibliometrics. The AAS was not correlated with TC numbers in the ACL research. Higher AAS and Twitter popularity of an article were related to receiving high-volume RC performance. The DBV represented a stronger correlation with traditional metrics than the AAS.
Keywords
In search of instantaneous information on the impact of an article, scholars have begun exploring new nonformal online platforms that have emerged in the past few decades.
4,19
These innovative bibliometric approaches, or alternative metrics, provide a more recent interaction with the published articles compared with traditional citation-based metrics such as journal rankings and article citation performance, which require 2 or 3 years to reach their real value to accumulate.
5
One of these alternative metrics—known as Altmetrics— is a web-based quantitative representation of online attention to a research paper.
20
Different databases are analyzed and combined to produce an Altmetric donut—a visual representation (Figure 1)—and the Altmetric Attention Score (AAS)—a numerical value. Altmetrics mainly evaluates the online performance of a study beyond medical databases and may be considered a tool to measure the influence of publications on the community.
11,21
Demonstration of an Altmetric donut, with an Altmetric Attention Score inside the donut. Each color in the donut corresponds to a different source that is available for a given manuscript. The additive value of each source is also indicated at each line.
Another alternative performance metric for published research is dimensions. This includes many sources that belong to each step in the production and postpublishing processes, including Altmetrics.
18
It takes into account the citation performance of the latest 2 years, also called recent citations (RC). It combines data from publications, data sets, patents, and policy documents to calculate a score called a dimensions badge (Figure 2) and produces an interconnecting dimensions map.
9
In this sense, dimensions may be considered as an effort to unite the scholarly and nonscholarly performance of scientific research papers. Demonstration of a dimensions badge value (DBV). The DBV is given in the geometric figure. The values of total citations, recent citations (in the past 2 calendar years), field citation ratio, and recent citation ratios are also available aside. In the given example, the number of recent citations is 157 and the DBV is 709.
Twitter is a popular social media tool not only for the instantaneous sharing of information but also for diverse scientific activities. AlFaris et al 2 reported that 35% of medical students use Twitter for vocational training, as an increasing number of medical journals publish their Twitter handle on the internet. The notification rate of research papers on Twitter was reported 14 to be ≥20%. The total number of tweets (TN) and retweets for a study is now considered an early estimator of its impact in terms of Altmetrics. 10
Costas et al 6 published a study in 2015 on the relationship between alternative metrics and traditional metrics. Batooli et al 4 reported a strong positive correlation between the number of views of the articles in ResearchGate and Mendeley and the number of citations of the articles in Scopus. Thelwall et al 25 reported statistically significant associations between higher metric scores and higher citations for articles with positive Altmetric scores in forums such as Twitter, Facebook wall posts, research highlights, blogs, the mainstream media, and forums (P < .05). Scarlat et al 23 mentioned that high-volume publications also have a high-volume social media impact. Silva et al 24 reported that the AAS exerted a stronger relationship with the number of citations than the journal impact factor and open access status in sports sciences journals. Hughes et al 15 demonstrated that journals with higher numbers of retweets had higher AAS and were positively associated with impact factor.
On the other hand, Kunze et al 17 did not find a strong correlation between traditional metrics and the AAS in orthopaedic studies. De Gregori et al 8 reported that alternative metrics showed a statistically weak correlation with Facebook and TN, with the exception of those for Mendeley. These studies suggest that although alternative metrics have interrelations with citation performance, they may not represent them in the same way and may instead have a potential additive value in general.
The anterior cruciate ligament (ACL) is a popular subject not only for researchers but also for the growing sportive population, promising a high AAS in addition to good citation performance. There is a continuing interest in the improvement of ACL treatment modalities to provide an early return to previous sport levels. A good example of this is the resurgence in anterolateral ligament (ALL) reconstruction techniques as an adjunct to ACL reconstruction, which has been reported to decrease revision rates significantly. 12,13 The purpose of the present study was to analyze the alternative metric scores of highly cited recently published articles on the ACL and to assess the relationship between alternative and traditional metrics. We postulated the following hypotheses:
Alternative modes of performance scores of the scientific works—such as AAS, TN, and dimensions badge value (DBV)—might be linked to each other and, at the very least, to some parameters of the traditional metrics.
Alternative metrics—such as the AAS and DBV—are under the influence of social media. Thus, the research focus of alternative metrics and scholarly metrics may be different considering their different data sources.
Methods
Data Collection and Analytical Tools
The present study was conducted in accordance with the Helsinki Declaration of ethical principles. No ethical approval was required from the local committee of ethics regarding the data extracted from the literature. No personal data were included. On July 31, 2021, we conducted a search for articles published between 2011 and 2021 using the search term “anterior cruciate ligament” in the Web of Science Core Collection—including the Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, Conference Proceedings Citation Index–Science, Conference Proceedings Citation Index–Social Science & Humanities, Book Citation Index–Science, Book Citation Index–Social Sciences & Humanities, and Emerging Sources Citation Index databases.The first 100 full-text articles having the highest citations were selected by the authors. Only the articles under the human health category on the epidemiology etiology, pathophysiology, histology, diagnosis, therapy, and prognosis related to the ACL were selected. Articles with only abstracts were excluded.
The impact factor for each included journal was retrieved from the 2019 Clarivate Journal Citation Reports. All journals were searched in the 2020 Scimago Journal and Country Rank (SJCR) for their h-index and quartile index.
For all articles, we recorded the publication year, author names, publication year study design, research category and topic, level of evidence, impact factor, field citation ratio (FCR), total citation (TC), average citation per year (ACY), RC, DBV, AAS, and TN. The FCR is available for a field with >500 articles that have been published in the past 2 years. It is calculated by the division of the TC value of a given article by the ACY value in the same field. Study types and evidence levels were reviewed using the 7-grade list provided by Ackley et al. 1
The Altmetric it tool on the Altmetric website was used to delineate the AAS and TN values of the articles included (https://www.altmetric.com/products/free-tools/bookmarklet/). Given an article’s digital object identifier number, the AAS of each article was calculated automatically by a weighted average process, evaluating the attention of the article in alternative sources and producing an Altmetric donut, with each color symbolizing a distinct source of attention. The DBV for each article was gathered from the dimensions website (http://www.digital-science.com/product/dimensions).
Mapping knowledge domains is a method for visual representations of bibliometric data. VOSviewer (https://www.vosviewer.com) is an online tool that calculates co-occurrence data and builds visual networks in the form of interconnected hotspots. 16 In this study, visual bibliometric coupling analyses on the country of origin and article keywords of the top 100 cited publications were performed with VOSviewer software Version 1.6.17. A threshold value of 2 (ie, the number of publications in which ≥2 keywords were shared) was used in the keyword co-occurrence analysis maps.
Statistical Analysis
Descriptive statistics for continuous data are shown as mean ± SD or median and range or interquartile range (IQR). Categorical data are shown as percentages. The Shapiro-Wilk test was used to test the distributions of continuous variables for normality. The differences in parameters between ≥3 groups were compared using the Kruskal-Wallis (post hoc: Dunn) or analysis of variance (post hoc: Duncan) tests. Spearman or Pearson correlation coefficients were calculated to detect any linear relationships between the variables of interest, in which a correlation coefficient (r) between 0.8 and 1 was considered very strong, between 0.6 and 0.79 was considered strong, between 0.4 and 0.59 was considered moderate, 0.2 and 0.39 was considered weak, and between 0 and 0.19 as very weak. 22 Beta coefficients were estimated by univariate linear regression analysis. All analyses were performed by using SPSS Statistics for Windows Version 23.0 (IBM). P < .05 was considered statistically significant.
Results
A total of 25,001 articles using the term “anterior cruciate ligament” were identified in our Web of Science search. Limiting the search to articles published between 2011 and 2021 resulted in 14,868 papers in the preferred indexes. English language limitation resulted in 14,537 papers. A total of 13,124 articles—11,751 reviews and 1373 original articles—were selected. The articles in the top 100 list are presented in the Supplemental Material, available separately.
The overall characteristics of the top 100 articles are provided in Table 1. The included articles had a mean TC value of 181.34 ± 78.15 and were considered high-impact studies, with a mean level of evidence of 4.31 ± 1.40 (Table 1). The included articles were cited 18,134 times by 6877 papers. Excluding self-citations, the search revealed 17,824 citations in 6805 articles. All articles were published between 2011 and 2017, with 90% of the studies published between 2011 and 2015 (Table 2). Cohort studies (n = 18) and systematic reviews (n = 22) were dominant (Table 2). The mean 2019 impact factor of the journals in which these articles were published was 6.10 ± 4.23.
General Characteristics of the Top 100 Articles on ACL Research a
a ACL, anterior cruciate ligament; AAS, Altmetric Attention Score; ACY, average citations per year; DBV, dimensions badge value; IF, impact factor; LoE, level of evidence; RC, recent citations; TC, total citations; TN, number of tweets.
b According to Ackley et al. 1
Characteristics of the Top 100 Articles a
a Values are presented as No. of articles.
b Articles with >1 topic.
Journals That Published Articles in the Top 100 List
According to the publication year, the RC was strongly correlated with the AAS and TN (P < .05). The AAS, TN, and ACY increased significantly with publication time (Figure 3 and Table 4).
Progression of the mean TC, RC, AAS, and TN by publication year, 2011-2017. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.
Comparison of TC, RC, AAS, and TN According to Publication Year, 2011-2017 a
a Data are reported as median [interquartile range]. Bold P values indicate significant differences within the column (P < .05, Dunn or Duncan test). AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.
The study design and research category were not correlated with any of the traditional or alternative citation metrics (Tables 5 and 6). Regarding research topics, studies on return to sports (RTS) were found to be related to increased TC, AAS, and TN (P < .05), and studies on anatomy and on the ALL had higher AAS and TN compared with others (P < .05), RC was statistically correlated with meniscal and cartilage lesions and return to sport (P < .05) (Table 7).
Comparison of TC, RC, AAS, and TN According to Study Design a
a Data are reported as median [interquartile range]. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.
Comparison of TC, RC, AAS, and TN According to Research Category a
a Data are reported as median [interquartile range]. AAS, Altmetric Attention Score; RC, recent citations; TC, total citations; TN, number of tweets.
Comparison of Research Topic According to the Distribution of TC, RC, AAS, and TN a
a Data are reported as median [interquartile range]. Bold P values indicate significant differences within the column (P < .05). AAS, Altmetric Attention Score; ALL, anterolateral ligament; RC, recent citations; RTS, return to sports; TC, total citations; TN, number of tweets.
TC and AAS Analyses
The median values for the TC and AAS were 160 (IQR, 117-561) and 24.50 (IQR, 1-730), respectively. The top study, “Return to Sport Following Anterior Cruciate Ligament Reconstruction Surgery: A Systematic Review and Meta-analysis of the State of Play” was conducted by Ardern et al 3 and was published in 2011; it achieved a TC of 561 and an AAS of 96, ranking it 16th in terms of AAS performance. “Evidence-based Clinical Practice Update: Practice Guidelines for Anterior Cruciate Ligament Rehabilitation Based on a Systematic Review and Multidisciplinary Consensus” conducted by van Melick et al 26 was published in 2016 and had the highest AAS (n = 730). The AAS was not available for 7 studies in the 100 list (Table 8). Annual cumulative totals of TC demonstrated less growth over the captured time compared with the increasing performance of the AAS and TN after 2015 (Figure 3).
TC, TN, AAS, and DBV of the Top 10 Articles According to AAS a
a AAS, Altmetric Attention Score; DBV, dimensions badge value; TC, total citations; TN, number of tweets.
Twitter Analysis
The analysis of the TN revealed that 87 articles from the top 100 list were shared on Twitter. Of the top 10 articles with the highest TN, 4 dealt with ACL rerupture, 3 with RTS and rehabilitation, and 1 with ALL (Tables 4 -7).
Journal Analysis
According to the 2020 SJCR, all journals had a Quartile 1 index ranking, indicating that they occupied the top 25% of journals within their subject area. Among the 20 journals in which the articles in the top 100 list were published, The American Journal of Sports Medicine (AJSM) published the most cited articles (n = 49) (see Table 3). In contrast, the first 3 articles in the top 100 list were published in the British-based journals (Supplemental Material).
The articles with the highest TC and AAS were published in the British Journal of Sports Medicine—the journal with the highest impact factor in the top 100 list. There was a British journal predominance, with 6 articles in the list of the first 10 papers according to their AAS in the ACL research (Table 8).
Research Topics
The research category with the most cited and tweeted articles was postoperative knee function and concomitant injuries, whereas the main research topics were RTS and rehabilitation. When listed according to their AAS performance, the main research topics of the top 10 articles were RTS (n = 3) and ACL rerupture (n = 3) (Table 8).
Correlation Analysis
The results of the correlation analysis are displayed in Table 9. The AAS was found to have a strong positive correlation with the TN (r = 0.971; P = .001). While the AAS and TN did not have a significant correlation with the TC, they did show a weak positive correlation with the ACY (r = 0.363 and P = .001; r = 0.393 and P = .001, respectively) and a moderate correlation with the RC (r = 0.459 and P = .001; r = 0.438 and P = .001, respectively). The AAS had a weak negative correlation with the publication year (r = –0.315; P = .002).
Results of the Correlation Analysis a
a The r values were obtained from the Spearman rank correlation or the Pearson correlation (n = 100). AAS, Altmetric Attention Score; ACY, average citation per year; DBV, dimensions badge value; FCR, field citation ratio; IF, impact factor; LoE, level of evidence; RC, recent citation; TC, total citation; TN, number of tweets.
b Publication year.
c Correlation is significant at the .01 level.
d Correlation is significant at the .05 level.
There was a weak but positive correlation between the FCR and TN (r = 0.279; P = .008). However, the FCR was strongly correlated with the TC, ACY, and RC. The FCR showed a weak significant negative correlation with the publication year (P < .05). The TN had a weak positive correlation with the ACY and dimensions. On the other hand, it was negatively correlated with the publication year (P < .05).
The journal h-index and journal impact factor were strongly correlated (r = 0.604; P = .001). The journal impact factor was weakly correlated with the AAS, FCR, TN, RC, ACY, and DBV (P < .05). The journal h-index was correlated significantly but weakly with the level of evidence (r = –0.242; P = .016).
The DBV showed strong correlations with traditional citation metrics such as the TC (r = 0.935; P = .001), FCR (r = 0.850; P = .001), and ACY (r = 0.840; P = .001) but weak correlations with alternative metrics such as the AAS (r = 0.234; P = .025) and TN (r = 0.224; P = .032).
Results of Regression Analysis
There was a moderate positive correlation between the AAS and RC (r = 0.459; P = .001) (Figure 4A). According to univariate linear regression analysis, ∼21% of the variation in the AAS was due to recent citations. A 1-unit increase in recent citations resulted in a 1.54 increase in the AAS.
There was a moderate positive correlation between the RC and TN (r = 0.438; P = .001) (Figure 4B). According to univariate linear regression analysis, ∼19% of the variation in the RC was explained by the TN. A 1-unit increase in the TN resulted in a 0.11 increase in the RC. There was a strong positive correlation between the AAS and TN (r = 0.971; P = .001) (Figure 4C). According to univariate linear regression analysis, ∼94% of the variation in the AAS was explained by the TN. A 1-unit increase in the TN resulted in an 0.8 increase in the AAS.
Scatterplots of the relationship between (A) the AAS and RC, (B) the RC and TN, and (C) the AAS and TN. There is a varying degree of correlation between the AAS, RC, and TN, and the AAS and TN showed the highest correlation. AAS, Altmetric Attention Score; RC, recent citations; TN, number of tweets.
Visualization Analysis
The overall strength of bibliographic coupling linkages with other countries was calculated and visualized for each of the 23 countries in the top 100 list (Figure 5). The magnitude of the nodes corresponds to their publication number and citation performance. The power of interrelation and collaboration among countries was indicated by the thickness and distance of links between nodes (Figure 5). The biggest spot in the country table was occupied by the United States, which was the most competing country in terms of the number of publications and citations (n = 55 documents), TC received (n = 9659 citations), and international cooperation index (n = 6826 link strength). In terms of the number of publications, TC, and international cooperation index, the United States was followed by Australia ([13] [3153] [2814]), Sweden ([10] [1751] [2586]), Norway ([7] [1751] [2853]), Belgium ([6] [1320] [1826]), Canada ([6] [859] [1024]), England ([4] [677] [1223]), France ([4] [660] [771]), Denmark ([3] [529] [555]), and Austria ([2] [378] [1387]).
VOSviewer output for countries. Schematic representation of the distribution of papers according to countries allows for visualizing the impact of each country. The United States is in the center and has the highest number and strongest interconnections.
The hotspots of the ACL research were produced by the VOSviewer tool by calculating co-occurrence levels of high-frequency keywords. The keyword co-occurrence limit of ≥2 was met by 53 of the 181 extracted keywords, and the network of keywords produced 6 major clusters, represented by different colors in Figure 6. The most prevalent keywords after “anterior cruciate ligament” were as follows: “ACL reconstruction”; “knee”; “anterolateral ligament”; “return to sport”; “revision”; “epidemiology”; and “biomechanics,” with a total occurrence of 36, 29, 11, 8, 7, 6, 5, 4, respectively.
The visual representation of links between the keywords demonstrated that the anatomic studies concentrated around ALL studies. RTS was considered one of the key elements of knee function. Injury was the most recurring keyword, appearing in 3 different clusters, “ACL injury” in the knee injury cluster, “reinjury” in the allograft cluster, and “injury” in the epidemiological studies and meniscectomy cluster (Figure 6).
Visualization of VOSviewer keywords co-occurrence. The distribution and interrelation of keywords with ≥2 co-occurrences produced 6 major clusters: knee, knee injury, re-injury, revision, anterolateral ligament, epidemiology, each represented by a different color. ACL, anterior cruciate ligament; KOOS, Knee injury and Osteoarthritis Outcome Score.
Discussion
This manuscript was based on the bibliometric information of the top 100 most cited ACL studies between 2011 and 2021 and evaluated the relationship between traditional metrics—including TC, journal h-index, and impact factor—and alternative metrics. To the best of our knowledge, this is the first study to compare traditional metrics with alternative metrics, including Twitter, Altmetrics, and dimensions.
The most striking finding of this study may be the strong correlation between the RC, AAS, and TN according to the publication year, supporting the first hypothesis of this manuscript. RCs are those that are not taken into consideration in journal index performances. Therefore, the parallelism of the AAS performance with the RC renders the AAS a good indicator of the future value of a manuscript. Moreover, being a component of the AAS, Twitter’s popularity may represent the AAS performance, supporting the idea of sharing publications among scholars on Twitter to augment the TC of a given study. According to another study, 7 journals with a considerable Twitter performance present a high AAS performance as well.
The journal impact factor is considered to be one of the most trusted bibliometric power indicators. 5 It is a measure of the total performance of all of the articles in the journal rather than a specific one. In this study, the impact factor was found to be weakly correlated with the citation-based and alternative metrics in ACL studies and strongly correlated only with journal h-index. Moreover, journal h-index was weakly correlated with the level of evidence. In other words, the scientific level of a journal is not enough to attract the attention of the online community and does not guarantee the level of evidence for a highly cited article.
One of the initial hypotheses of this paper—the possible difference in the interest fields of patients and scholars—was evaluated in this study. Although the research category and research topic of the studies were not correlated with the citation metrics in this study, research on RTS was found to have a positive influence on the TC and AAS, whereas the ALL research was found to be correlated with the AAS and TN. Thus, RTS is still one of the most important issues for both researchers and social media users, possibly athletic patients. On the other hand, the correlation of ALL studies with increased AAS may be considered as the tendency of scholars to share those articles on the online platforms. Therefore, the AAS may be considered the bridge where the 2 major populations of interest meet, as it represents both scholar and nonscholar interest fields simultaneously.
The DBV was found to be weakly correlated with the AAS and TN but strongly correlated with traditional metrics in this research. In other words, dimensions may have a different category than Altmetrics by being closer to the citation-based metrics. This is probably because of its stronger connections with research data than Altmetrics, which is only a small component of its database. From this view, dimensions value may be regarded as an alternative metric that preserves the density of scholarly content while not ignoring nonscholar sources of attention.
The results of this study reaffirm those of Zhu et al 27 who reported that in the past 20 years, the most influential journal in ACL surgery was the American Journal of Sports Medicine and the most productive country was the United States. According to a visual representation of interconnections between countries, the United States is not only the strongest country in publication numbers but also the one with the strongest international relationships with other countries. On the other hand, although the United Kingdom was ranked sixth in the number of publications, journals from the United Kingdom were also successful, especially in terms of publishing papers with the highest TC and AAS values.
Limitations
This study was based on the bibliometric data of the ACL research in a given time, and the variables are time-dependent and thus may show differences with the progression of time. Twitter, AAS, and dimension values were selected to represent alternative metrics. However, more databases exist to evaluate alternative attention sources that were not included in this study. Furthermore, not all social media platforms are open to data searches, such as those used in China, Russia, and North Korea. Additionally, the selected language of this study is English, and thus, the literature in other languages was not included.
Conclusion
Alternative metrics are a new approach to representing the value of a research paper in a broader sense. Altmetrics is not a direct representation of bibliometrics, and the AAS is not correlated with TC numbers, impact factor, or journal h-index. The TN value is a good estimator of the RC value, suggesting that articles that have a high social media performance may be expected to maintain receipt of high-volume citations. Dimensions are an example of the new perspective in the metric systems through the combination of alternative metrics with traditional metrics. Rather than being a complementary tool to traditional metrics that measure the scholarly impact of a research paper, alternative metrics—including attention to social media such as Twitter—seem to change the rules of the “citation race” with their power to reflect the interest of a greater population. Future metrics will be more powerful and instantaneous with the accumulation of alternative metrics data.
Supplemental Material for this article is available at https://journals.sagepub.com/doi/full/10.1177/23259671231166701
Supplemental Material
Supplemental Material, sj-pdf-1-ojs-10.1177_23259671231166701 - Relationship of Altmetric Attention Score, Twitter Performance, and Dimensions Badge Value With Traditional Metrics in Top-Cited Anterior Cruciate Ligament Research Studies
Supplemental Material, sj-pdf-1-ojs-10.1177_23259671231166701 for Relationship of Altmetric Attention Score, Twitter Performance, and Dimensions Badge Value With Traditional Metrics in Top-Cited Anterior Cruciate Ligament Research Studies by Hasan Basri Sezer, Yoann Bohu, Alexandre Hardy, Nicolas Lefevre and Aysun Sezer in Orthopaedic Journal of Sports Medicine
Footnotes
Final revision submitted January 5, 2023; accepted January 27, 2023.
The authors declared that there are no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
