Abstract
Background
Alternative metrics to traditional, citation-based metrics are increasingly being used. These are complementary to traditional metrics, like downloads and citations, and give information on how often a given journal article is discussed and used in professional (reference managers) and social networks, such as mainstream media and Twitter. Altmetrics is used in most journals and is available in all indexed headache medicine journals. Whether Altmetrics have an input on traditional, citation-based metrics or whether it is a stand-alone metric system is not clear. Actively promoting a paper through media channels will probably increase the Altmetric score but the question arises whether this will also increase citations and downloads of this individual paper.
Methods
Focusing on this point we performed a randomized study in order to test the hypothesis that a promotion intervention would improve citations and other science metric scores. We selected 48 papers published in Cephalalgia from July 2019 to January 2020 and randomized them to either receive an active promotion through social media channels or not. The primary outcome used was the difference between mean article citations with versus without intervention 12 months after the intervention period.
Results
The results show that the alternative metrics significantly increased for those papers randomly selected to receive an intervention compared to those who did not. This effect was observed in the first 12 months, right after the boosting strategy was performed. The higher promoted paper diffusion in social media lead to a significantly higher number of citations and downloads.
Conclusion
Further promotion strategies should be studied in order to tailor the best cost-benefit intervention.
Introduction
The traditional way of measuring the scientific impact of an article or a journal is by its number of citations in peer-reviewed journals, the so-called impact factor (IF) which was introduced in 1975 (1). Today, the IF is used worldwide as a journal level-metric, i.e. to rank journals for their relative importance within their field and this ranking is used by universities and funding bodies. Using the scores based on the number of journal citations to decide on promotion and research proposals has been criticised (2). The rationale is that the IF does not necessarily reflect a greater dissemination of clinical or scientific content to clinicians, patients and society. A finding or paper may be highly interesting for society and may be frequently discussed, but still little cited. Additionally, the IF may disincentivize the pursuit of risky and potentially ground-breaking science, since areas already highly populated are more likely to have large numbers of scientists expected to reference one’s work, even if only mainstream (3).
Until recently, the impact of scientific articles on social media and reference managers was not quantified (4). Alternative and complementary to traditional, citation-based metrics other measures of impact have been introduced recently to measure the impact spread of scientific papers (5,6). The so-called alternative metrics, in contrast to the traditional metrics, report how often a given scientific paper is communicated to fellow users, measuring indicators for research impact such as the influence of science in public awareness and health care policies. To date, four methods have been used as an alternative metrics: Altmetric (4), Plum X (2,7), PLOS ALM (Article-Level Metrics) (8) and ImpactStory (9).
Altmetric (http://www.altmetric.com) is currently the most used. It is a tool developed by a group of British researchers, condensing the metric information in the so-called Altmetric Donut (see Figure 1 as an example). The Altmetric Donut is in essence a coloured ring in which each colour represents a different source with a particular scoring system (10). The Altmetric score is calculated for a single article, reflecting how often it has been mentioned in social media (Facebook, Twitter), newspapers, encyclopaedias (Wikipedia), online platforms (Faculty1000 and Publication Peer-Reviews), videos on YouTube, questions and answers websites (Q&A stack overflow), and policy documents available online. It is also used by many publishers, including SAGE, for the journals Cephalalgia and Cephalalgia Reports.
Example of Altmetric donut in a Cephalalgia paper.
Studies have demonstrated that traditional journal citations correlate with Altmetric scores in different scientific areas (11–14). One study suggested that the number of tweets in the first days of publication predicted the number of article citations (15). This study needs to be repeated to understand whether indeed tweets have an impact on citations, given that tweets are not usually based on the scientific rigor of the content tweeted but the awareness the content creates in the individual twitter user. It is also unknown, whether a specific intervention in order to increase social media/Altmetrics citation would improve the number of traditional citations.
Cephalalgia (subscription journal) and Cephalalgia Reports (open access) are under SAGE Publishing management, and beside the IF Altmetric is also provided by SAGE for all published papers. However, little is known about the influence of Altmetric on the journal’s impact factor which is why we studied the effect of Altmetric score on Cephalalgia citations using a prospective, randomized design. We hypothesized that an intervention directed to improve Altmetric scores would increase paper citations over time compared to papers without this intervention (see Table 2).
Papers distribution according to number of authors in each paper, their countries and regions; baseline Altmetric score, download number and citations in control and intervention groups.
*Total accounts for more than 25 papers due to cooperation between countries in certain papers.
**co-operations were between USA-Europe, Europe-Asia, Asia-Europe.
Downloads, Altmetric scores and citations (dimensions, crossref and web of science) 12 and 24 months after intervention in both groups.
*p < 0,001; **p < 0,05; #p = ns.
Altmetric scores and its components. Note that papers from the control group were naturally also noted in the respective news and media, but significantly less than in the intervention group.
*Each News score accounts for 7 Altmetric points.
Methods
This is a prospective, randomized, parallel-arm, superiority trial, comparing two groups of papers published in Cephalalgia from July 2019 to January 2020 (online first). Papers with original research data, reviews, opinion pieces, case series were chosen, 48 published articles were randomized and all were included in the analysis. Editorials, simple case reports and papers written by pharmaceutical companies were excluded. Papers were randomly allocated, on a (1:1) ratio, to the intervention or control group (no intervention, i.e. no boosting of the article), totalling 24 articles in each group. All randomized manuscripts were given the intended intervention, 48 eligible manuscripts were available for randomization during the study time-frame of six months. No power calculation was performed.
Each month papers were paired according to manuscript type, each of the articles paired were then randomized to the control or intervention group. One author (MFPP) selected the papers pairing by article type, then sent these papers to a third party who randomized them using a computer generated list of random numbers. No post randomization exclusions occurred.
Number of authors, countries, and institutions who participated in the study were collected. Enrolment ceased when the first six-month time frame ended. The study statistician generated the allocation sequence, one of the authors (MFPP) assigned and enrolled papers into the study. Blinding was not used since it is not relevant for the study methodology.
Twelve months after the paper was published online, each article was measured by its Altmetric scores and its components, number of tweets, number of citations (measured by Dimensions, Crossref and Web of Science) and number of article downloads. A second round of measures was performed 18 to 24 months after the online publication date.
The intervention was focused on boosting Altmetric scores throughout the campaign for the randomly allocated papers. Most of the work was done during the first two months after paper publication, but limited to six months.
Interventions followed a standardized procedure, authors were asked about their papers’ main findings, what aspects should be relevant to headache medicine and its impact to society. One of the authors (MFPP) wrote a press release about each selected article. Images were selected, articles without illustrations received a customized illustration for the press release. A Twitter-tailored text about each selected article was posted one to four weeks after its publication online on the International Headache Society (IHS) social media account and their members were invited to retweet. A campaign was performed on the platform in order to promote the posts.
Third parties were contracted to publish and promote the press release on Facebook, Reddit, news outlets, blogs, Publon reviews, F1000, Wikipedia, and YouTube (16). Some authors agreed to record videos about their papers, the material was edited and posted at the IHS YouTube channel.
The study variables were: Article Altmetric score – defined as the Altmetric score 12 and 24 months after its publication online; and Article Citation – defined as the number of citations a given article had 12 and 24 months after its publication online.
The primary outcome used was the difference between mean article citations with versus without intervention 12 months after the intervention period.
Secondary outcomes were: mean articles Altmetric scores and its components; and mean article citations 12 months after intervention phase in intervention and control groups.
The trial was conducted in accordance to the original protocol, no changes to the eligibility criteria or protocol occurred after the trial started (see Table 3).
Statistical analysis
Statistical analysis was performed in order to compare the least square means of article citations 12 and 24 months after intervention phase between the intervention group versus control group (Welch’s t-test). Altmetric scores were correlated with number of tweets, number of article downloads, views, number of authors, countries, and institutions using Pearson or Spearman’s correlation test. A p < 0.05 was considered significant.
Because of the nature of this trial, it was not registered to the available registries. The full trial protocol is available upon request.
Results
Forty-eight papers were selected. Table 1 shows the papers’ distribution according to author number in each paper and regions in both groups. Number of authors in each group were equivalent.
The majority of papers were from authors with European institutional affiliation (n = 24), followed by North America (n = 14), and Asia (n = 9). The number of authors in each paper did not correlate with number of downloads (r −0.088; p = 0.54), Mendeley readers (r 0.073; p = 0.62), Altmetric scores (r −0.082; p = 0.57), or citations (r −0.015; p = 0.92). Co-operative studies between continents (i.e. authors from more than one continent on the same paper, n = 6) showed relatively higher citation mean (10.2 ± 8.4) when compared to studies from North America (7.7 ± 6.2) vs Europe (6.1 ± 4.5).
The intervention group (n = 24) had a significantly higher Altmetric score, after 12 (total score 1296, mean 54 ± 29.8 vs 172, mean 7.2 ± 6.3 in controls), and 24 months (1334, 55.6 ± 32.3 vs 194 ± 8.1)), p < 0.001. Number of downloads were 54% higher in the intervention group at 12 months (p = 0.04), and still 27% higher at 24 months (p = 0.06). Citations were higher in the intervention group, from both the scores provided by Dimension Citations (26% higher at 12 months, 17% at 24 months (p = 0.04)), Crossref (20% at 12 months, and 19% at 24 months (p = 0.04)), and Web of Science (17% at 12 months, 11% at 24 months, p = 0.18) databases. Baseline Altmetric scores were balanced between groups (see Table 1).
Discussion
Previous studies showed a positive correlation between Altmetrics scores and citations (17–19), but limited information is available as to whether improving the alternative metrics with a specific strategy is worthwhile for improving citation number. Our data strongly suggests that this is the case.
Our study showed that the alternative metrics of scientific citation significantly increased in those papers randomly selected to receive an intervention targeted to augment the Altmetric score, compared to those who did not receive such an intervention. This effect was pronounced in the first 12 months right after the boosting strategy was performed. The increased number in the Altmetric score was associated with an increase in citations. The number of citations were higher in the intervention group, mostly in the first 12 months, suggesting the effect is more related to the intervention, as opposed to the natural citation increase observed overtime. A continuous effort might be beneficial for papers’ improvement in alternative scores and traditional citations after 12 months, but our pilot project focused the intervention only on a limited six-month time frame.
Some studies found that the number of authors correlated with citations and scores (20). We did not find such a correlation between number of authors and any of the endpoints, citations, downloads and Altmetric scores in our sample, including Mendeley readers. This is probably due to some papers with very high number of authors (around 20), from countries without social media or high usage of Mendeley reference software.
It is noteworthy that authors did not participate actively in promoting their papers during the study period, although we have asked them. By the time we started the interventions not many authors had social media accounts and/or did not see a possible benefit of promoting their papers in public domains. It is likely that more engagement in the social media activities by authors and researchers would achieve better paper scores. We note that we cannot make assumptions of a possible modulation of the impact factor through the augmentation of the Altmetric score. Given that the individual papers of the intervention group were more cited, this is however likely if done in a programmed and consequent process over years.
One may consider if our findings are applicable to other journals beyond headache medicine and neurology (21). Further experience may confirm this assumption.
Limitations of our study are the relatively small sample size of this pilot study. Including more papers in a longer observation frame would open the chance for analysing factors such as manuscript type, comparison between clinic versus experimental papers, population-based or clinic-based studies. We did not include clinical trials published by industry since industry papers are promoted with a lot more investment than the one available in our study. We also wanted to avoid criticism in promoting selected industry papers. Future studies targeting specific aspects of the alternative score such as Mendeley readers, Wikipedia, Twitter, Facebook or other social media would give better results. Most of our intervention related Altmetric scores came from Twitter and news outlets, and we did not investigate the role of other specific scores.
Conclusion
Promoting papers diffusion in social media led to a significant improvement in Altmetric score, which in turn increased the number of citations and downloads.
Further promotion strategies should be studied in order to tailor the best cost-benefit intervention.
Article highlights
Actively promoting a paper through media channels increases the Altmetric score significantly. A higher promoted paper diffusion in social media leads to a significantly higher number of citations and downloads.
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors are grateful for the funding of this study from the International Headache Society. Funding did not influence the study conduction in any way.
