Investigating the transparency and influenceability of altmetrics using the example of the RG score and the ResearchGate platform

Abstract

The present paper investigates the transparency and influenceability of the RG score. This altmetric specially developed by the best-known academic social network ResearchGate is intended to indicate a researcher’s academic perception in one single figure. We conducted a self-experiment to demonstrate that the indications of the RG score are difficult for the user to understand and are not transparent. They therefore do not fulfil the requirements of the Leiden Manifesto for research metrics. The results of our investigation show that activity in social networks appears to have a great impact on the RG score and can strategically and selectively influence this result. Furthermore, we succeeded in entering publications by other authors as our own, thus dramatically improving our RG score. On the whole, with a little effort and without any academic publications of our own we were able to achieve an RG score which was higher than almost half the scores of all ResearchGate users. This self-experiment should be interpreted as a pilot study and can be implemented in an expanded form in future.

1. Introduction

In the present era of Science 2.0, researchers have numerous opportunities of drawing attention to their output and putting this output online in the form of full texts, of exchanging views with other researchers, and managing their bibliographic data. For this purpose, researchers can make use of portals such as Academia.edu, ResearchGate, Mendeley, Bibsonomy, CiteULike, and Zotero ([13, pp. 39–40]; [19, p. 721]). The significance and relevance of such academic networks for researchers’ work is still unclear, with little research having been conducted on the matter. Moreover, there is an ongoing transformation of publication habits in the various disciplines ([10]; [19, p. 722]; [4]). Altmetrics are gaining increasing significance in this context since a consideration of classical citation figures related to journal publications is often one-dimensional and thus incomplete with respect to the impact of research on society beyond the classical peer-review journals [3]. Academic social networks offer various possibilities of solving this problem by employing their own metrics. Number of downloads, number of clicks, or specially developed metrics, such as ResearchGate’s RG score, are just a few possible ways in which a researcher’s social media impact [7] can be measured. The term altmetrics was first coined in 2010 [14] and there is still no generally recognized definition for this approach to measuring academic impact [4]. In addition, there is no consensus in the scientometric community on the value of altmetrics and what information such metrics could offer. Apart from the lack of a clear theory, there is considerable criticism of the fact that only the output of users of Web 2.0 systems can be measured as well as the wide variety of different types of metrics and opportunities for manipulating such metrics [15]. The present paper focuses on the latter problem, which will be investigated on the basis of the above-mentioned RG score. A self-experiment was conducted to show which factors have the greatest influence on the RG score and what strategy can be employed to influence the RG score most effectively.

2. ResearchGate and the RG score

The academic social network ResearchGate was founded in 2008 and is one of the pioneers of Science 2.0 platforms [7, p. 2]. With more than 12 million profiles, ResearchGate has a large number of users and according to a recent survey by the science journal Nature is currently very popular. 88% of the 3,500 scholars surveyed said that they were aware of ResearchGate, whereas 29% visited it regularly and also had their own profile [23]. A special feature that distinguishes ResearchGate from other academic social networks is that it has its own impact metric, the RG score. This metric consists of various components and – similar to the impact factor – is expressed in a single figure. The components include the “contributions” that can be controlled by registered users themselves by posting their own publications and composing questions and answers. Furthermore, “interactions” with other researchers on ResearchGate have an impact on the score [7]. In this context, the way in which questions and answers are perceived by other users is significant since these interactions can be marked by users as either positive (upvote) or negative (downvote). The evaluation system has now been changed to include only positive markings (“recommendations”). In addition, the RG scores of the users who evaluate the contributions and with whom you interact is an important factor: the higher their score the greater is the positive influence on your own score [16]. Thelwall & Kousha [20] describe the RG score therefore as “a hybrid scholarly achievements and site use indicator”. The authors note that “a large bias towards academics and institutions” is given by “its [RG score’s] activity component”. Tausch [18] criticizes the RG score as a contributor to “a real ranking-mania” and the fact that contributions of users with a lower RG score seem to have a less important value than those of users with a higher RG score.

All users can obtain a good overview of the breakdown of their own score as soon as they take a closer look at the RG score on their own profile. The breakdown of your score can be found there in the form of a pie chart [6, p. 2]. Although ResearchGate indicates which factors are included in the calculation of the RG score, no further information is given about these details and therefore the values and the associated algorithm remain a mystery [7, p. 2]. Furthermore, on its homepage ResearchGate claims that the RG score is being continuously refined and that the algorithm is therefore continually modified [16], which makes it more difficult to reconstruct the score. In spite of these difficulties, Jordan [6] attempted to reconstruct the RG score by calculating the correlations between the RG score and their factors. She was largely able to reproduce the score, but subscribed to the findings of [7] that the RG score is not transparent and that it is merely possible to reproduce the score to a certain degree [6, p. 2]. Orduna-Malea et al. [12] investigated which components of ResearchGate seem to have the biggest influence on the RG score. The results of the study show that asking and answering questions has a significant impact on the RG score and that “it seems impossible to get a high RG score solely through publications”. Neuhold [11] compared the trend of the RG score with the rise of the number of views, the asking and answering of questions and the number of citations. He came also to the same conclusion as [6,7] that the calculation of the RG score is irreproducible. This lack of transparency and the impossibility of correctly recalculating your own score contradicts the Leiden Manifesto for research metrics of Hicks et al. [5]. The authors are of the opinion that there are ten basic principles on which proponents of scientific evaluation should base their work. Amongst others, principle five says that those being evaluated should have the opportunity to verify the data on which this evaluation is based (“Allow those evaluated to verify data and analysis”, [5, p. 430]). In order to facilitate this verification, principle four moreover says that the data acquisition and evaluation should be kept open, transparent, and simple (“Keep data collection and analytical processes open, transparent and simple”, [5, p. 430]). ResearchGate’s RG score does not fulfil these two important principles so that according to [5] it cannot represent a reliable scientific metric for evaluating a researcher. Thelwall & Kousha [21] claim that “the potential for gaming the figures, would rule out ResearchGate data from formal evaluations”. In February 2016, Academia.edu also introduced its own metrics – the AuthorRank and the PaperRank. Whereas the AuthorRank appears in a user’s profile, every paper posted also has its own PaperRank. While the latter is influenced by recommendations from other users with respect to the paper in question, the AuthorRank is influenced by the PaperRank [1]. In contrast to the RG score, the algorithm for calculating Academia.edu’s two metrics was revealed when it was introduced.

Other critical voices that do not regard the RG score as a suitable metric for academic evaluations point out that the score is easy to manipulate. One possibility of boosting your own score is to personally increase the number of reads, that is to say the number of clicks obtained for your own publications on ResearchGate. According to ResearchGate [17], it is not possible to increase the number of reads from one’s own profile or from artificially generated traffic sources. Nevertheless, Haugen [2] succeeded in increasing his own score by means of a second profile. By a simple automated updating of the publications posted under his first profile, Haugen succeeded in multiplying the weekly reads so that he became the most read author in his discipline. Within a period of just three weeks, he was able to increase his RG score from 20.97 to 21.95 without any great effort. And by continuing this procedure he was able to increase his score even further [2, pp. 1–2]. Jordan [6, p. 3] found out that the RG score correlates with the quantity of answers given by the respective accounts. In this context Li et al. [9] found out that the “peer-judged answer quality” is associated with the responder’s RG score, answer length and response time. The authors were able to verify that the earlier an answer to a question is posted and the longer the content of this answer is, the more is the RG score influenced by these factors.

Only a few publications such as that by [2,6,9,11,12] have investigated the breakdown of the RG score in practical terms in order to illustrate the theoretical problems. In the present paper, we take a further step in this direction by investigating the RG score on the basis of a self-experiment. Some of the existing RG profiles already imply that the social activity seem to have a major influence on the RG score ¹ because of the fact that the users have a bigger proportion of social activity than publications while having RG scores that are higher than 97,5% of all users. The study performed by Orduna-Malea et al. [12] already showed that social activity has a significant impact on the RG score by investigating different author samples. The difference to this study is that the profiles from the samples have already been active for some time and therefore the RG scores had a certain extent. The aim of this study was to show what score a new user starting from scratch can achieve without the influence of a single academic publication and simply by making active use of the Q&A function. This work is limited to one case study due to the fact that only real persons are allowed to create profiles on ResearchGate. Furthermore, the previously shown examples of already existing profiles show tendencies toward the extent of influence that social activity seem to have. Our goal was to investigate what time it takes to achieve a certain score, if it is possible to develop strategies to maximize the growth of the score and which factors of social activity are the most promising in order to be able to do that. Additionally, we investigated whether it was possible to claim publications by other authors as one’s own in order to raise one’s RG score.

This experiment will be used as a basis to discuss the question of which requirements for stability, transparency, and clarity are fulfilled by altmetrics and their platforms, thus enabling altmetrics to function as reliable sources of information.

3. Research questions/method

The self-experiment aimed to answer the following questions with respect to the RG score:

How rapidly can your own RG score be increased by Q&A activity?

Which of the factors has the greatest influence on the RG score?

Can you achieve a score similar to or even higher than “real” academics?

Is it possible to assign publications by others to your own profile to boost your RG score?

In order to answer (1), Q&A activity was undertaken in ResearchGate with a newly created profile (Andreas Meier) without any publications at all. The level of activity was varied each week (ranging from regular to moderate or no activity) in order to observe and document fluctuations in the RG score. Furthermore, the factors were also varied in order to find an answer to question (2). In one case only questions were answered, in another interval questions were exclusively posed, and in another time slot a mixture of the two activities. In the study, the percentage by which the RG score exceeded other scores on ResearchGate was always documented. It should therefore be possible to answer question (3), i.e. the extent to which a good RG score can be achieved by investing a little time in order to become established as a real academic on ResearchGate. In order to answer question (4), we examined whether it is possible to assign publications to your profile that had been written by other academics.

The self-experiment was started on 7 December 2015 and concluded three months later on 7 March 2016.

4. Results and implementation

Whereas the RG score is influenced by a number of factors such as the impact of your own academic publications, social activities, and evaluation by other members of the social network, this self-experiment was designed to examine the extent to which you can increase your own RG score without the influence of publications. The weekly development of the RG score is shown in Fig. 4 and additionally a summary of weekly activities in Table 1.

At the beginning of the self-experiment, we first answered some questions and posed some questions ourselves until the RG score was activated. We restricted ourselves to questions and answers from our own areas such as bibliometrics, scientometrics, bibliographic databases etc., which were entered in the newly created profile under “Skills and Expertise”. The RG score was first activated on 21 December 2015, i.e. two weeks after the start of the self-experiment, and at this point in time amounted to 1.69. To achieve this, we had to answer a total of 41 questions and post three of our own. At this point, our social network already included five followers and the RG score was already higher than 5% of all ResearchGate users. It was also interesting to see that according to ResearchGate 82% of the score was based on the answers and 18% on the questions. The three questions only accounted for ∼7% of the total previous activity and these questions were moreover only answered a total of seven times.

In the following week, the activity was reduced to just eight further answers and the RG score reached 2.78, which was already higher than 10% of the entire ResearchGate community. No questions were posed this time, but the three existing questions received 13 additional responses. In spite of this, the composition of the score was shifted in a positive direction since these questions now accounted for 23% of the score and the answers only 77%. In order to investigate what changes occurred in the case of no activity, we did not use ResearchGate the following week and did not receive any further answers to the questions we had posed. Nevertheless, the score still increased slightly by 0.01 to 2.79 and the distribution was shifted by another percentage point in favour of the questions (now accounting for 24% in comparison to 76% for the answers). However, the further course of the experiment showed that inactivity can also have a different effect on the RG score.

It also became subsequently apparent that questions seem to have more influence on the RG score than answers. Up to 11 January 2016, only two additional questions were posed and nine answers given (a total of five questions and 58 answers). Nevertheless, the previous composition of the RG score was unchanged, consisting of 24% for the questions and 76% for the answers.

One single downgrading of the RG score was observed during an update on 1 February 2016. In the previous week, however, activity had not been reduced but rather intensified, with a total of four questions and eleven answers. Nevertheless, the score was reduced from 4.19 to 4.11. The following increase to 5.11 does not reflect this downgrading in any way since the activity in this week was reduced in comparison to the previous week by only answering six additional questions.

Up to this point in time, the social activity took place in our own research fields so that we posed relevant academic questions and gave the most competent and helpful answers possible. From 8 February, a different approach was applied in order to increase the RG score. First of all, we began to selectively increase the number of persons whose updates we intended to follow (that is to say, we became a follower of these users). Until that point, we had not been a follower of any other user, but we had a total of eleven followers, who had opted to follow us as a result of our previous social activities and with whom we had no personal or professional contact. We now began to selectively follow users with the highest possible RG scores. Our aim was to participate in discussions in which these persons were also involved in the hope that these persons would positively evaluate our own contributions. In order to increase the likelihood of this occurring, the persons had to fulfil the criterion of being relatively active on ResearchGate themselves as well as having a high RG score. Furthermore, this increased the probability that these persons could also become our followers, either due to the quality of our participation in discussions in which they also participated or in the hope that these persons would automatically “follow us back” so that in future our own questions and answers would appear on their homepage. A total of six users with RG scores between ∼94 and ∼211 were selected for this purpose who had each previously left between 920 and more than 9,000 answers on ResearchGate. In fact, all six users soon became followers and we were thus able to increase the probability that these persons would in future take part in discussions in which we also participated. In this way, our own visibility on the social network increased enormously since these users and their total of 8,600 followers represented a huge number of potential participants (see Fig. 1). For example, if all six of these users were to answer a question this would become visible to all their followers since they would all receive a message on their respective homepages.

Now that our own network had been selectively expanded, we began to take part in discussions in which persons with high RG scores also participated. By 22 February 2016, we answered another 16 questions in the hope that those persons who we had recently started to selectively follow would positively label our answers. Furthermore, the probability that the questions and thus also the answers would be seen by a great many users was considerably greater than answering questions selected at random due to these academics’ high number of followers. Indeed, this method enabled us to obtain significantly more upvotes than previously – nine in comparison to a maximum of four. This also gave rise to the desired effect that the users being followed also labelled our contributions as positive ² . We therefore succeeded in increasing our RG score within two weeks from 5.11 to 6.45 and additionally increasing our visibility by another ten followers.

Although the effort invested from 8 to 22 February was relatively low with on average just over one answer per day, we then decided to test how the RG score could be increased if just one question were posed. The general question: “Who or what inspired you to join your scientific discipline or science in general?” was intended to address all users and not just a certain group of academics. In addition, we also made use of the function of sharing our own question by sending the link to this question to the previously selected six active users in a private message in order to increase the probability that they would leave an answer to this question. Two of these six users took part in the discussion and thus considerably expanded our visibility with their total of ∼2,750 followers. The seven questions posed to date received a total of 39 answers. Only one question was labelled positively on one occasion and the question that was looked up most frequently was viewed 163 times in the three months after it was posted. Due to the improved conditions, the most recent question with 170 reads, 35 answers, and 25 upvotes within just one week became the most successful question we posed. Until the next update on 29 February 2016, our RG score increased by 0.64 points to 7.09 and we gained five new followers. In the subsequent two weeks, we did not undertake any further activities and in this period the question was only answered one more time. Nevertheless, it received a total of 294 reads. The RG score continued to rise to 7.35 on 7 March 2016 and finally to 7.49 on 14 March 2016. After the question had been posed, the breakdown of the RG score was shifted in favour of the questions and after the final update it was found that 69% was due to the influence of the answers and 31% to the influence of the questions (before the question was posed the ratio was 77% to 23%). Altogether, it was possible to increase the RG score within three weeks with the aid of one question by 1.04, so that this score exceeded 35% of the scores of all users, whereas before the question was posed the figure was 30% of all scores.

The self-experiment was thus concluded with respect to the influence of Q&A activity on ResearchGate on one’s own RG score. On this basis, we then proceeded to examine how easy it would be to additionally improve our own score by assigning to ourselves publications written by other authors.

ResearchGate provides one relatively simple opportunity of doing this. If you have not deposited any publications in your ResearchGate profile, ResearchGate asks you whether you are the author of publications that can be found on ResearchGate but which could not be assigned to any active profile. No distinction is made as to whether the content of the publications fits your profile but rather whether these are publications by authors whose name matches your own. ResearchGate seems to use an algorithm to match a large number of publications with an author of the same name to particular authors. You are asked whether you are one of the authors and whether you would like to confirm authorship of the relevant publications (see Fig. 2).

Since one of the titles of the publications is always presented for orientation purposes, it is relatively easy to use citation databases to determine how frequently this article has been cited so that you can roughly estimate what the influence on your RG score will be. To this end, we scrolled through the ResearchGate suggestions (see Fig. 2) and searched for the relative impact of the proposed publications in the Web of Science. A publication entitled “LAN emulation on an ATM network” [22], which was co-authored by an Andreas Meier, was found to be relatively frequently cited (30 citations). Furthermore, ResearchGate had identified a second publication for this author, although it was not possible to view this proposal since only one title was given for orientation purposes. Nevertheless, it was possible to specify that both publications were written by the author of the test profile and they were promptly added to our profile without any further evidence being required. The second publication entitled “Connectionless data service in an ATM-based customer premises network” [8] contributed a further 17 citations, so that we arrived at 47 citations with just two publications by other authors. The next update on 21 March 2016 brought the greatest increase so far in the RG score amounting to a plus of 4.11 points from 7.49 to 11.60. The profile also received an h-index of 2 and the RG score was now higher than 47.5% of all users, so that we had managed to outperform almost half of the users (see Fig. 3).

5. Discussion of results

If scientometric indicators (irrespective of whether they are indicators based on bibliometric or altmetric data) are to be increasingly used by science management for purposes of science policy then it is extremely important that those evaluated and those making the evaluation can readily obtain information about the content, breakdown, and informative value of such metrics. This is the context in which the results of the present study should be discussed and interpreted.

ResearchGate is one of the largest and most frequently used academic social media platforms. The platform is used by individuals to create and publish a publications profile and also for communication purposes. Both activities lead to a corresponding change in the RG score and are thus components of a combined reward system. As the study showed, the communication component alone can be used to achieve an RG score within a few weeks – without any publications at all – equal to that of other academics in the network with 30 to 40 publications – without using the communication components ³ . This result corresponds with the results of [12] that social activity is crucial for achieving a high RG score. The metric is claimed to “measure scientific reputation based on how all of your research is received by your peers” [16]. It seems impossible to discern the ”scientific reputation” with the help of the RG score as the results show. The interactions are not evaluated content-related, can be about plain non-scientific topics and still have a weighty effect on the score. If we compare the effort required for communication with the effort that would have to be invested in about 40 publications to ultimately achieve the same score then it is apparent that it is much easier to increase one’s score by using the communications components than by authoring publications. This naturally raises the question as to the proportionality of cost and benefit. However, if one approaches the question the other way round and asks why ResearchGate weights its communications component so excessively then one may conclude that this is a deliberate device to make the platform more attractive to users. In order to reach a fair assessment, the two parts of the platform must be evaluated separately since the comparability of publications, citations, and reads on the one hand and social attractiveness on the other must in general be regarded as low. Kraker & Lex [7] argue that “the RG score has important deficiencies that seem to prevent it from being used as a scientific reputation measure”. This reveals a recurrent problem of altmetrics: How should the weighting of evaluations of data from different sources be handled if they are to be combined into one overall score? And also the question: Does one overall score make sense? With respect to this second question, we can refer to the current state of the discussion of the altmetrics literature and the present state of the art at conferences. If it is not yet quite clear what altmetrics actually measure, how can a sensible overall score be developed which fulfils the requirements of the Leiden Manifesto? In this context, it is very important to consider the transparency and clarity of the metrics and the data on which they are based. This highlights one of the great differences to bibliometrics. Whereas citation databases such as Web of Science or Scopus consist solely of the same type of data (published literature with footnotes), altmetrics involve a number of different sources, which can in part be influenced by the users themselves. The question “How much is a blog entry worth compared to a tweet?” is very difficult to answer. And another point should not be neglected. From bibliometrics it is well known that different methods of publication and communication are favoured in the different disciplines. This naturally also applies to altmetrics. However, as yet no altmetrics platform provides discipline-specific benchmarks. This is also the case with ResearchGate, which is not represented in any altmetrics portal due to its company policy of not providing any information with respect to its data. This also shows that the familiar questions from bibliometrics and information science concerning completeness, precision, and recall of data infrastructures are just as relevant as ever.

6. Summary and outlook

This self-experiment shows how rapidly and relatively easily the RG score can be manipulated. Whereas at first we had to feel our way towards identifying which factors could have the greatest influence on the RG score, ultimately we developed a type of strategy to specifically inflate our RG score. Asking questions proved to be much more effective than answering them. In our self-experiment, we found that the former had a demonstrably greater influence on the breakdown of the RG score and involves much less effort. Furthermore, we developed a simple but effective strategy to attract as many followers as possible so that our own visibility was enormously magnified within the social network and we were thus able to entice more participants to our specially posed questions. ResearchGate additionally offers the opportunity of sharing questions with your followers, which also increases the probability that they would take part in our discussions and thus enhance our RG score. Another problem is the lack of checks on the real authorship of academic articles. In our self-experiment, there was no difficulty in declaring publications by other authors to be our own, which also had a massive impact on our RG score. The simple method of manipulating the number of reads without any great effort could also have been used for the purpose of increasing the RG score. However, this approach had already been identified by [2] and was therefore not part of the self-experiment applied in the present study. Altogether, it can be said that there is a wide range of possibilities for manipulating the RG score and a combination of these options can rapidly accelerate the increase.

The self-experiment implemented here on the RG score, which can be regarded as a pilot study, could be taken up and expanded in further research work. For example, it could be investigated in more detail which factors influence the RG score in what form by, for example, creating several profiles and having each of these profiles undertake just one type of activity.