Sage Journals: Discover world-class research

Abstract

Evaluating academic papers and groups is important in scholar evaluation and literature retrieval. However, current evaluation indices, which pay excessive attention to the citation number rather than the citation importance and unidirectionality, are relatively simple. This study proposes new evaluation indices for papers and groups. First, an improved PageRank (PR) algorithm introducing citation importance is proposed to obtain a new citation-based paper index (CPI) via a pre-ranking and fine-tuning strategy. Second, to evaluate the paper’s influence inside and outside its research field, the focus citation-based paper index (FCPI) and diversity citation-based paper index (DCPI) are proposed based on topic similarity and diversity, respectively. Third, aside from the statistical indices for academic papers, we propose a foreign academic degree of dependence (FAD) to characterise the dependence between two academic groups. Finally, artificial intelligence (AI) papers from 2005 to 2019 are utilised for a case study.

Keywords

Group evaluation PageRank Paper evaluation

Get full access to this article

View all access options for this article.

References

Zhang

. Predicting future influence of papers, researchers, and venues in a dynamic academic network. J Inform 2020; 14(2): 101035.

Chu

Evans

. Slowed canonical progress in large fields of science. PNAS 2021; 118(41): e636118.

Schubert

. Using the h-index for assessing single publications. Scientometrics 2009; 78(3): 559–565.

Garfield

. The impact factor. Curr Content 1994; 25(20): 3–7.

Garfield

. Citation indexing for studying science. Nature 1970; 227: 669–671.

Garfield

. Citation indexes for science: a new dimension in documentation through association of ideas. Science 1955; 122: 108–111.

Siudem

Żogała-Siudem

Cena

et al. Three dimensions of scientific impact. PNAS 2020; 117(25): 13896–13900.

Singh

Shubhankar

Pudi

. An efficient ranking algorithm for scientific research papers. In: Proceedings of the 2011 3rd conference on data mining and optimization (DMO), Putrajaya, Malaysia, 28–29 June 2011, pp. 88–95. Piscataway, NJ: IEEE.

Gerrish

Blei

. A language-based approach to measuring scholarly impact. In: Proceedings of the ICML 2010, http://www.cs.columbia.edu/~blei/papers/GerrishBlei2010.pdf

10.

Jiang

Sun

Yang

et al. Exploiting heterogeneous scientific literature networks to combat ranking bias: evidence from the computational linguistics area. Journal of the Association for Information Science and Technology 2016; 67(7): 1679–1702.

11.

Wang

Xie

Zhang

et al. Coranking the future influence of multiobjects in bibliographic network through mutual reinforcement. ACM Trans Intell Syst Technol 2016; 7(4): 1–28.

12.

Bai

Liu

. PaperRank: a ranking model for scientific publications. In: Proceedings of the 2009 WRI world congress on computer science and information engineering, Angeles, CA, 31 March–2 April 2009, pp. 277–281. New York: IEEE.

13.

Page

Brin

Motwani

et al. The PageRank citation ranking: bringing order to the web. Technical Report, 1998, http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf

14.

Chen

Xie

Maslov

et al. Finding scientific gems with Google’s PageRank algorithm. J Inform 2007; 1(1): 8–15.

15.

Guan

Zhao

. Bringing PageRank to the citation analysis. Inform Process Manage 2008; 44(2): 800–810.

16.

Ding

Yan

Frazho

et al. PageRank for ranking authors in co-citation networks. J Am Soc Inform Sci Technol 2009; 60(11): 2229–2243.

17.

Yan

Ding

. Weighted citation: an indicator of an article’s prestige. J Am Soc Inform Sci Technol 2010; 61(8): 1635–1643.

18.

Wang

Tong

Zeng

. Ranking scientific articles by exploiting citations, authors, journals, and time information. AAAI 2013; 27(1): 933–939.

19.

Fiala

. Time-aware PageRank for bibliographic networks. J Inform 2012; 6(3): 370–388.

20.

Nykl

Ježek

Fiala

et al. PageRank variants in the evaluation of citation networks. J Inform 2014; 8(3): 683–692.

21.

Hirsch

. An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship. Scientometrics 2010; 85(3): 741–754.

22.

Waltman

Van Eck

. The inconsistency of the h-index. J Am Soc Inf Sci 2012; 63(2): 406–415.

23.

Leydesdorff

. The ‘academic trace’ of the performance matrix: a mathematical synthesis of the h-index and the integrated impact indicator (i3). J Assoc Inf Sci Technol 2014; 65(4): 742–750.

24.

Ding

Liu

Kandonga

. Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors. Scientometrics 2020; 122: 1303–1322.

25.

Min

Sun

Ding

. Quantifying the evolution of citation cascades. Proc Assoc Inform Sci Technol 2017; 54(1): 761–763.

26.

Gotelli

Chao

. Measuring and estimating species richness, species diversity, and biotic similarity from sampling data. Encycloped Biodiversity 2013; 5: 195–211.

27.

Garfield

. Citation analysis as a tool in journal evaluation. Science 1972; 178(4060): 471–479.

28.

Pajíc

. On the stability of citation-based journal rankings. J Inform 2015; 9(4): 990–1006.

29.

Moed

. Measuring contextual citation impact of scientific journals. J Inform 2010; 4(3): 265–277.

30.

González-Pereira

Guerrero-Bote

Moya-Anegón

. A new approach to the metric of journals’ scientific prestige: the SJR indicator. J Inform 2010; 4(3): 379–391.

31.

Meyérs

Quan

. The use of the h-index to evaluate and rank academic departments. J Mater Res Technol 2017; 6(4): 304–311.

32.

Ding

Zhang

. Understanding the impact change of a highly cited article: a content-based citation analysis. Scientometrics 2017; 112: 927–945.

33.

Ding

Liu

Guo

et al. The distribution of references across texts: some implications for citation. J Inform 2013; 7(3): 583–592.

34.

Chen

Liu

. Where are citations located in the body of scientific articles? A study of the distributions of citation locations. J Inform 2013; 7(4): 887–896.

35.

Devlin

Chang

Lee

et al. BERT: pre-training of deep bidirectional transformers for language understanding, https://aclanthology.org/N19-1423.pdf

36.

Blei

Jordan

. Latent Dirichlet allocation. J Mach Learn Res 2003; 3: 993–1022.

37.

Pennington

Socher

Manning

. GloVe: global vectors for word representation. Proc Conf Empir Meth Nat Lang Process 2014; 14: 1532–1543.

38.

Priem

Taraborelli

Groth

et al. Altmetrics: a manifesto, https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1187&context=scholcom

39.

Yuan

Gao

et al. LightLDA: big topic models on modest computer clusters. In: Proceedings of the 24th international conference on world wide web, http://davidwd.org/papers/lightlda.pdf

40.

Ding

et al. Interdisciplinary scholarly communication: an exploratory study for the fiel. Scientometrics 2019; 119: 1597–1619.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

4.53 MB

Improved PageRank and New Indices for Academic Impact Evaluation Using AI Papers as Case Studies

Abstract

Keywords

Get full access to this article

References

Supplementary Material