Citation metrics and evaluation of journals and conferences

Abstract

Citation analysis aims at evaluating the published scientific manuscripts, their authors and the publication venues (journals/conferences). There are several popular metrics for measuring the impact of the journals, the Impact Factor (IF) being the most popular. Similarly, the $H$ -index is a popular metric for evaluating and ranking conferences. We have presented a review of metrics for citation analysis, categorised according to their applicability for evaluating journals and conferences. The citation metrics may also be categorised as popularity measuring and prestige measuring. Prestige measuring indicators like SCImago Journal Rank (SJR) and Eigenfactor have already gained popularity for evaluating journals. We discuss their role in evaluating the conferences. Indeed, some conferences have already started mentioning their prestige score in terms of the SJR of their conference proceedings. We also propose a Normalised Immediacy Index ( $I I_{norm}$ ), a variant of the Immediacy Index ( $II$ ), to measure the immediate relevance of articles published in a journal/conference. It is shown that the proposed metric can be used for immediacy relevance comparison irrespective of the publication schedule of the articles. Spearman correlation was run to determine the relationship between the values of the proposed $I I_{norm}$ and traditional metrics ( $H$ -index for conferences and IF for journals). A strong, positive monotonic correlation was observed between $I I_{norm}$ and H-index ( $r_{s}$ = .67, $n$ = 17, $p$ < .01) for conferences and between $I I_{norm}$ and IF ( $r_{s}$ = .65, $n$ = 20, $p$ < .01) for journals.

Keywords

Bibliometrics citation metrics conference journal publication ranking scientometrics venue evaluation

Get full access to this article

View all access options for this article.

References

Garfield

The agony and the ecstasy: the history and meaning of the journal impact factor. Int Congress Peer Rev Biomed, 2005, http://garfield.library.upenn.edu/papers/jifchicago2005.pdf

Hirsch

JE.

An index to quantify an individual’s scientific research output. Proc Nat Acad Sci USA 2005; 102(46): 16569–16572.

Bergstrom

Measuring the value and prestige of scholarly journals. College Res Librar News 2007; 68(5): 314–316.

Group

SR.

Description of SCImago journal rank indicator, 2008, https://www.scimagojr.com/SCImagoJournalRank.pdf

Agarwal

Durairajanayagam

Tatagari

et al. Bibliometrics: tracking research impact by selecting the appropriate metrics. Asian J Androl 2016; 18(2): 296.

Mingers

Yang

Evaluating journal quality: a review of journal citation indicators and ranking in business and management. Euro J Operat Res 2017; 257(1): 323–337.

Van Raan

. Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics 2005; 62(1): 133–143.

Moed

HF.

Citation analysis in research evaluation, vol. 9. Berlin: Springer Science and Business Media, 2006.

Moed

HF.

The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review. Sci Public Policy 2007; 34(8): 575–583.

10.

Adams

The use of bibliometrics to measure research quality in UK higher education institutions. Archivum Immunologiae et Therapiae Experimentalis 2009; 57(1): 19.

11.

Abramo

D’Angelo

CA.

Evaluating research: from informed peer review to bibliometrics. Scientometrics 2011; 87(3): 499–514.

12.

Wouters

Thelwall

Kousha

et al. The metric tide. Literature review Supplementary report I to the independent review of the role of metrics in research assessment and management, http://www.dcscience.net/2015_metrictideS1.pdf

13.

Nicolaisen

Citation analysis. Ann Rev Inform Sci Technol 2007; 41(1): 609–641.

14.

Bornmann

Daniel

HD.

What do citation counts measure? A review of studies on citing behavior. J Document 2008; 64(1): 45–80.

15.

Alonso

Cabrerizo

Herrera-Viedma

et al. H-index: a review focused in its variants, computation and standardization for different scientific fields. J Inform 2009; 3(4): 273–289.

16.

Panaretos

Malesios

Assessing scientific research performance and impact with single indices. Scientometrics 2009; 81(3): 635.

17.

Egghe

The Hirsch index and related impact measures. Ann Rev Inform Sci Tech 2010; 44(1): 65–114.

18.

Norris

Oppenheim

The h-index: a broad review of a new bibliometric indicator. J Document 2010; 66(5): 681–705.

19.

Vinkler

The evaluation of research by scientometric indicators. Amsterdam: Elsevier, 2010.

20.

Mingers

Leydesdorff

A review of theory and practice in scientometrics. Euro J Operat Res 2015; 246(1): 1–19.

21.

Wildgaard

Schneider

Larsen

A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics 2014; 101(1): 125–158.

22.

Cronin

Sugimoto

CR.

Beyond bibliometrics: harnessing multidimensional indicators of scholarly impact. Cambridge, MA: MIT Press2014.

23.

Kousha

Thelwall

Web indicators for research evaluation. Part 3: books and non-standard outputs. El Profesional de la Información 2015; 24(6): 724–726.

24.

Rijcke

Wouters

Rushforth

et al. Evaluation practices and effects of indicator use – a literature review. Res Evaluat 2016; 25(2): 161–169.

25.

Thelwall

Kousha

Web indicators for research evaluation. Part 1: citations and links to academic articles from the web. El Profesional de la Información 2015; 24(5): 587–606.

26.

Thelwall

Kousha

Web indicators for research evaluation. Part 2: social media metrics. El Profesional de la Información 2015; 24(5): 607–620.

27.

Waltman

A review of the literature on citation impact indicators. J Inform 2016; 10(2): 365–391.

28.

Haunschild

Beyond bibliometrics: harnessing multidimensional indicators of scholarly impact. J Scientometric Res 2015; 4(1): 40.

29.

O’Gara

. Review of ‘beyond bibliometrics: harnessing multidimensional indicators of scholarly impact’, edited by Blaise Cronin and Cassidy R. Sugimoto. Collect Manage 2015; 40(2): 114–116.

30.

Karanatsiou

Misirlis

Vlachopoulou

Bibliometrics and altmetrics literature review. Perform Measure Metric 2017; 18: 16–27.

31.

Aksnes

Langfeldt

Wouters

Citations, citation indicators, and research quality: an overview of basic concepts and theories. SAGE Open 2019; 9(1): 2158244019829575.

32.

Glänzel

Chi

PS.

The big challenge of scientometrics 2.0: exploring the broader impact of scientific research in public health. Scientometrics 2020; 125(2): 1011–1031.

33.

Jeyasekar

Saravanan

Innovations in measuring and evaluating scientific information. New York: IGI Global, 2018.

34.

Citeseer. http://citeseer.ist.psu.edu (accessed 27 February 2020).

35.

Dhammi

Haq

RUI

. What is indexing. Indian J Orthop 2016; 50(2): 115–115.

36.

Awesome Scholarly Data Analysis. https://shubhanshu.com/awesome-scholarly-data-analysis (accessed 9 January 2020).

37.

Web of science core collection: the citation report & the h-index, https://clarivate.libguides.com/woscc/citationreport (accessed 22 June 2020).

38.

Immediacy Index. http://adminapps.webofknowledge.com/JCR/help/h_immedindex.htm (accessed 22 April 2016).

39.

Scopus Journal Metrics. https://www.elsevier.com/solutions/scopus/how-scopus-works/metrics (accessed 13 September 2019).

40.

Using Research Indicators. https://libguides.jcu.edu.au/research-indicators/snip-and-sjr (Accessed 1 January 2022).

41.

Google Scholar Metrics. https://scholar.google.co.in/intl/en/scholar/metrics.html (accessed 13 September 2019).

42.

Delgado-López-Cózar

Cabezas-Clavijo

Á.

Ranking journals: could Google scholar metrics be an alternative to journal citation reports and SCImago Journal rank?

Learn Publish 2013; 26(2): 101–113.

43.

Hug

Brändle

MP.

The coverage of Microsoft academic: analyzing the publication output of a university. Scientometrics 2017; 113(3): 1551–1571.

44.

Conference ranks. http://www.conferenceranks.com/ (accessed 28 July 2018).

45.

Effendy

Yap

RHC

. Investigations on rating computer sciences conferences. In WWW – World Wide Web Consortium (W3C). URL https://www.microsoft.com/en-us/research/publication/investigations-on-rating-computer-sciences-conferences/

46.

Aminer. https://aminer.org/ (accessed 28 July 2018).

47.

Franceschet

The role of conference publications in cs. Commun ACM 2010; 53(12): 129–132.

48.

Core rankings portal. http://www.core.edu.au/conference-portal (accessed 28 July 2018).

49.

Australian Business Deans Council. https://abdc.edu.au/research/ (accessed 23 August 2019).

50.

Monastersky

The number that’s devouring science. Chronicle of Higher Education, 2005, https://www.chronicle.com/article/the-number-thats-devouring-science/

51.

Rossner

Van Epps

Hill

Show me the data. J Cell Biol 2007; 179(6): 1091–1092.

52.

McVeigh

Mann

SJ.

The journal impact factor denominator: defining citable (counted) items. JAMA 2009; 302(10): 1107–1109.

53.

Eigenfactor. http://www.eigenfactor.org/whyeigenfactor.php (accessed 17 April 2016).

54.

West

Althouse

Rosvall

et al. Eigenfactor score and article influence score: detailed methods, http://www.eigenfactor.org/methods.pdf

55.

Franceschet

The difference between popularity and prestige in the sciences and in the social sciences: a bibliometric analysis. J Inform 2010; 4(1): 55–63.

56.

Fersht

The most influential journals: impact Factor and Eigenfactor. Proc Nat Acad Sci 2009; 106(17): 6883–6884.

57.

Falagas

Kouranos

Arencibia-Jorge

et al. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 2008; 22(8): 2623–2628.

58.

Guerrero-Bote

Moya-Anegón

A further step forward in measuring journals’ scientific prestige: the SJR2 indicator. J Inform 2012; 6(4): 674–688.

59.

Caires

Again, the role of conference papers in computer science and informatics, 2015, http://ctp.di.fct.unl.pt/~lcaires/papers/conferencesCS.v4.pdf

60.

Freyne

Coyle

Smyth

et al. Relative status of journal and conference publications in computer science. Commun ACM 2010; 53(11): 124–132.

61.

Eckmann

Rocha

Wainer

Relationship between high-quality journals and conferences in computer vision. Scientometrics 2011; 90(2): 617–630.

62.

Rahm

Thor

Citation analysis of database publications. ACM Sigmod Record 2005; 34(4): 48–53.

63.

Chen

Konstan

JA.

Conference paper selectivity and impact. Commun ACM 2010; 53(6): 79–83.

64.

Bowyer

KW.

Mentoring advice on ‘conferences versus journals’ for CSE faculty. Notre Dame: University of Notre Dame, 2012.

65.

Davis

PM.

Eigenfactor: does the principle of repeated improvement result in better estimates than raw citation counts?

J Am Soc Inform Sci Technol 2008; 59(13): 2186–2188.

66.

Donner

Effect of publication month on citation impact. J Inform 2018; 12(1): 330–343.