Journal assessment is of great significance to promote the development of academic platforms. Citation analysis is a recognised tool to assess the citation performance of journals. However, due to the shortcomings such as the inflated Journal Impact Factor and heterogeneity of the citations, additional dimensions are necessary to be considered to balance with the article citation. This article constructs a three-dimensional journal assessment framework to measure the comprehensive influence of a journal based on article citation, author and institution influence. The CRITIC-Entropy weighting method is employed to calculate the weighted average scores for the three dimensions, respectively. Then, a newly defined Pareto-dominated set-based sum of TOPSIS score (PDS-based STS) approach is developed to assess the comprehensive influence of journals. A sample of 76 journals in Economics field is selected to demonstrate the effectiveness and validity of the proposed assessment method. The Chartered Association of Business Schools’ Academic Journal Guide 2021 (CABS-AJG 2021) which is an expert-based journal rating is chosen as a baseline model. It has been found that the assessment method using PDS-based STS shows a more rational journal ranking than that based on the Journal Impact Factor if using the CABS-AJG 2021’s ratings as the benchmark model.
CaiLTianJLiuJ, et al. Scholarly impact assessment: a survey of citation weighting solutions. Scientometrics2019; 118(2): 453–478.
2.
FengLZhouJLiuSL, et al. Analysis of journal evaluation indicators: an experimental study based on unsupervised Laplacian score. Scientometrics2020; 124(1): 233–254.
3.
HalimZKhanS. A data science-based framework to categorize academic journals. Scientometrics2019; 119(1): 393–423.
4.
ThelwallM. Are classic references cited first? an analysis of citation order within article sections. Scientometrics2019; 120(2): 723–731.
5.
YuDWangWZhangS, et al. A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals. Scientometrics2017; 111(1): 521–542.
6.
CurrieRRPandherGS. Finance journal rankings and tiers: an active scholar assessment methodology. J Bank Financ2011; 35(1): 7–20.
7.
XuNChanKCChangCH. A quality-based global assessment of financial research. Rev Quant Finance Account2016; 46(3): 605–631.
8.
GarfieldE. Citation analysis as a tool in journal evaluation: journals can be ranked by frequency and impact of citations for science policy studies. Science1972; 178(4060): 471–479.
9.
KleijnenJPCVan GroenendaalW. Measuring the quality of publications: new methodology and case study. Inf Process Manag2000; 36(4): 551–570.
10.
WernerR. The focus on bibliometrics makes papers less useful. Nature2015; 517(7534): 245–245.
11.
IoannidisJPAThombsBD. A user’s guide to inflated and manipulated impact factors. Eur J Clin Invest2019; 49(9): e13151.
12.
PostmaE. Inflated impact factors? the true impact of evolutionary papers in non-evolutionary journals. PLoS ONE2007; 2(10): e999.
13.
WilsonAE. Journal impact factors are inflated. Bioscience2007; 57(7): 550–551.
14.
ZengZShiL. A two-dimensional journal classification method based on output and input factors: perspectives from citation and authorship related indicators. Scientometrics2021; 126(5): 3929–3964.
15.
DingYCroninB. Popular and/or prestigious? measures of scholarly esteem. Inf Process Manag2011; 47(1): 80–96.
16.
MartinBR. The use of multiple indicators in the assessment of basic research. Scientometrics1996; 36(3): 343–362.
17.
MoedHFHaleviG. Multidimensional assessment of scholarly research impact. J Assoc Inf Sci Technol2015; 66(10): 1988–2002.
18.
YangSXingXWolframD. Difference in the impact of open-access papers published by China and the USA. Scientometrics2018; 115(2): 1017–1037.
19.
YuLShenXPanY, et al. Scholarly journal evaluation based on panel data analysis. J Informetr2009; 3(4): 312–320.
20.
HausteinS. Multidimensional journal evaluation: analyzing scientific periodicals beyond the impact factor. 1st ed. Berlin/Boston, MA: De Gruyter Saur, 2012.
21.
FryTDDonohueJMSaladinBA, et al. The origins of research and patterns of authorship in the International Journal of Production Research. Int J Prod Res2013; 51(23–24): 7470–7500.
22.
SanfilippoPHewittAWMackeyDA. Plurality in multi-disciplinary research: multiple institutional affiliations are associated with increased citations. PeerJ2018; 6: e5664.
GormanMFKanetJJ. Evaluating operations management – related journals via the author affiliation index. Manuf Serv Oper Manag2005; 7(1): 3–19.
25.
AhlgrenPWaltmanL. The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments. J Informetr2014; 8(4): 985–996.
26.
SaarelaMKarkkainenTLahtonenT, et al. Expert-based versus citation-based ranking of scholarly and scientific publication channels. J Informetr2016; 10(3): 693–718.
27.
LiuXZFangH. A comparison among citation-based journal indicators and their relative changes with time. J Informetr2020; 14(1): 101007.
28.
NeuhausCMarxWDanielHD. The publication and citation impact profiles of Angewandte Chemie and the journal of the American chemical society based on the sections of chemical abstracts: a case study on the limitations of the Journal Impact Factor. J Am Soc Inf Sci2009; 60(1): 176–183.
29.
YuTYuGWangMY. Classification method for detecting coercive self-citation in journals. J Informetr2014; 8(1): 123–135.
30.
ReedijkJMoedHF. Is the impact of journal impact factors decreasing?J Doc2008; 64(2): 183–192.
31.
WaltersWH. Citation-based journal rankings: key questions, metrics, and data sources. IEEE Access2017; 5: 22036–22053.
HarlessDReillyR. Revision of the journal list for doctoral designation. Unpublished report, Virginia Commonwealth University, Richmond, VA, 25June2019.
HirschJE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A2005; 102(46): 16569–16572.
36.
SchreiberM. An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the A-index, and the R-index. J Am Soc Inf Sci Tec2008; 59(9): 1513–1522.
37.
BornmannLDanielHD. The state of h index research is the h index the ideal way to measure research performance?Embo Rep2009; 10(1): 2–6.
38.
OppenheimC. Using the h-index to rank influential British researchers in information science and librarianship. J Am Soc Inf Sci Tec2007; 58(2): 297–301.
39.
SaraykarSSalehASelekS. The association between NIMH funding and h-index in psychiatry. Acad Psychiatry2017; 41(4): 455–459.
40.
SviderPFHusainQFolbeAJ, et al. Assessing National Institutes of Health funding and scholarly impact in neurological surgery. J Neurosurg2014; 120(1): 191–196.
41.
BallP. Index aims for fair ranking of scientists. Nature2005; 436(7053): 900–900.
42.
HarzingAWvan der WalR. A Google Scholar h-index for journals: an alternative metric to measure journal impact in economics and business. J Am Soc Inf Sci2009; 60(1): 41–46.
43.
LapidowAScudderP. Shared first authorship. J Med Libr Assoc2019; 107(4): 618–620.
44.
HagenNT. Harmonic publication and citation counting: sharing authorship credit equitably – not equally, geometrically or arithmetically. Scientometrics2009; 84(3): 785–793.
45.
AgrawalVKAgrawalVRungtusanathamM. Theoretical and interpretation challenges to using the author affiliation index method to rank journals. Prod Oper Manag2011; 20(2): 280–300.
46.
Abdul-MajeedGMahmoodWNamerNSM. Measuring research performance of Iraqi universities using Scopus data. Scientometrics2021; 126(3): 2349–2363.
47.
ShannonCEWeaverW. The mathematical theory of communication. 10th ed.Urbana, IL: The University of Illinois Press, 1964.
48.
RenZ. Evaluation method of port enterprise product quality based on entropy weight TOPSIS. J Coast Res2020; 103(sp1): 766–769.
49.
DiakoulakiDMavrotasGPapayannakisL. Determining objective weights in multiple criteria problems – the CRITIC method. Comput Oper Res1995; 22(7): 763–770.
50.
MingersJLipitakisEAECG. Counting the citations: a comparison of Web of Science and Google Scholar in the field of business and management. Scientometrics2010; 85(2): 613–625.
51.
AmaraNLandryR. Counting citations in the field of business and management: why use Google Scholar rather than the Web of Science. Scientometrics2012; 93(3): 553–581.
52.
BartneckCKokkelmansS. Detecting h-index manipulation through self-citation analysis. Scientometrics2011; 87(1): 85–98.
53.
HarzingAWAlakangasS. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics2016; 106(2): 787–804.
54.
SaarelaMKaerkkaeinenT. Can we automate expert-based journal rankings? analysis of the Finnish publication indicator. J Informetr2020; 14(2): 101008.
55.
WaltersWHMarkgrenS. Do faculty journal selections correspond to objective indicators of citation impact? results for 20 academic departments at Manhattan College. Scientometrics2019; 118(1): 321–337.
56.
DaudAAhmadMMalikMSI, et al. Using machine learning techniques for rising star prediction in co-author network. Scientometrics2015; 102(2): 1687–1711.
