Assessing the Quality of Universities: A Gedankenexperiment Derived from Creating Stakeholder Value

Abstract

World University Ranking (WUR) systems play a significant role in how universities are funded and whom they can attract as faculty and students. Yet, for the purpose of comparing universities as institutions of higher education, current systems are readily gamed, provide little guidance about what needs to be improved, and fail to allow for the diversity of stakeholder needs in making comparisons. We suggest a list of criteria that a WUR system should meet, and which none of the current popular systems appears satisfy. By using as a starting point the goal of creating value for the diverse and sometimes competing stakeholder requirements for a university, we suggest via a thought experiment a rating process that is consistent with all the criteria, and a way in which it might be trialled. Also, the resulting system itself adds value for individual users by allowing them to tune it to their own particular circumstances. However, an answer to the simple question ‘Which is the best university’ may well be: there is no simple answer.

Keywords

Bibliometrics performance measurement stakeholder value student satisfaction SV rating system world university ranking systems

Get full access to this article

View all access options for this article.

References

Adler

, Ewing

, & Taylor

(2009). Citation statistics (with discussion). Statistical Science , 24, 1–28. https://doi.org/10.1214/09-sts285

Anowar

, Helal

M. A.

, Afroj

, Sultana

, Sarker

, & Mamun

K. A.

(2015). A critical review on world university ranking in terms of top four ranking systems. In Elleithy

& Sobh

(Eds.), New trends in networking, computing, e-learning, systems sciences, and engineering, lecture notes in electrical engineering (Vol. 312, pp. 559–566). Springer. https://doi.org/10.1007/978-3-319-06764-3_72

Calderon

(2020). New ranking results show how some are gaming the system. University World News . https://www.universityworldnews.com/post.php?story=20200612104427336.

Espeland

W. N.

, & Sauder

(2007). Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology , 113(1), 1–40. https://doi.org/10.1086/517897.

Fauzi

M. A.

, Tan

C. N. L.

, Daud

, & Awalludin

M. M. N.

(2020). University rankings: A review of methodological flaws. Issues in Educational Research , 30(1), 79–96.

Fisher

N. I.

(2013). Analytics for leaders . A performance measurement system for business success . Cambridge University Press. https://doi.org/10.1017/cbo9781107053779

Fisher

N. I.

(2018). Stakeholder value as an organising principle for strategic planning, with application to a university department. Journal of Creating Value , 2(1), 1–10. https://doi.org/10.1177/2394964318771251.

Fisher

N. I.

(2019). A comprehensive approach to problems of performance measurement. Journal of the Royal Statistical Society Series A , 182(3), 755–803. https://doi.org/10.1111/rssa.12424

Fisher

N. I.

(2021). Performance measurement: Issues, approaches, and opportunities. Harvard Data Science Review , 3.4. https://doi.org/10.1162/99608f92.c28d2a68.

10.

Fisher

N. I.

, Lunn

P. D.

, & Sasse

S. M.

(2021). A comparison of possible approaches to continuously improving safety culture. Journal of Creating Value , 7(2), 1–23. https://doi.org/10.1177/23949643211044048.

11.

Goldratt

E. M.

(1990). The Haystack Syndrome. Sifting information out of the data ocean . Croton-on-Hudson Inc, North River Press.

12.

Hazelkorn

(2019). University rankings: There is room for error and ‘malpractice’ . Elephant in the Lab. https://doi.org/10.5281/zenodo.2592196

13.

Jacobs

, & Fisk

(2020). ARTU: The aggregate ranking of top universities. https://research.unsw.edu.au/artu/full-rankings.

14.

Johnes

(2018). University rankings: What do they really show? Scientometrics , 115, 585–606. https://doi-org/10.1007/s11192-018-2666-1.

15.

Kauppi

(2018). The global ranking game: Narrowing academic excellence through numerical objectification. Studies in Higher Education , 43(10), 1750–1762. https://doi.org/10.1080/03075079.2018.1520416

16.

Kordupleski

(2003). Mastering customer value management . Pinnaflex Educational Resources.

17.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology , 140, 1–55.

18.

Luque-Martínez

, & Faraoni

(2020). Meta-ranking to position world universities. Studies in Higher Education , 45(4), 819–833. https://doi.org/10.1080/03075079.2018.1564260.

19.

Mahat

(2017). Gerak appalled by enforced citation stacking in Universiti Malaya . https://www.malaysiakini.com/letters/394746.

20.

Moed

H. F.

(2017), A critical comparative analysis of five world university rankings. Scientometrics , 110, 967–990. https://doi-org/10.1007/s11192-016-2212-y.

21.

Pusser

, & Marginson

(2013). University rankings in critical perspective. The Journal of Higher Education , 84(4), 544–568. https://doi.org/10.1080/00221546.2013.11777301.

22.

Selten

, Neylon

, Huang

C. K.

, & Groth

(2020). A longitudinal analysis of university rankings. Quantitative Science Studies , 1(3), 1109–1135. https://doi.org/10.1162/qss_a_00052

23.

Soh

(2017). The seven deadly sins of world university ranking: A summary from several papers. Journal of Higher Education Policy and Management , 39(1), 104–115. https://doi.org/10.1080/1360080x.2016.1254431.

24.

Taylor

, & Braddock

(2007). International university ranking systems and the idea of university excellence. Journal of Higher Education Policy and Management , 29(3), 245–260. https://doi.org/10.1080/13600800701457855.

25.

Zhang

, Xiao

, Wu

, & Lu

Xin

(2020). Comprehensive world university ranking based on ranking aggregation. Computational Statistics , 36. https://doi.org/10.1007/s00180-020-01033-8