The Research Assessment Exercises (RAE) conducted in the UK have attractedvarious types of published response. These include citation analyses and a review of the public reception of the RAE 1996, which included a brief critique of the citation studies.
This paper develops the critique. Largely unexplored issues in the theory or assumptions of bibliometrics, e.g. the level of citation which corresponds to a quantum of research quality, are found to emerge in the studies. A weak, and unsatisfactorily treated, correlation between citation aggregates and research quality for individual entities is revealed. The proposal to replace informed peer review by citation analysis is regarded as highly unrealistic.
Productive uses for citation analysis in research evaluation are suggested. A historically rare instance of correlation between rankings derived from citation aggregates and from real world peer review has been revealed by the studies. The future value of citation analysis could be to inform, but not to determine, judgements of research quality.
A combination of methods is advocated for future studies of the RAEs. Information science must attend to considerations of value, as well as using established techniques, if it is to avoid marginalisation.
Get full access to this article
View all access options for this article.
References
1.
[1] R.N. Kostoff , The Handbook of Research Impact Assessment (DTIC Report ADA296021, 7th ed., 1997). Available at: http://www.dtic.mil/dtic/kostoff/Handweb71.html
2.
[2] Department of Education Northern Ireland, Higher Education Funding Council for England, Higher Education Funding Council for Wales and Scottish Higher Education Funding Council, RAE 2001. Available at: http://www.rae.ac.uk/
3.
[3] G. Whittington , The 1996 Research Assessment Exercise , British Accounting Review29(2) (1997) 181-197 .
4.
[4] J. Elkin and D. Law , The 1996 Research Assessment Exercise: the library and information management panel , Journal of Librarianship and Information Science29(3) (1997) 131-141 .
5.
[5] L.B. Seng and P. Willett , The citedness of publications by United Kingdom library schools , Journal of Information Science21 (1995) 68-71 .
6.
[6] C. Oppenheim , The correlation between citation counts and the 1992 Research Assessment Exercises ratings for British library and information science departments , Journal of Documentation51(1) (1995) 18-27 .
7.
[7] C. Oppenheim , The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology , Journal of Documentation53(5) (1997) 477-487 .
8.
[8] The Impact of the 1992 Research Assessment Exercise on Higher Education Institutions in England (Report Ref: M 6/97) ( Higher Education Funding Council for England , 1997).
9.
[9] I. McNay , The Impact of the 1992 RAE on Institutional and Individual Behaviour in English Higher Education: The Evidence from a Research Project (Ref: M 6/97) ( Higher Education Funding Council for England, Bristol , 1997).
10.
Also in Information Research3(4) (1998), available at: http://www.shef.ac.uk/%7Eis/publications/infres/paper45.html
11.
[11] B. Latour , We Have Never Been Modern, translated by C. Porter ( Harvard University Press, Cambridge, MA , 1993).
12.
[12] N. Roberts , A search for information man , Social Science Information Studies2(2) (1982) 93-104 .
13.
[13] J. Warner , Retrieval performance tests in relation to online bibliographic searching . In: D. Shaw (ed.), ASIS ’92: Celebrating Change: Information Management on the Move (Proceedings of the 55th ASIS Annual Meeting, Pittsburgh, PA, 26-29 October 1992) (Learned Information for the American Society for Information Science, Medford, NJ, 1992).
14.
[14] M. Foucault , What is an author? In: P. Rabinow (ed.), The Foucault Reader ( Penguin Books , 1984).
15.
[15] L. Fleck , Genesis and Development of a Scientific Fact, translated by F. Bradley and T.J. Treen and edited by T.J. Treen and R.K. Merton ( Chicago University Press, Chicago, IL , 1979).
16.
[16] J. Warner , From Writing to Computers ( Routledge, London and New York , 1994).
17.
[17] S.C. Bradford , Documentation ( C. Lockwood, London , 1948).
18.
[18] S.C. Bradford , Romance of Roses ( London , 1946).
19.
[19] W. Weaver , Alice in Many Tongues: The Translations of Alice in Wonderland ( University of Wisconsin Press, Madison, WI , 1964).
20.
[20] W. Weaver and C.E. Shannon , The Mathematical Theory of Communication ( University of Illinois Press, Urbana, IL , 1949).
21.
[21] Y. Lindholm-Romantschuk and J. Warner , The role of monographs in scholarly communication: an empirical study of philosophy, sociology and economics , Journal of Documentation52(4) (1996) 389-404 .
22.
[22] E. Garfield , From citation indexes to infometrics: is the tail now wagging the dog?Libri48(2) (1998) 67-80 .
23.
[23] S. Hodges , B. Hodges , A.J. Meadows , M. Beaulieu and D. Law , The use of an algorithmic approach for the assessment of research quality , Scientometrics35(1) (1996) 3-13 . 1
24.
[24] V. Rosenberg , The scientific premises of information science , Journal of the American Society for Information Science25(4) (1974) 263-269 .
