Recent reductions in research budgets have led to the need for greater selectivity in resource allocation. Measures of past performance are still among the most promising means of deciding between competing interests. Bibliometry, the mea surement of scientific publications and of their impact on the scientific community, assessed by the citations they attract, provides a portfolio of indicators that can be combined to give a useful picture of recent research activity. In this state-of-the- art review the various methodologies that have been developed are outlined in terms of their strengths, weaknesses and par ticular applications. The present limitations of science indica tors in research evaluation are considered and some future directions for developments in techniques are suggested.
Get full access to this article
View all access options for this article.
References
1.
S. Aaronson, The footnotes of science, Mosaic6 (2) (1975) 22-27.
2.
Abrc, Evaluation of National Performance in Basic Research: a review of techniques for evaluating national performance in basic research, with case studies in genetics and solid state physics, ABRC Science Policy Studies No. 1, Advisory Board for the Research Councils, London, 1986.
3.
A. Anderson , Research gradings stir emotions, Nature322 (1986) 299.
4.
R.C. Anderson , F. Narin and P. McAllister, Publication ratings versus peer ratings of universities, J. Amer. Soc. Info. Sci.29 (2) (1978) 91-103.
5.
S. Bauin, J.-P. Courtial and J. Law, Policy and the interpretation of co-word data: notes on the acid rain study , Paper prepared for CNRS/SPSG Seminar on Science Indicators and Policy, London, September 17-18, 1986.
6.
W. Broad, The publishing game: getting more for less, Science211 (1981) 1137-1139.
7.
M. Callon , J.-P. Courtial, W.A. Turner and S. Bauin, From translations to problematic networks: an introduction to co-word analysis, Social Science Information22 (2) ( 1983) 191-235.
8.
M. Callon, J. Law and A. Rip, eds., Mapping the Dynamics of Science and Technology (MacMillan , London, 1986).
9.
M.P. Carpenter and F. Narin, The adequacy of the Science Citation Index as an indicator of international scientific activity, J. Amer. Soc. Info. Sci.32 (1981) 430-439.
10.
K. Clayton , Measuring the cost of research, Times Higher Education Supplement (12th September 1986) 1.
11.
J.R. Cole and S. Cole, The Ortega Hypothesis, Science178 (1972) 368-375.
12.
S. Cole, L. Rubin and J.R. Cole, Peer review and the support of science, Scientific American237 (4) ( 1977) 34-41.
13.
S. Cole, J.R. Cole and G.A. Simon, Chance and consensus in peer review, Science214 (1981) 881-886.
14.
J.P. Courtial , M. Callon, S. Bauin and W.A. Turner, Co-Word Analysis, Paper presented at the CNRS/SPSG Seminar on Science Indicators and Policy, Science Policy Support Group, London, September 17-18, 1986.
15.
B. Cronin, The Citation Process (Taylor Graham, London, 1984).
16.
D. Crouch , J. Irvine and B.R. Martin, Bibliometric analysis for science policy: an evaluation of the United Kingdom's research performance in ocean currents and protein crystallography, Scientometrics9 (5-6) (1986) 239-267.
17.
D.J. De SollaPrice, Little Science, Big Science ( Columbia University Press, 1963).
18.
D. Edge, Quantitative measures of communication in science: a critical review, History of Science17 (1979) 102-134.
19.
G. Folly , B. Hajtman, J. Nagy and I. Ruff, Methodological problems in ranking scientists by citation analysis, Scientometrics3 (1981) 135-147.
20.
R. Fransson , Resource allocation based on evaluation of research , Internat. J. Inst. Manag. Higher Educ.9 (1985) 67-71.
21.
E. Garfield, Mapping the structure of science, in: Citation Indexing, Its Theory and Application in Science, Technology and the Humanities ( Wiley-Interscience, New York, 1979 ) 98-147.
22.
E. Garfield , Is citation analysis a legitimate evaluation tool?Scientometrics1 (4) (1979) 359-375.
23.
E. Garfield , Uses and misuses of citation frequency, Current Contents43 (1985) 403-409.
24.
H.H. Gee and F. Narin, An Analysis of Research Publications supported by NIH 1973-76 and 1977-80 , National Institutes of Health Evaluation Report, NIH, Bethesda, MA, 1986.
25.
M. Gordon , Evaluating the evaluators, New Scientist (10 February 1977) 342-343.
26.
D. Greenberg , Fraud and the scientific method, New Scientist (6 November 1986) 64.
27.
P. Hassanaly and H. Dou, Information systems and scientometric study in chemical oceanography, NATO ASI Series G9 (1986) 9-31.
28.
D. Hicks , B.R. Martin and J. Irvine, Bibliometric techniques for monitoring performance in strategic research: the case of integrated optics, R&D Management16 (3) (1986) 211-224.
29.
D. Hicks , Limitations of co-citation analysis as a tool for science policy, Social Studies of Science, to appear.
30.
R. Hooghiemstra , An Atlas of Science: sidestepping discipline limitations a great advantage of cluster analysis, Science Policy in the Netherlands7 (2) (1985) 12-14.
31.
J. Irvine and B.R. Martin, Assessing basic research: the case of the Isaac Newton Telescope, Social Studies of Science13 (1983) 49-86.
32.
J. Irvine and B.R. Martin, What directions for basic research? in: M. Gibbons, P. Gummett and B.M. Udgaonkar, eds., Science and Technology Policy in the 1980's and Beyond (Longman, London/New York, 1984) 67-98.
33.
J. Irvine , B.R. Martin, T. Peacock and R. Turner, Charting the decline in British science , Nature316 (1985 ) 587-90.
34.
J. Irvine and B.R. Martin, EvaluatingBig Science: CERN's past performance and future prospects, Scientometrics7 (1985) 281-308.
35.
L.V. Jones, The assessment of scholarship, New Directions for Program Evaluation6 (1980) 1-20.
36.
M.M. Kessler , Comparison of results of bibliographic coupling and analytic subject indexing, American Documentation16 (1965) 223-233.
37.
S.M. Lawani and A.E. Bayer, Validity of citation criteria for assessing the influence of scientific publications: new evidence with peer assessment, J. Amer. Soc. Info. Sci.34 (1983) 59-74.
D. Lindsey , Production and citation measures in the sociology of science: the problem of multiple authorship, Social Studies of Science10 (1980) 145-162.
40.
B.R. Martin and J. Irvine, Assessing basic research: some partial indicators of scientific progress in radio astronomy, Research Policy12 (1983) 61-90.
41.
B.R. Martin , J. Irvine and N. Minchin, An International Comparison of Government Funding of Academic and Academically Related Research, ABRC SciencePolicy Studies No. 2, Advisory Board for the Research Councils, London, 1986.
42.
P.R. McAllister , R.C. Anderson and F. Narin, Comparison of peer and citation assessment of the influence of scientific journals, J. Amer. Soc. Info. Sci.31 (1980) 147-152.
43.
P.R. McAllister and D.A. Wagner, Relationship between R&D expenditures and publication output for US colleges and universities, Research in Higher Education15 ( 1981) 3-29.
44.
P.R. McAllister and F. Narin, Characterization of the research papers of US medical schools, J. Amer. Soc. Info. Sci.34 (2) (1983) 123-131.
45.
R. McCulloch , International indicators of science and technology: how does the US compare?Scientometrics2 (1980) 355-367.
46.
D.H. McQueen and J.T. Wallmark, Innovation output and academic performance at Chalmers University of Technology, Omega Internat. J. Management Science12 (1984) 457-464.
47.
I.I. Mitroff and D.E. Chubin, Peer review at the NSF: a dialectical policy analysis, Social Studies of Science9 (1979) 199-232.
48.
H.W. Moed , W.J.M. Burger, J.G. Frankfort and A.F.J. VanRaan, The use of bibliometric data for the measurement of university research performance, Research Policy14 (1985) 131-149.
49.
H.F. Moed , W.J.M. Burger, J.G. Frankfort and A.F.J. VanRaan, The application of bibliometric indicators: important field- and time-dependent factors to be considered, Scientometrics8 (3-4) ( 1985) 177-203.
50.
P. Murugesan and M.J. Moravcsik, Variation of the nature of citation measures with journals and scientific specialities, J. Amer. Soc. Info. Sci.29 (1978) 141-147.
51.
F. Narin, Concordance between subjective and bibliometric indicators of the nature and quality of performed biomedical research, A Program Evaluation Report for the Office of Program Planning and Evaluation, NIH, National Institutes of Health, US Department of Health and Human Services, Bethesda, MA, 1983.
52.
E. Noma, Subject Classification and Influence Weights for 3000 Journals, Report prepared by Computer Horizons Inc. for the National Institutes of Health, USA and the Advisory Board for the Research Councils, UK, 1986.
53.
Nsf, Report of the NSF Advisory Committee on Merit Review, National Science Foundation, Washington, DC, 1986.
54.
OECD, The Measurement of Scientific and Technical Activities 'Frascati Manual' 1980, Organisation for Economic Cooperation and Development, Paris, 1981.
55.
OECD, OECD Science and Technology Indicators, Organisation for Economic Cooperation and Development, Paris, 1986.
56.
P.G. Pardey, PublicSector Production of Agricultural Knowledge, PhD Thesis, University of Minnesota, June 1986.
57.
A.L. Porter , Citation analysis: queries and caveats, Social Studies of Science7 (1977 ) 257-267.
58.
H. Rothman, ABRC Science Policy Study: further studies on the evaluation and measurement of scientific research, Report for the Economic and Science Research Council , Bristol Polytechnic, 1985.
59.
H. Rothman and G. Lester, The use of bibliometric indicators in the study of insecticide research, Scientometrics8 (3-4) (1985) 247-262.
60.
A. Schubert , S. Zsindely and T. Braun, Scientometric analysis of attendance at international scientific meetings, Scientometrics5 (1983) 177-187.
61.
Science Indicators; An Analysis of the State of US Science, Engineering and Technology, Reports of the National Science Board, National Science Foundation, Washington, DC, 1972, 1974, 1976, 1978, 1980, 1982.
62.
H. Small and E. Sweeney, Clustering the Science Citation Index using co-citations: I; A comparison of methods, Scientometrics7 (1985) 391-409.
63.
H. Small , E. Sweeney and E. Greenlee, Clustering the Science Citation Index using co-citations: II; Mapping Science, Scientometrics8 (1985) 321-340.
64.
L.G. Soete and S.M.E. Wyatt, The use of foreign patenting as an internationally comparable science and technology output indicator, Scientometrics5 (1983) 31-54.
65.
I. Spiegel-Rosing , Bibliometric and content analysis, Social Studies of Science7 (1977 ) 97-113.
66.
D. Sullivan , D.H. White and E.J. Barboni, The state of science: indicators in the speciality of Weak Interactions, Social Studies of Science7 (1977) 167-200.
67.
D. Sullivan , D.H. White and E.J. Barboni, Co-citation analyses of science: an evaluation , Social Studies of Science7 (1977) 223-240.
68.
P. Vinkler , Management system for a scientific research institute based on the assessment of scientific publications, Research Policy15 (1986) 77-87.
69.
G. Vladutz and J. Cook, Bibliographic coupling and subject relatedness, in: B. Flood, J. Witiak and T.H. Hogan, eds., 1984: Challenges to an Information Society, Proceedings of the 47th ASIS Annual Meeting (Knowledge Industry Publications, New York , 1984) 204-207.
70.
T. Von Waldkirch , Ten years of project-oriented allocation of resources at E.T.H. Zurich: review and evaluation of experiences, Internat. J. Institutional Management in Higher Education9 (1) (1985) 52-57.
71.
A. Weinberg, Criteria for scientific choice, Minerva1 (1963) 159-171.
72.
W.S. Wise, Agricultural R&D, Nature313 (1985) 8.
73.
J. Ziman, Criteria for national priorities in research, in: Rise and Fall of a Priority Field, Proceedings of the ESRC/NSF Symposium, Paris, 22-24 September, 1985 (European Science Foundation, Strasbourg, 1985) 95-125.
