The aim of this article is to review the potential of Science and Technology (S&T) Indicators for the evaluation of research programs. We suggest that indicators are a useful complement to other evaluation methodologies (surveys, case studies, panels) for summative evaluations, where the focus is on measuring program results and the degree of achievement of program objectives. However, we argue that indicators have a much broader potential to support formative evaluation, where the focus is on learning from past experience in order to design future research programs. Indicators, for example, are a valuable support for debates between social actors concerning strategic choices about research funding. We conclude by suggesting what needs to be done to realize this potential.
AbmaT (2006) The practice and politics of responsive evaluation. American Journal of Evaluation27(1): 31–43.
2.
ArnoldE (2004) Evaluating research and innovation policy: a systems world needs systems evaluation. Research Evaluation13(1): 3–17.
3.
ArnoldEClarkJMuscioA (2005) What the evaluation record tells us about European Union Framework Programme performance. Science and Public Policy32(5): 385–97.
4.
AVEDAS, CWTS, FAS. Research and Stockholm School of Economics (2009) Structuring Effects of Community Research – The Impact of the Framework Programme on Research and Technological Development (RTD) on Network Formation. Brussels, Report for the European Commission, Directorate-General for Research.
5.
BarréR (2001) Sense and nonsense of S&T productivity indicators. Science and Public Policy28(4): 259–66.
6.
BarréR (2004) S&T indicators for policy making in a changing science-society relationship. In: MoedHFGlänzelWSchmochU (eds) Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publishers, 115–32.
7.
BlochC (2007) Assessing recent developments in innovation measurement: the third edition of the Oslo Manual. Science and Public Policy34(1): 23–34.
8.
BonaccorsiADaraioCLeporiBSlipersaeterS (2007) Indicators on individual higher education institutions: addressing data problems and comparability issues. Research Evaluation16(2): 66–78.
9.
BraamRVan den BesselaarP (2010) Lyfe cycles of research groups: the case of CWTS. Research Evaluation19(3): 173–84.
10.
BraunD (2003) Lasting tensions in research policy-making – a delegation problem. Science and Public Policy30(5): 309–21.
11.
BraunDGustonD (2003) Principal-agent theory and research policy: an introduction. Science and Public Policy30(5): 302–8.
12.
BreschiSLissoniF (2004) Knowledge networks from patent data. In: MoedHFGlänzelWSchmochU (eds) Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer, 613–44.
13.
CampbellDFJ (2003) The evaluation of university research in the United Kingdom and The Netherlands, Germany and Austria. In: ShapiraPKuhlmannS (eds) Learning from Science and Technology Policy Evaluation. Cheltenham: Edward Elgar, 98–131.
14.
CozzensSE (1997) The knowledge pool: measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning20(1): 77–89.
15.
DelangheHMuldurU (2007) Ex-ante impact assessment of research programmes: the experience of the European Union’s 7th Framework Programme. Science and Public Policy34(3): 169–83.
16.
DesrosièresA (2008) Gouverner par les nombres. Paris: Presses de l’école des mines.
17.
European Commission (2011) Innovation Union Competitiveness Report. Brussels: European Commission.
18.
European Court of Auditors (2007) Evaluating the EU Research and Technological Development (RTD) Framework Programmes. Brussels: European Commission.
19.
FellerI (2007) Mapping the frontiers of evaluation of public-sector R&D programs. Science and Public Policy34(10): 681–90.
20.
GeorghiouLLarédoP (2006) Evaluation of publicly funded research: recent trends and perspectives. In: OECD (eds) OECD Science, Technology and Industry Outlook. Paris: OECD, 177–99.
21.
GlänzelWSchubertA (2005) Analysing scientific networks through co-authorship. In: MoedHFGlänzelWSchmochU (eds) Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publications, 257–76.
22.
GodinB (2005) Measurement and Statistics on Science and Technology. London: Routledge.
23.
GruppH (2000) Indicator-assisted evaluation of R&D programmes: possibilities, state of the art and case studies. Research Evaluation8(2): 87–99.
24.
GruppHSchmochUKuntzeU (1991) Patents as potential indicators of the utility of EC Research Programmes. Scientometrics21(3): 417–45.
25.
GubaEGLincolnYS (1989) Fourth Generation Evaluation. Newbury Park, CA: SAGE.
26.
GulbrandsenMSlipersaeterS (2007) The third mission and the entrepreneurial university model. In: BonaccorsiADaraioC (eds) Universities and Strategic Knowledge Creation. Specialization and Performance in Europe. Cheltenham: Edwar Elgar, 112–43.
27.
GustonDH (1999) Stabilizing the boundary between US politics and science: the role of the Office of Technology Transfer as a boundary organization. Social Studies of Science29(1): 87–111.
28.
GuyKNauwelaarsC (2003) Benchmarking STI Policies in Europe: In Search of Good Practice. Seville: Institute for Prospective Technological Studies.
29.
HEFCE (Higher Education Funding Council for England) (2007) Research Excellence Framework. Consultation on the Assessment and Funding of Higher Education Research Post 2009. London: HEFCE.
30.
Heller-SchuhBBarberMHenriquesLPaierMPontikakisDScherngellTVeltriGAWeberM (2011) Analysis of Networks in European Framework Programmes (1984-2006). Luxembourg: Publications Office of the European Union.
31.
IPTS (2002) RTD Evaluation Toolbox. Brussels: Institute for Prospective Technological Studies, EUR 20382N.
32.
JolyPBMangematinV (1996) Profile of public laboratories, industrial partnerships and organisation of R&D: the dynamics of industrial relationships in a large research organization. Research Policy25: 901–22.
33.
KlerkxLLeeuwisC (2008) Delegation of authority in research funding to networks: experience with a multiple goal boundary organization. Science and Public Policy35(3): 183–96.
34.
KuhlmannS (1998) Moderation of policy-making? Science and technology policy evaluation beyond impact measurement: the case of Germany. Evaluation4(2): 130–48.
35.
KuhlmannS (2003) Evaluation of research and innovation policies: a discussion of trends with examples from Germany. Journal of Technology Management26(2–4): 131–49.
36.
KuhlmannSBoekholtPGeorghiouLGuyKHéraudJLarédoPLemolaTLoveridgeDLuukkonenTPoltWRipASanz MenéndezLSmitsR (1999) Improving Distributed Intelligence in Complex Innovation Systems. Karlsruhe: Frauenhofer Institute – Systems and Innovation Research.
37.
LarédoPMustarP (2000) Laboratory activity profiles: an exploratory approach. Scientometrics47(3): 515–39.
38.
LatourBWoolgarS (1979) Laboratory Life. The Construction of Scientific Facts. Princeton, NJ: Princeton University Press.
39.
LeporiB (2011) Coordination modes in public funding systems. Research Policy40(3): 355–67.
40.
LeporiBBarréRFilliatreauG (2008) New perspectives and challenges for the design and production of S&T indicators. Research Evaluation17(1): 33–44.
41.
LeporiBDingesMRealeESlipersaeterSThevesJVan den BesselaarP (2007) Comparing the evolution of national research policies: what patterns of change?Science and Public Policy34(6): 372–88.
42.
LissoniFLlerenaPMcKelveyMBulatS (2008) Academic patenting in Europe: new evidence from the KEINS database. Research Evaluation17(2): 87–102.
43.
LuukkonenT (1998) The difficulties in assessing the impact of EU Framework Programs. Research Policy27(6): 599–610.
44.
MartinBRIrvineJ (1983) Assessing basic research: some partial indicators of scientific progress in radio astronomy. Research Policy12(2): 61–90.
45.
MerkxFVan den BesselaarP (2008) Positioning indicators for cross-disciplinary challenges: the Dutch coastal defense research case. Research Evaluation17(1): 4–16.
46.
MoedHFGlänzelWSchmochU (eds) (2004) Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publishers.
47.
MogoutovAKahaneB (2007) Data search strategy for science and technology emergence: a scalable and evolutionary query for nanotechnology tracking. Research Policy36(6): 893–903.
48.
Molas-GallartJ (2011) Tracing ‘productive interactions’ to identify social impacts: an example from the social sciences. Research Evaluation20(3): 219–26.
49.
Molas-GallartJDaviesA (2005) Toward theory-led evaluation: the experience of european science, technology and innovation policies. American Journal of Evaluation20(10): 1–18.
50.
NederhofAJ (2006) Bibliometric monitoring of research performance in the social sciences and the humanities: a review. Scientometrics66(1): 81–100.
51.
OECD (1995) The Measurement of Human Resources Devoted to S&T – Canberra Manual. Paris: OECD.
52.
OECD (2002) Frascati Manual. Proposed Standard Practice for Surveys on Research and Experimental Development. Paris: OECD.
53.
OECD (2005) Guidelines for Collecting and Interpreting Innovation Data — The Oslo Manual. Paris: OECD.
54.
OECD (2006) Government R&D Funding and Company Behaviour. Measuring Behavioural Additionality. Paris: OECD.
55.
PorterT (1995) Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton, NJ: Princeton University Press.
56.
Proneos, OST, NIFU-Step and CERIS-CNR (2009) Tools and Indicators for the Evaluation and Monitoring of European Research Framework Programmes. Brussels: European Commission.
57.
RealeEInzeltALeporiBVan den BesselaarP (2011) Indicators for the evaluation of the internationalization of Government Funding Agencies: results from an exploratory and participatory project at European level. ENID STI Indicator Conference, 7–9 SeptemberRome, Italy.
58.
RoessnerD (2000) Quantitative and qualitative methods and measures in the evaluation of research. Research Evaluation8(2): 125–32.
59.
RogersJJordanG (2011) Introduction to a special section on new approaches to predicting and tracing R&D program effects for policy learning. Research Evaluation20(4): 263–6.
60.
RueggRFellerI (2003) A Toolkit for Evaluating Public R&D Investment: Findings from ATP’s First Decade. NIST GCR 03-857.
61.
ScherngellTBarberM (2011) Distinct spatial characteristics of industrial and public research collaborations: evidence from the fifth EU Framework Programme. The Annals of Regional Science46(2): 247–66.
62.
SpaapenJVan DroogeL (2011) Introducing ‘productive interaction’ in social impact evaluation. Research Evaluation20(3): 211–18.
StameN (2004) Theory-based evaluation and types of complexity. Evaluation10(1): 58–76.
65.
Technopolis and OST (2009) Bibliometric Profiling of Framework Programme Participants. Brussels: EPEC.
66.
Technopolis and OST Paris (2004) Future Priorities for Community Research Based on Bibliometric Analysis of Publication Activity for the Five-Year Assessment (1999-2003) of Community Research Activities. Brussels: EPEC.
67.
ThoenigJ (2000) Evaluation as usable knowledge for public management reforms. Evaluation6(2): 217–29.
68.
TijssenR (2001) Global and domestic utilization of industrial relevant science: patent citation analysis of science–technology interactions and knowledge flows. Research Policy30(1): 35–54.
69.
TrochimWMMarcusSEMasseLCMoserRPWeldPC (2008) The evaluation of large research initiatives: a participatory integrative mixed methods approach. American Journal of Evaluation29: 8–28.
70.
Van RaanAFJ (2004) Measuring science. In: MoedHFGlänzelWSchmochU (eds) Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publishers, 19–50.
71.
WeissC (2004) Theory-based evaluation: past, present, future. New Directions for Evaluation76: 41–55.
72.
WhitleyRGlaserJ (2007) The Changing Governance of the Sciences: The Advent of Research Evaluation Systems. Dordrecht: Springer.
