The most appropriate criteria and the best approach for evaluating centres concerned with research into health and health services, are matters of academic and policy debate. There is interest in assessing the impact made by research on policy and practice as well as using more traditional peer review of the knowledge produced. In this context the article describes the development of a multidimensional categorization of benefits, or payback, from research and development (R&D) and a model for conducting evaluations of impact. This categorization and model were used during an assessment of two R&D centres funded by a regional office of the National Health Service in the UK. The acceptability of such an approach is discussed in the light of relevant proposals from other authors, changes in the nature of knowledge production and the consequent role of multiple stakeholders.
Anderson, J. (1989) ‘New Approaches to Evaluation in UK Research Funding’, Concept Paper No. 9. London: Science Policy Support Group.
2.
Antman, E. , J. Lau, B. Kupelnick, F. Mosteller and T. Chalmers (1992) ‘A Comparison of Results of Meta-analyses of Randomised Control Trials and Recommendations of Clinical Experts’, Journal of American Medical Association268: 240–248.
3.
Bach, L. and L. Georghiou (1998) ‘The Nature and Scope of RTD Impact Measurement’, a discussion paper for the International Workshop on Measurement of RTD Results/Impacts, Brussels, 28–29 May.
4.
Backer, T. (1993) ‘Information Alchemy: Transforming Information Through Knowledge Utilization’, Journal of American Society for Information Science44: 217–221.
5.
Bryan, S. , M. Buxton, R. Sheldon and A. Grant (1998) ‘Magnetic Resonance Imaging for the Investigation of Knee Injuries: An Investigation of Preferences’, Health Economics7: 595–603.
6.
Buxton, M. (1994) ‘The Costs and Benefits of Heart Transplant Programmes at Harefield and Papworth Hospitals’, in M. Buxton, R. Elliott, S. Hanney, M. Henkel, J. Keen, M. Sculpher and P. Youll (eds) Assessing Payback from Department of Health Research and Development: Preliminary Report: Eight Case Studies, Research Report No. 19, Vol. 2. Uxbridge: HERG, Brunel University.
7.
Buxton, M. and S. Hanney (1994) ‘Assessing Payback from Department of Health Research and Development: Preliminary Report: Main Report’, Research Report No. 19, Vol. 1. Uxbridge: HERG, Brunel University.
8.
Buxton, M. and S. Hanney (1996) ‘How can Payback from Health Services Research be Assessed?’, Journal of Health Services Research & Policy1: 35–43.
9.
Buxton, M. and S. Hanney (1997) ‘Assessing Payback from Department of Health Research and Development: Second Report: Main Report’, Research Report No. 24, Vol. 1. Uxbridge: HERG, Brunel University.
10.
Buxton, M. , S. Hanney, T. Packwood, S. Roberts and P. Youll (1999) ‘Assessing the Benefits from North Thames Research and Development’, Research Report No. 25. Uxbridge: HERG, Brunel University.
11.
Cave, M. and S. Hanney (1990) ‘Performance Indicators for Higher Education and Research’, in M. Cave, M. Kogan and R. Smith (eds) Output and Performance Measurement in Government. London: Jessica Kingsley.
12.
Cave, M. and S. Hanney (1996) ‘Assessment of Research Impact on Non-Academic Audiences’, unpublished report to the ESRC.
13.
Cave, M. , D. Burningham, M. Buxton, S. Hanney, C. Pollitt, M. Scanlan and M. Shurmer (1994) The Valuation of Changes in Quality in the Public Services. London: HMSO.
14.
Cave, M. , S. Hanney, M. Henkel and M. Kogan (1997) The Use of Performance Indicators in Higher Education, 3rd edn. London: Jessica Kingsley.
15.
Chalmers, I. (1992) ‘Assessing the Effects of Health Technologies’, report from the Advisory Group on Health Technology Assessment chaired by Iain Chalmers. London: Department of Health.
16.
Commission of the European Communities (1991) Patents as Indicators of the Utility of European Community R&D Programmes. Luxembourg: Office for Official Publications of the European Community.
17.
Committee on Science, Engineering and Public Policy (1999) Evaluating Federal Research Programs. Washington, DC: National Academy Press.
18.
Department of Health (1992) Review of the Role of Department of Health Funded Research Units, The Williams Report. London: Department of Health.
19.
Department of Health (1993) A Centre for Research and Development in Primary Health Care: A Statement of Requirements. London: Department of Health.
Flynn, N. (1986) ‘Performance Measurement in Public Sector Services’, Policy and Politics14: 389–404.
22.
Gambardella, A. (1992) ‘Competitive Advantages from In-house Scientific Research: The US Pharmaceutical Industry in the 1980s’, Research Policy21: 391–407.
23.
Georghiou, L. (1995) ‘Assessing the Framework Programmes: A Meta Evaluation’, Evaluation1(2): 171–188.
24.
Gibbons, J. and J. Tunstill (1993) ‘Review of Good Dissemination Initiatives. Report to Department of Health’, Social Work Development Unit: University of East Anglia.. Mimeo
25.
Gibbons, M. and L. Georghiou (1987) Evaluation of Research – A Selection of Current Practices. Paris: OECD.
26.
Gibbons, M. , C. Limoges, J. Nowotny, S. Schwartzman, P. Scott and M. Trow (1994) The New Production of Knowledge. London: Sage.
27.
Godin, B. (1998) ‘Writing Performative History: The New New Atlantis?’, Social Studies of Science28(3): 465–483.
28.
Guba, E. and Y. Lincoln (1989) Fourth Generation Evaluation. Newbury Park, CA: Sage.
29.
Haas, P. (1992) ‘Introduction: Epistemic Communities and International Policy Co-ordination’, International Organization46: 1–36.
30.
Henkel, M. and M. Kogan (1981) The DHSS Funded Research Units: The Process of Review. Uxbridge: Department of Government, Brunel University.
31.
IMPEL (1993) Proposals for Intellectual Property Handling and Technology Transfer in the National Health Services. A report prepared for the NHS Executive. London: IMPEL.
32.
Klein, R. and N. Carter (1988) ‘Performance Measurement: A Review of Concepts and Issues’, in D. Beeton (ed.) Performance Measurement: Getting the Concepts Right. London: Public Finance Foundation.
33.
Knott, J. and A. Wildavsky (1980) ‘If Dissemination Is The Solution, What Is The Problem?’, Knowledge: Creation, Diffusion, Utilization1(4): 537–578.
34.
Kogan, M. and M. Henkel (1983) Government and Research: The Rothschild Experiment in a Government Department. London: Heinemann.
35.
Kogan, M. and A. Tuijnman (1995) Educational Research and Development. Paris: OECD.
36.
Koopmanschap, M. and F. Rutten (1993) ‘Indirect Costs in Economic Studies: Confronting the Confusion’, PharmacoEconomics4: 446–454.
37.
Kuhlmann, S. (1998) ‘Moderation of Policy-Making? Science and Technology Policy Evaluation Beyond Impact Measurement – The Case of Germany’, Evaluation4(2): 130–148.
38.
Mansfield, E. (1991) ‘Academic Research and Industrial Innovation’, Research Policy20: 1–12.
39.
Martin, B. and A. Salter (1996) ‘The Relationship Between Publicly Funded Basic Research and Economic Performance’, a Science Policy Research Unit Report for HM Treasury. Brighton: SPRU, University of Sussex.
40.
Mays, N. (1990) ‘Improving the Dissemination and Likelihood of Use of Research in the Social Medicine and Health Services Research Unit: General Issues and Proposals for Change’, unpublished report prepared for the then Chief Scientist at the Department of Health.
41.
Mushkin, S. (1979) Biomedical Research: Costs and Benefits. Cambridge, MA: Ballinger.
42.
National Institutes of Health (1993) Cost Savings Resulting from NIH Research Support, 2nd edn. Bethesda, MD: NIH.
43.
Office of Science and Technology and Programme for Policy Research in Engineering Science and Technology (1993) Returns to Research and Development Spending. London: HMSO.
44.
Office of Technology Assessment (1986) Research Funding as an Investment: Can We Measure the Returns?Washington, DC: Government Printing Office.
45.
Peckham, M. and R. Smith (eds) (1996) Scientific Basis of Health Services. London: BMJ Publishing Group.
46.
Platt, J. (1987) ‘Research Dissemination: A Case Study’, The Quarterly Journal of Social Affairs3: 181–198.
47.
Ratcliffe, J. and M. Buxton (1999) ‘Patient's Preferences Regarding the Process and Outcomes of High Technology Medicine: An Application of Conjoint Analysis to Liver Transplantation’, International Journal of Technology Assessment in Health Care5(2): 340–351.
48.
Review Advisory Committee (1993) Special Health Authorities: Research Review. London: HMSO.
49.
Robertson, D. (1995) ‘Ex-Post Evaluation of Selected Transport Research Projects’, Research Report No. 86. Crowthorne: Transport Research Laboratory.
50.
Rosenburg, N. (1990) ‘Why do Firms do Basic Research (With Their Own Money)?’, Research Policy19: 165–174.
51.
Smith, K. (1991) ‘Economic Returns to R&D: Methods, Results and Challenges’, Review Paper No. 3. London: Science Policy Support Group.
52.
Spaapen, J. and C. Sylvian (1994) Societal Quality of Research. London: Science Policy Support Group.
53.
Swann, P. (1996) ‘The Economic Value of Publicly Funded Basic Research: A Framework for Assessing the Evidence’, a report for DTI. Manchester: Policy Research in Engineering, Science and Technology, University of Manchester.
54.
Thomas, P. (1985) The Aims and Outcomes of Social Policy Research. London: Sage.
55.
Van der Meulen, B. and A. Rip (1995) ‘Assessing Societal Quality of Research in Environment Sciences’, in B. van der Meulen and A. Rip (eds) Indicatoren en indicaties voor de beoordeling vans maatschappelijke kwaliteit van onderzoek. Eindrapport. Rapport voor de commissie Overleg Sectorraden. Enschede: University of Twente.
56.
Weisbrod, B. (1983) Economics and Medical Research. Washington, DC: American Enterprise Institute.
57.
Weiss, C. (1979) ‘The Many Meanings of Research Utilization’, Public Administration Review39: 426–431.
58.
Weiss, C. (1980) ‘Knowledge Creep and Decision Accretion’, Knowledge: Creation, Diffusion, Utilization1: 381–404.
59.
Weiss, C. and M. Bucuvalas (1980) ‘Trust Tests and Utility Tests: Decision-Makers’ Frames of Reference for Social Science’, American Sociological Review45: 302–313.
60.
Whiston, T. (1990) The Evaluation of ESRC Research Centres. SPRU: University of Sussex. Mimeo.
