The last five years have witnessed the increasing use of evaluated pilot programmes as a way of developing health services policy in the UK. Total purchasing was the first major quasi-market development in the National Health Service to be independently evaluated from the outset. The initiative allowed local, volunteer pilots considerable freedom to implement extensions to existing arrangements for general practitioner budget holding for specialist services in the National Health Service. The experience indicates that future evaluations of similarly complex innovations should give attention to developing explanatory frameworks (i.e. theories), which include consideration of the impact of the context in which interventions are introduced on their potential outcomes. Such an approach should help in ensuring the generalizability of evaluations through theory building and thus increase their relevance for policy development. In addition, evaluations must be designed to be capable of accommodating the needs of changing policy imperatives if they are to have long-term usefulness.
Berk, R. and P. Rossi (1990) Thinking about Program Evaluation.Newbury Park, CA: Sage.
2.
Booth, T. (1988) Developing Policy Research. Aldershot: Gower.
3.
Bowns, I. R. and T. McNulty (1999) Re-engineering Leicester Royal Infirmary: an Independent Evaluation of Implementation and Impact. Sheffield: School of Health and Related Research.
4.
Clarence, E. and C. Painter (1998) ‘Public Services under New Labour: Collaborative Discourses and Local Networking’, Public Policy and Administration13: 8–22.
5.
Connell, J. P., A. C. Kubisch, L. B. Schorr and C. H. Weiss (eds) (1995) New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts. Washington, DC: The Aspen Institute.
6.
Department of Health (1993) Research for Health. London: HMSO.
7.
Department of Health Research and Development Directorate (1995) National Evaluation of Total Purchasing Pilot Schemes: Research Brief. London: Department of Health, mimeo.
8.
Faddis, B. J. and P. Ahrens-Gray (2000) Evaluation of Head Start Family Child Care Demonstration.http://www2.acf.dhhs.gov/programs/hsb/core/dox/fcc_final.txt.
9.
Farrington, D. P. (1998) ‘Evaluating “Communities that Care”: Realistic Scientific Considerations’, Evaluation4: 204–210.
10.
Flynn, R. and G. Williams (1997) Contracting for Health: Quasi-markets and the National Health Service.Oxford: Oxford University Press.
11.
Fulbright-Anderson, K., A. C. Kubisch and J. P. Connell (eds) (1998) New Approaches to Evaluating Community Initiatives. Vol. 2: Theory, Measurement, and Analysis.Washington, DC: The Aspen Institute.
12.
Gask, L., J. Lee, S. Donnan and M. Roland (1998) Total Purchasing and Extended Fund-holding of Mental Health Services.London: King's Fund.
13.
Glass, N. (1999) ‘Sure Start: The Development of an Early Intervention Programme for Young Children in the United Kingdom’, Children and Society13: 257–264.
14.
Goodwin, N., S. Abbott, K. Baxter, D. Evans, A. Killoran, G. Malbon,N. Mays,J. Scott and S. Wyke (2000) The Dynamics of Primary Care Commissioning: a Close Up of Total Purchasing Pilots.London: King's Fund.
15.
Guba, Y. and E. Lincoln (1989) Fourth Generation Evaluation.London: Sage.
16.
Ham, C., D. Hunter and R. Robinson (1995) ‘Evidence Based Policy Making’, British Medical Journal310: 71–72.
17.
Harrison, S. and B. Wood (1999) ‘Designing Health Service Organization in the UK, 1968 to 1998: from Blueprint to Bright Idea and “Manipulated Emergence”’, Public Administration77: 751–768.
18.
Judge, K. (2000) ‘Testing Evaluation to the Limits: the Case of English Health Action Zones’, Journal of Health Service Research Policy5(1): 3–5.
19.
Judge, K., M. Barnes, L. Bauld, M. Benzeval, A. Killoran and R. Robinson (1999) Health Action Zones: Learning to Make a Difference.PSSRU Discussion Paper 1546. Canterbury: University of Kent at Canterbury.
20.
Julnes, G., M. M. Mark and G. T. Henry (1998) ‘Promoting Realism in Evaluation. Realistic Evaluation and the Broader Context’, Evaluation4(4): 483–504.
21.
Killoran, A., N. Mays, S. Wyke and G. Malbon (1999) Total Purchasing. A Step Towards Primary Care Groups.London: King's Fund.
22.
Klein, R. (1995) ‘Self-inventing Institutions: Institutionalised Design and the UK Welfare State’, in R. Goodin (ed.) The Theory of Institutional Design. Cambridge: Cambridge University Press.
23.
Klein, R. (2000) ‘From Evidence-based Medicine to Evidence-based Policy’, Journal of Health Service Research Policy5(2): 65–66.
24.
Labour Party (1995) Renewing the NHS: Labour's Agenda for a Healthier Britain.London: Labour Party.
25.
Le Grand, J., N. Mays and J. A. Mulligan (eds) (1998) Learning from the NHS Internal Market. A Review of the Evidence. London: King's Fund.
26.
McLeod, H. and J. Raftery (2001) ‘Managing Emergency Hospital Activity’, in N. Mays, S. Wyke, G. Malbon and N. Goodwin (eds) The Purchasing of Health Care by Primary Care Organizations: An Evaluation and Guide to Future Policy. Buckingham: Open University Press.
27.
Martin, S. and I. Sanderson (1999) ‘Evaluating Public Policy Experiments: Measuring Outcomes, Monitoring Performance, or Managing Pilots?’, Evaluation5(3): 245–258.
28.
Maynard, A. (1986) ‘Performance Incentives’, in G. Teeling-Smith (ed.) Health, Education and General Practice. London: Office of Health Economics.
29.
Maynard, A., M. Marinker and D. P. Grey (1986) ‘The Doctor, the Patient and their Contract: Alternative Contracts: Are They Viable?’, British Medical Journal292: 1438–1440.
30.
Mays, N. (1997) ‘Key Findings to Date From the National Evaluation of First Wave Total Purchasing Pilot Projects in England and Scotland’, Paper prepared as Briefing for the Department of Health/NHSE (unpublished).
31.
Mays, N., B. Goodwin, G. Bevan and S. Wyke (1997) ‘What is Total Purchasing?’, British Medical Journal315: 652–655.
32.
Mays, N., N. Goodwin, A. Killoran and G. Malbon (1998) Total Purchasing: A Step towards Primary Care Groups. London: King's Fund.
33.
Mays, N., S. Wyke, G. Malbon and N. Goodwin (eds) (2001) The Purchasing of Health Care by Primary Care Organizations: An Evaluation and Guide to Future Policy. Buckingham: Open University Press.
34.
Mechanic, D. (1993) ‘Social Research in Health and the American Sociopolitical Context: the Changing Fortunes of Medical Sociology’, Social Science and Medicine36: 95–102.
35.
Miller, C. (1998) Joint Action on Health Inequalities: The Policy Drivers for the Health Service and for Local Authorities.London: Health Education Authority.
36.
NHS Executive (1997) Personal Medical Services under the NHS (Primary Care) Act 1997, The Contractual Framework for PMS Provider Pilots. Leeds: NHS Executive.
37.
Øvretveit, J. (1998) Evaluating Health Interventions.Buckingham: Open University Press.
38.
Packwood, T., M. Buxton and J. Keen (1990) ‘Resource Management in the National Health Service: a First Case Study’, Policy and Politics18(4): 245–255.
39.
Packwood, T., J. Keen and M. Buxton (1991) Hospitals in Transition: the Resource Management Experiment. Milton Keynes: Open University Press.
40.
Packwood, T., C. Pollitt and S. Roberts (1998) ‘Good Medicine? A Case Study of Business Process Re-engineering in a Hospital’, Policy and Politics26: 401–415.
41.
Patton, M. (1997) Utilisation-focussed Evaluation: the New Century Text. London: Sage.
42.
Pawson, R. and N. Tilley (1997) Realistic Evaluation.London: Sage.
43.
Pettigrew, A., E. Ferlie and L. McKee (1992) Shaping Strategic Change.London: Sage.
44.
Posnett, J., N. Goodwin, J. Griffiths, A. Killoran, G. Malbon, N. Mays, M. Place and A. Street (1998) The Transaction Costs of Total Purchasing.National Evaluation of Total Purchasing Pilots Working Paper. London: King's Fund.
45.
Raine, R. (1998) ‘Evidence-based Policy: Rhetoric and Reality’, Journal of Health Services Research and Policy3(4): 2251–2253.
46.
Robinson, R. (1994) ‘Introduction’, in R. Robinson and J. Le Grand (eds) Evaluating the NHS Reforms. London: King's Fund.
47.
Robinson, R. and J. Le Grand (eds) (1994) Evaluating the NHS Reforms.London: King's Fund.
48.
Secretaries of State for Health, Wales, Northern Ireland and Scotland (1989) Working for Patients. Cmnd 555. London: HMSO
49.
Secretary of State for Health (1997) The New NHS: Modern, Dependable.London: The Stationery Office.
50.
Shadish, W., T. Cook and L. Leviton (1991) Foundations of Programme Evaluation: Theories of Practice. London: Sage.
51.
Total Purchasing National Evaluation Team (1999) Developing Primary Care in the New NHS. Lessons from Total Purchasing.London: King's Fund.
52.
Townsend, P. and N. Davidson (eds) (1982) The Black Report.Harmondsworth: Penguin.
53.
Walt, G. (1994) Health Policy: An Introduction to Process and Power.London: Zed Books.
54.
Weiss, C. H. (1999) ‘The Interface between Evaluation and Public Policy’, Evaluation5(4): 468–486.
55.
Weiss, C. and M. Bucuvalas (1980) Social Science Research and Decision-making.New York: Columbia University Press.
56.
Wilkin, D., S. Gillam and B. Leese (eds) (2000) The Nation Tracker Survey of Primary Care Groups and Trusts. Progress and Challenges 1999/2000.University of Manchester: NPRDC Publications.
57.
Wolff, N. (2001) ‘Randomized Trials of Socially Complex Interventions: Promise or Peril?’, Journal of Health Services Research and Policy6(2): 1–4.
58.
Wyke, S., G. Malbon, N. Mays and N. Goodwin (2001) ‘How Was Change Achieved?’, in N. Mays, S. Wyke, G. Malbon and N. Goodwin (eds) The Purchasing of Health Care by Primary Care Organizations: An Evaluation and Guide to Future Policy. Buckingham: Open University Press.
59.
Zigler, E. F. and S. Styfo (1996) ‘Head Start and Early Childhood Intervention: the Changing Course of Social Science and Social Policy’, in E. F. Zigler, S. L. Kagan and N. W. Hall (eds) Children, Familes and Government. New York: Cambridge University Press.
