Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions

Abstract

This article proposes ways to use programme theory for evaluating aspects of programmes that are complicated or complex. It argues that there are useful distinctions to be drawn between aspects that are complicated and those that are complex, and provides examples of programme theory evaluations that have usefully represented and address both of these. While complexity has been defined in varied ways in previous discussions of evaluation theory and practice, this article draws on Glouberman and Zimmerman's conceptualization of the differences between what is complicated (multiple components) and what is complex (emergent). Complicated programme theory may be used to represent interventions with multiple components, multiple agencies, multiple simultaneous causal strands and/or multiple alternative causal strands. Complex programme theory may be used to represent recursive causality (with reinforcing loops), disproportionate relationships (where at critical levels, a small change can make a big difference — a `tipping point') and emergent outcomes.

Cet article présente des moyens pour utiliser la théorie du programme afi n d’évaluer des aspects compliqués et complexes d’un programme. Il montre que l’on peut établir des distinctions utiles entre les aspects compliqués et complexes et fournit des exemples d’évaluations fondées sur la théorie du programme qui ont utilement représenté et traité ces deux aspects. Alors que la complexité a été défi nie de différentes façons dans des discussions antérieures, cet article s’appuie sur la conceptualisation des différences entre complication (multiplicité des composants) et complexité (émergence) proposée par Glouberman et Zimmerman (2002). Une théorie du programme compliquée peut être utilisée pour représenter des interventions avec de multiples composants, de multiples acteurs, de multiples chaînes causales simultanées ou alternatives. Une théorie de programme complexe peut être utilisée pour représenter une causalité récursive (avec des boucles de renforcement), des relations disproportionnées (avec des seuils critiques pour lesquels un petit changement peut faire une grande différence) et des effets émergents.

Keywords

collaboration complexity performance measurement programme theory theory of change

Get full access to this article

View all access options for this article.

References

Abma, T.A. and M. Noordegraaf (2003) ` Public Managers Amidst Ambiguity: Towards a Typology of Evaluative Practices in Public Management ', Evaluation 9(3): 285—306 .

Arnkil, R. , T. Spangar , J. Nieminen , P. Rissanen and J. Konkinen (2002) Emergent Evaluation and Learning in Multi-Stakeholder Situations . Paper presented at the 2002 Conference of the European Evaluation Society, Seville. Available at http://www.europeanevaluation.org/docs/ArnkilSevilleslides13.pdf

Bakewell, O. and A. Garbutt (2005) The Use and Abuse of the Logical Framework Approach . Stockholm: SIDA .

Barnes, M. , E. Matka and H. Sullivan (2003) ` Evidence, Understanding and Complexity: Evaluation in Non-Linear Systems ', Evaluation 9(3): 265—84 .

Barnes, M. , H. Sullivan and E. Matke (2004) The Development of Collaborative Capacity in Health Action Zones: A Final Report from the National Evaluation. Birmingham: University of Birmingham . URL: http://www.socialresearch.bham.ac.uk/downloads/HAZ_FINAL_REPORT_JAN_11.pdf (downloaded 11 October 2007).

Batterham, R. , D. Dunt and P. Disler (1996) ` Can we Achieve Accountability for Long-Term Outcomes in Rehabilitation? ', Archives of Physical Medicine Rehabilitation 77(12): 1219—25 .

Chen, H.T. (1990) Theory-driven evaluation. Thousand Oaks, CA: Sage .

Collaborative Institute for Research, Consulting and Learning in Evaluation (CIRCLE) (2006) Evaluation of the Stronger Families and Communities Strategy 2000—2004 . Third newsletter. Available at www.facsia.gov.au

Davies, R. (2004) ` Scale, Complexity and the Representation of Theories of Change (Part I) ', Evaluation 10: 101—21 .

10.

Davies, R. (2005) ` Scale, Complexity and the Representation of Theories of Change (Part II) ', Evaluation 11: 133—49 .

11.

Donaldson, S. (2005) ` Using Program Theory-Driven Evaluation Science to Crack the Da Vinci Code ', in M. C. Alkin and C. A. Christie (eds) Theorists' Models in Action, New Directions in Evaluation , 106, pp. 65—84 .

12.

Douthwaite, B. , R. Delve , J. Ekboir and S. Twomlow (2003a) ` Contending with Complexity: The Role of Evaluation in Implementing Sustainable Natural Resource Management ', International Journal of Agricultural Sustainability 1(1): 51—66 .

13.

Douthwaite, B. , T. Kuby , E. van de Fliert and S. Schulz (2003b) ` Impact Pathway Evaluation: An Approach for Achieving and Attributing Impact in Complex Systems ', Agricultural Systems 78: 243—65 .

14.

Douthwaite, B. and S. Schulz (2001) ` Spanning the Attribution Gap: The Use of Program Theory to Link Project Outcomes to Ultimate Goals in Integrated Natural Resource Management and Integrated Pest Management ', paper presented at the INRM Workshop, 2001, Cali , Colombia. Available at http://www.ciat.cgiar.org/inrm/workshop2001/docs/titles/4—1APaperBDouthwaite.pdf.

15.

Eoyang, G. and T. Berkas (1998) `Evaluation in Complex Adaptive System', in M. Lissack and H. Gunz (eds) Managing Complexity in Organizations. Westport, CT: Quorum Books .

16.

Eoyang, G. , L. Yellowthunder and V. Ward (1998) ` A Complex Adaptive Systems (CAS) Approach to Public Policy Decision Making', paper presented to the Society for Chaos Theory in Psychology in the Life Sciences, August.

17.

Funnell, S. (1997) ` Program Logic: An Adaptable Tool for Designing and Evaluating Programs ', Evaluation News and Comment 6(1): 5—7 .

18.

Funnell, S. (2000) `Developing and Using a Program Theory Matrix for Program Evaluation and Performance Monitoring', in P. Rogers , T. Hacsi , A. Petrosino and T. Huebner (eds) Program Theory in Evaluation: Challenges and Opportunities, New Directions for Evaluation, 87, pp. 91—101. San Francisco, CA: Jossey-Bass .

19.

Giuliani, R. (2002) Leadership. London: Little, Brown .

20.

Glouberman, S. (2001) `Towards a New Perspective on Health Policy', CPRN Study No. H/03, Canadian Policy Research Networks Inc., Ottawa .

21.

Glouberman, S. and B. Zimmerman (2002) Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Commission on the Future of Health Care in Canada, Discussion Paper 8. Available at http://www.hc-sc.gc.ca/english/pdf/romanow/pdfs/8_Glouberman_E.pdf

22.

Gregory, R.J. (1995) `Accountability, Responsibility, and Corruption: Managing the “Public Production Process”', in J. Boston (ed.) The State Under Contract, pp. 56—77. Wellington , NZ: Bridget Williams Books .

23.

Judge, K. and L. Bauld (2001) ` Strong Theory, Flexible Methods: Evaluating Complex Community-Based Initiatives ', Critical Public Health 11(1): 19—38 .

24.

Kankare, I. (2004) ` Treading on Fuzzy Grounds: Evaluation Practices in Managing Complexities ', paper presented at the European Evaluation Society conference, Berlin.

25.

Kurtz, C.F. and D.J. Snowden (2003) ` The New Dynamics of Strategy: Sense-making in a Complex and Complicated World ', IBM Systems Journal 42(3): 462—83 . URL: http://www.research.ibm.com/journal/sj/423/kurtz.pdf (accessed 11 October 2007).

26.

Martin, C.M. and J.P. Sturmberg (2005) ` General Practice: Chaos, Complexity and Innovation ', Medical Journal of Australia 183: 106—9 .

27.

Nagarajan, N. and M. Vanheukelen (1997) Evaluating EU Expenditure Programmes: A Guide. Brussels : Directorate-general for Budgets of the European Union .

28.

Patton, M.Q. (1994) ` Developmental Evaluation ', American Journal of Evaluation 15: 311—19 .

29.

Patton, M.Q. (1997) Utilization-focused Evaluation (3rd edn). Thousand Oaks, CA: Sage .

30.

Patton, M.Q. (2003) ` New Dimensions and Meanings of Useful Evaluation ', paper presented at the 2003 conference of the Australasian Evaluation Society, Auckland.

31.

Pawson, R. (2006) `Simple Principles for the Evaluation of Complex Programmes', in A. Killoran , C. Swann and M. Kelly (eds) (2006) Public Health Evidence: Tackling Health Inequalities, pp. 223—38. Oxford: Oxford University Press .

32.

Pawson, R. and N. Tilley (1997) Realistic Evaluation. London : SAGE .

33.

Perrin, B. (2003) Implementing the Vision: Addressing the Challenges of Results-Based Management. Paris: OECD . Available at http://www.oecd.org/dataoecd/4/10/2497163.pdf

34.

Pinnegar, S. (2006) Are Complex Programs the Best Response to Complex Policy Issues? City Futures Research Centre Issues paper. Kensington, NSW : University of New South Wales .

35.

Potter, P. (2004) ` Generalizing from the Unique: Bridging the Gap between Case-Based and Policy-Level Evaluation in the Governance of Neighbourhood Renewal ', paper presented at the 2004 Conference of the European Evaluation Society. Available at www.europeanevaluation.org

36.

Regine, B. and R. Lewin (n.d.) `Complexity Science in Human Terms: A Relational Model of Business '. Available at http://www.harvest-associates.com/pubs/human.html

37.

Riggan, J.M. (2005) `Looking Where the Key is: Surfacing Theories of Change in Complex, Collaborative Initiatives', PhD dissertation, University of Pennsylvania . Available at http://repository.upenn.edu/dissertations/AAI3179796

38.

Rogers, P. , A. Petrosino , T. Hacsi and T. Huebner (2000) `Program Theory Evaluation: Practice, Promise and Problems', in P. Rogers , A. Petrosino , T. Hacsi and T. Huebner (eds) Program Theory Evaluation: Challenges and Opportunities, pp. 5—13, New Directions in Evaluation series. San Francisco, CA: Jossey-Bass .

39.

Rogers, P.J. (2000) `Causal Models in Program Theory Evaluation', in P. J. Rogers , A. J. Petrosino , T. Hacsi and T. A. Huebner (eds) Program Theory Evaluation: Challenges and Opportunities, pp. 47—55, New Directions in Evaluation series. San Francisco, CA: Jossey-Bass .

40.

Rogers, P.J. (2004) ` Logic Models ', in S. Mathison (ed.) Encyclopedia of Evaluation , pp. 232—4 . Newbury Park, CA : SAGE.

41.

Sanderson, I. (2000) ` Evaluation in Complex Policy Systems ', Evaluation 6(4): 433—54 .

42.

Schorr, L. (1997) Common Purpose: Strengthening Families and Neighborhoods to Rebuild America. New York: Anchor Books Doubleday .

43.

Sibthorpe, B. , N. Glasgow and D. Longstaff (2004) Complex Adaptive Systems: A Different Way of Thinking about Health Care Systems. A Brief Synopsis of Selected Literature for Initial Work Program — Stream 1. Canberra: Australian Primary Healthcare Research Institute, Australian National University .

44.

Smith, M.F. (1989) Evaluability Assessment: A Practical Approach. Boston: Kluwer Academic Publishers .

45.

Sparks, D. (2003) `NSDC's Theory of Change', Results September 2003. URL: http://www.nsdc.org/library/publications/results/res9-03spar.cfm (accessed 13 November 2007).

46.

Stacey, R. (1996) Complexity and Creativity in Organizations. San Francisco, CA: Berrett-Koehler .

47.

Stame, N. (2004) ` Theory-Based Evaluation and Types of Complexity ', Evaluation 10(1): 58—76 .

48.

Stufflebeam, D.L. (2004) `The 21st Century CIPP Model', in M. C. Alkin (ed.) Evaluation Roots, pp. 245—66. Newbury Park, CA: SAGE .

49.

Uusikylä, P. and V. Valovirta (2004) ` Three Spheres of Performance Governance: Spanning the Boundaries from Single-Organisation Focus towards a Partnership Network ', Evaluation 13(4): 399—419 .

50.

Weiss, C.H. (1995) `Nothing as Practical as a Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families', in J. P. Connell , A. C. Kubisch , L. B. Schorr and C. H. Weiss (eds) New Approaches to Evaluating Community Initiatives, pp. 65—9. Washington, DC: Aspen Institute .

51.

Weiss, C. (1998) Evaluation: Methods for Studying Programs and Policies . Englewood Cliffs, NJ: Prentice Hall .

52.

Wholey, Joseph S. (2003) ` Using Policy Evaluation to Support Decision Making and Improve Program Performance', Japanese Journal of Evaluation Studies 3(2): 6—21 .

53.

Winston, J. (1999) ` Understanding Performance Measurement: A Response to Perrin ', American Journal of Evaluation 20(1): 95—100 .

54.

W. K. Kellogg Foundation (2004) Logic Model Development Guide. Battle Creek, MI: W. K. Kellogg Foundation . Available at http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

55.

World Bank (2004) `Pages from World Bank History: The Fight Against Riverblindness'. Available at http://web.worldbank.org/WBSITE/EXTERNAL/EXTABOUTUS/EXTARCHIVES/0,,contentMDK:20098846~pagePK:36726~piPK:36092~theSitePK:29506,00.html