Evaluators are yet to define systemic evaluation adequately. With respect to impact evaluation, systemic evaluators are critical of current methods such as simple logic models, or logframes, for capturing the contextual complexity in which programs operate. Despite a lack of clarity about what constitutes a systemic evaluation, many evaluators are suggesting it is a better alternative for capturing the true complexity of the context in which a program operates. However, capturing complexity is itself complex, resulting in amorphous and ambiguous approaches and results. This can create problems in relation to evaluation capacity building, usability of results, and evaluator credibility. This article demonstrates how methods such as simple logic models or logic models grounded in situational awareness can be an ally in bridging clients' understanding, so that systemic evaluation is not perceived to be new, but rather something that continues to build on past investments.
Get full access to this article
View all access options for this article.
References
1.
BoydA., GeerlingT., GregoryW.J., KaganC., MidgleyG., MurrayP., & WalshM.P.2007, ‘Systemic evaluation: a participative, multi-method approach’, Journal of the Operational Research Society, vol. 58, no. 10, pp. 1306–1320, doi: 10.1057/palgrave.jors.2602281.
2.
BrouselleA.2010, ‘Assessing the validity of program theory: logic analysis’, paper presented at the 9th Annual European Evaluation Society International Conference, Prague, 6-8 October.
3.
CabreraD., ColosiL., & LobdellC.2008, ‘Systems thinking’, Evaluation and Program Planning, vol. 31, no. 3, pp. 299–310.
4.
ChenH.T.1990, Theory-driven evaluations, Sage, Newbury Park, California.
5.
ChenH., & RossiP.H.1983, ‘Evaluating with sense: the theory-driven approach’, Evaluation Review, vol. 7, no. 3, pp. 283–302.
6.
ColeG.E.1999, ‘Advancing the development and application of theory-based evaluation in the practice of public health’, American Journal of Evaluation, vol. 20, no. 3, pp. 453–470.
7.
DonaldsonS.I.2002, ‘Theory-driven program evaluation in the new millennium’, in DonaldsonS.I., & ScrivenM (eds), Evaluating social programs and problems, Lawrence Erlbaum Mahwah, New Jersey.
8.
FieldenS.J., RuschM.L., MasindaM.T., SandsJ., FrankishJ., & EvoyB.2007, ‘Key considerations for logic model development in research partnerships: a Canadian case study’, Evaluation and Program Planning vol. 30, no. 1, pp. 115–124.
9.
FettermanD.M.2001, Foundations of empowerment evaluation, Sage, Thousand Oaks, California.
10.
FloodR.L.1999, Rethinking fifth discipline: learning within the unknowable, Routledge, London.
11.
GoodyearL., & Dahler-LarsenP.2010, ‘Evaluation systems in the public interest?’, paper presented at the 9th Annual European Evaluation Society International Conference, Prague, 6-8 October.
12.
GubaE.G., LincolnY.S.1989, Fourth generation evaluation, Sage, Newbury Park, California.
13.
HollidayR.2010, ‘A case study on adaptive action in education’, paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, Texas, 10-13 November.
14.
HummelbrunnerR.2010a, ‘Beyond logframe: critique, variations and alternatives’, in FujitaN (ed.), Beyond logframe: using systems concepts in evaluation, Foundation for Advanced Studies on International Development, Tokyo.
15.
HummelbrunnerR.2010b, ‘Changing institutional approaches to using logic models’, paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, Texas, 10-13 November.
16.
Joint Committee on Standards for Educational Evaluation1994, The program evaluation standards: how to assess evaluations of educational programs, 2nd edn, Sage, Thousand Oaks, California.
LeeuwF.L.2003, ‘Reconstructing program theories: methods available and problems to be solved’, American Journal of Evaluation, vol. 24, no. 1, pp. 5–20.
19.
LithmanE.2010, ‘Evaluability assessment 2.0—or: on a reflexive approach to evaluation system in the knowledge society’, paper presented at the 9th Annual European Evaluation Society International Conference, Prague, 6-8 October.
20.
MarkM.M., HenryG.T., & JulnesG.2000, Evaluation: an integrated framework for understanding, guiding, and improving policies and programs, Jossey-Bass, San Francisco.
21.
McLaughlinJ.A., & JordanG.B.1999, ‘Logic models: a tool for telling your programs performance story’, Evaluation and Program Planning, vol. 11, no. 2, pp. 54–61.
22.
MillarA., SimeonebR.S., & CarnevalecJ.T.2001, ‘Logic models: a systems tool for performance management’, Evaluation and Program Planning, vol. 24, no. 4, pp. 73–81.
23.
MorellJ.2010, Unintended consequences: Jonny Morell on logic models and unintended consequences, AEA365, a Tip a Day for Evaluators, viewed 1 December 2011, <http://aea365.org/blog/?tag=unintended-consequences>.
24.
PattonM.Q.2000, ‘Utilization-focused evaluation’, in StufflebeamD.L., MadausG.F., & KellaghanT (eds), Evaluation models: viewpoints on educational and human services evaluation, 2nd edn, Kluwer, Norwell, Massachussets.
25.
PinnegarS.2006, Are complex programs the best response to complex policy issues? City Futures Research Centre issues paper, University of New South Wales, Kensington, NSW.
26.
PreskillH., & BoyleS.2008, ‘A multidisciplinary model of evaluation capacity building’, American Journal of Evaluation, vol. 29, no. 4, pp. 443–459.
27.
RengerR.2011, ‘Systemic evaluation: much too soon?’, paper presented at the GIZ International Conference on Systemic Approaches in Evaluation, Eschborn, Germany, 25-26 January.
28.
RengerR., & HurleyC.2006, ‘From theory to practice: lessons learned in the application of the ATM approach to developing logic models’, Evaluation and Program Planning, vol. 29, no. 2, pp. 106–119.
29.
RengerR., & TitcombA.2002, ‘A three-step approach to teaching logic models’, American Journal of Evaluation, vol. 23, no. 4, pp. 493–503.
30.
RengerR., WilliamsonS., & WoodS.2011, ‘Systemic evaluation: using logic models to bridge capacity building gaps’, roundtable presentation at the Australasian Evaluation Society 2011 International Conference, Sydney, 31 August – 2 September.
31.
RogersP.J.2008, ‘Using programme theory to evaluate complicated and complex aspects of interventions’, Evaluation, vol. 14, no. 1, pp. 29–48, doi: 10.1177/1356389007084674.
32.
RogersP.2010, ‘Representing simple, complicated, and complex aspects in logic models for evaluation quality’, paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, Texas, 10-13 November.
33.
RosasS.R.2005, ‘Concept mapping as technique to program theory development: an illustration using family support program’, American Journal of Evaluation, vol. 26, no. 3, pp. 389–401.
34.
RosenbergL., & PosnerL.1979, The logical framework: a manager's guide to a scientific approach to design and evaluation, Practical Concepts Incorporated, Washington, DC.
ScrivenM.1991, Evaluation thesaurus, Sage, Newbury Park, California.
37.
StufflebeamD.L., ShinkfieldA.J.2007, Evaluation theory, models, and applications, Jossey-Bass, San Francisco.
38.
TytelM.2010, ‘Translating lessons learned into lessons applied’, paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, Texas, 10-13 November.
39.
WholeyJ.S.2003, ‘Using policy evaluation to support decision making and improve program performance’, Japanese Journal of Evaluation Studies, vol. 3, no. 2, pp. 6–21.
40.
WilliamsB.2010, ‘Fitting the key to the lock: matching systems methods to evaluation questions’, paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, Texas, 10-13 November.
41.
YangE.2011, ‘National efforts to build coalition and evaluator capacity to track community and systems change’, American Evaluation Association Annual Conference, Anaheim, California, 2-5 November.
42.
ZürcherD.2011, ‘Why the (development) world is not a system’, paper presented at the GIZ International Conference on Systemic Approaches in Evaluation, Eschborn, Germany, 25-26 January.
