This article presents a guiding framework for evaluators to conduct systems evaluation: system evaluation theory (SET). The article firstly defines ‘systems’, ‘system theory’, and ‘system thinking’; it then discusses the confusion in the evaluation literature surrounding the use of these terms (noting their application is primarily in evaluating programs, not evaluating modern day systems). Three guiding principles for evaluating modern day systems are then presented demonstrating: a) their relationship to systems theory; and b) how they assist the evaluator in accomplishing an evaluation.
Get full access to this article
View all access options for this article.
References
1.
AbdulmalekF., & RajgopalJ.2007, ‘Analyzing the benefits of lean manufacturing and value stream mapping via simulation: a process sector case study’, International Journal of Production Economics, vol. 107, no. 1, pp. 223–236, DOI: 10.1016/j.ijpe.2006.09.009.
2.
AdairJ.2009, Not bosses but leaders how to lead the way to success, 3rd edn, Kogan Page, London.
3.
AdamsK.M., HesterP.T., BradleyJ.M., MeyersT.J., & KeatingC.B.2014, ‘Systems theory as the foundation for understanding systems’, Systems Engineering, vol. 17, no. 1, pp. 112–123, DOI: 10.1002/sys.21255.
4.
AlkinM., & ChristieC.2004, ‘An evaluation theory tree’, in Evaluation roots: tracing theorists’ views and influences, 1st edn, pp. 12–65, Sage Publications, Thousand Oaks, California.
5.
AndrewsH.A.1985, Evaluating for excellence: addressing the need for responsible and effective faculty evaluation, New Forums Press, Inc., Stillwater, Oklahoma.
6.
ArveyR.D., CampbellJ.P., DunnetteM.D., & HellervikL.V.1973, ‘The development and evaluation of behaviorally based rating scales’, Journal of Applied Psychology, vol. 57, no. 1, pp. 15, DOI: 10.1037/h0034185.
7.
BakerA.H., & ManojB.S.2007, ‘Communication challenges in emergency response’, Communications of the ACM, vol. 50, no. 3, pp. 51–53, DOI: 10.145/1226736.1226765.
8.
BakosJ.Y., & TreacyM.E.1986, ‘Information technology and corporate strategy: a research perspective’, MIS Quarterly, vol. 10, no. 2, pp. 107–119, DOI: 10.2307/249029.
9.
BanathyB.H.1992, A systems view of education: concepts and principles for effective practice, Educational Technology, Englewood Cliffs, New Jersey.
BeckerK., RengerR., & McPhersonM.2015, ‘Indicators of buy-in to gauge evaluation success’, Australasian Journal of Evaluation, vol. 15, no. 2, pp. 12–21.
12.
BorichG.D., & JemelkaR.P.1982, Programs and systems: an evaluation perspective, Academic Press, San Diego, CA.
13.
BoyleR., & LemaireD.1999, Building effective evaluation capacity: lessons from practice, vol. 8, Transaction Publishers, New Brunswick, New Jersey.
14.
BrandonR.N.2014, Adaptation and environment, Princeton University Press, Princeton, New Jersey.
15.
BuckleyW.F.1998, Society—a complex adaptive system: essays in social theory, Gordon and Breach, Australia.
16.
BuckleyW.F.1967, Sociology and modern systems theory, Prentice-Hall, Englewood Cliffs, New Jersey.
17.
BurnsT.R.2007, ‘System theories’, in RitzerG (ed), The Wiley-Blackwell Encyclopedia of Sociology, Malden, MA and Blackwell Publishing, Oxford.
18.
BuznaL., HelbingD., & PetersK.2008, ‘Modelling of cascading effects and efficient response to disaster spreading in complex networks’, International Journal of Critical Infrastructures, vol. 4, nos. 1-2, pp. 46–62, DOI: 10.1504/UCIS.2008.016091.
19.
CashinW.E., & DowneyR.G.1992, ‘Using global student rating items for summative evaluation’, Journal of Educational Psychology, vol. 84, no. 4, pp. 563.
20.
ChecklandP., & PoulterJ.2010, ‘Soft systems methodology’, in Systems approaches to managing change: a practical guide, 1st edn, pp. 191–242, Springer, London.
21.
ChecklandP.1999, Systems thinking. Rethinking management information systems, Oxford University Press, New York.
22.
ChenH., HaileyD., WangN., & YuP.2014, ‘A review of data quality assessment methods for public health information systems’, International Journal of Environmental Research and Public Health, vol. 11, no. 5, pp. 5170–5207.
23.
ChristieC.A., & AlkinM.C.2008, ‘Evaluation theory tree reexamined’, Studies in Educational Evaluation, vol. 34, no. 3, pp. 131–135.
24.
CohenE., & TichyN.M.2003, ‘Why are leaders important?’, in Business Leadership: A Jossey-Bass reader, pp. 4–28, Jossey-Bass, San Francisco.
25.
CohenA.Y., & KibelB.M.1993, The basics of open systems evaluation: a resource paper, The Pacific Institute for Research and Evaluation, Chapel Hill, North Carolina.
26.
ComfortL.K., DunnM., JohnsonD., SkertichR., & ZagoreckiA.2004, ‘Coordination in complex systems: increasing efficiency in disaster mitigation and response’, International Journal of Emergency Management, vol. 2, nos. 1-2, pp. 62–80.
27.
CoşkunR., AkandeA., & RengerR.2012, ‘Using root cause analysis for evaluating program improvement’, Evaluation Journal of Australasia, vol. 12, no. 2, pp. 4–14.
28.
CurphyG.J., HoganJ., & HoganR.1994, ‘What we know about leadership: effectiveness and personality’, American Psychologist, vol. 49, no. 6, pp. 493.
29.
DenehyT.E., & HsuW.W.2003, Duplicate management for reference data, research report RJ10305, IBM Research.
30.
Department of Homeland Security2013, Homeland Security Exercise and Evaluation Program, Federal Emergency Management Agency.
31.
DeyoD., JulianD.A., & JonesA.1995, ‘Open systems evaluation and the logic model: program planning and evaluation tools’, Evaluation and Program Planning, vol. 18, no. 4, pp. 333–341.
32.
DonaldsonS.I., & LipseyM.W.2006, ‘Roles for theory in contemporary evaluation practice: developing practical knowledge’, in ShawI, GreeneJC, & MarkMM (eds), The handbook of evaluation: policies, programs, and practices, pp. 56–75, Sage Publications, London.
33.
DonnellyJ.1984, ‘Cultural relativism and universal human rights’, Human Rights Quarterly, vol. 6, no. 4, pp. 400–419.
34.
EisenbergM.S.2013, Resuscitate: how your community can improve survival from sudden cardiac arrest, University of Washington Press, Washington.
35.
EricsonC.A.2011, Concise encyclopedia of system safety: definition of terms and concepts, John Wiley & Sons, Hoboken, New Jersey.
36.
FainterJ.1991, ‘Quality assurance not equal to quality improvement’, Journal of Quality Assurance, vol. 13, no. 1, pp. 8–9.
FlynnB.B., SchroederR.G., & SakakibaraS.1994, ‘A framework for quality management research and an associated measurement instrument’, Journal of Operations Management, vol. 11, no. 4, pp. 339–366.
39.
FullanM.2004, Leadership & sustainability: system thinkers in action, Corwin Press, Thousand Oaks, California.
40.
FoltysovaJ.2013, Validation of reconstructed program theory, University of Arizona, Tucson, Arizona.
41.
GershonR., StoneP., BakkenS., & LarsonE.2004, ‘Measurement of Organizational Culture and Climate in Healthcare’, The Journal of Nursing Administration, vol. 34, no. 1, pp. 33–40.
42.
GibsonJ.J.1960, ‘The concept of the stimulus in psychology’, American Psychologist, vol. 15, no. 11, pp. 694.
43.
GreeneJ.C.1994, ‘Qualitative program evaluation: practice and promise’, in DenzinNK, & LincolnYS (eds), Handbook of qualitative research, Sage, Thousand Oaks, California, pp. 530–544.
44.
HeifetzR.A., & LaurieD.L.1997, ‘The work of leadership’, Harvard Business Review, vol. 75, no. 1, pp. 124–134.
45.
HolwegM.2007, ‘The genealogy of lean production’, Journal of Operations Management, vol. 25, no. 2, pp. 420–437, DOI: 10.1016/j.jom.2006.04.001.
46.
InnesJ.E.2004, ‘Consensus building: Clarifications for the critics’, Planning Theory, vol. 3, no. 1, pp. 5–20.
47.
JeliazkovaM., & WesterheijdenD.F.2002, ‘Systemic adaptation to a changing environment: towards a next generation of quality assurance models’, Higher Education, vol. 44, nos. 3-4, pp. 433–448.
48.
JuergensH.F.1977, ‘Attributes of information system development’, MIS Quarterly, vol. 1, pp. 31–41.
49.
KasunicM.2001, Measuring systems interoperability: challenges and opportunities, Carnegie-Mellon University Pittsburgh Software Engineering Institute, Pittsburgh, Pennsylvania.
50.
KatzenbachJ.R., SteffenI., & KronleyC.2012, ‘Cultural change that sticks’, Harvard Business Review, vol. 90, no. 7, pp. 8.
51.
KazdinA.E.1979, ‘Unobtrusive measures in behavioral assessment’, Journal of Applied Behavior Analysis, vol. 12, no. 4, pp. 713–724.
52.
KotterJ.P.2001, What leaders really do, Harvard Business Review, vol. 71, no. 11, pp. 3–11.
53.
LandauM.1969, ‘Redundancy, rationality, and the problem of duplication and overlap’, Public Administration Review, vol. 1, no. 4, pp. 346–358.
54.
LanzendörferM., RubensR., VahlhausM., & ZintlM.2011, ‘Final panel discussion: conclusions’, paper presented at the Deutsche Gesellschaft für Internationale Zusammenarbeit Systemic Approaches in Evaluation Conference, Eschborn, Germany, January.
55.
LaszloA.2009, ‘The nature of evolution’, World Futures, vol. 65, no. 3, pp. 204–221.
56.
LeCompteM.D.2000, ‘Analyzing qualitative data’, Theory into Practice, vol. 39, no. 3, pp. 146–154.
57.
LiuJ.J., BellamyG.R., & McCormickM.2007, Patient bypass behavior and critical access hospitals: implications for patient retention’, Journal of Rural Health, vol. 23, no. 1, pp. 17–24.
58.
MarkM.M., HenryG.T., & JulnesG.2000, Evaluation: an integrated framework for understanding, guiding, and improving policies and programs, Jossey-Bass, San Francisco, California.
59.
McDavidJ.C., HuseI., & HawthornL.R.L.2013, ‘Key concepts and issues in program evaluation and performance measurement’, in Program evaluation and performance measurement: an introduction to practice, 2nd ed., Sage, Thousand Oaks, California.
60.
McLaughlinJ.A., & JordanG.B.1999, ‘Logic models: a tool for telling your programs performance story’, Evaluation and Program Planning, vol. 22, no. 1, pp. 65–72.
61.
MeadowsD.H., & WrightD.2008, Thinking in systems: a primer, Chelsea Green Pub, White River Junction, Vermont.
62.
Miller-WilliamsS.L., & KritsonisW.A.2009, ‘A systems approach to comprehensive school reform: using the realms of meaning and the Baldridge model as a systems framework’, National Forum of Applied Educational Research Journal, vol. 23, nos. 1-2, pp. 1–8.
63.
MohanR.2014, ‘Evaluator advocacy it is all in a day's work’, American Journal of Evaluation, vol. 35, no. 3, pp. 397–403.
64.
MorellJ.A.2005, ‘Why are there unintended consequences of program action, and what are the implications for doing evaluation?’, American Journal of Evaluation, vol. 26, no. 4, pp. 444–463.
65.
OdumE.1971, Fundamentals of ecology, 3rd edn, Saunders, Philadelphia, Pennsylvania.
66.
ØvretveitJ., & GustafsonD.2002, ‘Evaluation of quality improvement programmes’, Quality and Safety in Health Care, vol. 11, no. 3, pp. 270–275.
PhillipsD.C.1971, ‘James, Dewey, and the reflex arc’, Journal of the History of Ideas, vol. 32, no. 4, pp. 555–568.
71.
RadnorZ, & WalleyP., StephensA., & BucciG.2006, Evaluation of the Lean approach to business management and its use in the public sector, Scottish Executive Social Research, Edinburgh, http://www.gov.scot/Resource/Doc/129627/0030899.pdf.
72.
ReedG.E.2006, ‘Leadership and systems thinking’, Defense AT&L, vol. 35, no. 3, pp. 10–13.
73.
ReedM.S.2008, ‘Stakeholder participation for environmental management: a literature review’, Biological Conservation, vol. 141, no. 10, pp. 2417–2431.
74.
RengerR., FoltysovaJ., BeckerK., & SouvannasacdE. (in press a), ‘The power of the context map: designing realistic outcome evaluation strategies and other unanticipated benefits’, Evaluation and Program Planning, vol. 52, pp. 118–125.
75.
RengerR., BeckerK., RoganM., & Kontz-BartelsT. (in press b), ‘Process flow mapping for systems improvement: lessons learned’, Canadian Journal of Program Evaluation.
76.
RengerR., BartelG., & FoltysovaJ.2013, ‘The reciprocal relationship between implementation theory and program theory in assisting program design and decision-making’, Canadian Journal of Program Evaluation, vol. 28, no. 1, pp. 27–41.
77.
RengerR., WoodS., WilliamsonS., & KrappS.2012, ‘Systemic evaluation, impact evaluation and logic models’, Evaluation Journal of Australasia, vol. 11, no. 2, pp. 24–30.
78.
RengerR.2011, ‘Constructing and verifying program theory using source documentation’, The Canadian Journal of Program Evaluation, vol. 25, no. 1, pp. 51–67.
79.
RengerR., & BourdeauB.2004, ‘Strategies for values inquiry: an exploratory case study’, American Journal of Evaluation, vol. 25, no. 1, pp. 39–49.
80.
RengerR., & TitcombA.2002, ‘A three-step approach to teaching logic models’, American Journal of Evaluation, vol. 23, no. 4, pp. 493–503.
81.
RogersP.J.2011, ‘Program theory and logic models for systemic evaluation’, paper presented at the Deutsche Gesellschaft für Internationale Zusammenarbeit Systemic Approaches in Evaluation Conference, January, Eschborn, Germany.
82.
RogersP.J.2008, ‘Using programme theory to evaluate complicated and complex aspects of interventions’, Evaluation, vol. 14, no. 1, pp. 29–48.
83.
RollandC.1993, ‘Modeling the requirements engineering process’, paper presented at the 3rd European-Japanese Seminar on Information Modelling and Knowledge Bases, Budapest, Hungary.
84.
SandersJ.R.1994, The program evaluation standards: how to assess evaluations of educational programs, 2nd edn, Sage Publications, Thousand Oaks, California.
85.
ScheinE.H.1990, ‘Organizational culture’, American Psychologist, vol. 45, no. 2, pp. 109–119, DOI: 10.1037/0003-066X.45.2.109.
86.
SchwabD.P., HenemanH.G.III, & DeCotiisT.A.1975, ‘Behaviorally anchored rating scales: a review of the literature’, Personnel Psychology, vol. 28, pp. 549–562.
87.
ScottT., MannionR., DaviesH., & MarshallM.2003, ‘The quantitative measurement of organizational culture in health care: a review of the available instruments’, Health Services Research, vol. 38, no. 3, pp. 923–945, DOI: 10.1111/1475-6773.00154.
88.
SpeerD.C.1998, Mental health outcome evaluation, Academic Press, San Diego, California.
89.
StameN.2004, ‘Theory-based evaluation and types of complexity’, Evaluation, vol. 10, no. 1, pp. 58–76.
90.
VartanianT.P.2010, Secondary data analysis, Oxford University Press, New York.
91.
Von BertalanffyL.1968, General system theory: foundations, development, applications, George Braziller, New York.
92.
WeissC.H.1995, ‘Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families’, in ConnellJP, KubischAC, SchorrLB, WeissCH (eds), New approaches to evaluating community initiatives. Concepts, methods, and contexts, Aspen Institute, Washington DC.
93.
WilliamsJ.C.2015, ‘A systems thinking approach to analysis of the Patient Protection and Affordable Care Act’, Journal of Public Health Management and Practice, vol. 21, no. 1, pp. 6–11, DOI: 10.1097/PHH.0000000000000150.
94.
WilliamsB., & HummelbrunnerR.2010, Systems concepts in action: a practitioner's toolkit, Stanford University Press, Palo Alto, California.
