This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no literature describing how to define these system elements. Using an example from the evaluation of several United States cardiac care systems, the paper shares a number of methods used to define the system boundaries, components, and relationships. The paper describes how each of these elements informs the evaluation of step two of SET—evaluating system efficiency. The discussion shares lessons learned, and notes the relationship between methods used in system and program evaluation.
Get full access to this article
View all access options for this article.
References
1.
AdamsK. M., HesterP. T., BradleyJ. M., MeyersT. J., & KeatingC. B. (2014). Systems theory as the foundation for understanding systems.Systems Engineering, 17(1), pp. 112–123. doi:10.1002/sys.21255
BeckerK., RengerR., & McPhersonM. (2015). Indicators of buy-in to gauge evaluation success.Evaluation Journal of Australasia, 15(2), pp. 12–21.
4.
CoşkunR., AkandeA., & RengerR. (2012). Using root cause analysis for evaluating program improvement.Evaluation Journal of Australasia, 12(2), pp. 4–14.
5.
EisenbergM. S. (2013). Resuscitate: How your community can improve survival from sudden cardiac arrest.Seattle: University of Washington Press.
6.
EricsonC. A. (2011). Concise encyclopedia of system safety: Definition of terms and concepts.Hoboken, NJ: John Wiley & Sons.
7.
FettermanD. (2001). Foundations of empowerment evaluation.Thousand Oaks, CA: Sage Publications.
8.
GenaC. (2005). Methods and techniques for the evaluation of user-adaptive systems.The Knowledge Engineering Review, 20(1), pp. 1–37. doi: 10.1017/S0269888905000299
9.
GreenL. W., & KreuterM. (2005). Health program planning: An educational approach.New York: McGraw-Hill.
10.
GuestG., BunceA., & JohnsonL. (2006). How many interviews are enough? An experiment with data saturation and variability.Field Methods, 18(1), pp. 59–82. doi.org/10.1177/1525822X05279903
11.
HargreavesM. B., & PodemsD. (2012). Advancing systems thinking in evaluation: a review of four publications.American Journal of Evaluation, 33(3), pp. 462–470.
LiuJ. J., BellamyG. R., & McCormickM. (2007). Patient bypass behavior and critical access hospitals: implications for patient retention.The Journal of Rural Health, 23(1), pp. 17–24.
14.
MorrelJ. A. (2010). Logic models: Uses, limitations, links to methodology and data. American Evaluation Association Annual Meeting, San Antonio, Texas, November 10-13. Retrieved 26 July 2017 from: http://jamorell.com/documents/LM_Workshop_AEA_2010_11_06_2010.pdf
15.
ParsonsT. (1961). An outline of the social system.Classical Sociological Theory, (2), pp. 421–440.
PetersK., BuznaL., & HelbingD. (2008). Modelling of cascading effects and efficient response to disaster spreading in complex networks.International Journal of Critical Infrastructures, 4(1-2), pp. 46–62. doi:10.1504/IJCIS.2008.016091
RadcliffT. A., BrasureM., MoscoviceI. S., & StenslandJ. T. (2003). Understanding rural hospital bypass behavior.The Journal of rural health, 19(3), pp. 252–259.
20.
RengerR. (2015). System Evaluation Theory (SET).Evaluation Journal of Australasia, 15(4), pp. 16–28.
21.
RengerR. (2016). Illustrating the evaluation of system feedback mechanisms using system evaluation theory (SET).Evaluation Journal of Australasia, 16(4), pp. 15–21.
22.
RengerR., BartelG., & FoltysovaJ. (2013). The reciprocal relationship between implementation theory and program theory in assisting program design and decision-making.The Canadian Journal of Program Evaluation, 28(1), pp. 27–41.
23.
RengerR., & BourdeauB. (2004). Strategies for values inquiry: An exploratory case study.American Journal of Evaluation, 25(1), pp. 39–49.
24.
RengerR., & HurleyC. (2006). From theory to practice: lessons learned in the application of the ATM approach to developing logic models.Evaluation and Program Planning, 29(2), pp. 106–119.
25.
RengerR., FoltysovaJ., IenusoS., RengerJ., & BoozeW. (2017). Evaluating system cascading failures.Evaluation Journal of Australasia, 17(2), pp. 29–36.
26.
RengerR., McPhersonM., Kontz-BartelsT., & BeckerK. (2016). Process flow mapping for systems improvement: Lessons learned.The Canadian Journal of Program Evaluation, 31(1), pp. 109–121.
27.
RengerR., WoodS., WilliamsonS., & KrappS. (2012). Systemic evaluation, impact evaluation, and logic models.Evaluation Journal of Australasia, 11(2), pp. 24–30.
SandersJ. R. (1994). The program evaluation standards: how to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: SAGE Publications.
30.
ShneidermanB., FischerG., CzerwinskiM., MyersB., & ResnickM. (2005). Creativity Support Tools: A workshop sponsored by the National Science Foundation.Washington, D.C., June 13-14.
31.
SpaiteD. W., HanlonT., CrissE. A., ValenzuelaT. D., WrightA. L., KeelyK. T., & MeislinH. W. (1990). Prehospital cardiac arrest: the impact of witnessed collapse and bystander CPR in a metropolitan emergency medical service system with short response times.Annals of Emergency Medicine, 19(11), pp. 1264–1269.
ThomasD. R. (2006). A general inductive approach for analyzing qualitative evaluation data.American Journal of Evaluation, 27(2), pp. 237–246.
34.
WehipeihanaN. (2011). He Oranga Poutama case study: Applying developmental evaluation and systems thinking in indigenous contexts. International conference on systemic approaches in evaluation, Eschborn, Germany, January 25-26. Retrieved 26 July 2017 from: http://www.evaluation-conference.de/downloads/58_A4_Presentation_2.pdf
35.
WeissC. H. (1995). Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. In ConnellJ. P., KubischA. C., SchorrL. B., and WeissC. H. (eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65–92). Washington, DC: Aspen Institute.
36.
WilliamsJ. C. (2015). A systems thinking approach to analysis of the patient protection and affordable care act.Journal of Public Health Management and Practice, 21(1), pp. 6–11. doi:10.1097/PHH.0000000000000150
37.
WilliamsB., & HummelbrunnerR. (2010). Systems concepts in action: a practitioner's toolkit.Stanford, CA: Stanford University Press.
