This article presents the Renger System Test (RST) as a method for assessing whether a system evaluation approach is suitable for evaluating complex interventions. The RST has three criteria: (1) the intervention includes multiple components, (2) these components operate interdependently, and (3) their interdependence produces an outcome that no single component can achieve alone. An NIH-funded case illustrates how to apply the RST to program source documentation. Findings suggest several key terms can be used to infer whether each of the RST criteria is met. The article discusses the pitfalls of using an intervention's label to select the most appropriate evaluation approach, as well as the challenges when there is a disconnect between the funder-mandated evaluation approach and a better fit-for-purpose system approach.
AckoffR. (1994). If Russ Ackoff had given a TED talk. https://www.youtube.com/watch?v=OqEeIG8aPPk.
2.
ArnoldR. D.WadeJ. (2015). A definition of systems thinking: A systems approach. 2015 Conference on Systems Engineering Research. Procedia Computer Science, 1–10. Retrieved on August 23, 2020 from. https://www.researchgate.net/publication/273894661
3.
AtkinsonL.CollinsB. (2023). Is this strategy working?: The essential guide to strategy development, evaluation, and goal achievement. Systems Thinking Press.
4.
AtkinsonL.CollinsB. A. (2024). Using evaluative thinking to achieve better results. Strategy Magazine, 39, 8–11.
5.
CabreraD. (Ed.). (2006). Systems evaluation and evaluation systems whitepaper series. Ithaca, NY: Cornell University Dspace Open Access Repository. National Science Foundation Systems Evaluation Grant No. EREC-0535492.
FunnellS. C.RogersP. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. John Wiley & Sons.
8.
GuiseJ. M.ChangC.ButlerM.ViswanathanM.TugwellP. (2017). AHRQ Series on complex intervention systematic reviews—Paper 1: An introduction to a series of articles that provide guidance and tools for reviews of complex interventions. Journal of Clinical Epidemiology, 90, 6–10. https://doi.org/10.1016/j.jclinepi.2017.06.011
9.
HainesS. G. (2000). The systems thinking approach to strategic planning and management. St. Lucie Press.
10.
HainesS. G. (2011). Systems thinking: The new frontier: Discovering simplicity in an age of complexity. Systems Thinking Press.
11.
HargreavesM. B.PodemsD. (2012). Advancing systems thinking in evaluation: A review of four publications. American Journal of Evaluation, 33(3), 462–470. https://doi.org/10.1177/1098214011435409
12.
HenryG. T.MarkM. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293–314. https://doi.org/10.1177/109821400302400302
McLaughlinJ. A.JordanG. B. (1999). Logic models: A tool for telling your programs performance story. Evaluation and Program Planning, 22(1), 65–72. https://doi.org/10.1016/S0149-7189(98)00042-1
17.
MeadowsD. (2008). Thinking in systems: A primer. Chelsea Green Publishing.
NicholsL. (2002). Participatory program planning: Including program participants and evaluators. Evaluation and Program Planning, 25(1), 1–14. https://doi.org/10.1016/S0149-7189(01)00044-1
22.
PattonM. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford press.
23.
PattonM. Q.Campbell-PattonC. E. (2022). Utilization-focused evaluation, (5th ed.). Sage.
24.
PetticrewM. (2011). When are complex interventions ‘complex’? When are simple interventions ‘simple’?. European Journal of Public Health, 21(4), 397–398. https://doi.org/10.1093/eurpub/ckr084
25.
RengerR. (2011). Constructing and verifying program theory using source documentation. The Canadian Journal of Program Evaluation, 25(1), 51–67. https://doi.org/10.3138/cjpe.025.003
26.
RengerR. (2015). System evaluation theory (SET): A practical framework for evaluators to meet the challenges of system evaluation. Evaluation Journal of Australasia, 15(4), 16–28. https://doi.org/10.1177/1035719X1501500403
27.
RengerR. (2022). System evaluation theory: A blueprint for practitioners evaluating complex interventions operating and functioning as systems. IAP.
28.
RengerR. (2025). Systems evaluation theory: Developing and implementing an evaluation approach for complex interventions meeting the system test. Canadian Evaluation Society e-Institute.
29.
RengerR.BartelG.FoltysovaJ. (2013). The reciprocal relationship between implementation theory and program theory in assisting decision-making. The Canadian Journal of Program Evaluation, 28(1), 27–41. https://doi.org/10.3138/cjpe.028.002
30.
RengerR.FoltysovaJ.RengerJ.DonaldsonS. I.HartG.HawkinsA. (2020). Comparing and contrasting a program versus system approach to evaluation: The example of a cardiac care system. Canadian Journal of Program Evaluation, 35(2), 240–257. https://doi.org/10.3138/cjpe.68127
31.
RengerR.HurleyC. (2006). From theory to practice: Lessons learned in the application of the ATM approach to developing logic models. Evaluation and Program Planning, 29, 106–119. https://doi.org/10.1016/j.evalprogplan.2006.01.004
32.
RengerR.McPhersonM.Kontz-BartelsT.BeckerK. (2016). Process flow mapping for systems improvement: Lessons learned. The Canadian Journal of Program Evaluation, 31, 109–121. https://doi.org/10.3138/cjpe.267
33.
RengerR.RengerJ.PescudM. T.Van EckR. N.BassonM. D.RengerJ. (2024). Four suppositions about emergent properties of complex interventions that operate and function as systems: A call for evaluation research. Evaluation Journal of Australasia, 24(1), 56–69. https://doi.org/10.1177/1035719X231217335
34.
RengerR.RengerJ.Van EckR. N.BassonM. D.RengerJ. (2023). A comparative case study illustrating the influence of a system perspective on the outcome evaluation of simple versus complex interventions. Evaluation Journal of Australasia, 23(2), 101–110. https://doi.org/10.1177/1035719X231160584
35.
RogersP. J. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14, 29–48. https://doi.org/10.1177/1356389007084674
36.
SouvannasacdE.EngblomH.Van EckR.RengerR. (2022). Lessons learned in evaluating system interdependencies using qualitative methods. Evaluation Journal of Australasia, 22(2), 108–125. https://doi.org/10.1177/1035719X211066961
37.
Von BertalanffyL. (1968). General system theory. New York, 41973(1968), 40.
38.
von Thiele SchwarzU.NielsenK.EdwardsK.HassonH.IpsenC.SavageC., Simonsen Abildgaard, J., Richter, A., Lornudd, C., Mazzocato, P., & ReedJ. E. (2020). How to design, implement and evaluate organizational interventions for maximum impact: The sigtuna principles. European Journal of Work and Organizational Psychology, 30(3), 415–427. https://doi.org/10.1080/1359432X.2020.1803960
39.
WaltonM.GatesE. F.VidueiraP. (2021). Insights and future directions for systems and complexity-informed evaluation. New Directions for Evaluation, 2021(170), 159–171. https://doi.org/10.1002/ev.20459
40.
WatkinsD. C. (2017). Rapid and rigorous qualitative data analysis: The “RADaR” technique for applied research. International Journal of Qualitative Methods, 16(1), 1609406917712131. https://doi.org/10.1177/1609406917712131
WilliamsB. (2010). Fitting the key to the lock: Matching systems methods to evaluation questions. The 24th annual conference of the American Evaluation Association, San Antonio, Texas.
43.
YarbroughD. B.ShulhaL. M.HopsonR. K.CaruthersF. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd edn.). Sage Publications.
