This article describes how system evaluation theory (SET) guided the evaluation of cardiac care response systems efficiency in seven rural United States. Specifically, the article focuses on the approach and methods used to evaluate system feedback mechanisms; one key factor affecting system efficiency. Mixed methods were applied to evaluate five criteria of system feedback efficiency: frequency, timeliness, credibility, specificity, and relevance. Examples from the cardiac care response system evaluation are used to illustrate each of the evaluation criteria. The discussion contrasts the role of the evaluator in system versus program evaluation, notes the post-hoc support of SET system attributes in affecting system efficiency, and offers additional consideration in evaluating system feedback mechanisms.
Get full access to this article
View all access options for this article.
References
1.
AshJ. S., BergM., & CoieraE. (2004). Some unintended consequences of information technology in health care: The nature of patient care information system-related errors.Journal of the American Medical Informatics Association, 11(2), pp. 104–112. doi: 10.1197/jamia.M1471
2.
BanathyB. H. (1992). A systems view of education: Concepts and principles for effective practice.Englewood Cliffs, NJ: Educational Technology.
ChenH. T. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness.Thousand Oaks, CA: Sage Publications.
8.
ChenH., HaileyD., WangN., & YuP. (2014). A review of data quality assessment methods for public health information systems.International Journal of Environmental Research and Public Health, 11(5), pp. 5170–5207. doi: 10.3390/ijerph110505170
9.
ClowD. (2012). The learning analytics cycle: closing the loop effectively. In LAK ‘12 Proceedings of the 2nd international conference on learning analytics and knowledge.New York: ACM, pp. 134–138. doi: 10.1145/2330601.2330636
10.
EisenbergM. S. (2013). Resuscitate! How your community can improve survival from sudden cardiac arrest.2nd ed.Seattle: University of Washington Press.
11.
FlynnB. B., SchroederR. G., & SakakibaraS. (1994). A framework for quality management research and an associated measurement instrument.Journal of Operations Management, 11(4), pp. 339–366. doi: 10.1016/S0272-6963(97)90004-8
GrahamB. B. (2004). Detail process charting: Speaking the language of process.Hoboken, NJ: John Wiley & Sons.
14.
GrossmanD. C., KimA., MacdonaldS. C., KleinP., CopassM. K., & MaierR. V. (1997). Urban-rural differences in prehospital care of major trauma.The Journal of Trauma: Injury, Infection, and Critical Care, 42(4), pp. 723–729.
15.
McDavidJ. C., HuseI., & HawthornL. R. L. (2013) Key concepts and issues in program evaluation and performance measurement. In Program evaluation and performance measurement: An introduction to practice, (2nd ed.). Thousand Oaks, CA: Sage.
16.
NickolsP. (2000). Standard operating procedures.The Quality Assurance Journal, 4(2), pp. 91–94. doi: 10.1002/1099-1786(200006)4: 2<91: : AID-QAJ78>3.0.CO;2-P
17.
RengerR. (2011). Constructing and verifying program theory using source documentation.The Canadian Journal of Program Evaluation, 25(1), pp. 51–67.
18.
RengerR. (2015). System Evaluation Theory (SET): A practical framework for evaluators to meet the challenges of system evaluation.Evaluation Journal of Australasia, 15(4), pp. 16–28.
19.
RengerR., McPhersonM., Kontz-BartelsT., & BeckerK. (2016). Process flow mapping for systems improvement: Lessons learned.The Canadian Journal of Program Evaluation, 31(1). Retrieved 22 October 2016 from http://hdl.handle.net/10515/sy56t0hd0
20.
RengerR., QinX., RiceD., FoltysovaJ., & RengerJ. (2016a). National data collection efforts pose challenges for many EMS agencies.EMSInsider.
21.
RengerR., QinX., RiceD., FoltysovaJ., SouvannasacdE., RengerJ., MarkwartB., HartG., & BjerkeM. (2016b). Challenges and solutions facing EMS in supporting the IOM recommendation for a national cardiac arrest registry: A system perspective. White Paper. Retrieved 22 October 2016 from https://ruralhealth.und.edu/pdf/ems-supporting-iom-recommendation.pdf
22.
SaundersR. P., EvansM. H., & JoshiP. (2005). Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide.Health Promotion Practice, 6(2), pp. 134–147. doi: 10.1177/1524839904273387
23.
SlifkinA. B., & NewellK. M. (1998). Is variability in human performance a reflection of system noise?Current Directions in Psychological Science, 7(6), pp. 170–177. doi: 10.1111/1467-8721.ep10836906
24.
WilliamsB., & HummelbrunnerR. (2010). Systems concepts in action: A practitioner's toolkit.Stanford, CA: Stanford University Press.
