A common evaluation purpose is to determine whether a policy or program was implemented as intended: referred to as formative evaluation, process evaluation, or evaluating program improvement. A well-designed formative evaluation is important in: detecting program drift; providing timely feedback to program staff to make cost-saving mid-course corrections; reassuring the sponsor that quality assurance measures are implemented to protect investments; and interpreting impact/outcome evaluation. A formative evaluation should not just gather data on deviations from an anticipated course of action, but provide recommendations for improvement. Current methods for program improvement vary in their ability to solicit targeted recommendations. Root cause analysis (RCA) is a well-established, robust methodology used in a variety of disciplines. RCA has been primarily used by evaluators operating from a theory-driven orientation to evaluate the merit and worth of a program or policy. Surprisingly, a review of the literature suggests that RCA's utility as a program improvement tool has remained largely unrecognised in evaluation. This article illustrates the application of RCA in evaluating program improvement. The conditions under which RCA might be preferred over other formative evaluation methods are discussed.
Get full access to this article
View all access options for this article.
References
1.
AdamsM., ComponationP., CzarneckiH., & SchroerB.J.1999, ‘Simulation as a tool for continuous process improvement’, in FarringtonP.A., NembhardH.B., SturrockD.T., & EvansG.W. (eds), Proceedings of the 1999 Winter Simulation Conference, ACM Press, Piscataway, New Jersey.
2.
AladwaniA.M.2001, ‘Change management strategies for successful ERP implementation’, Business Process Management Journal, vol. 7, no. 3, pp. 266–275.
3.
AnjardR.P.1998, ‘Total quality management: key concepts’, Work Study, vol. 47, no. 7, pp. 238–247.
4.
AshJ.S., & GuapponeK.P.2007, ‘Qualitative evaluation of health information exchange efforts’, Journal of Biomedical Information, vol. 40, no. 6, suppl., pp. S33–39.
5.
BondG.R.1991, ‘Variations in an assertive outreach model’, New Directions for Mental Health Services, no. 52, pp. 65–80.
6.
CaleL., & HarrisJ.2009, ‘Fitness testing in physical education: a misdirected effort in promoting healthy lifestyles and physical activity?’, Physical Education & Sport Pedagogy, vol. 14, no. 1, pp. 89–108.
7.
CaracelliV.J., & RigginL.J.C.1994, ‘Mixed-method evaluation: developing quality criteria through concept mapping’, American Journal of Evaluation, vol. 15, no. 2, pp. 139–152.
8.
CareyR.G., & SeibertJ.H.1993, ‘A patient survey system to measure quality improvement: questionnaire reliability and validity’, Medical Care, vol. 31, no. 9, pp. 834–845.
9.
Centers for Disease Control and Prevention2008, Data collection methods for program evaluation: questionnaires, Evaluation briefs, no. 14, Centers for Disease Control and Prevention, Atlanta, Georgia, viewed 3 August 2012, <http://www.cdc.gov/healthyyouth/evaluation/pdf/brief14.pdf>.
10.
ChenH.T.1990, Theory-driven evaluations, Sage, Newbury Park, California.
11.
ChienC., WangW., & ChengJ.2007, ‘Data mining for yield enhancement in semiconductor manufacturing and an empirical study’, Expert Systems with Applications, vol. 33, no. 1, pp. 192–198.
12.
ColeG.1999, ‘Advancing the development and application of theory-based evaluation in the practice of public health’, American Journal of Evaluation, vol. 20, no. 3, pp. 453–470.
13.
CreswellJ.W.2009, Research design: qualitative, quantitative, and mixed method approaches, 3rd edn, Sage, Thousand Oaks, California.
14.
De GraveW., BoshuizenH., & SchmidtH.1996, ‘Problem-based learning: cognitive and metacognitive processes during problem analysis’, Instructional Science, vol. 24, no. 5, pp. 321–341.
15.
DoggettA.M.2005, ‘Root cause analysis: a framework for tool selection’, Quality Management Journal, vol. 12, no. 4, pp. 34–45.
16.
DonaldsonS.I.2003, ‘Theory-driven program evaluation in the new millennium’, in DonaldsonS.I., & ScrivenM (eds), Evaluating social programs and problems: visions for the new millennium, Lawrence Erlbaum, Mahwah, New Jersey.
GaskellG.2000, ‘Individual and group interviewing’, in BauerM.W., & GaskellG (eds), Qualitative researching with text, image and sound: a practical handbook, Sage, London.
19.
GlesneC.2011, Becoming qualitative researchers: an introduction, Pearson Education, Boston.
HolzerP.J., HigginsJ.R., BromfieldL.M., RichardsonN., & HigginsD.J.2006, ‘The effectiveness of parent education and home visiting child maltreatment prevention programs’, Child Abuse Prevention Issues, no. 24, pp. 1–24, viewed 2 August 2012, <http://www.aifs.gov.au/nch/pubs/issues/issues24/issues24.html>.
22.
HuntingtonC., & RengerR.2003, NAO's response to the OMB-PART report: April–June, NAO News of the National AHEC Organization, Oak Creek, Wisconsin.
23.
LewisJ.2003, ‘Design issues’, in RitchieJ, & LewisJ (eds), Qualitative research practice: a guide for social science students and researchers, Sage, London.
24.
LiangY., ZhangY., JetteM., SivasubramaniumA., & SahooR.2006, ‘BlueGene/L failure analysis and prediction models’, Proceedings of the 2006 international conference on dependable systems and networks, vol 3, Philadelphia, Pennsylvania, pp. 425–434.
25.
MarkM.M., HenryG.T., & JulnesG.2000, Evaluation: an integrated framework for understanding, guiding, and improving policies and programs, Jossey-Bass, San Francisco.
26.
MayT.1997, Social research: issues, methods and process, 2nd edn, Open University Press, Buckingham, UK.
27.
McDavidJ.C., & HawthornL.R.L.2006, Program evaluation and performance measurement: an introduction to practice, Sage, Thousand Oaks, California.
28.
McLaughlinJ.A., & JordanG.B.1999, ‘Logic models: a tool for telling your program's performance story’, Evaluation and Program Planning, vol. 22, no. 1, pp. 65–72.
29.
MillsS.C., & RaganT.J.2000, ‘A tool for analyzing implementation fidelity of an integrated learning system’, Educational Technology Research and Development, vol. 48, no. 4, pp. 21–41.
30.
MorellJ.2000, ‘Internal evaluation: a synthesis of traditional methods and industrial engineering’, American Journal of Evaluation, vol. 21, no. 1, pp. 41–52.
31.
OhnoT.1988, Toyota production system, Productivity Press, Portland, Oregon.
32.
PattonM.Q.1987, How to use qualitative methods in evaluation, Sage, Newbury Park, California.
PosavacE.J.2011, Program evaluation: methods and case studies, 8th edn, Prentice Hall, Boston.
35.
RasmussenJ.1997, ‘Risk management in a dynamic society: a modeling problem’, Safety Science, vol. 27, no. 2/3, pp. 183–213.
36.
ReijersH.A., & MansarS.L.2005, ‘Best practices in business process design: an overview and qualitative evaluation of successful redesign heuristics’, The International Journal of Management Science, vol. 33, no. 4, pp. 283–306.
37.
RengerR.2010, ‘Constructing and verifying program theory using source documentation’, The Canadian Journal of Program Evaluation, vol. 25, no. 1, pp. 51–67.
38.
RengerR., & HurleyC.2006, ‘From theory to practice: lessons learned in the application of the ATM approach to developing logic models’, Evaluation and Program Planning, vol. 29, no. 2, pp. 106–119.
39.
RengerR., & TitcombA.2002, ‘A three-step approach to teaching logic models’, American Journal of Evaluation, vol. 23, no. 4, pp. 493–503.
40.
RengerR., DavisM., & GranilloB. in press, ‘Using root cause analysis (RCA) to facilitate corrective actions, after action reports (AARs), and improvement plans’, Journal of Emergency Management.
41.
RosasS.R.2005, ‘Concept mapping as technique to program theory development: an illustration using family support program’, American Journal of Evaluation, vol. 26, no. 3, pp. 389–401.
42.
SaundersR.P., EvansM.H., & JoshiP.2005, ‘Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide’, Health Promotion Practice, vol. 6, no. 2, pp. 134–147.
43.
ScrivenM.1991, Evaluation thesaurus, 4th edn, Sage, Newbury Park, California.
44.
SendersJ.W.2004, ‘FMEA and RCA: the mantras of modern risk management’, Quality and Safety in Healthcare, vol. 13, no. 4, pp. 249–250.
45.
ShernD.L., TrochimW.M.K., & LaCombC.A.1995, ‘The use of concept mapping for assessing fidelity of model transfer: an example for psychiatric rehabilitation’, Evaluation and Program Planning, vol. 18, no. 2, pp. 143–153.
46.
SpathP.2003, ‘Don't get caught in the activity trap’, Hospital Peer Review, vol. 28, no. 9, pp. 130–132.
47.
SpeelmanE.N., Lopez-RidauraS., ColomerN.A., AstierM., & MaseraO.R.2007, ‘Ten years of sustainability evaluation using the MESMIS framework: lessons learned from its application in 28 Latin American case studies’, International Journal of Sustainable Development and World Ecology, vol. 14, no. 4, pp. 345–361.
48.
TrochimW.1989, ‘An introduction to concept mapping for planning and evaluation’, A Special Issue of Evaluation and Program Planning, vol. 12, no. 1, pp. 1–16.
49.
VenkatasubramanianV., RengaswamyR., KavuriS.N., & YinK.2003, ‘A review of process fault detection and diagnosis part III: process history based methods’, Computers and Chemical Engineering, vol. 27, no. 3, pp. 327–346.
50.
WilliamsP.M.2001, ‘Techniques for root cause analysis’, paper presented at the pathology fall symposium, disaster and emergency management: knowledge gained, experience applied, Baylor University Medical Center, 2 November, Baylor University Medical Center Proceedings, vol. 14, pp. 154–157.
51.
YampolskayaS., NesmanT.M., HernandezM., & KochD.2004, ‘Using concept mapping to develop a logic model and articulate a program theory: a case example’, American Journal of Evaluation, vol. 25, no. 2, pp. 191–207.
