Practitioners implementing and evaluating humanitarian programs in conflict zones face unique challenges requiring context-sensitive guidance. Conflict zones—geographical areas with militarized and non-militarized violence, widespread political instability, and state-sanctioned intimidation and genocidal violence—are rife with logistical, methodological, and ethical challenges that impact implementation and evaluation. Although challenges have been well-documented, few solutions or evaluation frameworks exist to help evaluators in conflict zones confront these challenges. This article examines how the application of implementation science frameworks, specifically, Reach, Effectiveness, Adoption, Implementation, and Maintenance and the Consolidated Framework for Implementation Research, offers useful strategies to mitigate some challenges in conducting evaluations in conflict zones. Areas for future research include collaborative team approaches to using these frameworks, ethical guidance, and reporting and dissemination.
AkinyemiOOAdebayoABasseyC, et al. (2021) Assessing community engagement in Nigeria polio eradication initiative: Application of the Consolidated Framework for Implementation Research. BMJ Open11(8): e048694.
2.
AltareCWeissWRamadanM, et al. (2022) Measuring results of humanitarian action: Adapting public health indicators to different contexts. Conflict and Health16(1): 54.
3.
AronssonILHassnainH (2021) Evaluation and ethics in contexts of fragility, conflict and violence. In: HassnainHKellyLSommaS (eds) Ethics for Evaluation. London: Routledge, 80–100.
4.
AugustinaviciusJLGreeneMCLakinDP, et al. (2018) Monitoring and evaluation of mental health and psychosocial support programs in humanitarian settings: A scoping review of terminology and focus. Conflict and Health12: 9.
5.
AutesserreS (2014) Peaceland: Conflict Resolution and the Everyday Politics of International Intervention. Cambridge: Cambridge University Press.
6.
BakraniaSBalvinNDaidoneS, et al. (2021) Impact evaluation in settings of fragility and humanitarian emergency. Innocenti Discussion Paper 2021-02. Florence: UNICEF Office of Research—Innocenti.
7.
Barbrook-JohnsonPCastellaniBHillsD, et al. (2021) Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus. Evaluation27(1): 4–17.
8.
BauerMSDamschroderLHagedornH, et al. (2015) An introduction to implementation science for the non-specialist. BMC Psychology3(1): 32.
9.
BerwickDM (2008) The science of improvement. Journal of the American Medical Association299(10): 1182–4.
10.
BlanchetKRameshAFrisonS, et al. (2017) Evidence on public health interventions in humanitarian crises. The Lancet390(10109): 2287–96.
11.
BollettinoV (2008) Understanding the security management practices of humanitarian organizations. Disasters32(2): 263–79.
12.
BoltonPLeeCHarozEE, et al. (2014) A transdiagnostic community-based mental health treatment for comorbid disorders: Development and outcomes of a randomized controlled trial among Burmese refugees in Thailand. PLoS Medicine11(11): e1001757.
13.
BornsteinL (2010) Peace and conflict impact assessment (PCIA)in community development: A case study from Mozambique. Evaluation16(2): 165–76.
14.
BowmanN (2020) Nation-to-nation in evaluation: Utilizing an indigenous evaluation model to frame systems and government evaluations. New Directions for Evaluation2020(166): 101–18.
15.
BozzoliCBruckTWaldN (2013) Evaluating programs in conflict-affected areas. Households in conflict network. Working Paper No. 142. Brighton: The Institute of Development Studies.
16.
BraithwaiteJChurrucaKLongJC, et al. (2018) When complexity science meets implementation science: A theoretical and empirical analysis of systems change. BMC Medicine16(1): 63.
17.
BrussetEDe ConingCHughesB (eds) (2016) Complexity Thinking for Peacebuilding Practice and Evaluation. New York: Springer.
18.
BushKDugganC (2013) Evaluation in conflict zones: Methodological and ethical challenges. Journal of Peacebuilding & Development8(2): 5–25.
19.
CardenFAlkinMC (2012) Evaluation roots: An international perspective. Journal of Multidisciplinary Evaluation8(17): 102–18.
ChambersDAGlasgowREStangeKC (2013) The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science8: 117.
23.
ChaudriSCordesKMillerN, et al. (2019) Humanitarian health programming and monitoring in inaccessible conflict settings: A literature review. Journal of International Humanitarian Action4: 9.
24.
ChonkaPEdle AliAStuvøyK (2022) Eyes on the ground and eyes in the sky: Security narratives, participatory visual methods and knowledge production in “danger zones.”Security Dialogue53(6): 567–88.
25.
ChouinardJACousinsJB (2013) Participatory evaluation for development: Examining research-based knowledge from within the African context. African Evaluation Journal1(1): 9.
26.
CohenFYaegerL (2021) Task-shifting for refugee mental health and psychosocial support: A scoping review of services in humanitarian settings through the lens of RE-AIM. Implementation Research and Practice2: 1–13.
27.
CorralPIrwinAKrishnanN, et al. (2020) Fragility and Conflict: On the Front Lines of the Fight against Poverty. Washington, DC: World Bank.
28.
CurranGMBauerMMittmanB, et al. (2012) Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care50(3): 217–26.
29.
DamschroderLJ (2020) Clarity out of chaos: Use of theory in implementation research. Psychiatry Research283: 112461.
30.
DamschroderLJLoweryJC (2013) Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implementation Science8: 51.
31.
DamschroderLJAronDCKeithRE, et al. (2009) Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science4(1): 50.
32.
DamschroderLJReardonCMOpra WiderquistMA, et al. (2022) Conceptualizing outcomes for use with the consolidated framework for implementation research (CFIR): The CFIR Outcomes Addendum. Implementation Science17(1): 7.
33.
DickinsonH (2006) The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future. Health & Social Care in the Community14: 375–83.
DuffieldM (2012) Challenging environments: Danger, resilience and the aid industry. Security Dialogue43(5): 475–92.
36.
DugganCBushK (2014) The ethical tipping points of evaluators in conflict zones. American Journal of Evaluation35(4): 485–506.
37.
EasterlyWPfutzeT (2008) Where does the money go? Best and worst practices in foreign aid. Journal of Economic Perspectives22(2): 29–52.
38.
EbrahimA (2016) The many faces of nonprofit accountability. In: RenzDHermanRD (eds) The Jossey-Bass Handbook of Nonprofit Leadership and Management. Hoboken, NJ: John Wiley, 102–23.
39.
EylerA (2021) Operationalization and measurement. In: EylerAA (ed.) Research Methods for Public Health. New York: Springer, 63–76.
40.
FastLA (2007) Characteristics, context and risk: NGO insecurity in conflict zones. Disasters31(2): 130–54.
41.
FastLA (2010) Mind the gap: Documenting and explaining violence against aid workers. European Journal of International Relations16(3): 365–89.
42.
FastLAFreemanCFO’NeillM, et al. (2013) In acceptance we trust? Conceptualising acceptance as a viable approach to NGO security management. Disasters37(2): 222–43.
43.
FitzpatrickJL (2012) Commentary: Collaborative evaluation within the larger evaluation context. Evaluation and Program Planning35: 558–63.
44.
FrelinJ (2016) Analysing complex conflicts: Requirements for complexity-aware conflict analysis. In: BrussetEDe ConingCHugheB (eds) Complexity Thinking for Peacebuilding Practice and Evaluation. London: Palgrave Macmillan, 207–37.
45.
FynnJFHardemanWMiltonK, et al. (2020) A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health20(1): 1000.
46.
GatesEF (2016) Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning59: 62–73.
47.
GatesEFWaltonMVidueiraP, et al. (2021) Introducing systems-and complexity-informed evaluation. New Directions for Evaluation2021(170): 13–25.
48.
GlasgowREHardenSMGaglioB, et al. (2019) RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health7: 64.
49.
GlasgowREVogtTMBolesSM (1999) Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health89(9): 1322–7.
50.
GuanTHHtutHNDavisonCM, et al. (2021) Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS One16(12): e0261470.
HaqueSSFreemanMC (2021) The applications of implementation science in water, sanitation, and hygiene (WASH) research and practice. Environmental Health Perspectives129(6): 65002.
53.
HardenSMSmithMLOryMG, et al. (2018) RE-AIM in clinical, community, and corporate settings: Perspectives, strategies, and recommendations to enhance public health impact. Frontiers in Public Health6: 71.
54.
HassnainHKellyLSommaS (eds) (2021) Evaluation in Contexts of Fragility, Conflict and Violence: Guidance from Global Evaluation Practitioners. Exeter: International Development Evaluation Association.
55.
HoltropJSEstabrooksPAGaglioB, et al. (2021) Understanding and applying the RE-AIM framework: Clarifications and resources. Journal of Clinical and Translational Science5(1): e126.
56.
HurlburtMAaronsGAFettesD, et al. (2014) Interagency Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice. Children and Youth Services Review39: 160–168. https://doi.org/10.1016/j.childyouth.2013.10.005
KwanBMMcGinnesHLOryMG, et al. (2019) RE-AIM in the real world: Use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Frontiers in Public Health7: 345.
59.
LandesSJMcBainSACurranGM (2020) Reprint of: An introduction to effectiveness-implementation hybrid designs. Psychiatry Research283: 112630.
60.
Lengnick-HallRStadnickNADicksonKS, et al. (2021) Forms and functions of bridging factors: Specifying the dynamic links between outer and inner contexts during implementation and sustainment. Implementation Science16(1): 34.
61.
LingT (2012) Evaluating complex and unfolding interventions in real time. Evaluation18(1): 79–91.
62.
MadianouMOngJCLongboanL, et al. (2016) The appearance of accountability: Communication technologies and power asymmetries in humanitarian aid and disaster recovery. Journal of Communication66(6): 960–81.
63.
Makan-LakhaP (2016) Monitoring and evaluation in peacebuilding: Reclaiming the lost opportunities for improving peacebuilding in practice. In: BrussetEDe ConingCHugheB (eds) Complexity Thinking for Peacebuilding Practice and Evaluation. London: Palgrave Macmillan, 177–204.
64.
MarkMMAllenJBGoodwinJL (2021) Stakeholder involvement in evaluation: Does it affect observers’ perceptions of an evaluation? And which stakeholder group(s) do people think should to participate?Evaluation Review45(3–4): 166–90.
65.
MarraM (2011) Some insights from complexity science for the evaluation of complex policies. In: ForssKMarraMSchwartzR (eds) Evaluating the Complex: Attribution, Contribution, and Beyond, vol. 18. New Brunswick, NJ: Transaction Publishers, 315–34.
66.
MeansARKempCGGwayi-ChoreMC, et al. (2020) Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: A systematic review. Implementation Science15(1): 17.
67.
MidgleyG (2016) Four domains of complexity. Emergence: Complexity & Organization18(2): 137–50.
68.
MoullinJCDicksonKSStadnickNA, et al. (2019) Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science14(1): 1.
69.
MoullinJCDicksonKSStadnickNA, et al. (2020) Ten recommendations for using implementation frameworks in research and practice. Implementation Science Communications1(1): 42.
70.
MurphyABiringanineMRobertsB, et al. (2017) Diabetes care in a complex humanitarian emergency setting: A qualitative evaluation. BMC Health Services Research17(1): 431.
71.
OlswangLBPrelockPA (2015) Bridging the gap between research and practice: Implementation science. Journal of Speech, Language, and Hearing Research58(6): S1818–26.
72.
Organisation for Economic Cooperation and Development (OECD) (2012) Evaluating peacebuilding activities in settings of conflict and fragility—Improving learning for results. DAC guidelines and reference series, OECD, Paris.
73.
PawsonR (2013) The Science of Evaluation: A Realist Manifesto. London: SAGE.
74.
PaytonCKumarGSKimballS, et al. (2022) A logic model framework for planning an International Refugee Health Research, Evaluation, and Ethics Committee. Health Promotion Practice23: 852–60.
75.
PhillipsTDe WetJP (2017) Towards rigorous practice: A framework for assessing naturalistic evaluations in the development sector. Evaluation23(1): 102–20.
76.
RamalingamB (2013) Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World, 1st edn. Oxford: Oxford University Press.
77.
RapportFClay-WilliamsRChurrucaK, et al. (2018) The struggle of translating science into action: Foundational concepts of implementation science. Journal of Evaluation in Clinical Practice24(1): 117–26.
78.
ReynoldsMGatesEHummelbrunnerR, et al. (2016) Towards systemic evaluation. Systems Research & Behavioural Science33: 662–73. https://doi.org/10.1002/sres.2423
79.
RiddeVGoossensSShakirS (2012) Short-term consultancy and collaborative evaluation in a post-conflict and humanitarian setting: Lessons from Afghanistan. Evaluation and Program Planning35(1): 180–8.
80.
RiddeVPérezDRobertE (2020) Using implementation science theories and frameworks in global health. BMJ Global Health5(4): e002269.
81.
RogersPJ (2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation14(1): 29–48.
82.
RosenthalJArkuREBaumgartnerJ, et al. (2020) Systems science approaches for global environmental health research: Enhancing intervention design and implementation for household air pollution (HAP) and water, sanitation, and hygiene (WASH) programs. Environmental Health Perspectives128(10): 105001.
83.
RossignoliCMGianiADi IacovoF, et al. (2017) Enhancing participatory evaluation in a humanitarian aid project. Evaluation23(2): 134–51.
84.
Sa’DaCADurochFTaitheB (2013) Attacks on medical missions: Overview of a polymorphous reality: The case of Médecins Sans Frontières. International Review of the Red Cross95(890): 309–30.
85.
SaezPKonyndykJWordenR (2021) Financing the Humanitarian Public Good: Towards a More Effective Humanitarian Financing Model. Washington, DC: Center for Global Development.
86.
SarkerMSahaAMatinM, et al. (2020) Effective maternal, newborn and child health programming among Rohingya refugees in Cox’s Bazar, Bangladesh: Implementation challenges and potential solutions. PLoS One15(3): e0230732.
87.
SchliepMEAlonzoCNMorrisMA (2017) Beyond RCTs: innovations in research design and methods to advance implementation science. Evidence-Based Communication Assessment and Intervention11(3–4): 82–98.
88.
SchonJ (2021) How narratives and evidence influence rumor belief in conflict zones: Evidence from Syria. Perspectives on Politics19(2): 539–52.
89.
StameNHawkinsPVan Den BergRD (2021) Evaluation contributing to the common good and to tackling bad: 259ethical processes in evaluation. In: Van Den BergRDHawkinsPStameN (eds) Ethics for Evaluation. London: Routledge, 258–84.
SwindleT (2020) What’s in a name: Is implementation science repackaged process evaluation? The Society for Implementation Science in Nutrition. Available at: https://www.implementnutrition.org/impsci_and_process_eval/ (accessed 21 April 2022).
92.
TabakRGKhoongECChambersDA, et al. (2012) Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine43(3): 337–50.
93.
TemboDHickeyGMontenegroC, et al. (2021) Effective engagement and involvement with community stakeholders in the co-production of global health research. British Medical Journal372: n178.
94.
TolWAAgerABizouerneC, et al. (2020) Improving mental health and psychosocial wellbeing in humanitarian settings: Reflections on research funded through R2HC. Conflict and Health14(1): 71.
UrbanJBTrochimW (2009) The role of evaluation in research-practice integration. American Journal of Evaluation30(4): 538–53.
97.
US Agency for International Development (USAID) (2006) Monitoring & evaluation in post-conflict settings. Management Systems International, Washington, DC.
VijayakumarLMohanrajRKumarS, et al. (2017) CASP—An intervention by community volunteers to reduce suicidal behaviour among refugees. International Journal of Social Psychiatry63(7): 589–97.
100.
WanzerDL (2021) What is evaluation? Perspectives of how evaluation differs (or not) from research. American Journal of Evaluation42(1): 28–46.
