An Integrated MERLA (Monitoring,Evaluation,Research,Learning,and Adapting) Framework for Evidence-Based Program Improvement

Abstract

Monitoring, evaluation, and research activities generate important data, but they often fail to change policies or programs. In addition, local program staff and partners often feel disconnected from these activities, which undermines their ownership of data and results. To bridge the gaps between monitoring, evaluation, and research and to give them a common purpose through locally owned, actionable learning, we developed an integrated Monitoring, Evaluation, Research, Learning, and Adapting (MERLA) framework. Implemented and refined in over 20 countries, the framework integrates data-gathering methodologies with collaborative techniques for translating evidence into knowledge that influences program practice and government policy. In this publication, we describe the MERLA framework and present two case studies demonstrating how its application enables learning and adaptation led by in-country teams.

Keywords

monitoring evaluation research learning adapting evidence-based local ownership global health international development

Although most global health programs gather data associated with their work, all too often, they do so without defining how those data will influence future program activities and policies. Historically, donors dictated the routine monitoring and evaluation (M&E) frameworks that program implementors and evaluators used to track progress, and few programs gathered data beyond those mandates (Edmunds & Marchant, 2008; Shukla et al., 2016). Over the past few decades, however, more program implementers and evaluators have cooperated on developmental evaluations and research studies prompted by gaps in M&E data (Bamberger, 2000; Copestake et al., 2018; Patton, 2010, 2016; Pouw et al., 2017; Rey et al., 2014). Several teams of researchers, evaluators, implementers, and donors have highlighted best practices and frameworks for improving evaluation and research and for using results to improve programs (Kim et al., 2018; Nkwake & Morrow, 2016; Oliver et al., 2014; Theobald et al., 2018). More broadly, organizations have developed guidance empowering local staff and partners to use data for programmatic adaptations. These approaches and tools include learning agendas; data to action frameworks; pause and reflect sessions; and collaborating, learning, and adapting (CLA) guidance documents (Edelstein et al., 2018; U.S. Agency for International Development [USAID], n.d.; WHO, 2018; Zambruni et al., 2017).

Despite the progress on the individual components of M&E, research, learning, and adapting, however, programs often still conduct these activities in isolation (AbouZahr & Macfarlane, 2019; Edmunds & Marchant, 2008; Rey et al., 2014). Many program teams still see M&E and research as time-consuming burdens rather than useful tools (Holvoet et al., 2012; LaRocca et al., 2012; Oliver et al., 2014; Shukla et al., 2016). In particular, they feel that researchers prioritize journal publications over program improvements (AbouZahr & Macfarlane, 2019; Ridde, 2016). Compounding this disconnect between those who implement programs and those who gather data about them, M&E and research teams often consist of experts from wealthier donor countries who “parachute” into programs rather than of people with deep ties to the program country (Kalibala & Nutley, 2019; Ridde, 2016). This approach inhibits the development of in-country researchers’ and evaluators’ skills and undercuts their ownership of the results. In turn, these barriers impede the long-term quality, usefulness, and sustainability of M&E and research (Kalibala & Nutley, 2019; Lescano et al., 2019; Schwerdtle et al., 2017).

These challenges forced us, the authors, to reexamine how and why we carry out M&E and research within our programs. Through discussions with M&E, research, management, and operations staff from our health programs in over 20 countries, as well as through literature reviews, we identified two common requirements. First, we needed a framework for integrating information from M&E and research for use in program adaptation and policy development (Bill & Melinda Gates Foundation, n.d.; United Nations Children’s Fund, n.d.; USAID, n.d.; Young, 2019). Second, we needed to improve the sustainability of those activities and increase in-country teams’ ownership over them, in line with the 2030 Agenda for Sustainable Development (Belle et al., 2017; Gilson & Raphaely, 2008; United Nations, 2015).

To fill those needs, we first examined approaches, best practices and recent advances in routine monitoring, developmental evaluation, research, CLA, and capacity building (Bamberger, 2000; Belle et al., 2017; Copestake et al., 2018; Edelstein et al., 2018; Gilson & Raphaely, 2008; Kim et al., 2018; Nkwake & Morrow, 2016; Oliver et al., 2014; Patton, 2010, 2016; Pouw et al., 2017; Rey et al., 2014; Theobald et al., 2018; United Nations, 2015; USAID, n.d.; WHO Department of Reproductive Health and Research et al., 2018; Young, 2019; Zambruni et al., 2017). We then synthesized and integrated these best practices to create an integrated Monitoring, Evaluation, Research, Learning, and Adapting (MERLA) framework. The MERLA framework was initially piloted in 2017 in two countries, Guinea and the Philippines, and gradually scaled up over the next 3 years to over 20 countries across sub-Saharan Africa and Asia. This deliberate, iterative scale-up allowed us to adapt and refine the framework through continuous feedback from multiple in-country stakeholders.

In this article, we describe the components of the MERLA framework and their interactions. We then provide two case studies of global health programs that used the MERLA framework to inform locally driven programmatic improvements: one, a malaria prevention program in Guinea, and the other, a family planning program in the Philippines. We began implementing the MERLA framework in these countries, and so these case studies reflect the benefits of local stewardship and ownership of in-country learning and adapting through the full MERLA framework over the longest possible period of time.

The MERLA Framework

The MERLA cycle (Figure 1) illustrates the complementary activities that produce data and the processes for using those data to improve programs and policies, which together constitute the MERLA framework. Although the MERLA cycle graphic suggests that each activity in the process occurs in sequence, programs often conduct activities in multiple parts of the framework simultaneously.

Figure 1.

RTI International’s Monitoring, Evaluation, Research, Learning, and Adapting cycle.

The process of developing and implementing a MERLA framework for a specific program begins with creating feedback loops between local champions—including program staff and leadership, technical experts, local government partners, donors, clients, and beneficiaries—as well as external experts. In selecting local champions, the program team seeks to include people who have access to the required data, will be affected by the findings of the framework, and/or have the social capital to generate support for MERLA activities. Through collaboration, these different groups develop a common understanding of not only what M&E and research data the program will collect but how the conclusions drawn from those data will be analyzed and used to improve the program’s outcomes. This collaborative effort accentuates the value of proposed MERLA activities, which in turn develops other program staff and partners into local champions. While the efforts of dedicated MERLA staff can provide the impetus for MERLA work, the backing of leadership from the beginning of the program ensures the appropriate allocation of time and financial resources to MERLA activities so that they can create meaningful improvements to program implementation. By considering MERLA from the beginning of the program, the team also includes scheduled “pause and reflect” sessions in the workplan, during which the program team and other local champions discuss program activities and MERLA work. These sessions ensure continued engagement with MERLA by all program staff, and they also create space for deeper questioning of and revisions to the program’s original theory of change.

Programs then formalize their MERLA questions into a Learning Agenda, which summarizes the gaps in knowledge for effective implementation, the questions that those gaps raise, and the proposed means of answering those questions. Those means can include informal processes, such as leveraging existing routine M&E data or having conversations with key stakeholders, but can also include more formal research studies. To best address implementation needs, even formal studies are small and quick so that they can rapidly provide evidence to supplement routine M&E data while the program is active. Such studies might examine the reasons for a trend in the M&E data, evaluate a piloted intervention to address an identified problem, or illuminate the context of a proposed activity.

During and after these data-gathering activities, teams use pause and reflect sessions, roundtable discussions, after-action reviews, and other collaborative practices to further share their data with internal and external stakeholders. During these conversations, program teams use CLA methodologies to discuss both challenging and successful activities and identify areas requiring change or growth, based on the synthesized evidence. Program teams adjust these practices as required for their context and for their specific needs, such as investigating the effectiveness of a specific activity or discussing overall program results. Through scheduled and ad hoc feedback loops, the discussions then drive changes and additions to the Learning Agenda, which in turn encourages future data gathering and use. The detailed, frequent use of data encourages local champions to see data collection activities as necessary for informing their decisions rather than as an externally imposed, burdensome requirement. By including many voices, the MERLA framework fosters a culture of openness and evolution in which those who gather and analyze the data feel connected to its use and capable of improving their program based on the evidence they generate.

The strength of the MERLA framework lies in the integrated and complementary nature of its parts. It breaks from the conventional siloed nature of M&E and research, and instead challenges programs to tie data collection directly to learning needs and to collaborate with many stakeholders to make meaningful changes based on their findings. Through iterative learning and continuous adaptation, the MERLA framework helps build flexible, effective, evidence-based programs that are owned and led by local stakeholders and collaborators.

Case Studies

The following case studies show the MERLA framework in action. In both, local stakeholders in a global health program used the MERLA framework to conduct integrated M&E and research, learn from their findings, adapt their program implementation, and influence policy discussions.

MERLA Case Study 1: Malaria Prevention During Pregnancy in Guinea

Background

The World Health Organization (WHO) recommends delivering preventive treatment using sulfadoxine–pyrimethamine for malaria during antenatal care (ANC) visits (WHO, 2016), and it recently updated its ANC guidelines (WHO, 2016) and malaria treatment guidelines (WHO, 2015) to increase the recommended number of ANC visits from four to eight. The guidance changed in part to increase the opportunities to receive intermittent preventive treatment in pregnancy (IPTp) during ANC visits starting in the second trimester, with a goal of at least three doses, each at least 1 month apart. In Guinea, a malaria-endemic country, 80.9% of pregnant women receive ANC from a trained provider. However, only 28.6% of pregnant women attend their first ANC visit in the first 4 months of pregnancy, and only 35.3% attend at least four ANC visits (Institut National de la Statistique & ICF, 2019). These data suggest that half of pregnant women in Guinea do not receive a full WHO-recommended course of IPTp.

The evidence gap

The Guinean national health information system includes data on the number of IPTp doses delivered at a facility level in the health facility monthly malaria reports. The StopPalu program and its follow-on, StopPalu+, both funded by the President’s Malaria Initiative and implemented by RTI International, use these routine data to understand the malaria situation in Guinea. In reviewing these cross-sectional data, however, the in-country MERLA team noted that the data lacked a true denominator, that is, a measure of the number of pregnant women served by the facilities, or the overall target population of the intervention. The MERLA team therefore could not use the routine data to identify areas of low or high IPTp coverage in terms of the proportion of pregnant women who received a full course of IPTp. Working with the wider program team and external technical experts, the MERLA team identified that they needed more information on how often pregnant women attended ANC visits and received the recommended three doses of IPTp. They also decided to investigate how to increase the number of people receiving the full course of IPTp during ANC visits.

Integrated monitoring, evaluation, and research to synthesize program learning

The process of understanding and increasing IPTp use in Guinea involved three integrated steps: enhancing monitoring data, holding informal community conversations, and conducting a research study. To supplement routine monitoring data, the in-country MERLA team, in concert with local stakeholders, developed a new monitoring form to track cohorts of pregnant women based on the month that they attended their first ANC visit. The first cohorts showed that the percentage of pregnant women receiving at least three doses of IPTp varied significantly by district and health facility. For example, among those who attended their first ANC visit in January 2019, Koubia district reported that 86.0% received three doses of IPTp, compared with only 24.9% in Boffa district. In addition to developing a system to improve the accuracy of routine monitoring data, in-country program staff facilitated informal learning conversations with community members and health providers to better understand why pregnant women might not attend at least four ANC visits. During these conversations, the program team learned that pregnant women did not always know that they should attend follow-up ANC visits, which illuminated an opportunity for improving IPTp coverage. The program team decided to explore communication methods to address the knowledge gaps identified in the community discussions. Since 82.9% of households in Guinea owned a mobile phone (Institut National de la Statistique & ICF, 2019), the team developed a short message service (SMS) system reminding people of their upcoming ANC visits.

To test out the system, the in-country MERLA team, supported by RTI International’s US-based MERLA experts, conducted a small pilot study in the Conakry and Kindia regions (Flueckiger et al., 2019). The study compared ANC attendance and IPTp coverage among pregnant women who attended an initial ANC visit at selected health facilities and received an SMS reminder 1 day before the next scheduled ANC visit with pregnant women who attended an initial ANC visit but did not receive an SMS reminder at comparison health facilities in the same region. The study found that women enrolled in the pilot were three times more likely to receive second dose of IPTp and 12 times more likely to receive a third dose of IPTp than those not enrolled (Flueckiger et al., 2019).

Adaptation driven by learning

After gathering contextual information through enhanced monitoring and informal conversations and piloting a solution through the research study, the in-country program team gained national support for using SMS reminders to increase IPTp coverage. First, the program team shared and discussed the results with external collaborating organizations, including USAID and the National Malaria Control Program (NMCP), using a pause and reflect session, during which evidence from the research was shared along with routine M&E data to show the possible effects of the intervention. The NMCP agreed to begin using SMS reminders for ANC visits and used the monitoring data to prioritize facilities for additional interventions.

Also, the program team reviewed the monitoring results with staff from health facilities where a smaller proportion of pregnant people received three doses of IPTp. In those conversations, the health facility staff agreed with the importance of closely monitoring IPTp coverage. They also helped to identify potential means of increasing IPTp use in addition to SMS reminders, including communication activities using community health workers and closer follow-up by providers if pregnant women did not return after their initial ANC visit. Moving forward, the program and health outcomes of these changes will appear in both the routine and enhanced monitoring data, which will then allow program teams to further refine the interventions and identify new means of preventing malaria in Guinea.

Sustainability of learning and adapting

The use of the MERLA framework gave the program team and health facility staff new insights into the populations they serve and encouraged them to utilize their data to develop interventions to address those populations’ needs. It has also given the program team the tools to lead a national conversation on using SMS reminders to reduce the risk of malaria during pregnancy. The NMCP now supports the use of SMS reminders to increase IPTp coverage, and this activity has been included in the national malaria Global Fund proposal for reducing the spread of malaria in Guinea.

MERLA Case Study 2: Improving Family Planning in the Philippines

This case study focuses on how the Legazpi city health department in the Philippines used the MERLA framework in 2018 to identify a gap in routine monitoring data, design an operations research study to fill the evidence gap, and use learnings to inform programmatic adaptations and policy decisions.

Background

The Lactational Amenorrhoea Method (LAM), commonly referred to as exclusive breastfeeding, can effectively prevent pregnancy under three conditions: (1) the person who gave birth is exclusively or nearly exclusively breastfeeding, (2) the person who gave birth is amenorrheic, or not currently experiencing menstrual periods, and (3) the baby is six months old or younger (WHO Department of Reproductive Health and Research et al., 2018). In the Philippines, the government promotes LAM as one of many effective methods of family planning commonly used and accepted by residents. The city health department in Legazpi City, Albay Province, embraced LAM because it both promotes infant health through encouraging breastfeeding and gives people control over their reproductive choices (WHO Department of Reproductive Health and Research et al., 2018). The city health department worked with LuzonHealth, a program funded by USAID and implemented by RTI international, to train community health workers in promoting LAM.

The evidence gap

In-country MERLA staff gathered routine monitoring data on family planning outcome indicators at district level, but not at the city level. Community health workers, however, shared stories of people presenting with unintended pregnancies who reported that they had been exclusively breastfeeding their less-than-6-month-old infants. The Legazpi City health department therefore wanted to know what percentage of exclusively breastfeeding people were in fact meeting the LAM criteria for protection against an unintended pregnancy.

Integrated monitoring, evaluation, and research to synthesize programmatic learning

With technical input from LuzonHealth staff, city health officials and community health workers designed and implemented two research studies to fill the evidence gap with quantitative and qualitative data. The quantitative study used a prospective design in which community health workers tracked compliance with LAM and uptake of modern family planning methods over a 9-month period among 521 people who had recently given birth. The final analysis included people who gave birth from August 1, 2016, through October 15, 2016; resided in Legazpi City during that time; and participated in at least one interview per month for 9 months, with complete and consistent responses.

The data were collected with a paper-based tracking tool developed in collaboration with the Legazpi City Health Department and community health workers. The tracking tool was pretested in selected health centers in Legazpi City for clarity and efficacy. The community health workers verbally administered the tool, which included questions about breastfeeding practices, use of modern family planning methods, return of menses, and family planning counseling referral status. Community health works recorded responses onto the questionnaires, which were then collated into a Microsoft Excel workbook. The study found that at 6 months postpartum, 87.3% of participants no longer met LAM criteria and, of those, only 30.7% reported shifting to an alternative modern family planning method. At 6 months postpartum, therefore, 56.6% of participants were not protected from pregnancy through any modern family planning method. To contextualize these findings, city health department staff conducted structured interviews and focus group discussions with community health workers and people who had recently given birth. These qualitative studies found that the participants who reported exclusive breastfeeding thought themselves protected from unintended pregnancy, even though many of them did not actually meet all three LAM criteria. Due to their assumed protection from unintended pregnancy, however, they had not adopted other modern methods of family planning.

Adaptation driven by learning

During start-up of the LuzonHealth program, we established a feedback loop and learning platform that included a range of stakeholders, including health department staff, community health workers, program beneficiaries (breastfeeding people), their partners, local NGOs working in nutrition and family planning, and the LuzonHealth MERLA team. Following completion of the LAM-related activities described above, the program facilitated a series of pause and reflect discussion workshops in which research findings were presented alongside routine health data and current guidance for community health workers and health department staff. During these sessions, health officials worked with people who could become pregnant and community health workers to develop evidence-based program adaptations and policy changes to improve modern family planning use and LAM compliance. Initially, the LuzonHealth MERLA team introduced and managed these workshops. Over the course of the sessions, however, as city health department staff realized the value of the workshops for their own learning and adaptation, they adopted and—with mentoring from the LuzonHealth MERLA team—began to lead the workshops themselves.

In the pause and reflect sessions, stakeholders discussed that although people who had recently given birth always had access to a range of modern family planning methods, the almost exclusive emphasis on LAM meant that few sought out non-LAM methods. Consequently, the city health department revised its community health worker trainings to emphasize that, in addition to LAM, community health workers should provide information and support for other modern family planning methods. That way, someone exclusively breastfeeding but not meeting all LAM criteria would be more empowered to shift to another modern family planning method.

Sustainability of learning and adapting

While the application of the MERLA framework resulted in meaningful learning and adaptations, its true success lies in the fact that it was initiated and led not by professionals in the United States or Manila, but by city health officials, community health workers, and people enrolled in the program. As these local stakeholders identified problems, they used the MERLA framework to shape their own learning and translate it into practical, locally owned change. A Legazpi city health department officer shared that “[thanks] to the Pause and Reflect sessions that we now lead, we feel empowered to understand our data, see our gaps, and identify our solutions. We feel confident to advocate for policy changes because we are champions of our learning” (V. Belbes, personal communication, October 2018). The workshops, with their focus on reflection and evidence-based decision-making, have established an ongoing positive feedback loop of data gathering, learning, and adapting within the Legazpi city health department.

Conclusion and Future Applications

Using evidence to inform program and policy improvement requires a holistic, locally led approach. The MERLA framework intentionally links M&E and research with learning and adapting through a lens of local ownership and championship, which improves implementation in ways not possible with any one piece of the framework alone. In our case studies and the application of the MERLA framework in over 20 countries, the information and understanding generated drove collaborative, locally led program adaptations and allowed teams operating at local levels to influence national and international health policies. The flexibility of the MERLA framework, which reflects the dynamic and iterative processes of data gathering, analysis, and use, makes it applicable in many contexts.

The MERLA approach requires more resources than traditional, standalone M&E due to the time required for learning activities such as pause and reflect sessions, the skills required to collaborate with a broad range of stakeholders in analyzing data, and the resources required for operational research. In addition, the relative complexity of MERLA activities can overwhelm program staff, whose day-to-day implementation responsibilities can prevent their finding time to participate in MERLA activities. To strengthen and sustain local MERLA expertise where we work, we have institutionalized peer-to-peer capacity building workshops within local governments, embedded program staff within ministries of health and vice versa, and provided seed financial assistance with matched commitments from governments and private stakeholders to sustain capacity building at the local level. Our Philippines case study illustrates the utility of our approach, as the city health department adopted the MERLA lens and took over the management of pause and reflect sessions from our program team.

To galvanize internal resources for MERLA, we integrate MERLA into program design from the beginning. Our program staff prioritize learning and adapting within their job descriptions, which enables leadership to have realistic resource requirement conversations during the program design phase. We have found particular value in engaging program management and finance staff in these discussions; when implementation, management and finance staff see the value of MERLA in improving program effectiveness and contributing to sustainable achievements, they too become MERLA champions and support the adequate allocation of resources for MERLA. In our Guinea case study, the MERLA approach allowed a small study and set of learning conversations to influence national and international policy, which helped our program team realize the value of their MERLA work beyond the program’s scope of work.

In countries where we implement MERLA, we have learned that setting up structures and processes for the smooth, collaborative flow of learning helps ensure that learning leads to program and policy change. This approach is most necessary in the common case when learning originates at the community level but policy change happens at higher regional and national levels. Application of the MERLA framework empowers community champions to influence higher level program and policy change through regular, collaborative structured Pause and Reflect sessions between all stakeholders and partners. These regular sessions foster respect and trust between local champions and policy makers, so when communities present their learnings and ideas for adaptation, policy makers value their input and allow it to influence policy.

In summary, the application of the MERLA framework requires sound technical approaches as well as intentional efforts and resources to strengthen and sustain local ownership of learning and adapting. The rewards of MERLA, however, in terms of a well-understood, locally led program that changes to meet the needs of the populations it serves and to better achieve its outcomes, make these resources worth investing. The MERLA framework presented here enables the locally-driven, sustainable development and use of monitoring, evaluation and research evidence for program and policy learning and adaptation.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Rachel D. Stelmach

References

AbouZahr

Macfarlane

S. B.

(2019). Global health data: An unfinished agenda. In Macfarlane

S. B.

AbouZahr

(Eds.), The Palgrave handbook of global health data methods for policy and practice (pp. 487–508). Palgrave Macmillan UK. https://doi.org/10.1057/978-1-137-54984-6_25

Bamberger

(2000). The evaluation of international development programs: A view from the front. American Journal of Evaluation, 21(1), 95–102. https://doi.org/10.1177/109821400002100108

Belle

S. V.

van de Pas

Marchal

(2017). Towards an agenda for implementation science in global health: There is nothing more practical than good (social science) theories. BMJ Global Health, 2(2), e000181. https://doi.org/10.1136/bmjgh-2016-000181

Bill & Melinda Gates Foundation. (n.d.). Evaluation policy. https://www.gatesfoundation.org/How-We-Work/General-Information/Evaluation-Policy

Copestake

Allan

Bekkum

W. van

Belay

Goshu

Mvula

Remnant

Thomas

Zerahun

(2018). Managing relationships in qualitative impact evaluation of international development: QuIP choreography as a case study. Evaluation, 24(2), 169–184. https://doi.org/10.1177/1356389018763243

Edelstein

Lee

L. M.

Herten-Crabb

Heymann

D. L.

Harper

D. R.

(2018). Strengthening global public health surveillance through data and benefit sharing. Emerging Infectious Diseases, 24(7), 1324–1330. https://doi.org/10.3201/eid2407.151830

Edmunds

Marchant

(2008). Official statistics and monitoring and evaluation systems in developing countries: Friends or foes? The Partnership in Statistics for Development in the 21st Century. https://paris21.org/sites/default/files/3638.pdf

Flueckiger

R. M.

Thierno

D. M.

Colaço

Guilavogui

Bangoura

Reithinger

Fitch

E. R.

Taton

J.-L.

Fofana

(2019). Using short message service alerts to increase antenatal care and malaria prevention: Findings from implementation research pilot in guinea. The American Journal of Tropical Medicine and Hygiene, 101(4), 806–808. https://doi.org/10.4269/ajtmh.19-0202

Gilson

Raphaely

(2008). The terrain of health policy analysis in low and middle income countries: A review of published literature 1994–2007. Health Policy and Planning, 23(5), 294–307. https://doi.org/10.1093/heapol/czn019

10.

Holvoet

Gildemyn

Inberg

(2012). Taking stock of monitoring and evaluation arrangements in the context of poverty reduction strategy papers: Evidence from 20 aid-dependent countries in sub-Saharan Africa (SSRN Scholarly Paper ID 2161148). Social Science Research Network. https://doi.org/10.1111/j.1467-7679.2012.00597.x

11.

Institut National de la Statistique, & ICF. (2019). Guinea demographic and health survey (EDS V) 2016-18. https://dhsprogram.com/publications/publication-fr353-dhs-final-reports.cfm

12.

Kalibala

Nutley

(2019). Engaging stakeholders, from inception and throughout the study, is good research practice to promote use of findings. AIDS and Behavior, 23(2), 214–219. https://doi.org/10.1007/s10461-019-02574-w

13.

Kim

Wilcher

Petruney

Krueger

Wynne

Zan

(2018). A research utilisation framework for informing global health and development policies and programmes. Health Research Policy and Systems, 16(1), 9. https://doi.org/10.1186/s12961-018-0284-2

14.

LaRocca

Yost

Dobbins

Ciliska

Butt

(2012). The effectiveness of knowledge translation strategies used in public health: A systematic review. BMC Public Health, 12(1), 751. https://doi.org/10.1186/1471-2458-12-751

15.

Lescano

A. G.

Cohen

C. R.

Raj

Rispel

Garcia

P. J.

Zunt

J. R.

Hamer

D. H.

Heimburger

D. C.

Chi

B. H.

A. I.

Bukusi

E. A.

(2019). Strengthening mentoring in low- and middle-income countries to advance global health research: An overview. The American Journal of Tropical Medicine and Hygiene, 100(1_Suppl), 3–8. https://doi.org/10.4269/ajtmh.18-0556

16.

Nkwake

A. M.

Morrow

(2016). Clarifying concepts and categories of assumptions for use in evaluation. Evaluation and Program Planning, 59, 97–101. https://doi.org/10.1016/j.evalprogplan.2016.05.014

17.

Oliver

Innvar

Lorenc

Woodman

Thomas

(2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research, 14(1), 2. https://doi.org/10.1186/1472-6963-14-2

18.

Patton

M. Q

. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.

19.

Patton

M. Q.

(2016). What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. American Journal of Evaluation, 37(2), 250–265. https://doi.org/10.1177/1098214015626295

20.

Pouw

Dietz

Bélemvire

de Groot

Millar

Obeng

Rijneveld

Van der Geest

Vlaminck

Zaal

(2017). Participatory assessment of development interventions: Lessons learned from a new evaluation methodology in Ghana and Burkina Faso. American Journal of Evaluation, 38(1), 47–59. https://doi.org/10.1177/1098214016641210

21.

Rey

Tremblay

M.-C.

Brousselle

(2014). Managing tensions between evaluation and research: Illustrative cases of developmental evaluation in the context of research. American Journal of Evaluation, 35(1), 45–60. https://doi.org/10.1177/1098214013503698

22.

Ridde

(2016). Need for more and better implementation science in global health. BMJ Global Health, 1(2), e000115. https://doi.org/10.1136/bmjgh-2016-000115

23.

Schwerdtle

Morphet

Hall

(2017). A scoping review of mentorship of health personnel to improve the quality of health care in low and middle-income countries. Globalization and Health, 13(1), 77. https://doi.org/10.1186/s12992-017-0301-1

24.

Shukla

Teedon

Cornish

(2016). Empty rituals? A qualitative study of users’ experience of monitoring & evaluation systems in HIV interventions in western India. Social Science & Medicine, 168, 7–15. https://doi.org/10.1016/j.socscimed.2016.08.041

25.

Theobald

Brandes

Gyapong

El-Saharty

Proctor

Diaz

Wanji

Elloker

Raven

Elsey

Bharal

Pelletier

Peters

D. H.

(2018). Implementation research: New imperatives and opportunities in global health. The Lancet, 392(10160), 2214–2228. https://doi.org/10.1016/S0140-6736(18)32205-0

26.

United Nations Children’s Fund. (n.d.). AGORA. https://agora.unicef.org/

27.

United Nations. (2015). Transforming our world: The 2030 agenda for sustainable development. https://sustainabledevelopment.un.org/post2015/transformingourworld

28.

U.S. Agency for International Development. (n.d.). CLA tool kit [Text]. USAID Learning Lab. https://usaidlearninglab.org/cla-toolkit

29.

World Health Organization. (2015). Guidelines for the treatment of malaria (3rd ed.). https://apps.who.int/iris/bitstream/handle/10665/162441/9879241549127_eng.pdf?sequence=1

30.

World Health Organization. (2016). WHO recommendations on antenatal care for a positive pregnancy experience. World Health Organization. https://apps.who.int/iris/bitstream/handle/10665/250796/9789241549912-eng.pdf

31.

World Health Organization. (2018). Driving progress towards rabies elimination: Results of Gavi’s Learning Agenda on rabies and new WHO position on rabies immunization. Meeting Report, 1-3 May 2018, Kathmandu, Nepal. https://apps.who.int/iris/bitstream/handle/10665/279494/WHO-CDS-NTD-NZD-2019.01-eng.pdf

32.

World Health Organization Department of Reproductive Health and Research, Johns Hopkins Bloomberg School of Public Health/Center, & for Communication Programs. (2018). Family planning: A global handbook for providers (2018 update). CCP and WHO. https://apps.who.int/iris/bitstream/handle/10665/260156/9780999203705-eng.pdf?sequence=1

33.

Young

(2019). How USAID is building the evidence base for knowledge management and organizational learning. Knowledge Management for Development Journal, 14(2), 60–82.

34.

Zambruni

J. P.

Rasanathan

Hipgrave

Miller

N. P.

Momanyi

Pearson

Rio

Romedenne

Singh

Young

Peterson

(2017). Community health systems: Allowing community health workers to emerge from the shadows. The Lancet Global Health, 5(9), e866–e867. https://doi.org/10.1016/S2214-109X(17)30268-1