Sage Journals: Discover world-class research

Abstract

While evaluations play a critical role in accounting for and learning from context, it is unclear how evaluations can take account of climate change. Our objective was to explore how climate change and its interaction with other contextual factors influenced One Health food safety programs. To do so, we integrated questions about climate change into a qualitative evaluation study of an ongoing, multi-sectoral program aiming to improve pork safety in Vietnam called SafePORK. We conducted remote interviews with program researchers (n = 7) and program participants (n = 23). Based on our analysis, researchers believed climate change had potential impacts on the program but noted evidence was lacking, while program participants (slaughterhouse workers and retailers) shared how they were experiencing and adapting to the impacts of climate change. Climate change also interacted with other contextual factors to introduce additional complexities. Our study underscored the importance of assessing climate factors in evaluation and building adaptive capacity in programming.

Keywords

climate change complex program food safety One Health research on evaluation

Introduction

Food safety risks are exacerbated by shocks to the food system, including environmental degradation, market distortions, political upheavals, and now the COVID-19 pandemic (Fanzo et al., 2020). Increasingly coming into sharper focus is the complex challenge of climate change, which has multifaceted impacts on food-borne disease (Kerr et al., 2022). For instance, rising temperatures can promote the growth and survival of food-borne pathogens (WHO, 2021). And during flooding, food can become unsafe to eat if the food encounters contaminated floodwater (Mbow et al., 2019). The 2021 report on Food Security and Nutrition in the World highlights that the number of people experiencing food insecurity is increasing after decades of steady declines, contributed by, among other factors, climate change (FAO et al., 2021). These shocks underscore the need for holistic action that addresses the different drivers of food safety risk.

Multicomponent programs focused on health promotion, incentives for behavioral change, and/or the enabling environment show promise in addressing the different drivers of food safety risks (Grace, 2017). Such programs can be considered complex programs,¹ defined as programs with “several interacting components” (Moore et al., 2014). Complex programs exhibit a number of characteristics, including, but not limited to: the number of components involved; the number of groups, settings, or levels targeted; and, the level of flexibility of the program (Hawe, 2015). Evaluating the progress and performance of complex programs can provide important insights to help programs achieve their food safety objectives (Hendriks, 2020). Several frameworks exist to guide such evaluations (e.g. Craig et al., 2018; Paparini et al., 2021; Skivington et al., 2021).

Evaluators have long recognized that the success of a complex program depends on context, of which there are many different understandings (Rog, 2012). Context can be defined as the conditions—social, cultural, political, environmental, economic—within which programs take place (Craig et al., 2018; Pawson and Tilley, 1997). These conditions are often referred to as contextual factors. Because there are different dimensions of context, which can manifest similarly across programs, there have been attempts to classify context. For example, Pawson (2013) operationalized context by classifying it at four different levels: individual, interpersonal relationships, institutional setting, and infrastructure. In this article, we adopt the definition of context in implementation research referring to both broader context (e.g. social, political) within which an organization resides and the setting through which implementation process will proceed (e.g. interpersonal, institutional) (Damschroder et al., 2009). Assessing the influence of context on program implementation and outcomes is important for understanding how programs work, why they sometimes fail, and how they can be adapted to different settings (Moore et al., 2014; Skivington et al., 2021).

An important yet often overlooked factor introducing additional complexity² within already complex food safety programs is external shocks (Fanzo et al., 2021; Hendriks, 2020). Climate change is increasingly placing large pressures on the social and natural systems within which complex food safety programs are situated. These pressures can have profound implications on programming; they can affect the phenomenon of food safety, interrupt program delivery, and undermine program effectiveness (Lam et al., 2021a; Mbow et al., 2019). External shocks are rarely considered in programming perhaps because they are perceived to be unlikely (Craig et al., 2018). However, by assessing the influences of shocks on programs, evaluations can both generate important contextual insights and potentially contribute to shock-resilient programs. For instance, evaluations could suggest recommendations to respond to climate change, that if feasible could lead to climate action integrated into future programs.

While evaluations can help to consider context, how evaluations consider the influence of climate change on food safety and other development programs remains understudied (Hendriks, 2020; Lam et al., 2021b). The observed risks of climate change—which are projected to intensify this century—make it important to address climate change in program planning and evaluation (Intergovernmental Panel on Climate Change [IPCC], 2022). As such, the goal of this study was to better understand how key actors involved in food safety programs experience and respond to climate change. To do so, we drew on a qualitative evaluation study of an ongoing program aiming to improve pork safety in Vietnam called SafePORK. Specifically, our objective was to characterize how climate change and its interaction with other contextual factors influence SafePORK. Through this, we identified insights to help account for this influence in evaluation.

Methods

Food systems in Vietnam

Understanding how food safety programs in Vietnam work requires an understanding of Vietnamese food systems and their context (Table 1). Because many Vietnamese have moved from the status of “having enough food to eat” to that of “needing safe food to eat” (Nguyen-Viet et al., 2017), food safety has become one of the most pressing issues among citizens (The World Bank, 2016). Repeated episodes of unsafe food practices have received widespread media attention in Vietnam (Nguyen-Viet et al., 2017). A particular area of concern has been pork safety, the main animal source food in Vietnamese diets. Pork is often produced by smallholder systems and sold fresh in traditional “wet” markets. Although wet markets are the primary access point for fresh meat, wet markets often lack mechanisms for food safety control (Dang-Xuan et al., 2016).

Table 1.

Pork food safety context in Vietnam.

Examples	Description
Social	The roles of smallholder producers in the pork value chain in Vietnam are gendered. Pig collection and slaughtering activities are mostly done by men, whereas women are largely responsible for pork processing and retail (Mitchell, 2019). Because of gendered roles, pork value chain actors have differential access to credit, resources, and agricultural training (Tran et al., 2018).
Environmental	Vietnam has experienced and will continue to experience dramatic climate change impacts, which introduce challenges to Vietnamese agriculture (Tuyet Hanh et al., 2020). Temperature changes, increased weather and climate extremes, salinity intrusion, and increased rainfall variability are already threatening rice and aquaculture production in the Mekong Delta (Anh and Nghiep, 2020).
African swine fever	Since February 2019, African swine fever has caused significant pig deaths across Vietnam. Given the limited supply of pigs, pork prices more than doubled (e.g. from 40.000 to 100.000 VNĐ per kg), resulting in shifts in pork production and consumption patterns. For example, pig farmers adopted heightened awareness of where pigs were sourced (Chau, 2020).
COVID-19	Since January 2020, Vietnam has been responding to COVID-19. Attention from local authorities toward food safety programming declined as their priorities shifted to COVID-19 control. Pork value chain actors are more concerned about public health now compared to before the pandemic (ILRI, 2020).

The SafePORK program

SafePORK is a five-year program (2017–2022) aiming to reduce the burden of food-borne disease in informal markets of Vietnam through designing, piloting, and evaluating food safety interventions (Lam et al., 2020). SafePORK builds on a previous five-year program called PigRISK (2012–2017) (Lam et al., 2016). There are five core objectives of SafePORK: (1) generate actionable evidence on the efficacy of approaches for improving pork safety; (2) develop and test light–touch interventions toward food safety; (3) validate the theory of change for market-based interventions; (4) identify recommendations for enhanced engagement of pig value chain actors; and (5) build capacity in food safety risk communication (Figure 1). Of note, climate change is not an explicit objective of the program, presenting an opportunity to ask questions about climate change and reflect on our experiences in doing so. This evaluation study focused on climate change builds on a previous evaluation study aimed at capturing program outcomes on a real-time basis using Theory of Change and Outcome Mapping approaches (Lam et al., 2021c). In addition, this study contributes in-depth, qualitative insights from SafePORK participants (e.g. slaughterhouse workers, retailers), whereas the previous study focused on perspectives from SafePORK researchers only. Furthermore, this study primarily focuses on program context using an implementation science framework.

Figure 1.

SafePORK program objectives and their interrelationships.

SafePORK is guided by One Health, a conceptual framework considered promising for addressing food safety in low- and middle-income countries (LMICs) (Grace, 2017). Recognizing that human, animal, and environmental health are inextricably linked, One Health calls for multiple disciplines to work together on complex health challenges (Nguyen-Viet et al., 2022; Zinsstag et al., 2011). Responding to this call, SafePORK mobilized researchers from different fields and institutions including veterinary epidemiology (International Livestock Research Institute), public health (Hanoi University of Public Health), agricultural economics (Vietnam National University of Agriculture), and livestock and farming systems (National Institute of Animal Science). SafePORK has also been working with program participants (local authorities, slaughterhouse workers, retailers, market managers) to shape the direction of the research (Baum et al., 2006).

SafePORK locations

To date, SafePORK has taken place in three provinces (Figure 2). All SafePORK locations were selected to build on previous activities and relationships (e.g. from PigRISK). Hung Yen and Nghe An provinces represented rural areas characterized by small-scale (<10 pigs/day) and medium-scale (11–50 pigs/day) pig slaughterhouses, respectively (Dang-Xuan et al., 2017). Hoa Binh represented a mountainous province inhabited mostly by ethnic minorities. Ethnic minority groups in Hoa Binh typically farmed Ban pigs (lợn Bản), which are an indigenous breed raised on a free scavenging feeding system. Undernutrition and stunting have been widespread among ethnic minority women and children in Vietnam, requiring targeted efforts to reduce health inequities (World Bank Group, 2019).

Figure 2.

Map of Vietnam showing locations of Hung Yen, Nghe An, and Hoa Binh provinces and their proximity to Hanoi.

SafePORK project activities

The SafePORK program has been implementing multiple interventions addressing community priorities within the three locations (Figure 3). In Hung Yen and Nghe An, SafePORK has primarily focused on interventions within traditional slaughterhouses and wet markets, which were critical points in the pork value chain. In Hoa Binh, SafePORK has focused on strengthening the Ban pig value chain by linking rural smallholder farmers to informal markets. SafePORK activities have been implemented in different stages (Figure 4).

Figure 3.

Photos of SafePORK interventions. (a) Slaughterhouse intervention involving the provision of grids, boots, soap stations, educational materials, and training. (b) Market intervention involving the provision of cutting boards, disinfection sprays, trays, cloths, aprons, educational materials, and training. (c) Facilitating linkages between Ban pig producers and markets.

Figure 4.

Timeline of SafePORK program implementation.

Data collection and analysis

We developed two separate semi-structured interview guides—one for researchers and another for program participants (Table S1). We selected the Consolidated Framework for Implementation Research (CFIR) to explore the implementation and evaluation of SafePORK because of its comprehensive multilevel constructs and flexibility offered in recommendations of its use (Damschroder et al., 2009). This framework focuses primarily on implementation context and more recently implementation outcomes (Damschroder et al., 2022). The authors of CFIR note that although CFIR is a determinant framework, CFIR does not link determinants to implementation outcomes. They add that users can design studies to establish such linkages. For example, to uncover causal mechanisms as to why certain outcomes might occur within specific contexts, Yakovchenko et al. (2021) paired CFIR with realist-informed evaluation. The authors noted the combined CFIR/realist-informed approach applied to the context of complex psychosocial interventions helped to better understand why some contexts are more conducive than others to the implementation of complex interventions. This experience suggests CFIR and realist-informed evaluation could be productively combined.

To better understand the influences of context on SafePORK, we incorporated a realist lens by adding mechanisms and outcomes to existing constructs of context. We conceptualize mechanisms as participant reaction to program components (Lemire et al., 2020). Produced by mechanisms, outcomes are changes in participant attitudes, knowledge, and behavior (Pawson and Tilley, 1997). Importantly, mechanisms (generative processes) and context (that within which generative processes occur) are inextricably linked because context is involved in generative processes (Maxwell, 2004). Still, we distinguish the two because it is analytically useful to do so in building causal claims (Greenhalgh and Manzano, 2021). For researchers, we asked questions about program implementation, with a focus on contextual factors influencing implementation. For program participants, we asked questions about their experiences of participating in the program, including what changed (outcome), what influenced these changes (context), and how the program contributed to this change (mechanism).

Having a stand-alone evaluation question around climate change is considered a promising strategy for ensuring its consideration (Lam et al., 2021a). Doing so provides a starting point for prioritizing questions and answers around climate change in the evaluation. Furthermore, evaluations will also have to consider how climate change is associated with other contextual factors. In our study, we asked questions that were both retrospective (e.g. how have climate change and other factors interacted to influence the program) and forward-looking (e.g. how might program processes and outcomes change in the context of evolving and interacting risks like climate change). Our goal was not to attribute program changes to climate impacts, but to explore the ways in which climate impacts have contributed (and could contribute) to changes in program context, mechanisms, and outcomes. As such, for both groups of interviewees, we explicitly asked questions how climate change influenced their activities. First, we used the framing of weather and climate extremes with prompts for heat, flooding, and droughts; then, we asked questions about the role of climate change in intensifying weather and climate extremes. This framing of climate change aligns with the IPCC (IPCC, 2021). Of note, because this line of inquiry was situated within a larger evaluation of implementation, this questioning represents one entry point for climate integration into evaluation. While this questioning helps to ensure climate is considered throughout the evaluation, there exist other opportunities to integrate climate into evaluation processes (e.g. integrating climate into program theory) that should be tested and reflected on to advance climate-responsive evaluation practice.

Interviews were conducted between August and September 2021 via Microsoft Teams (for researchers) and phone (for program participants). For SafePORK researchers, interviews were carried out in English by the lead author (S.L.). For program participants, interviews were conducted in Vietnamese by team members (S.D.-X., T.L.). Team members had previous experience conducting qualitative interviews and received further training from SL before data collection, as well as ongoing support throughout the study. Participants were selected by team members based on occupation, age, and location, to ensure a diversity of experiences. Interviews were audio-recorded with verbal informed consent and varied in duration by group (i.e. 60–120 minutes with researchers and 15–20 minutes with program participants). Interviews conducted in Vietnamese were transcribed verbatim and then translated into English before analysis. For key quotations of program participants used in the findings, we went back to the original transcripts to ensure the nuances of Vietnamese were captured (van Nes et al., 2010).

We conducted thematic analysis to identify patterns or themes in the data (Braun et al., 2018). Specifically, we applied several sequential phases of deductive coding, with results from one phase informing the focus on coding in the subsequent phase. In phase 1, we examined contextual factors influencing the SafePORK program. To do so, we followed guidelines supporting the systematic assessment of implementation contexts in complex health programs (Damschroder et al., 2009). Building from these findings, phase 2 focused on characterizing climate change as a key contextual factor that influenced program implementation and outcomes. Specifically, we made connections between climate change and other contextual factors. To facilitate this analysis, we examined narratives around climate change, held debrief meetings with program leadership, and drew on our collective experiences from our long-term and ongoing engagement with the program. Finally, in phase 3, we explored how programming experiences varied by gender, considering that gender shapes people’s experiences with climate change and programs (Lam et al., 2019a). Regular discussions among the authors were held to co-develop the themes. NVivo^© software (QSR International, Burlington, MA, USA) was used to facilitate the coding of text segments, develop themes, and organize quotations. As a way of assuring research quality for this study, we kept an audit trail (Table S2) of decisions made during data collection and analysis. We also wrote reflective memos to document contextual information and reflect on the data collection and analysis process. To protect the identity of interviewees, while also allowing readers to see the breadth of interviews, we identified quotations from each interviewee with a number assigned. The research findings around barriers, facilitators, and outcomes of implementation were reported separately and used by the team to adjust ongoing activities (Figure S1).

A note on COVID-19 and ethics

At the time of this study, Vietnam experienced a surge in COVID-19 cases. Due to public health measures and restrictions, all interviews were conducted remotely. Rapport was also pre-established between S.D.-X., T.L., and program participants. As rapport facilitated access to research participants, the authorship team continually reflected on whether interviewees felt pressure to participate, especially during a pandemic. All interviewees verbally provided their informed consent to participate before the interview. Program participants also received compensation for their time at a rate recommended by local partners and reflected local norms. This work received research ethics approval from the University of Guelph (20-02-003) and Hanoi University of Public Health (018-110/DD-YTCC).

Results

A total of 30 program respondents were interviewed. The seven researchers (4 men, 3 women) of the SafePORK team interviewed came from public health (n = 3), agricultural economics (n = 3), veterinary science (n = 2), and farming systems (n = 1) backgrounds. The 23 program participants interviewed included slaughterhouse workers (n = 10), retailers (n = 9), local authorities (n = 3), and market managers (n = 1) from Hung Yen (n = 13), Nghe An (n = 7), and Hoa Binh provinces (n = 3). The market manager and two out of three local authorities were men. All slaughterhouse workers we interviewed were men and nearly all retailers were women (eight out of nine). Participants worked in their occupation anywhere from 2 to 35 years and engaged with SafePORK anywhere from 1–3 years. Apart from two ethnic minority interviewees in Hoa Binh, all program participants were members of the ethnic majority (n = 22) (Table S3).

In reviewing contextual factors influencing SafePORK, we developed four overarching themes: context (environmental, organizational, socio-cultural, political) and external shocks (African swine fever, COVID-19, climate change) (Table S4). Of note, there may be overlaps between context and external shocks; for example, we considered climate change to be both parts of the environmental context (if influences were anticipated) and external shock (if influences were unanticipated). The relationship between these themes is presented in Figure 5.

Figure 5.

Contextual factors and their interrelations affecting the SafePORK program.

Context

The environmental, organizational, social-cultural, and political context in which SafePORK was situated influenced SafePORK implementation and outcomes. In examining these contextual factors, this study identified climate change as an important element that shaped contextual influences. Incorporating climate change adaptation into programming—either explicitly or implicitly, directly, or indirectly—can help to reduce the impact of climate change on SafePORK.

Environmental

Features of the built environment of SafePORK interventions influenced the extent to which SafePORK was implemented. The different extents of implementation suggested complex food safety programs require tailoring of activities to various physical settings rather than a one-size-fits-all approach. For example, in Nghe An, only one grid was considered necessary to accommodate the specific physical location of small-scale traditional slaughterhouses. In Hung Yen, SafePORK co-invested with slaughterhouse owners in an additional grid to support more-intensive pork processing practiced at the medium scale. One researcher explained:

You can imagine from 2 am to 5 am, in Hung Yen they have a few hours to slaughter seven or eight pigs . . . Not like in Nghe An where there is sometimes only one pig. (Interviewee #4)

When asked whether the changing climate was a factor influencing the program, most researchers believed weather and climate extremes had potential impacts on program outcomes but noted evidence was lacking. For example, several researchers mentioned heatwaves as a possible driver of microbial growth in pork, though could not attribute this to directly climate change versus seasonal influences. Researchers also noted how flooding made it sometimes difficult to travel to communities but considered program implementation to remain relatively unchanged. Because researchers considered these impacts of climate change to be minimal, they did not develop climate risk management strategies. Researchers also added that the program encompassed many activities already and did not have the time or resources to add a climate change focus. This sentiment was captured by one researcher who said “we have a lot of things to do” (Interviewee #3).

While nothing was specifically done to improve adaptive capacity, we found some SafePORK activities likely indirectly contributed to climate change adaptation. In turn, these activities might have protected food safety from climate change impacts, thus ensuring SafePORK outcomes. For example, recalling a risk communication activity conducted earlier in 2021, one researcher alluded to the important role of keeping meat chilled during heatwaves, which reportedly increased in severity and frequency in recent years. The researcher stated:

Some small wet markets operate during the afternoon, between 4 to 7 pm. But most wet markets operate early in the morning. We encouraged consumers to buy the pork early morning and put it in the refrigerator. So that in the evening they use it for cooking. (Interviewee #2)

Organizational

SafePORK is committed to testing, adapting, and evaluating innovative strategies in dynamic environments. This organizational culture of openness to learning was identified as an important factor influencing program implementation and outcomes. For example, one researcher explained how he was surprised when a slaughterhouse owner expressed interest in co-investing in additional equipment. The researcher did not anticipate this outcome and linked it to more frequent community visits built into the program as requested by SafePORK participants. Recognizing that program activities and outcomes shift, researchers emphasized the need to systematically track changes on a real-time basis to support adaptive management. To do so, the team drew on evaluation approaches focused on outcomes, namely Theory of Change and Outcome Mapping. Applying such techniques reportedly helped researchers to develop an overview of the program theory, which in turn helped to identify areas for monitoring and outcomes and their contextual influences (e.g. climate change). One researcher explained:

We imagine how SafePORK can work in practice. How SafePORK can change the knowledge, attitudes, and practices of value chain actors, and even researchers. And then we develop a plan to monitor these changes. (Interviewee #1)

Socio-cultural

Researchers reported quickly learning that the socio-cultural context in which SafePORK interventions were put into practice influences SafePORK implementation. For example, 6 months after the launch of interventions (August 2020), researchers noted a slow uptake of intervention packages. To identify promising strategies, the team conducted formative research in Hanoi and found that the adoption of practices was quicker in urban areas than in rural areas. While risk communication was an essential component within the initial design of SafePORK, researchers emphasized the importance of developing additional community awareness activities. One researcher explained “as long as consumers are concerned about microbial contamination, slaughterhouse workers and retailers have to do it [better hygienic practice]” (Interviewee #6).

Researchers talked about how roles within the pork value chain in Vietnam were highly gendered, with pork processing viewed as a man’s job, whereas pork retailing was mostly done by women. Researchers emphasized that slaughterhouse workers and retailers had different motivations for participating in the program. For slaughterhouse workers, adopting safe hygienic practices was a way to protect their health (e.g. from rusty knives, dirty water) and the economic well-being of their families. Retailers more often than slaughterhouse workers considered adopting safe hygienic practices as important for protecting consumers’ health but noted the changing climate posed problems in doing so. According to researchers, ensuring interventions addressed different priorities of different groups helped to ensure not only implementation effectiveness but also equitable benefits.

Paying attention to gender roles also helped the team identify key individuals who can facilitate program implementation. For example, researchers shared how joint decisions were made between husband and wife in upgrading their slaughterhouse (e.g. waste management, energy efficiency), particularly in Hung Yen and Nghe An. In this case, researchers emphasized the need to collaboratively develop the intervention with both the husband and wife, and in doing so, improve uptake of the intervention. The implementation of SafePORK interventions also varied within groups of retailers and slaughterhouse workers. Researchers acknowledged that participants were not a homogeneous group, which called for tailored interventions for different participants. Comparing the difference between family-run slaughterhouses and owner-run slaughterhouses, one researcher explained:

We have implemented the intervention in two types of slaughterhouses. The first one is a private slaughterhouse which means that the owner hires the workers. But for the smaller scale slaughterhouses, the owners are also the workers, and they just use the labor from their families. The way they operate is very different. So we need to adjust the way we work with them. (Interviewee #1)

Political

For researchers, politics played a key role in program implementation. Political priorities influenced different program components differently. For example, one researcher described her experience working on a previous project involving Ban pig production in another province (Son La) and reflected: “working in Hoa Binh was easier because local authorities were concerned about Ban pigs” (Interviewee #3). In contrast, support by local authorities in Hung Yen was likely not as strong considering “the district’s and province’s orientation are toward concentrated slaughtering points” rather than “small slaughterhouses” (Interviewee #4). The many different priorities likely left little room to prioritize climate change in this context.

External shocks

External shocks interacted with one another, resulting in compounding risks that impacted different program participants differently. For example, with African swine fever, slaughterhouse workers worried about sourcing pigs unaffected by the disease; in the context of climate change, many slaughterhouse workers worried not only about African swine fever in pigs but also about climate-related impacts such as heat stress. With COVID-19, retailers commonly experienced livelihood disruptions from COVID-19 preventive measures such as temporary market closures; together with climate-related impacts, many retailers faced both climate and market shocks. These interactions have been compounding risks, which likely created potential challenges for SafePORK implementation; for instance, as program participants dealt with multiple competing concerns, they might not have prioritized food safety.

Climate-related impacts

In asking about climate change, this study identified several impacts of climate shocks. Many interviewees described how climate change impacted food safety and food safety practices, which have potential implications on various SafePORK activities across the pork value chain.

At the farm level, several researchers mentioned hearing from farmers that pigs are “more sensitive” and “get sick more frequently” from climate variation. These impacts had implications on pork value chain actors’ practices. One local authority in Hung Yen explained some farmers would “do things that are not needed” (làm những thứ không cần thiết), like using veterinary drugs, which contribute to concerns of chemical residues in pork (Interviewee #10). Because SafePORK focuses on reducing microbial risks in pork, the increase in concern over chemical contamination—and subsequent decrease in concern over microbes—could hinder SafePORK implementation.

For slaughterhouse workers, extreme periods of drought created challenges when processing pork. One slaughterhouse worker in Hoa Binh with 22 years of experience explained that because he sourced water from the mountains, there was sometimes limited water availability, especially during the dry season. And during periods of extreme flooding, researchers discussed the possibility of rain contaminating waterways through run-off from nearby farms. Researchers reported that many slaughterhouse workers used these open water sources to process pig carcasses and/or clean tools. As such, in the context of climate change and increased flooding, the use of potentially contaminated floodwater could negatively impact outcomes for slaughterhouse workers.

Retailers explained how extreme heat and rain affected food safety. This awareness shaped their practices to protect food safety. For example, retailers were concerned about how heat spoiled pork faster. One retailer in Hung Yen with 15 years of experience explained “when the weather is hot, it can make your meat worse, it is not fresh and attractive anymore” (như thời tiết oi ả, thì có thể làm miếng thịt của mình xấu hơn đi, nó không được tươi, đẹp bắt mắt nữa) (Interviewee #15). Because the heat made meat “wet,” retailers reported using cardboard to help absorb moisture. This practice was observed by researchers, who emphasized that placing meat on cardboard was unsanitary. As such, outcomes for food safety actors could be negatively impacted by climate extremes and their influence on food safety practices.

African swine fever

African swine fever had a large influence on both SafePORK implementation and outcomes. Many researchers emphasized the need for program flexibility and adaptability in response to African swine fever and other external factors. For example, in response to travel restrictions to slaughterhouses resulting from African swine fever control measures, the team shifted implementation to focus on consumers through food safety awareness activities. This shift had positive, indirect implications for program participants. Slaughterhouse workers and retailers reported improved businesses because consumers were more aware of safe food handling practices and more likely to shop at places supported by SafePORK.

African swine fever also contributed to health behavioral changes of program participants. This influence confounded the effects of SafePORK interventions introduced at the same time. For example, one slaughterhouse worker explained how he was more careful to source healthy pigs and avoids contact with dirty water and knives. The SafePORK team mentioned how it was not possible to distinguish to what extent behavior change was caused by the intervention and what was caused by African swine fever.

COVID-19

Interviewees described many changes brought on by COVID-19, such as the inability to go to the field, switching from in-person to remote communications with program participants, and a shift in priorities of local authorities from food safety COVID-19 control. These changes influenced outcomes for food safety actors. For example, the practice of retailers shifted to incorporate COVID-19 preventive measures such as masking, handwashing, and sanitizing. While these practices complemented hygienic food handling practices, retailers commonly experienced livelihood disruptions from other COVID-19 preventive measures such as temporary market closures.

COVID-19 also made it difficult to maintain interventions. Working closely with program participants was a key implementation strategy influenced by COVID-19. One researcher emphasized, “when you are in contact with slaughterhouse workers and retailers it must be continuous” (Interviewee #3). The researcher explained that without close engagement, program participants were unlikely to sustain the adoption of intervention packages.

Discussion

Although contextual factors increasingly introduce complexity to food security programs, few studies explore contextual factors as their focus of inquiry, particularly the role of climate change (Greenhalgh and Manzano, 2021; Hendriks, 2020). In this article, we add to the limited evidence base on assessments of climate change and its interactions with other contextual factors within complex food safety programs through an analysis of an ongoing program aiming to reduce food-borne diseases in informal markets of Vietnam. To support the assessment of context, we discuss two strategies: integrating climate change into programming and theorizing programs. These strategies align with those to catalyze food systems transformations (Dinesh et al., 2021; Queiroz et al., 2021).

Integrating climate change into programming

Although the importance of integrating climate considerations into food safety and other development programs is increasingly recognized (Rowe, 2019; Ssekamatte, 2018; Uitto, 2019), how such programs account for climate change remains overlooked (Hendriks, 2020; Lam et al., 2021a). In our study, asking questions about climate change—for a food safety program where climate action was not an explicit objective—helped to identify ways that climate change affected food safety and food safety practices, which had implications for SafePORK implementation and outcomes. This finding demonstrates that climate change is important in all programs, regardless of whether the program has a specific climate change goal or not. In addition, integrating climate change into programs presents an opportunity to improve the resilience of outcomes for food safety actors. Frameworks supporting the assessment of context (e.g. UK Medical Research Council/National Institute for Health Research, Consolidated Framework for Implementation Research, Context and Implementation of Complex Interventions framework) (Damschroder et al., 2009, 2022; Moore et al., 2014; Pfadenhauer et al., 2017; Skivington et al., 2021) currently do not explicitly name climate change as a feature of context. Given the influence of climate change on food safety and other development programs—which is projected to intensify this century—we call for evaluators to explicitly consider climate change; contextual assessment is one entry point for doing so.

This study captured program activities that likely contributed to climate-related advantages outside of the scope of the targeted program outcomes. Inquiring about the difference that food safety programs are making to not only health but also the climate is important for ensuring programs are working toward change in climate-responsive ways. Importantly, our inquiries into climate change uncovered vulnerabilities of pork value chain actors to the changing climate. This finding highlights the need to explicitly integrate adaptive capacity activities in complex, multicomponent food safety programs. Furthermore, research on the impacts of climate change on food safety has typically focused on the “two ends” of food systems: production and consumption (Fanzo et al., 2021; Mbow et al., 2019). This study contributed knowledge on how intermediary links within the food value chain—such as processing and retailing—are influenced by climate change. This knowledge could inform programs applying holistic action toward food safety (WHO, 2021).

In assessing climate change in a program that did not have climate action as an explicit goal, this study contributes to efforts to mainstream environmental and climate change considerations into development programming (Lebel et al., 2012; Vincent and Colenbrander, 2018; World Resources Institute (WRI), 2018). For example, the United Nations Development Programme (UNDP, 2015) created a guide to support United Nations country teams in mainstreaming climate change into country programming. The suggested mainstreaming approach recommended steps to assess national policies, plans, and programs and integrate appropriate climate measures into their design, implementation, and monitoring. By directing attention toward climate actions, such assessments could encourage food safety programs to consider future measures to respond to climate change (Lam et al., 2021a).

Mainstreaming climate change into evaluation is especially important in contexts where climate change might interact with and compound other contextual factors. For SafePORK participants, climate change interacted with African swine fever and COVID-19 to exacerbate, compound, and create new socio-ecological vulnerabilities (e.g. multiple competing concerns). To reduce the impact of external factors on food safety actors, evaluations should explore not only how different drivers of vulnerabilities shape people’s experiences with the program but also the interactions of such drivers and their co-created challenges (Sultana, 2021). And because program components can be influenced by geographical, organizational, socio-cultural, and political factors at different stages of implementation, careful attention to changes in context over time could provide important, nuanced information about the role of context (Rasul, 2021). Importantly, flexibility and adaptability need to be a feature of programs experiencing external shocks so that programs can accommodate new needs and priorities created by the shock.

Mainstreaming climate change into evaluation should consider gender and other social dimensions. Our findings showed that women—most of whom were retailers—experienced more severe disruptions to their livelihoods from climate change compared to slaughterhouse workers, which highlights how climate change has substantial gendered impacts (Mbow et al., 2019). Feminist evaluators have acknowledged “there is tremendous but largely untapped potential for evaluation to deepening understanding around gender inequities” (Hay, 2012, p. 337). Several frameworks have been developed to support the assessment of both gender and climate considerations. For example, the International Fund for Agricultural Development (IFAD, 2021) created procedures to assess environmental, social, and climate in all projects. UN Women (2021) developed guidelines to mainstream gender into climate change projects. Considering gender and other social dimensions in climate assessments might help to ensure the perspectives, needs, and priorities of diverse social groups are addressed.

Theorizing what works and when

As demonstrated by our study, several contextual factors (e.g. climate change, organizational, socio-cultural, political)—many of which were anticipated in advance by the program team—influenced SafePORK implementation and outcomes. To account for these influences, the SafePORK team found it helpful to develop a program theory, which articulates a progression of outcomes and their contextual influences along with a program’s time frame. Doing so provided a structure for identifying, tracking, and responding to changes in outcomes and context in real-time (e.g. via Outcome Mapping). While researchers have advocated for theorizing how programs work and in what context to develop complex health programs (e.g. Skivington et al., 2021), we suggest theorizing these elements on an ongoing basis could enhance program development. In documenting SafePORK’s experiences in considering context, we expand on research calling for context to be positioned from the background into the foreground of our work (Rog, 2012).

SafePORK was also affected by several external shocks (e.g. compounding risks, such as African swine fever, COVID-19, climate change), which were not anticipated by SafePORK researchers in the initial program design. While thinking through emerging risks is an important exercise, operational constraints often mean food safety programs must focus on known, predictable, and well-defined risks (Hendriks, 2020). In programming contexts where it might not be feasible to analyze risks from external shocks at the planning stage, programs might consider theory-driven approaches to evaluation (e.g. Outcome Mapping). Such evaluations are designed to document the influence of and programmatic responses to external factors within the iterative program theory. However, to better prepare for the impact of shocks, programs might consider a structured risk assessment (Brown and Dimsdale, 2021). Generally, what determines whether a risk is low or high is its likelihood and its potential severity; shocks that are highly likely but might have a minimal impact may not need to be managed, while an event that is highly unlikely but could have a substantial impact may warrant preparation (IRGC, 2018). Such assessments should also consider how shocks might interact with one another to compound risks.

Guided by a realist-informed lens, we acknowledge that climate risks and other contextual factors may interact with mechanisms to produce different outcomes for different groups of participants. Although beyond the scope of our study, a “full” realist evaluation exploring the relationships between context and mechanisms might have enhanced the explanatory strength of our study. That said, operationalizing realist evaluation fully is challenging due to several broad issues, including identifying the most salient contextual conditions, differentiating between context and mechanisms, and describing the interplay between context, mechanisms, and outcomes (Nielsen et al., 2021). To respond to the latter issue, in our study, we found that explicitly asking “how does climate change connect with other contextual factors” during data collection and analysis yielded important insights into interactions between constructs of context. The position we hold is that applying this line of inquiry could be a promising analytic strategy to explore context and mechanism relationships, provided context and mechanism constructs are clearly defined. In addition, authors of the CFIR acknowledged that interactions between constructs exist but do not provide guidance on its exploration (Damschroder et al., 2009). The updated CFIR 2.0 now includes constructs for outcomes but still does not provide guidance on exploring construct interactions (Damschroder et al., 2022). Because we found climate change interacted with other contextual factors to compound risks to implementation, the CFIR should consider adding construct interactions or suggest tools for exploring these interactions (e.g. realist-informed evaluation).

We suggest several areas for future research. First, with the increasing climate crisis, mitigation and adaptation will need to be prioritized within food safety programs, while recognizing that different food safety actors might have diverse and conflicting needs and priorities. Importantly, integrated climate-food programs must consider trade-offs, co-benefits, and synergies between adaptation, mitigation, and food safety to help ensure sustainable development (Antle and Valdivia, 2021). Second, such programs often call for multidisciplinarity; however, the process and outcomes of multidisciplinary collaborations are not well examined within food security contexts (Gaihre et al., 2019; Lam et al., 2019b; Nguyen-Viet et al., 2021). To enhance multidisciplinary collaborations toward food security under a changing climate, future research should explore strategies for working across disciplines and sectors in this area. Finally, climate change may interact with other crises such as COVID-19, resulting in compounding risks that are experienced inequitably (Watts et al., 2021). We call for evaluations to consider how converging crises and their compounding risks are addressed in programs and suggest that this process include an equity-focused approach.

Limitations

We note a couple of limitations of our study. First, our study was part of a broader process evaluation exploring barriers and facilitators to SafePORK implementation. As such, questions around context—the focus of this study—were asked alongside other process evaluation questions. While this approach might have reduced time to inquire about context fully, we found the process evaluation provided a good entry point for exploring context. Second, because interviews were conducted remotely, many program participants were joining the interview from their homes. Children, customers, pets, and/or livestock sometimes created background noise that interrupted the flow of the interview. Despite these limitations, this research enhanced our understanding of how participants experienced food safety programs and how contextual factors influenced programming.

Conclusion

Complex food safety programs—those that have multiple interacting components, actors, and levels—are often key to achieving desired food safety outcomes. However, the process and outcomes of such programs can be affected by climate change, which is a contextual factor not often accounted for in program planning and evaluation. This study explored the experiences of the researchers and program participants involved in a complex food safety program in Vietnam. Developing a program theory was identified as a key strategy in understanding and addressing the influence of contextual factors. Asking about climate change—in a program where climate action was not a specific objective—helped to identify interactions between climate change and other contextual factors influencing the program, which provided important insights for informing climate-resilient programming moving forward. This study underscores the importance of assessing climate change in evaluations and strengthening adaptive capacity in programs.

Supplemental Material

sj-docx-1-evi-10.1177_13563890231156954 – Supplemental material for How can climate change and its interaction with other compounding risks be considered in evaluation? Experiences from Vietnam

Supplemental material, sj-docx-1-evi-10.1177_13563890231156954 for How can climate change and its interaction with other compounding risks be considered in evaluation? Experiences from Vietnam by Steven Lam, Warren Dodd, Hung Nguyen-Viet, Fred Unger, Trang T. H. Le, Sinh Dang-Xuan, Kelly Skinner, Andrew Papadopoulos and Sherilee L. Harper in Evaluation

Supplemental Material

sj-docx-2-evi-10.1177_13563890231156954 – Supplemental material for How can climate change and its interaction with other compounding risks be considered in evaluation? Experiences from Vietnam

Supplemental material, sj-docx-2-evi-10.1177_13563890231156954 for How can climate change and its interaction with other compounding risks be considered in evaluation? Experiences from Vietnam by Steven Lam, Warren Dodd, Hung Nguyen-Viet, Fred Unger, Trang T. H. Le, Sinh Dang-Xuan, Kelly Skinner, Andrew Papadopoulos and Sherilee L. Harper in Evaluation

Footnotes

Acknowledgements

The authors would like to thank SafePORK researchers and communities in Hung Yen, Nghe An, and Hoa Binh for their time and willingness to share their stories and experiences.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was part of the SafePORK project funded by the Australian Centre for International Agricultural Research (ACIAR, grant number LPS/2016/143). It also received support from the CGIAR One Health Initiative, which was supported by contributors to the CGIAR Trust Fund (). Steven Lam was supported by scholarships from the Canadian Institutes of Health Research and the Canadian Evaluation Society Educational Fund.

ORCID iDs

Steven Lam

Warren Dodd

Sinh Dang-Xuan

Supplemental material

Supplemental material for this article is available online.

Notes

Steven Lam holds a PhD from the University of Guelph. He is a health researcher and program evaluator whose research interests include food security, climate change, and social equity.

Warren Dodd is an Assistant Professor at the University of Waterloo. His research examines the intersections between the social and ecological determinants of global health and development.

Hung Nguyen-Viet is the co-leader of Animal and Human Health Program at the International Livestock Research institute. His research focuses on health, agriculture, and food safety.

Fred Unger is a veterinary epidemiologist and senior scientist at the International Livestock Research Institute. His work focuses on food safety risk assessments and management.

Trang T. H. Le is a research officer at the International Livestock Research Institute. Her research focuses on infectious disease and food safety using One Health approaches.

Sinh Dang-Xuan is a post-doc scientist at the International Livestock Research Institute. His research focuses on the link between health, agriculture, and food safety.

Kelly Skinner is an Associate Professor at the University of Waterloo where she led the online Master of Health Evaluation (2016–2020) and teaches several courses on evaluation.

Andrew Papadopoulos is a Professor and Coordinator, MPH at the University of Guelph. He works with public health agencies on public health policy and administration.

Sherilee L. Harper is a Canada Research Chair in Climate Change and Health, and Associate Professor at the University of Alberta. Her research investigates public health and climate change.

References

Anh

Nghiep

(2020) New Challenges for Food Security in Vietnam. Hanoi, Vietnam: FFTC Agricultural Policy Platform.

Antle

Valdivia

(2021) Trade-off analysis of agri-food systems for sustainable research and development. Q Open 1(1): qoa005.

Baum

MacDougall

Smith

(2006) Participatory action research. Journal of Epidemiology and Community Health 60(10): 854–7.

Braun

Clarke

Hayfield

, et al. (2018) Thematic analysis. In: Liamputtong

(ed.) Handbook of Research Methods in Health Social Sciences. Singapore: Springer, pp. 1–18.

Brown

Dimsdale

(2021) Climate Risk Management for International Organizations. London: Chatham House.

Chau

(2020) Scarcity sends Vietnam pork prices to 20-year high. Available at: https://e.vnexpress.net/news/business/industries/scarcity-sends-vietnam-pork-prices-to-20-year-high-4103700.html (accessed 30 November 2021).

Craig

Di Ruggiero

Frohlich

, et al. (2018) Taking Account of Context in Population Health Intervention Research: Guidance for Producers, Users and Funders of Research. Southampton: NIHR Evaluation.

Damschroder

Aron

Keith

, et al. (2009) Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science 4: 50.

Damschroder

Reardon

Opra Widerquist

, et al. (2022) Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): The CFIR Outcomes Addendum. Implementation Science 17: 7.

10.

Dang-Xuan

Nguyen-Viet

Meeyam

, et al. (2016) Food safety perceptions and practices among smallholder pork value chain actors in Hung Yen Province, Vietnam. Journal of Food Protection 79(9): 1490–7.

11.

Dang-Xuan

Nguyen-Viet

Unger

, et al. (2017) Quantitative risk assessment of human salmonellosis in the smallholder pig value chains in urban of Vietnam. International Journal of Public Health 62: 93–102.

12.

Dinesh

Hegger

DLT

Klerkx

, et al. (2021) Enacting theories of change for food systems transformation under climate change. Global Food Security 31: 100583.

13.

Fanzo

Bellows

Spiker

, et al. (2021) The importance of food systems and the environment for nutrition. American Journal of Clinical Nutrition 113(1): 7–16.

14.

Fanzo

Covic

Dobermann

, et al. (2020) A research vision for food systems in the 2020s: Defying the status quo. Global Food Security 26: 100397.

15.

FAO, IFAD, UNICEF, et al. (2021) The State of Food Security and Nutrition in the World 2021. Rome: FAO.

16.

Gaihre

Kyle

Semple

, et al. (2019) Bridging barriers to advance multisector approaches to improve food security, nutrition and population health in Nepal: Transdisciplinary perspectives. BMC Public Health 19: 961.

17.

Grace

(2017) Food Safety and the Sustainable Development Goals. Nairobi, Kenya: ILRI.

18.

Greenhalgh

Manzano

(2021) Understanding “context” in realist evaluation and synthesis. International Journal of Social Research Methodology 25(5): 583–95.

19.

Hawe

(2015) Lessons from complex interventions to improve health. Annual Review of Public Health 36(1): 307–23.

20.

Hay

(2012) Engendering policies and programmes through feminist evaluation: Opportunities and insights. Indian Journal of Gender Studies 19(2): 321–40.

21.

Hendriks

(2020) Food Security Policy, Evaluation and Impact Assessment. New York: Earthscan.

22.

ILRI (2020) Can COVID-19 spur transformations in food safety in traditional Vietnamese pork value chains? Available at: https://www.ilri.org/news/can-covid-19-spur-transformations-food-safety-traditional-vietnamese-pork-value-chains (accessed 2 December 2021).

23.

Intergovernmental Panel on Climate Change (IPCC) (2021) Summary for policymakers. In: Masson-Delmotte

Zhai

Pirani

, et al. (eds) Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge; New York: Cambridge University Press, pp.1–33.

24.

Intergovernmental Panel on Climate Change (IPCC) (2022) Climate Change 2022: Impacts, Adaptation and Vulnerability. Cambridge; New York: Cambridge University Press.

25.

International Fund for Agricultural Development (IFAD) (2021) Ifad’s Social, Environmental and Climate Assessment Procedures (SECAP). Rome: IFAD.

26.

IRGC (2018) IRGC Guidelines for the Governance of Systemic Risks. Lausanne: IRGC.

27.

Kerr

Hasegawa

Lasco

, et al. (2022) Chapter 5: Food, fibre, and other ecosystem products. IPCC WGII sixth assessment report, IPCC, Lausanne.

28.

Lam

Barot

Nguyen-Viet

, et al. (2016) Changes in Researcher Capacity in Assessing Food Safety Risks and Value Chains: Insights from Pigrisk Team. Hanoi, Vietnam: ILRI.

29.

Lam

Dodd

Berrang-Ford

, et al. (2021a) How are climate actions evaluated? A review of United Nations food security evaluations. Global Food Security 28: 100509.

30.

Lam

Dodd

Whynot

, et al. (2019a) How is gender being addressed in the international development evaluation literature? A meta-evaluation. Research Evaluation 28(2): 158–68.

31.

Lam

Dodd

Wyngaarden

, et al. (2021b) How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review. Evaluation and Program Planning 89: 102008.

32.

Lam

Nguyen

HTT

Nguyen-Viet

, et al. (2020) Mapping Pathways toward Safer Pork in Vietnam. Hanoi, Vietnam: ILRI.

33.

Lam

Nguyen

HTT

Tuan

HNH

, et al. (2021c) Unpacking the theory behind one health food safety programs : A Vietnam case study. Frontiers in Veterinary Science 8: 763410.

34.

Lam

Thompson

Johnson

, et al. (2019b) Toward community food security through transdisciplinary action research. Action Research 19(4): 656–73.

35.

Lebel

Krittasudthacheewa

(2012) Mainstreaming Climate Change Adaptation into Development Planning. Bangkok, Thailand: UNDP.

36.

Lemire

Kwako

Nielsen

, et al. (2020) What is this thing called a mechanism? Findings from a review of realist evaluations. New Directions for Evaluation 2020(167): 73–86.

37.

Maxwell

(2004) Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher 33(2): 3–11.

38.

Mbow

Rosenzweig

Barioni

, et al. (2019) Food security. In: Shukla

Skea

Calvo Buendia

, et al. (eds) Climate Change and Land: An IPCC Special Report on Climate Change, Desertification, Land Degradation, Sustainable Land Management, Food Security, and Greenhouse Gas Fluxes in Terrestrial Ecosystems. Cambridge; New York: Cambridge University Press, pp.437–455.

39.

Mitchell

MEV

(2019) A Review of the Literature: Gender, Food Safety and the Pork Value-chain in Vietnam. Sydney, NSW, Australia: IRLI.

40.

Moore

Audrey

Barker

, et al. (2014) Process evaluation in complex public health intervention studies: The need for guidance. Journal of Epidemiology and Community Health 68(2): 101–2.

41.

Nguyen-Viet

Lam

Nguyen-Mai

, et al. (2022) Decades of emerging infectious disease, food safety, and antimicrobial resistance response in Vietnam: The role of One Health. One Health 14: 100361.

42.

Nguyen-Viet

Pham

Lam

, et al. (2021) International, transdisciplinary, and ecohealth action for sustainable agriculture in Asia. Frontiers in Public Health 9: 592311.

43.

Nguyen-Viet

Tuyet-Hanh

Unger

, et al. (2017) Food safety in Vietnam: Where we are at and what we can learn from international experiences. Infectious Diseases of Poverty 6: 39.

44.

Nielsen

Lemire

Tangsig

(2021) Unpacking context in realist evaluations: Findings from a comprehensive review. Evaluation 28(1): 91–112.

45.

Paparini

Papoutsi

Murdoch

, et al. (2021) Evaluating complex interventions in context: Systematic, meta-narrative review of case study approaches. BMC Medical Research Methodology 21: 225.

46.

Pawson

(2013) The Science of Evaluation: A Realist Manifesto. London: SAGE.

47.

Pawson

Tilley

(1997) Realistic Evaluation. London: SAGE.

48.

Pfadenhauer

Gerhardus

Mozygemba

, et al. (2017) Making sense of complexity in context and implementation: The Context and Implementation of Complex Interventions (CICI) framework. Implementation Science 12: 21.

49.

Queiroz

Norström

Downing

, et al. (2021) Investment in resilient food systems in the most vulnerable and fragile regions is critical. Nature Food 2: 546–51.

50.

Rasul

(2021) Twin challenges of COVID-19 pandemic and climate change for agriculture and food security in South Asia. Environmental Challenges 2: 100027.

51.

Rog

(2012) When background becomes foreground: Toward context-sensitive evaluation practice. New Directions for Evaluation 2012(135): 25–40.

52.

Rowe

(2019) Sustainability-ready evaluation: A call to action. New Directions for Evaluation 2019(162): 29–48.

53.

Skivington

Matthews

Simpson

, et al. (2021) A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 374: n2061.

54.

Ssekamatte

(2018) The role of monitoring and evaluation in climate change mitigation and adaptation interventions in developing countries. African Evaluation Journal 6(1): a254.

55.

Sultana

(2021) Climate change, COVID-19, and the co-production of injustices: A feminist reading of overlapping crises. Social and Cultural Geography 22(4): 447–60.

56.

The World Bank (2016) Vietnam Food Safety Risks Management Challenges and Opportunities. Washington, DC: The World Bank.

57.

Thirsk

Clark

(2017) Using qualitative research for complex interventions: The contributions of hermeneutics. International Journal of Qualitative Methods 16: 1–10.

58.

Tran

Van Elahi

Zhang

, et al. (2018) Gender differences in formal credit approaches: Rural households in Vietnam. Asian-pacific Economic Literature 32(1): 131–8.

59.

Tuyet Hanh

Huong

LTT

Huong

NTL

, et al. (2020) Vietnam climate change and health vulnerability and adaptation assessment, 2018. Environmental Health Insights 14: 1–11.

60.

Uitto

(2019) Sustainable development evaluation: Understanding the nexus of natural and human systems. New Directions for Evaluation 2019(162): 49–67.

61.

UN Women (2021) Basics of Mainstreaming Gender into Climate Change and Disaster Risk Reduction Policies. New York: UN Women.

62.

United Nations Development Programme (UNDP) (2015) Mainstreaming Climate Change in National Development Processes and UN Country Programming. New York: UNDP.

63.

van Nes

Abma

Jonsson

, et al. (2010) Language differences in qualitative research: Is meaning lost in translation? European Journal of Ageing 7(4): 313–6.

64.

Vincent

Colenbrander

(2018) Developing and applying a five step process for mainstreaming climate change into local development plans: A case study from Zambia. Climate Risk Management 21: 26–38.

65.

Watts

Amann

Arnell

, et al. (2021) The 2020 report of The Lancet Countdown on health and climate change: Responding to converging crises. The Lancet 397: 129–70.

66.

WHO (2021) Draft WHO Global Strategy for Food Safety 2022-2030. Geneva: WHO.

67.

World Bank Group (2019) Persistent Malnutrition in Ethnic Minority Communities of Vietnam : Issues and Options for Policy and Interventions. Washington, DC: World Bank Group.

68.

World Resources Institute (WRI) (2018) From Planning to Action: Mainstreaming Climate Change Adaptation into Development. Washington, DC: WRI.

69.

Yakovchenko

McCullough

Smith

, et al. (2021) Implementing a complex psychosocial intervention for unstably housed Veterans: A realist-informed evaluation case study. Implementation Research and Practice 2: 1–11.

70.

Zinsstag

Schelling

Waltner-Toews

, et al. (2011) From “one medicine” to “one health” and systemic approaches to health and well-being. Preventive Veterinary Medicine 101: 148–56.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

12.21 MB