Better Evaluation Can Improve the Effectiveness of Agreements Between Public Health Organizations and Food and Beverage Companies

Abstract

Most Americans consume diets at odds with nutrition recommendations, contributing to our ongoing epidemics of noncommunicable diseases. One strategy for accelerating progress toward healthier diets involves formal agreements between companies and nongovernmental organizations to develop new products, reformulate existing products, and implement new marketing strategies. Although the general intention for industry agreements is to generate public health benefits, their goals and activities may not align with this intention. Based on a literature review and technical assistance we provided to organizations engaged in industry agreements, we believe that increasing scientific scope and rigor in evaluations of voluntary food and beverage industry agreements could improve potential public health benefits and understanding of their actual effects. We provide recommendations for external, unconflicted funding that permits comprehensive, independent, and rigorous evaluation of voluntary industry agreements.

Introduction

Reducing exposure to disease-promoting foods and increasing consumption of healthier options are critically important to public health. Substantial research implicates poor diet as a major factor in the etiology of noncommunicable diseases such as diabetes, cardiovascular disease, stroke, and some types of cancer, which are leading causes of disability and death (U.S. Burden of Disease 2017 Diet Collaborators, 2019; U.S. Burden of Disease Collaborators et al., 2018). Consumption of ultra-processed foods is inversely associated with dietary quality (Martínez Steele et al., 2017). Moreover, high intakes of sodium, processed meats, and sugary beverages and low intake of nuts and seeds, omega-3 fatty acids, and fruits and vegetables have direct relationships with the risk of death from heart disease, stroke, and diabetes (Micha et al., 2017).

Although public health can bring science to the forefront and policy can structure incentives and safety guidelines, the food and beverage supply chain—including commodity suppliers, manufacturers, distributors, and retail—has a dominant role in transforming the food marketplace through reformulation, new products, and changes in retail, pricing, and promotion practices. Product reformulation to alter ingredient profiles can reduce health risk through improved dietary intake without altering purchasing behavior (Hawkes et al., 2015; Lewin et al., 2006; Muth et al., 2019). In addition, pricing and promotional practices can facilitate healthy choices. However, the food industry can face substantial potential costs related to reformulation, new product development, and changes in retailer infrastructure.

Voluntary Industry Agreements as a Mechanism for Change in the Food and Beverage Industries

Strategies for accelerating food supply changes that mitigate industry risk and benefit public health are needed. One emerging strategy is the practice of embedding product, pricing, and promotion of healthier products within formal arrangements between industry and public health partners. In this article, we use the term “voluntary industry agreements” to describe such collaborations, but they could also be considered a type of industry self-regulation. These types of agreements can complement or, in some cases, substitute for public policies that come in the form of regulation or guidance to industry at the national or local level. Outside of the United States, many such agreements have been made between industries and public agencies and are commonly referred to as public–private partnerships. Within the United States, the public health partners have typically been nongovernmental organizations (NGOs). In a typical agreement, the industry and public health partners negotiate specific changes in product formulation, marketing, distribution, or portfolio mix and agree to terms for an evaluation.

Evaluation as a Means to Better Understand the Potential Health Benefits of Voluntary Industry Agreements

Much remains unknown about the potential public health benefits of voluntary industry agreements. Part of this knowledge gap stems from the state of evaluation in the field. In this article, we advocate for optimizing evaluation activities in industry agreements based on findings from a literature review and in-depth interviews with industry agreement participants. The utility of this analysis is at least threefold. First, the evaluation literature on this topic to date has drawn more from public–private partnerships outside of the United States, whereas less attention has been given to showing how more rigorous evaluation practices could improve domestic industry agreements with nonprofit public health partners. Second, we examine evaluations that focus on supplier behavior rather than on effects of programs and policies that focus predominately on consumer behavior. Third, anticipating rising interest in voluntary industry agreements in a range of topic areas, such as climate change, sustainability, and safety, the identification of core evaluation practices that promote evaluator independence and rigor in a range of settings is important.

Peer-Reviewed Evaluations of Voluntary Industry Agreements

To guide our efforts, we identified and synthesized the literature on the design and evaluation of voluntary industry agreements. We searched for articles published after 2000 in PubMed, Web of Science, AGRICOLA, Food Science and Technology Abstracts, and Business Source Corporate using the following search term: industry OR industries AND agreement* AND (health* OR food* OR nutrition* OR school* OR evaluation* OR sugar sweetened beverage* OR marketing). The original searches yielded 50 articles, and we focused our analysis on the 15 articles about evaluations of specific initiatives (see Supplemental Table 1) and 20 articles that were review articles, commentaries, or opinion pieces. Overall, few industry agreement evaluations have appeared in peer-reviewed journals, and many reports tend to remain proprietary within the domains of participants. Other reports describe agreements without providing detail on evaluation per se; for example, a review article describing the U.S.-based nonprofit Partnership for a Healthier America reviewed outcomes of a range of agreements but did not describe evaluation methods (Simon et al., 2017).

The Influence of Industry Partners on Public Health Aims of Voluntary Industry Agreements

Several studies raised concerns about industry partners’ influence on the design of initiatives. Articles evaluating the planning process for initiatives in England (Hanratty et al., 2012) and Mexico (Charvel et al., 2015) found that industry participation in planning weakened public health aims. Panjwani and Caraher (2014) concluded that the public sector had only nominal control over the United Kingdom’s Public Health Responsibility Deal (PHRD) design and implementation, describing the initiative as “fundamentally flawed” (p. 163). A related concern was that business partners can appropriate secular trends into partnership goals, and disentangling the effects of an initiative from other causes can be challenging. For example, Durand et al. (2015) and Knai et al. (2018) noted that business partners set PHRD goals that they would have achieved in the absence of the initiative. Knai et al.’s (2018) comprehensive analysis of the PHRD concluded that the systems driving the initiative included strong elements that mitigated against effectiveness; although the intentions and goals supported public health ideas, many action steps reflected industry self-interest and were inconsistent with evidence-based best practices for promoting individual behavior change (Knai et al., 2018). Mello et al. (2008) suggested the School Beverage Guidelines initiative capitalized on preexisting market trends. In a more complex example, evaluation of the Healthy Weight Commitment Foundation (HWCF) indicated that the 16 participating HWCF companies substantially exceeded their 2012 goal of selling 1 trillion fewer calories (Ng et al., 2014). Counterfactual modeling (Ng & Popkin, 2014) showed that a downward trend in purchasing products from HWCF preceded the initiative, and in fact, the rate of decline during the initiative was slower than what a simple continuation of the previous trend would have produced. Thus, evaluations need to be able to distinguish whether changes result from industry agreements or are coincidental with unrelated trends.

Limited Understanding of the Attribution and Causality of Effects of Voluntary Industry Agreements

Most of the evaluations we reviewed relied on pre–post or post-only designs, which have limited or no ability to determine effect attribution. Many used standard qualitative approaches such as analysis of documents and media (Gase et al., 2015; Knai, Petticrew, Durand, Eastmure, et al., 2015; Potvin Ken et al., 2014), product labels (Petticrew et al., 2016), direct observation (Beets et al., 2014), key informant interviews (Durand et al., 2015), and mixed qualitative approaches (Mason et al., 2014). Quantitative methods were rare and included analysis of industry data on product volume, content, and composition (Wescott et al., 2012).

Achievement of planned outputs is fundamental to establishing causality from interventions. At least four process evaluations indicated that the commitment they studied was implemented with fidelity to plans (Beets et al., 2014; Gase et al., 2015; Mason et al., 2014; Wescott et al., 2012). Wescott et al. (2012) conducted an independent evaluation of an agreement between the Alliance for a Healthier Generation and the beverage industry, specifically Coca-Cola, Dr Pepper Snapple Group, Pepsi, and the American Beverage Association, to implement the Alliance’s School Beverage Guidelines. Wescott et al. (2012) showed that the agreement led to a 90% reduction in sugary beverage calories shipped to schools. It is a unique example of a peer-reviewed study of a U.S.-based agreement between industry and an NGO (Healthier Generation). The Children’s Food and Beverage Advertising Initiative (CFBAI) in Canada was successful in reducing advertising of junk food to children on some television channels but is an example of how meeting goals may not translate to public health benefit; the net impact was offset by an increase in junk food marketing across all channels (Potvin Kent et al., 2014; Potvin Kent & Wanless, 2014). Indeed, additional reports cautioned that implementation success when present should be weighed against corporate activities that undermine public health, such as activities to protect interests and avoid regulation (Barquera et al., 2013; Freedhoff, 2014; Richards et al., 2015; Ronit & Jensen, 2014). Evaluations identifying initiatives that did not meet their implementation goals can potentially provide important information for other initiatives seeking stronger outcomes. Examples include evaluations of the U.S. CFBAI (Kunkel et al., 2014) and of an industry commitment for healthier vending machines in schools in Spain (Royo-Bordonada & Martínez-Huedo, 2014). Kunkel et al. (2014) found no significant improvement in the overall nutritional quality of foods marketed to children since adoption of the CFBAI standards, and Royo-Bordonada and Martínez-Huedo (2014) found that vending machines were still packed with high-calorie, low-nutrient-dense foods and were easily accessible to primary school pupils. Both papers suggested that other policy actions beyond voluntary approaches are needed.

A review of articles on public–private partnerships (Hernandez-Aguado & Zaragoza, 2016) showed that proponents believe voluntary agreements encourage productive dialogue and can lead to improvements in industry self-regulation and product healthfulness. But the authors cautioned that there is little evidence supporting these beliefs or the premise that agreements have had positive public health outcomes. One evidentiary problem, cited frequently in our literature review, is weak evaluation (Hawkes & Harris, 2011; Hernandez-Aguado & Zaragoza, 2016; Knai, Petticrew, Durand, Scott, et al., 2015; Petticrew et al., 2013; Ronit & Jensen, 2014).

Takeaways From Recent Technical Assistance and Research

In 2018, for a project on food and beverage industry transformation, we provided technical assistance to nonprofit organizations involved with food industry agreements and also conducted interviews with 15 individuals who were current or past participants in U.S.-based voluntary industry agreements. The interview participants were from 15 food or beverage companies, evaluation firms, and foundations. The food and beverage companies included food manufacturers, retailers, and foodservice providers involved in a broad range of commitments to offer and promote healthier products; the evaluation firms were involved in conducting numerous verifications and process evaluations of food and beverage industry agreements; and the foundations had experience funding collaborative efforts to establish industry agreements and to conduct evaluation efforts (due to confidentiality commitments, additional details are suppressed).

The takeaways from those discussions illuminated several issues with evaluation. In general, evaluations should assess progress against planned accomplishments. Among interview participants, we found that although industry agreements were usually driven by specific public health concerns, evaluation typically focused on implementation and not effectiveness. For example, most did not assess outcomes like changes in purchase or consumption behavior. Although interview participants agreed that progress toward public health and market outcome goals was important to assess, they noted barriers related to cost and accessibility of proprietary data and effect attribution.

A second takeaway concerned missed opportunities to conduct qualitative formative or process evaluation. We learned that some agreements set up by interview participants were designed without the intention or potential for public health impact per se, tending to serve more as proofs of concept or as small- to medium-scale pilot efforts. Although this latter type of initiative could be evaluated formatively for progress toward qualitative goals like developing trust or infrastructure for implementation, we did not hear of examples of this. Pilots that build trust and infrastructure could facilitate more ambitious future efforts with industry partners that merit market outcome assessment, so it could be quite valuable to identify partnership approaches that support these outcomes.

We also discussed evaluator roles and responsibilities in our interviews. There was clear consensus that third-party evaluators lend credibility to agreements. In addition, we learned that the structure of the relationship between evaluator and industry partner could affect evaluation rigor, scope, and objectivity. While interview participants stressed that evaluators need to have strong working relationships with industry partners, we also heard that data collection strategies were negotiated within those relationships. As an example, industry partners paid for some of the evaluations and determined the type and extent of proprietary data they would make available. Interview participants defended the quality of industry data while recognizing that it can appear nonobjective. Budget and design limits placed by industry partners can weaken evaluators’ capacity to obtain third-party sales data or to use rigorous quasi-experimental designs like time series. We thus inferred that the nature of the working relationship between evaluator and industry partner can affect or potentially weaken or appear to weaken evaluator objectivity and independence.

Recommendations

Evaluators can help ensure voluntary industry initiatives are operating as intended, having the desired effect on food industry practices, and leading to meaningful outcomes. To do this, evaluators must provide objective, data-driven input from the outset of the planning process through the end of the project. Developmental input can include helping public health partners identify suitable industry partners and avoiding partners whose business models undermine public health goals; identifying successful approaches to communication, negotiation, and implementation that build trust and support and promote successful implementation; and setting measurable, realistic, and meaningful objectives for implementation and outcomes. Subsequently, evaluators can design appropriate process and outcome evaluations that measure progress and assess barriers and facilitators to change, measure implementation, and assess actual outcomes and impact on sales and consumer knowledge, attitudes, and behavior. Ultimately, evaluators may be able to assess the impact of multiple agreements on the food industry and consumer behavior over time.

In designing the evaluation of industry agreements, we recommend using a broader array of data sources that move beyond observational data, such as store and restaurant audits, and company-provided records, such as product formulation, product sales, and advertising (Wiecha & Muth, in press). For example, consumer focus groups and surveys can more directly assess effects of the initiative on consumer knowledge, attitudes, and behavior. Commercial data sources such as store scanner data, label data, advertising data, and social media data can be used to independently assess behaviors of consumers and industry. For larger scale efforts, government data sets can be used to assess changes in food purchases, diet quality, and health status.

To carry out these recommendations, evaluators must be independent of both public health partner and industry partner influence and adequately funded to accomplish their work. With respect to independence, third-party evaluators of industry initiatives need to work without obligations to the industry partners. But freedom from obligation to the public health partners may also be important because a public health partner may be seeking to build rapport with industry partners, and an independent evaluator can help mitigate decisions that could weaken public health intentions. We have found that many evaluations in the United States employ a model where funding and data for the evaluation come from the industry partner. This model reduces evaluator capacity to plan and execute a rigorous design using independent, third-party data and, at a minimum, gives the appearance of nonindependence from the industry partner. Alternative funding strategies that increase evaluator independence and support building capacity for rigor are needed. For example, grantmakers who support transition to a healthier food and beverage marketplace could fund evaluators directly rather than giving grants to partners that then subcontract to evaluators.

Conclusion

Currently, we have limited evidence that voluntary industry agreements cause meaningful changes in consumer behavior or public health outcomes. This is particularly concerning in cases where voluntary industry agreements may be taking the place of government policy. Better process and outcome evaluations are clearly needed, and impact evaluation should be considered if circumstances permit. In practice, developing an evaluation within the context of an industry agreement is a process that can expose participants to an array of factors that may compromise rigor, including the structure of working relationships, the control of the evaluation budget and limitations on it, and the willingness of partners to negotiate objectives that support evidence-based public health actions.

In addition, we know little about how public health partners can best negotiate for effective agreements, even though this information could have transformational effects that foster change. Voluntary industry agreements have the potential to encourage product innovation that would not occur on the same timeline if the agreement was not in place, and early corporate adopters may demonstrate to other companies that there is a market for healthy products and encourage others to make changes to their own food lines. The positive public image that agreements bring to corporate partners could encourage companies to make additional changes or encourage new companies to focus on healthful products. In addition, congenial agreements could foster productive relationships between industry and public health partners. Conversely, the accumulation of rigorous evidence may indicate that voluntary industry agreements have only limited public health benefits and constitute a poor use of public health partners’ limited time, energy, and resources.

Our assessment was based on our qualitative assessment of a limited number of stakeholder interviews and available literature on evaluation of industry initiatives. However, these limitations reinforce the main conclusion that, at this time, we must not remain in a position of simply hoping that partnerships with industry are effective. Rather, it is time to move forward with an evaluation agenda that will help shape voluntary industry agreements that contribute to improving U.S. diets and reducing consumption of foods and beverages that cause chronic disease.

Supplemental Material

Supplemental Material, sj-docx-1-aje-10.1177_1098214020979782 - Better Evaluation Can Improve the Effectiveness of Agreements Between Public Health Organizations and Food and Beverage Companies

Supplemental Material, sj-docx-1-aje-10.1177_1098214020979782 for Better Evaluation Can Improve the Effectiveness of Agreements Between Public Health Organizations and Food and Beverage Companies by Jean L. Wiecha and Mary K. Muth in American Journal of Evaluation

Supplemental Material

Supplemental Material, sj-docx-2-aje-10.1177_1098214020979782 - Better Evaluation Can Improve the Effectiveness of Agreements Between Public Health Organizations and Food and Beverage Companies

Supplemental Material, sj-docx-2-aje-10.1177_1098214020979782 for Better Evaluation Can Improve the Effectiveness of Agreements Between Public Health Organizations and Food and Beverage Companies by Jean L. Wiecha and Mary K. Muth in American Journal of Evaluation