Bringing Evidence to Bear on Public Health in the United States

During the past 10 years, substantial advances have been made in the core components of evidence-based public health, described by Brownson et al as “making decisions on the basis of the best available scientific evidence, using data and information systems systematically, applying program-planning frameworks, engaging the community in decision making, conducting sound evaluation, and disseminating what is learned.” ¹ For example, clearinghouses and registries for evidence-based interventions and practices have been developed for many fields (eg, What Works Clearinghouse, Results First Initiative), ^2,3 and numerous commissions and organizations have dedicated themselves to promoting evidence-based practices (eg, the Commission on Evidence-Based Policymaking and GovEx/What Works Cities). ^4,5 In addition, numerous public health planning and evaluation frameworks that prioritize evidence-based decision making have taken root and proven valuable (eg, Results-Based Accountability ⁶ and Reach, Effectiveness, Adoption, Implementation, and Maintenance frameworks ⁷ ). Most importantly, the number of evidence-based public health interventions and programs has grown, with each being a result of deliberate and systematic studies, especially randomized trials and studies using other designs that provide rigorous results about intervention and program effects.

Despite these advances, however, the gap between scientific evidence and public health practice is still large. It persists for various reasons, including political pressures, resistance to change, lack of time, and difficulty in interpreting and accessing evidence. ^8,9 One common and persistent, but sometimes overlooked, reason that scientific evidence from rigorous research has not played a greater role in public health policy and practice is the gap between the answers policy makers and practitioners need for decision making and the answers that the most rigorous and reliable studies can provide. Public health practitioners and policy makers want to know if an intervention works, how it works, if it is appropriate and acceptable for their clients and constituents, and if it is cost-effective. However, the types of studies that can answer questions about an intervention’s appropriateness, acceptability, and delivery, such as process evaluations, qualitative studies, and cost-effectiveness studies, are not generally as reliable and rigorous as other types of studies used in public health research. Randomized controlled trials (RCTs) and systematic reviews are generally regarded as the most rigorous and reliable types of studies for determining the effectiveness of an intervention, but they are rarely designed to show how an intervention works or to evaluate its acceptability and appropriateness. RCTs provide accurate results about an intervention’s effectiveness by randomizing study participants to intervention and control groups; any difference in outcomes can then be attributed to the intervention rather than to any preexisting differences between the study participants in each group. ¹⁰ However, RCTs can pose challenges for directly informing public health practice because they are often conducted using groups of individuals that differ from the target population, tend to be too short to assess long-term effects, are not always able to identify variation in effects across subgroups, and, in some cases, may not be ethical or feasible. ^11,12

Systematic reviews also have limitations. Scientists place a high value on systematic reviews because they consolidate existing evidence and reflect the combined results of multiple studies. However, current approaches to conducting systematic reviews are not adept at synthesizing evidence for interventions that are complex, variable, and context dependent or for interventions examined through various study designs—including nonrandomized designs or studies conducted in diverse populations. These complex interventions are often essential for solving today’s complicated public health problems, such as opioid addiction and obesity (eg, see Bennett et al ¹³ ), but their complexity and varied designs make them difficult to evaluate using systematic reviews. ¹⁴

It is no surprise, then, that practitioners and policy makers turn to other types of studies to find answers to their practice-related questions. However, some of these approaches have limitations that many practitioners and policy makers do not appreciate fully. A good example of this is the pre-post study design, which compares a group of participants before and after they receive the intervention under study. ¹⁵ This study design is popular because it is inexpensive and easier to conduct than RCTs, and therefore can foster a culture of measurement, data collection, and analysis. However, because most pre-post studies do not compare the group that received the intervention with a group that did not receive the intervention, they cannot rule out the possibility that the outcome would have changed whether or not the group had received the intervention. ¹⁵ The result is that practitioners may conclude too quickly that the intervention was responsible for the change in the outcome, when in fact the outcome could have changed for other reasons. For example, several studies conducted in the 1990s using a pre-post study design found significant improvements over time in various measures of literacy in families who participated in the Even Start Family Literacy Program. However, a subsequent RCT found that children and parents in the control group made similar gains. ¹⁶ Other types of studies with limitations include pilot studies that are too small to produce reliable results and nonrandomized studies that use comparison groups that are too different from the group receiving the intervention. ¹⁷ An important step for advancing evidence-based public health, then, is to identify and advance strategies for building more evidence that is accurate, reliable, and relevant.

Strategies for Building More Evidence That is Accurate, Reliable, and Relevant

One way to build evidence that is accurate, reliable, and relevant is through greater use of alternative study designs. In this section, we focus on innovative study designs that can answer important questions about intervention efficacy and effectiveness but can also answer other questions that practitioners and policy makers ask without compromising scientific rigor. A good example is the randomized encouragement design, which is particularly useful when withholding an intervention to a control group is not possible or ethical. In this design, all participants in the study are informed about the availability of the intervention, but a random subset of participants receives extra encouragement to participate (eg, a reminder to get an influenza vaccination). ¹⁸ This design can yield accurate answers about the effect of the encouragement and the effect of the intervention. ¹⁹

A second design capable of building more reliable and relevant evidence, that can also be less costly and faster to implement than traditional designs, is the interrupted time series design. In this design, which generalizes and improves upon the pre-post design, ²⁰ outcomes are first modeled or measured for a period of time before a program is implemented. After the program is implemented, the outcomes are measured and compared with what would have been expected to happen if the program had not been implemented, thereby generalizing the pre-post design to more than 2 time points. An even stronger design is the comparative interrupted time series design, in which outcomes are measured over time in (at least) 2 groups: one group that received the program and a comparison group that did not receive the program. ²¹ The interrupted time series and comparative interrupted time series designs are often cost-effective because they can take advantage of already available administrative data (eg, monthly counts of domestic violence arrests) and can be particularly strong when multiple localities make the same policy change or when only a subset of localities makes a policy change. Examples include Kennedy-Hendricks et al, ²² who estimated the effects of Florida’s pill-mill law on rates of opioid overdose, and Rudolph et al, ²³ who examined the effects of Connecticut’s permit-to-purchase gun law on homicide rates. These designs also provide an opportunity for partnerships between researchers and practitioners to collect results for multiple localities and then aggregate the data by using methods similar to meta-analysis to generate more generalizable knowledge for other localities.

Another way to build evidence that is both relevant and reliable is to combine results from an RCT with population data. RCTs are often conducted in groups of people or sites that are not representative of the target population or location. ^24,25 Combining data from RCTs and populations can help predict the outcome of the program in the population, after accounting for differences between the trial participants and the population. Stuart et al ²⁴ illustrated application of these methods by using them to predict the effects of a statewide implementation of a schoolwide strategy for social, emotional, and behavioral supports called Positive Behavioral Interventions & Supports. Methods also exist for combining estimates of program effectiveness from RCTs and nonrandomized studies. ²⁶

Other promising approaches for building reliable, relevant evidence include systems-science approaches (to analyze how interventions might or might not work in specific contexts by modeling the system as a whole), ²⁷ dissemination and implementation research (to study how information about interventions is transmitted, interpreted, adopted, and improved), ²⁸ and adaptive trials (to answer questions about an intervention’s effectiveness more quickly and at a lower cost than traditional trials by modifying the trial design based on the analysis of accruing data). ²⁹ West et al ¹² provided an overview of rigorous alternatives to standard RCTs, such as nonexperimental comparison group designs and regression discontinuity designs, that could be used to estimate program effects. Finally, approaches such as process evaluations, qualitative studies, and cost-effectiveness studies ³⁰ can help complement rigorous designs (eg, Barry et al ³¹ ) by providing additional insight about what might be happening “on the ground” to explain or understand program effects.

Greater use of alternative study designs is just one way to build more reliable and relevant evidence. Increasing the availability of data that are current and applicable to local settings is another. For example, instead of collecting new data by using large-scale surveys and other intensive data-collection methods, researchers can retrieve existing data, such as Twitter hashtags, Google search terms, and administrative data collected for other purposes. Creative use of existing data can help localities obtain data on their own populations more quickly; Ridgeway ³² highlights such opportunities in criminology. Data sharing across and within locales is another method for faster data collection. For example, the Massachusetts Department of Public Health gained insight into the state’s opioid crisis by analyzing 10 data sets provided by 5 Massachusetts agencies. ³³

In any effort to build more evidence that is relevant to practitioners and policy makers, the first step may be to ask them what they need. One way to obtain this information is to include practitioners and policy makers in the design phase of research studies. In addition to increasing the likelihood that a study will be useful, including practitioners and policy makers in the initial stages of a research project may also help solidify their trust in the researchers, making them more likely to seek out the researchers’ expertise when they need help evaluating evidence. One example of such a partnership is the evaluation of a Rhode Island initiative to provide medications for addiction treatment within the correctional system. ³⁴ Thus, new, innovative approaches exist that can help researchers build evidence that is useful to practitioners.

Strategies for Increasing the Use of Evidence in Practice

Building evidence that is reliable and relevant to practitioners and policy makers does not guarantee that they will use it. We now highlight opportunities to increase the use of evidence in public health practice.

First, more evidence needs to be shared in ways that are sensitive to the demands on practitioners’ and policy makers’ time, resources, and expertise. For example, study results and evaluations should be summarized in short, targeted communications (eg, fact sheets, presentations, policy briefs) that are written in plain language. Some researchers may need training to communicate in these ways. Others can partner with communications professionals.

Second, policy makers and practitioners need assistance in judging evidence, regardless of the way in which it is communicated. Platforms that engage practitioners and policy makers in learning about the process of evidence building, the strengths and weaknesses of various types of studies, and sorting through evidence would be helpful. Promising examples of such resources include the Results First Clearinghouse Database ³⁵ and numerous research and evaluation clearinghouses made available by the US Department of Health and Human Services Office of Planning, Research, and Evaluation. ³⁶ Also needed are new and better approaches for integrating various types of evidence and communicating the rigor of the underlying studies and the strength of the accumulated evidence. Although some knowledge synthesis methods that aim to integrate evidence by statistically combining information from multiple sources have been described in the health sciences literature, these methods have not been fully reported, evaluated, or formalized. ³⁷

Finally, researchers and practitioners should work together to increase the use of evidence and promote more evidence-based decision making. Efforts along these lines include the Coalition for Evidence-Based Policy (http://coalition4evidence.org) and the Commission on Evidence-Based Policymaking, ⁴ but more work is needed. One vehicle for enabling this kind of work could be an entity with a mission to connect researchers with public health practitioners and policy makers. A possible model for this entity is the Patient-Centered Outcomes Research Institute, which engages a broad set of stakeholders in all phases of clinical care research, including identifying research questions, conducting studies, and translating findings into practice. ³⁸ Thus, several creative approaches can be applied to increase the use of available evidence in public health practice.

We are confident that greater use of these strategies will bring about a future in which researchers generate and communicate evidence that is useful for public health decision making and in which decision makers evaluate evidence more appropriately and systematically than today. A virtuous cycle of this nature, in which researchers increase their efforts to produce evidence that is relevant and accessible to policy makers and practitioners because they see policy makers and practitioners use such evidence and ask researchers for more of it, will yield more interventions, programs, and policies that are grounded in rigorous evidence and improve the health of individuals and populations.