Integrating Decision Science and Implementation Science to Inform Policy Decision Making

Abstract

Decision science and implementation science share the common goal of improving individual and population health through choosing and providing effective health innovations at scale. In this article, we summarize a symposium we hosted at the 45th Annual Society for Medical Decision Making North American meeting. The symposium aimed to illustrate how integrating implementation science and decision science can strengthen the real-world impact and practical utility of decision science methods. The symposium was attended by 51 individuals. It included 4 presentations by early-career researchers and a moderated discussion. Presentations covered innovative work at the intersection of implementation science and decision analytic modeling and focused on policy applications because of the presenters’ expertise and strong history of decision analytic modeling to inform policy decisions. The symposium’s moderated discussion indicated a need for developing collaborations between implementation and decision scientists to move this type of work forward. Suggested areas of future research are modeling to identify gaps in data and considering value-of-information methods, exploring how implementation could be incorporated into simulation methods beyond those discussed in the symposium (e.g., distributional and extended cost-effectiveness analyses), and integrating implementation science into other areas of decision science (e.g., preference and prioritization research, shared decision making). We urge decision science researchers to pursue interdisciplinary research integrating decision and implementation science to best inform policy decision making and drive the scale-up of promising policies across contexts.

Highlights

Implementation science concepts could strengthen the external validity and uptake of decision science methods such as decision analytic models.

The symposium discussed and highlighted innovative ways that decision science researchers could integrate implementation science frameworks and outcomes (e.g., cost, reach, fidelity) into decision analytic models to be more responsive to the multifaceted considerations of policy decision making.

Supporting interdisciplinary networking and collaboration between decision scientists and implementation scientists is critical to strengthen the real-world impact and practical utility of decision science methods.

Graphical Abstract

Keywords

decision science implementation science policy simulation modeling interdisciplinary research

Within decision science, decision analytic models aim to use simulation-based techniques to help inform decisions about medical, public health, and health-related resource allocations and health intervention delivery across contexts and populations.¹ Implementation science is the study of strategies to support the use of effective health innovations in practice.² Core to each of these fields is the common goal of improving individual and population health through choosing and providing effective health innovations at scale.

Decision scientists have a strong history of providing crucial information on, for example, the most cost-effective innovations for a given health issue, or understanding what various groups prefer and value.³ However, we argue that more attention to how innovations are put into practice is needed in decision science methods. Attending to such considerations can ensure decision analytic models reflect real-world processes and contexts, thereby strengthening the external validity and uptake of modeling by decision makers.⁴ This is precisely what implementation science can add.

We hosted a symposium at the 45th Annual Society for Medical Decision Making North American meeting focused on the integration of decision science and implementation science. The goal was to illustrate how decision science and implementation science are currently being integrated and spur discussion on ways to move such research forward. Because this symposium was attended by decision scientists at the annual meeting, we primarily aimed to highlight how implementation science concepts could be integrated within decision sciences, with a particular focus on decision analytic modeling. However, we note that there are similar arguments being made in the implementation science literature,^5–7 and research at this intersection could transcend traditional disciplinary boundaries.

Symposium presentations were given by 4 early-career researchers involved in novel research at this intersection, with specific attention to policy decision making and applications. We focus on policy because of our presenters’ expertise, the strong history of using models to inform policy decisions, and the overall importance of policy to population health. The symposium was attended by 51 people.

In this article, we summarize each presentation and key points raised by audience members. We aim to illustrate ways that decision scientists could begin integrating more implementation science concepts into their projects and identify key areas of future research for decision scientists interested in implementation science.

What Is Implementation Science?

The first presenter (G.C.) introduced the goals of dissemination and implementation science as well as key concepts, definitions, and frameworks of relevance to decision scientists. We specifically focus on implementation science, the study of how to scale-up and improve the delivery of “things” or innovations (pills, procedures, policies, programs, principles, practices) in routine, real-world settings and practices.^2,8–11

Three foundational implementation science concepts were introduced. We specifically selected these 3 distinct concepts because of their ability to be integrated within decision analytic models and thus their potential use to modelers: implementation determinants, strategies, and outcomes. Implementation determinants are factors that impede (barriers) or promote (facilitators) the uptake and successful integration of an innovation in real-world practice,¹² implementation strategies are specific ways to enhance the implementation of an innovation,¹³ and implementation outcomes are how the field measures and understands the effects of implementation efforts, separate from intervention effects.¹⁴ For each of these key concepts, we provide specific definitions, examples of how these concepts could be integrated into decision analytic models, and key references for further reading in Table 1.^15–55

Table 1

Key Implementation Science Concepts for Decision Analytic Modeling

Key Implementation Science Concept	Definitions(s)	Suggestions for Integrating into Decision Sciences	Key References
Implementation determinants	Factors that impede (barriers) or promote (facilitators) the uptake and successful integration of an innovation in real-world practice¹² • Common domains of determinants include: ○ Innovation: the “thing” being implemented (e.g., policy, program) ○ Individuals: people involved in implementing or receiving services ○ Inner setting: where the “thing” is being implemented ○ Outer setting: context surrounding the inner setting (e.g., hospital system around a clinic) As one example, determinants of implementing clinical decision support tools include³⁸: • Fit of the tools with clinic workflows (innovation and inner setting) • User trust in tool output (individuals) • Perceived benefit of the tool (individuals) • These determinants could be included as parameters in simulation models, affecting the uptake and sustainment of clinical decision support interventions over time in the model.	• Collaborate with implementation scientists to identify and include relevant determinants in decision analytic models using implementation theories, models, and frameworks and, potentially, community-engaged modeling methods (e.g., group model building). • Conduct sensitivity analyses to examine how modeled outcomes change across plausible ranges of determinants that may vary by implementation site. • Report which determinants were included and tested in model write-ups, and why.	^12,39–42
Implementation strategies	“Methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice”¹³ Examples of implementation strategies and how they might affect the costs included in modeling a clinical intervention: • Audit and feedback (regular collection and reporting of performance data to intervention providers): ongoing effort to maintain audit and feedback protocols introduces costs over time into modeling efforts • Create a learning collaborative (facilitate formal groups of providers to improve implementation): may require costs up front to start up, but if communities can self-sustain over time, then ongoing costs are minimal • Facilitation (process of interactive problem solving and support): ongoing costs will depend on the individuals and groups involved in the facilitation exercises Such strategies may also affect intervention effects over time.	• Collaborate with implementation scientists and key affected groups (e.g., frontline providers, organizational/system leaders, patients) to understand how an innovation would be implemented and incorporate that knowledge into model structures and parameters (e.g., cost, time horizon). • Conduct scenario analyses to examine how modeled implementation, service, and clinical outcomes change across different implementation strategies over time.	^13,43–45
Implementation outcomes	“Effects of deliberate and purposive actions to implement new treatments, practices, and services.”¹⁴ Outcomes: • Acceptability: agreement that intervention is satisfactory, desirable • Appropriateness: perceived fit of the intervention to those implementing or receiving the innovation • Adoption: whether and when an intervention is selected for implementation • Cost: of strategies to scale-up/integrate the intervention and start delivering it • Feasibility: whether intervention can be implemented with available resources • Fidelity: whether intervention is delivered as intended • Reach/penetration: proportion of intended population served • Sustainability: degree to which intervention is integrated into practice/a setting over time	• Measure both implementation and intervention costs across the full implementation process (e.g., preimplementation, implementation, sustainment) and include in economic evaluation calculations. • Simulate implementation costs over time with attention to scale-up and sustainment in new contexts. • Identify and include relevant implementation outcomes (costs, fidelity, penetration) in models, in addition to health and service outcomes. • Report modeled effects on implementation outcomes side-by-side with service and health outcomes.	^14,46–49 Measuring implementation costs: ^15–22 Measuring other implementation outcomes: ^50–54

Within implementation outcomes, we wish to bring readers’ attention to implementation costs (see the last row of Table 1). This is an active area of methodological research within implementation science, particularly with regard to conceptualizing and measuring implementation costs^15–22 and describing their wide-ranging applications.^23–32 In brief, implementation costs are an important complement to innovation costs because they quantify the costs and resources required to put the innovation into practice through implementation strategies.¹⁵ Similar to innovation costs, implementation costs include both fixed and variable costs that vary over the course of implementation and implementation scale.^4,17,20,33 Recent methodological articles have demonstrated how to apply common cost techniques such as micro-costing to understand implementation costs as well as innovation costs.^15,34,35 Within a modeling framework, these costs are crucial to understanding the true cost-effectiveness as well as the feasibility and affordability of innovations being considered for scale-up across various settings.^15,33,36,37

After this introduction, 3 speakers presented how they conduct interdisciplinary research integrating decision and implementation science. Each presentation approaches this integration slightly differently, using different aspects of implementation science and different modeling methods, effectively showcasing the breadth of potential applications. For each presentation, Figure 1 illustrates the specific components of implementation science and decision science included.

Figure 1

Overview of symposium presentations and components of decision science (DS) and implementation science (IS) included.

Three Applied Examples of Integrating Decision Science and Implementation Science

Developing a Multicriteria Decision Analysis Tool Using Implementation Science (Gracelyn Cruden)

To illustrate the integration of implementation science during the development of a decision analytic model to support policy planning, the presenter described the iterative development of a multicriteria decision analysis (MCDA) tool to support state-level child welfare system actors (e.g., leadership, middle managers, workgroups) make decisions about which programs to implement to prevent child maltreatment. MCDA approaches are broad but generally defined as approaches to analyzing (and comparing) decision options across multiple criteria.⁵⁵ To that end, approaches typically require identifying and defining which criteria matter, the relative importance of those criteria (i.e., weights), and how decision options will be compared (i.e., scoring).⁵⁵

MCDA criteria are the key factors that influence a decision or characterize the decision alternatives, such as potential resources required, outcomes affected, and costs. Criteria are typically presented as having acceptable ranges or thresholds. Weights quantify the relative influence of each criterion on the overall decision. Criteria and weights can be combined through a variety of quantitative methods to generate scores for each decision alternative (e.g., interventions) compared in the tool.^56,57 Scores are then used to identify which intervention alternatives are the best fit for the decision at hand. The team used implementation science concepts to define the tool’s criteria and develop strategies to support the tool’s use in practice.

The tool’s criteria integrated implementation science through the RE-AIM evaluation framework.^58,59 This framework specifies 5 key components linked to implementation success over time: reach, efficacy, adoption, implementation, maintenance.⁵⁹ After conducting formative qualitative work with 8 end users,⁵⁸ findings were mapped onto the RE-AIM framework to ensure that no implementation-relevant constructs were missing. Comparison revealed that concepts of implementation maintenance, or sustainment, were missing. A related criterion was added and then validated by end users in a second round of feedback. These criteria were further revised in a recent pilot, in which several criteria were modified for clarity or relevance and several were added, including one related to the potential impact on improving health equity.⁶⁰

Implementation science was also used to develop strategies that could support the ultimate utility and feasibility of the MCDA tool in practice. MCDA tools take a sociotechnical approach, meaning that the technical MCDA components are only part of the decision process. How the tool is integrated into decision making—particularly in group decisions that entail additional social aspects compared with individual decision making—is a critical part of an MCDA tool’s utility. Although there is substantial research on the methods for developing and validating the 3 core components of technical MCDA tools, the literature is sparse when it comes to specifying reproducible interventions for the “socio” component. Thus, in an ongoing clinical trial (K01MH128761; NCT06375551), the team is testing 2 modalities (live facilitation, self-guided) of a facilitation approach to incorporate the tool’s results into group decision discussions. The facilitation strategies are informed by systems thinking and systems science methods (group model building) and the decision hygiene approach—a process for minimizing bias and random variance in decision making.⁶¹ In addition to individual- and group-level decision-making outcomes, such as the decision makers’ perceptions of the quality of the decision experience and their commitment to decision making, implementation outcomes will also be assessed, including implementation fidelity—adherence to processes known to be critical for high-quality intervention implementation.⁶²

“Big P” Policy Implementation Considerations for Modeling (Natalie Riva Smith)

The second presentation discussed how concepts from implementation science can inform more traditional simulation modeling approaches. The discussion focused specifically on “Big P” policy, which refers to formal or written government actions, such as pieces of legislation or regulation.⁶³ Increasingly, health policy and public policy scholars are stressing the importance of studying how “Big P” policy is implemented, including the implementation process and strategies used, as well as measuring implementation outcomes in addition to traditional service and health outcomes.^63–67

Sugary drink tax policies provide a useful illustrative example of the importance of accounting for policy implementation.²³ Work evaluating the implementation of such policies in jurisdictions such as Berkeley and Oakland, California; Seattle, Washington; Philadelphia, Pennsylvania; and Cook County, Illinois, has shown that there are many actions that must happen after a policy has been formally passed to make it possible to achieve a policy’s intended impacts. For example, jurisdictions often need to undertake rulemaking to expand the specificity of legislation (e.g., determine responsibilities of key government partners, establish fine structures), create infrastructure to collect taxes, and educate regulated entities on how to pay taxes. The evidence also points to heterogeneity in implementation across jurisdictions.

Work accounting for implementation in standard policy evaluative work is emerging (both methodological and applied).^68–72 These types of studies often seek to understand how implementation mediates or moderates a policy’s health outcomes.^68,69 A necessary complement is for simulation modeling of policies to better incorporate implementation considerations (e.g., in cost-effectiveness modeling).⁴ This presentation addressed 3 potential theoretical issues that should be accounted for in such models:

Time lags: Implementation processes create a time lag between when the policy is adopted, when costs begin to accrue, and when health effects start to accrue.

Implementation costs: Not accounting for the costs of implementation processes will result in cost estimates that are too small.

Process heterogeneity: Implementation processes will unfold differently in different jurisdictions (e.g., different implementation strategies will be used).

Issues such as these can affect the accuracy of models and policy comparisons (particularly cost-effectiveness comparisons).⁴ These issues have relatively straightforward solutions: 1) explicitly modeling time lags, 2) including estimates of implementation costs, and 3) modeling heterogeneity in implementation processes via modifying a model’s structure or conducting scenario analyses. Sensitivity and uncertainty analyses focused on these issues are also critical: for example, examining the impact of different implementation time lags on cost-effectiveness results after a set time. At minimum, modelers should be explicit when models and results do not account for implementation.

As an applied example, this presentation drew on published work that assessed the potential costs to implement 2 sugary drink policies at the federal level in the United States: a sugary drink tax and warning label.²³ The team used a mixed-methods approach to identify and quantify implementation costs. First, they conducted interviews with individuals who were knowledgeable of federal policy-making procedures and had experience with federal rulemaking, public health law, the sugary drink industry, or public health advocacy. These interviews provided information on the possible way(s) that sugary drink policy implementation would unfold at the federal level. Specific actions that would be required of different groups included litigation to repeal legislation or hold up rulemaking, creating tax compliance or collection infrastructure, redesigning warning labels, and monitoring compliance with policies. This information was then used to drive the quantitative cost analyses. Cost estimates were developed using a variety of data sources including peer-reviewed literature, government documents and budget reports, or tools from nongovernmental organizations. As one example, the costs to relabel sugary drinks were estimated using an existing tool developed by the Research Triangle Institute.⁷³

The process and findings from this article underscore that policy is not costless to implement and speak to the importance of interdisciplinary, mixed-methods collaboration. Several important methodological questions and next steps were also raised, including what phases of the policy process are included in cost estimates, how to handle policy “packages” (i.e., multiple, related policies being implemented at once), and how to handle the significant uncertainties within cost estimation.

Identifying Implementation Strategies Using Participatory Systems Science and Modeling (Christina Yuan)

The third applied example focused on the application of systems science simulation modeling for implementation strategy decision making. In contrast to the to the prior 2 presentations where implementation science considerations informed decision science questions, this presentation illustrated how methods from decision science can help inform implementation science questions.

Ideally, implementation strategies are designed and selected to target implementation determinants (Table 1). However, in practice, there is often a mismatch between determinants and strategies,^74,75 and existing implementation strategy selection methods are often inadequate.^76–78 This session highlighted the potential for using systems science simulation models to serve as “virtual laboratories” to explore the potential effects of implementation strategies on outcomes prior to implementation. Such models can support decision making not only around which strategies to use in the first place but also how to operationalize and adapt strategies throughout the implementation process.

As one example, the Johns Hopkins ALACRITY Center for Health and Longevity in Mental Illness is planning to use a participatory systems science approach to identify implementation strategies to scale-up evidence-based smoking cessation treatment for people with serious mental illness.

The modeling approach involves triangulating the expertise of community partners (including policy makers), an extensive array of data sources, and model simulations. The process will begin by iteratively engaging with a diverse array of community partners to develop and refine a conceptual model that identifies candidate implementation strategies. The variables in the conceptual model will be rooted in the Exploration, Preparation, Implementation, Sustainment (EPIS) Implementation Framework,⁷⁹ which is an implementation model that highlights 4 phases of the implementation process and enumerates factors associated with the outer context (system), inner context (organization), factors that bridge the outer and inner context, and the nature of the innovation being implemented. This theory-informed conceptual model will provide the foundation for an integrated system model that will combine an agent-based model that simulates micro-level system behaviors (e.g., providers’ decision-making processes) with a system dynamics model that simulates macro-level system behaviors (e.g., staffing levels). The model will be calibrated using empirical data from prior research on smoking cessation treatment for people with serious mental illness, new primary data collection efforts collecting data across potential implementation contexts (including implementation costs), and a literature review of anticipated effect sizes of candidate implementation strategies.

The overall goal of the project is to create an interactive dashboard that includes “levers” to simulate implementation strategies at various levels of intensity and “switches” to simulate different combinations of strategies (which can be used to turn different strategies on and off, reflecting how different contexts may make decisions). The purpose of the dashboard will be to aid decision making by allowing users to “virtually test” how adjustments in the selection, dosing, and combination of implementation strategies would potentially affect intervention delivery and outcomes.

Cross-Cutting Theme: Data Equity Considerations when Integrating Decision Science and Implementation Science (Tran Thu Doan)

Each of the applied presentations touches on the data requirements of working at this intersection. Accounting for implementation contexts and processes in models, including costs and outcomes and comparing alternatives on a range of factors, requires detailed data. Especially when generating simulation results for specific contexts (often crucial in implementation-focused work), data must be sufficiently detailed to identify key factors relevant to implementing effective strategies for affected groups. Decision scientists will need data regarding implementation costs, clinical outcomes, intervention, and implementation outcomes as well as on how implementation processes intersect with usual care, disaggregated by populations of interest.

These data issues necessitate understanding data equity and related concepts, the focus of the final presentation. Data equity is a useful approach for identifying health similarities and differences within a population across contexts.^80–83 Data equity refers to the concept and practice of collecting and using data in ways that are fair and representative of populations served.^84,85 It is guided by principles of beneficence, justice, and fairness, as outlined in the Belmont Report,⁸⁶ for protecting human participants in ethical research and practice. It involves considerations of statistical power, bias, and discrimination in data collection methods, algorithms, and modeling to inform data-driven decision making. Data equity integrated with decision modeling can enable more context-specific modeling and help support precise implementation of programs and policies to promote overall health and quality of life.

A major component of data equity is data disaggregation, which involves the collection and use of data broken down by relevant categories. Thoughtful considerations are required regarding which, how, and when data categories should be disaggregated. For example, disaggregating sexual orientation and gender identity data by Medicaid coverage status could be extremely important for accurately simulating alternative policies or implementation processes.⁸⁷ Choices about the relevant data categories to use in each modeling effort should be made in collaboration with key decision partners, supplemented by the literature, and grounded in theory. When integrating decision modeling with implementation efforts, data can capture groups that fund the implementation, perform the implementation, or receive the implemented innovation. In disseminating research findings, products should fully and transparently justify and report on these choices, including what the variables are intended to measure and how the variables were measured.^88,89

A second major component of data equity is “equitable data collection,” which refers to the process of gathering information that ensures beneficence, justice, and fairness for all. This aims to prevent discrimination and promote equal representation and opportunities for all populations affected by the health issue under study. Strategies include innovative sampling techniques, respectful engagement with communities, and careful consideration of ethics, consent, and privacy. Data collection should be guided by data sovereignty and governance principles that are co-created in respectful collaboration with community members and other decision partners. Data collection procedures should honor community members’ expressed desires for ownership, control, access, and possession of such data.⁹⁰

Data equity and associated concepts are rapidly becoming institutionalized. There has been increasing national demand in recent years to improve the collection of granular data and to advance health for all. In a major update to the US Census data collection methods, the US Office of Management and Budget revised Statistical Policy Directive No. 15 in 2024, updating standards for maintaining, collecting, and presenting federal data on race and ethnicity.⁹¹ Key revisions included combining race and ethnicity into 1 question and establishing “Middle Eastern or North African” as a minimum category. In 2023, the White House released the Federal Evidence Agenda on lesbian, gay, bisexual, transgender, queer or questioning, intersex (LGBTQI+) people to support sexual orientation, gender identity, and variations in sex characteristics data collection.⁹² In 2024, the US Food and Drug Administration⁹³ and the National Academies of Sciences, Engineering, and Medicine⁹⁴ supported enhanced data collection of populations from different backgrounds through increasing enrollment in clinical studies or drug evaluations. The 2025 federal restrictions on diversity, equity, and inclusion programs or language introduced new uncertainty around the funding for and prioritization of future stratified analyses. Government agencies, health systems, and payers should be aware of recent policy changes, contextualize their work, and acknowledge challenges of conducting health equity analyses.

Data equity remains a critical tool to guide the integration of implementation and decision sciences because it allows researchers to investigate targeted actions that result in demonstrably improved health outcomes for disaggregated populations.

Moderated Discussion

There were 2 major points of discussion during the moderated discussion after the presentations.

Is Implementation Science Ready for Decision Science?

One point of discussion was whether implementation science concepts are “ready” to be integrated with decision analytic models and methods. Discussion considered whether there is adequate evidence on implementation strategy effectiveness and core components to parameterize models, whether generalizable quantitative estimates exist for modeling, and the extent to which implementation determinants affect intervention and implementation effectiveness in various contexts. These are important questions and likely have different answers depending on the substantive area or methods used. Relatedly, audience members questioned whether rigorous measures of implementation outcomes exist to generate data appropriate for model parameterization. While the field of implementation science recognizes the need for more measures, rigorous measures do exist and continue to develop (examples cited throughout Table 1). For example, measures of implementation determinants, such as organization climate,^95,96 are well-established.

These concerns about data also underscore the importance of collaboration between decision scientists and implementation scientists. For there to be appropriate data to inform models, decision scientists should collaborate with implementation scientists and ensure that implementation scientists know what data are needed for robust modeling efforts and can accordingly incorporate such measurement in implementation trial designs and measurement protocols.

Collaboration Is Crucial

A second major discussion point was the importance of improving collaboration between decision scientists and implementation scientists. Similar to how best practices for cost analyses entail engaging economic collaborators early, implementation science researchers can contribute knowledge of measurement challenges and possibilities and conduct the trials to collect data for simulation.^17,33,97,98 Ways to bring implementation science and decision science researchers together are needed. Recent work on ways to improve interdisciplinary and transdisciplinary collaboration between health economists and implementation scientists can help inform paths forward.⁹⁸ Possible processes to help establish and maintain collaborations include establishing a shared understanding of the central research goals, concepts, and methods from collaborators’ fields (e.g., decision science or implementation science), clarifying and appreciating the expertise of different team members, and discussing how approaches to questions might differ across disciplines.⁹⁸ Funding will be critical to protect researchers’ time for networking and to create collaboration activities.

Moving Forward

This symposium illustrated several ways that current work is integrating the fields of decision science and implementation science. Building on the presentations and discussion, we discuss some specific ideas for moving this type of work forward, with particular emphasis on actions that decision scientists can take.

To address concerns raised in the symposium’s discussion about limited implementation data, we propose that decision analytic modelers consider integrating value-of-information methods within models that include implementation concepts.^99–103 Modeling could apply value-of-information methods to understand which parameters are most critical to assess, the relative advantage of increased resource allocation to do so,¹⁰² or how implementation affects value-of-information results.⁹⁹

We also challenge decision scientists to consider that decision analytic modeling can be an effective learning tool, rather than just a predictive tool (although predictive modeling is certainly useful¹⁰⁴).^105–108 We envision that modeling to learn, rather than solely modeling to predict, could be particularly helpful to illuminate evidence gaps and define key priorities for implementation research (more broadly than with value-of-information methods). Overall, beginning to model can define evidence gaps, thereby helping to shape funding proposals and also serving the dual purpose of helping to formalize collaborations between decision and implementation scientists.

Another consideration for future work is how implementation science could be integrated with more advanced equity-focused simulation approaches, such as distributional or extended cost-effectiveness analysis.^4,109,110 For example, researchers have used distributional cost-effectiveness to examine how alternative engagement strategies within a cancer screening program influence outcomes across population groups.¹¹¹ Key strategies included targeted reminders to individuals living in income-deprived areas and a universal reminder to all individuals.¹¹¹ Notably, researchers allowed for heterogeneity in how different groups receiving the intervention would likely respond to each strategy.¹¹¹ Future work could build on these methodological approaches by incorporating findings from implementation research, such as alternative implementation strategies for engaging different groups or refining the costs of implementing different strategies. The need for stratified group-specific estimates in such approaches also underscores the need for attention to principles and practices of data equity.

While the work presented here focused on policy settings and decision analytic modeling, future work can and should expand this thinking into other areas of decision science, particularly shared decision making and preference research. Within shared decision making, scholars have suggested that incorporating implementation science concepts is a necessity to promote more widespread uptake of shared decision making in clinical practice.^112,113 Decision science work focused on preference and prioritization could also provide useful information for both decision analytic modeling and implementation science applications (e.g., which implementation outcomes are the most important include in models,¹¹⁴ which approaches are most acceptable to the population implementing an innovation or targeted by implemented interventions¹¹⁵).^116,117 Finally, the other side of dissemination and implementation science—dissemination—is also an important area for decision science researchers to consider. Even simple simulation models can involve massive amounts of assumptions, data, and results. Interrogating how to best communicate models to decision makers is extremely important to ensure model transparency and trust by users.¹¹⁸

Conclusion

Implementation and decision science share a broad goal of improving individual and population health by conducting research to inform the selection and scale-up of health innovations such as policies or programs. To realize this goal, our symposium argued that integrating these fields could increase the practical utility, usability, and uptake of decision analytic modeling approaches. To highlight the necessity and potential of this integration, the symposium included innovative work at the intersection of policy, implementation science, and decision analytic modeling. The symposium’s moderated discussion indicated a need for developing collaborations between implementation and decision scientists to move this type of work forward, and we outline potential areas for future research. Moving forward, we urge decision science researchers to undertake interdisciplinary research integrating implementation and decision science to help inform data-driven policy decision making and drive the scale-up of promising health policies across contexts.

Footnotes

Acknowledgements

The work presented in this article was presented at the 45th Annual Meeting of the Society for Medical Decision Making in Philadelphia, Pennsylvania, in October 2023. The authors gratefully acknowledge Dr. Mark Bouthavong, who supported the symposium as a moderator and served as an internal reviewer for this article.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this work was provided by the National Institutes of Health under grants K99CA277135 (NRS), K01MH128761 (GC), and P50MH115842 (CY). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report. Funding for open access for this article was made possible by K99CA277135.

Ethical Considerations

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

ORCID iDs

Natalie Riva Smith

Tran Thu Doan

Data Availability

There is no code or data associated with this article.

References

Roberts

Russell

Paltiel

Chambers

McEwan

Krahn

Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force-2. Value Health. 2012;15(6):804–11.

Brown

Curran

Palinkas

Aarons

Wells

Jones

, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22.

Hunink

MGM

Weinstein

Wittenberg

Drummond

Pliskin

Wong

, et al. Decision Making in Health and Medicine: Integrating Evidence and Values. 2nd ed. Cambridge (UK): Cambridge University Press; 2014.

Krebs

Nosyk

Cost-effectiveness analysis in implementation science: a research agenda and call for wider application. Curr HIV/AIDS Rep. 2021;18(3):176–85.

Smith

Knocke

Hassmiller Lich

Using decision analysis to support implementation planning in research and practice. Implement Sci Commun. 2022;3(1):83.

Sheldrick

Cruden

Schaefer

Mackie

TI.

Rapid-cycle systems modeling to support evidence-informed decision-making during system-wide implementation. Implement Sci Commun. 2021;2(1):116.

Wheeler

Leeman

Hassmiller Lich

Tangka

FKL

Davis

Richardson

LC.

Data-powered participatory decision making: leveraging systems thinking and simulation to guide selection and implementation of evidence-based colorectal cancer screening interventions. Cancer J. 2018;24(3):132–9.

PAR-19-274: Dissemination and Implementation Research in Health (R01 Clinical Trial Optional). Available from: https://grants.nih.gov/grants/guide/pa-files/PAR-19-274.html [Accessed 26 August, 2024].

Eccles

Mittman

BS.

Welcome to implementation science. Implement Sci. 2006;1(1):1–3.

10.

Brownson

Colditz

Proctor

, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York: Oxford University Press; 2018.

11.

Curran

GM.

Implementation science made too simple: a teaching tool. Implement Sci Commun. 2020;1(1):27.

12.

Damschroder

Reardon

Widerquist

MAO

Lowery

The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

13.

Proctor

Powell

McMillen

JC.

Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.

14.

Proctor

Silmere

Raghavan

, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

15.

Gold

McDermott

Hoomans

Wagner

TH.

Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17(1):11.

16.

Eisman

Quanbeck

Bounthavong

Panattoni

Glasgow

RE.

Implementation science issues in understanding, collecting, and using cost estimates: a multi-stakeholder perspective. Implement Sci. 2021;16(1):75.

17.

Dopp

Mundey

Beasley

Silovsky

Eisenberg

Mixed-method approaches to strengthen economic evaluations in implementation research. Implement Sci. 2019;14(1):2.

18.

O’Leary

Hassmiller Lich

, et al. Extending analytic methods for economic evaluation in implementation science. Implement Sci. 2022;17(1):27.

19.

Saldana

Ritzwoller

Campbell

Block

EP.

Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers. Implement Sci Commun. 2022;3(1):40.

20.

Saldana

Chamberlain

Bradford

Campbell

Landsverk

The Cost of Implementing New Strategies (COINS): a method for mapping implementation resources using the stages of implementation Completion. Child Youth Serv Rev. 2014;39:177–82.

21.

Wagner

Yoon

Jacobs

, et al. Estimating costs of an implementation intervention. Med Decis Making. 2020;40(8):959–67.

22.

Wagner

Dopp

Gold

HT.

Estimating downstream budget impacts in implementation research. Med Decis Mak. 2020;40(8):968–77.

23.

Smith

Hassmiller Lich

Hall

Trogdon

Frerichs

Implementation costs of sugary drink policies in the United States. J Public Health Policy. 2023;44(4):566–87.

24.

Smith

Simione

Farrar-Muir

, et al. Costs to implement a pediatric weight management program across 3 distinct contexts. Med Care. 2023;61(10):715–25.

25.

Donovan

Abell

Fernando

McPhail

Carter

HE.

Implementation costs of hospital-based computerised decision support systems: a systematic review. Implement Sci. 2023;18(1):7.

26.

Eisman

Hutton

Prosser

Smith

Kilbourne

AM.

Cost-effectiveness of the Adaptive Implementation of Effective Programs Trial (ADEPT): approaches to adopting implementation strategies. Implement Sci. 2020;15(1):109.

27.

Eisman

Whitman

Palinkas

, et al. A mixed methods partner-focused cost and budget impact analysis to deploy implementation strategies for school-based prevention. Implement Sci Commun. 2023;4(1):133.

28.

Acquilano

Forcino

Schubbe

, et al. The costs of implementing a conversation aid for uterine fibroids in multiple health care settings. Med Care. 2023;61(10):689.

29.

Kaplan

Bergman

, et al. Health system perspective on cost for delivering a decision aid for prostate cancer using time-driven activity-based costing. Med Care. 2023;61(10):681.

30.

Spees

Young

Rees

, et al. A cost analysis of rethink the strip: de-implementing a low-value practice in primary care. Med Care. 2023;61(10):708.

31.

Cidav

Mandell

Ingersoll

Pellecchia

Programmatic costs of project impact for children with autism: a time-driven activity based costing study. Adm Policy Ment Health Ment Health Serv Res. 2023;50(3):402–16.

32.

Ryan

Ekanayake

Evans

, et al. Cost analysis of MOUD implementation and sustainability in Massachusetts jails. Health Justice. 2025;13(1):9.

33.

Eisman

Kilbourne

Dopp

Saldana

Eisenberg

Economic evaluation in implementation science: Making the business case for implementation strategies. Psychiatry Res. 2020;283:112433.

34.

Cidav

Mandell

Pyne

Beidas

Curran

Marcus

A pragmatic method for costing implementation strategies using time-driven activity-based costing. Implement Sci. 2020;15(1):28.

35.

Cidav

Marcus

Mandell

, et al. Programmatic costs of the Telehealth Ostomy Self-Management Training: an application of time-driven activity-based costing. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2021;24(9):1245–53.

36.

Smith

Levy

DE.

Budget impact analysis for implementation decision making, planning, and financing. Transl Behav Med. 2024;14(1):54–9.

37.

Dopp

Kerns

SEU

Panattoni

, et al. Translating economic evaluations into financing strategies for implementing evidence-based practices. Implement Sci. 2021;16(1):66.

38.

Abell

Naicker

Rodwell

, et al. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci. 2023;18(1):32.

39.

Nilsen

Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

40.

Damschroder

Aron

Keith

, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

41.

Estrada-Magbanua

Huang

TTK

Lounsbury

, et al. Application of group model building in implementation research: a systematic review of the public health and healthcare literature. PLoS One. 2023;18(8):e0284765.

42.

ACCORDS Dissemination and Implementation Science Program at the University of Colorado-Denver, Dissemination and Implementation Research Core (DIRC) at the Washington University Institute for Clinical and Translational Science, Dissemination and Implementation Science Center (DISC) at UC San Diego. Dissemination and Implementation Models in Health. 2024. Available from: www.dissemination-implementation.org. [Accessed 5 August, 2024]

43.

Cook

Lyon

Locke

, et al. Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prev Sci Off J Soc Prev Res. 2019;20(6):914–35.

44.

Powell

Waltz

Chinman

, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.

45.

Waltz

Powell

Fernández

, et al. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.

46.

Proctor

Bunger

Lengnick-Hall

, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18(1):31.

47.

Damschroder

Reardon

Opra Widerquist

MA.

Conceptualizing outcomes for use with the consolidated framework for implementation research (CFIR): the CFIR outcomes addendum. Implement Sci. 2022;17(1):7.

48.

Lewis

Mettert

Dorsey

, et al. Measures and outcomes in implementation science. In: Advancing the Science of Implementation across the Cancer Continuum. Oxford (UK): Oxford University Press; 2018. p. 57–77.

49.

Mettert

Lewis

Dorsey

, et al. Measuring implementation outcomes: an updated systematic review of measures’ psychometric properties. Implement Res Pract. 2020;1:2633489520936644.

50.

Allen

Pilar

Walsh-Bailey

, et al. Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implement Sci. 2020;15(1):47.

51.

Chaudoir

Dugan

Barr

CH.

Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.

52.

Fishman

Yang

Mandell

Attitude theory and measurement in implementation science: a secondary review of empirical studies and opportunities for advancement. Implement Sci. 2021;16(1):87.

53.

Harachi

Abbott

Catalano

, et al. Opening the black box: using process evaluation measures to assess implementation and theory building. Am J Community Psychol. 1999;27(5):711–31.

54.

Weiner

Lewis

Stanick

, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):1–12.

55.

Thokala

Devlin

Marsh

, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health J Int Soc Pharmacoeconomics Outcomes Res. 2016;19(1):1–13.

56.

Frazão

TDC

Camilo

DGG

Cabral

ELS

, et al. Multicriteria decision analysis (MCDA) in health care: a systematic review of the main characteristics and methodological steps. BMC Med Inform Decis Mak. 2018;18(1):90.

57.

Belton

Pictet

A framework for group decision using a MCDA model: sharing, aggregating or comparing individual information?

J Decis Syst. 1997;6(3):283–303.

58.

Cruden

Frerichs

Powell

, et al. Developing a multi-criteria decision analysis tool to support the adoption of evidence-based child maltreatment prevention programs. Prev Sci. 2020;21(8):1059–64.

59.

Glasgow

Vogt

Boles

SM.

Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

60.

Cruden

Peterson

Vanness

, et al. User-centered refinement of a multi-criteria decision analysis tool for health intervention selection. Prev Sci. 2020;21(8):1059–64.

61.

Kahneman

Sibony

Sunstein

CR.

Noise: A Flaw in Human Judgment. London: Hachette UK; 2021. Available from: https://scholar.google.com/scholar?cluster=2643496849342872081&hl=en&oi=scholarr [Accessed 30 September, 2024].

62.

Alley

Chapman

Schaper

, et al. The relative value of pre-implementation stages for successful implementation of evidence-informed programs. Implement Sci. 2023;18(1):30.

63.

Chriqui

Asada

Smith

, et al. Advancing the science of policy implementation: a call to action for the implementation science field. Transl Behav Med. 2023;13(11):820–5.

64.

Purtle

Moucheraud

Yang

, et al. Four very basic ways to think about policy in implementation science. Implement Sci Commun. 2023;4(1):111.

65.

Emmons

Chambers

Abazeed

, et al. Embracing policy implementation science to ensure translation of evidence to cancer control policy. Transl Behav Med. 2021;11(11):1972–9.

66.

Howlett

Moving policy implementation theory forward: a multiple streams/critical juncture approach. Public Policy Adm. 2019;34(4):405–30.

67.

Hudson

Hunter

Peckham

Policy failure and the policy-implementation gap: can policy support programs help?

Policy Des Pract. 2019;2(1):1–14.

68.

McGinty

Seewald

Bandara

, et al. Scaling interventions to manage chronic disease: innovative methods at the intersection of health policy research and implementation science. Prev Sci. 2024;25(suppl 1):96–108.

69.

Smith

Levy

Falbe

, et al. Design considerations for developing measures of policy implementation in quantitative evaluations of public health policy. Front Health Serv. 2024;4:1322702.

70.

McLoughlin

Kumanyika

, et al. Mending the gap: measurement needs to address policy implementation through a health equity lens. Transl Behav Med. 2024;14(4):207–14.

71.

Purtle

Stadnick

NA.

Earmarked taxes as a policy strategy to increase funding for behavioral health services. Psychiatr Serv. 2019;70(5):392–4.

72.

Bullock

Lavis

JN.

Understanding the supports needed for policy implementation: a comparative analysis of the placement of intermediaries across three mental health systems. Health Res Policy Syst. 2019;17(1):82.

73.

Muth

Bradley

Brophy

, et al. 2014FDA labeling cost model. U.S. Food and Drug Administration; 2015. Available from: https://www.rti.org/publication/2014-fda-labeling-cost-model

74.

Lewis

Boyd

Walsh-Bailey

, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.

75.

Bosch

van der Weijden

Wensing

, et al. Tailoring quality improvement interventions to identified barriers: a multiple case analysis. J Eval Clin Pract. 2007;13(2):161–8.

76.

Powell

Beidas

Lewis

, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44(2):177–94.

77.

Kirchner

Smith

Powell

, et al. Getting a clinical innovation into practice: an introduction to implementation strategies. Psychiatry Res. 2020;283:112467.

78.

Damschroder

LJ.

Clarity out of chaos: use of theory in implementation research. Psychiatry Res. 2020;283:112461.

79.

Moullin

Dickson

Stadnick

, et al. Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.

80.

Karvonen

Bardach

NS.

Making lemonade out of lemons: an approach to combining variable race and ethnicity data from hospitals for quality and safety efforts. BMJ Qual Saf. 2024;33(2):74–7.

81.

Kwon

Suss

, et al. The mutually reinforcing cycle of poor data quality and racialized stereotypes that shapes Asian American health. Health Aff. 2022;41(2):296–303.

82.

Ponce

Shimkhada

Adkins-Jackson

PB.

Making communities more visible: equity-centered data to achieve health equity. Milbank Q. 2023;101(S1):302–32.

83.

Gee

Morey

Bacong

, et al. Considerations of racism and data equity among Asian Americans, Native Hawaiians, and Pacific Islanders in the context of COVID-19. Curr Epidemiol Rep. 2022;9(2):77–86.

84.

Panneton

Hill

Smith

. Principles for using public health data to drive equity. Available from: https://www.cdcfoundation.org/data-equity-principles?inline [Accessed 3 May, 2024].

85.

data.org. What is data equity and why does it matter? Available from: https://data.org/resources/what-is-data-equity-and-why-does-it-matter/ [Accessed 3 May, 2024]

86.

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont report: ethical principles and guidelines for the protection of human subjects of research. Superintendent of Documents; 1978.

87.

Tran

Nguyen

KH.

LGBTQI+ data collection in Medicaid to advance health equity. JAMA. 2024;332(4):277–8.

88.

Zikmund-Fisher

BJ.

Toward transparent demographic analyses: statement on the use and reporting of classification variables presented as measuring individual characteristics such as race, ethnicity, indigeneity, national origin, gender, sexual orientation, or socioeconomic status. Med Decis Making. 2022;42(3):277–9.

89.

Martinez

RAM

Andrabi

Goodwin

, et al. Conceptualization, operationalization, and utilization of race and ethnicity in major epidemiology journals, 1995-2018: a systematic review. Am J Epidemiol. 2023;192(3):483–96.

90.

The First Nations Information Governance Centre. The First Nations Principles of OCAP®. Available from: https://fnigc.ca/ocap-training/ [Accessed 19 September, 2024]

91.

Revisions to OMB’s Statistical Policy Directive No. 15: standards for maintaining, collecting, and presenting federal data on race and ethnicity]. Fed Regist. 2024;89(61):22182–90.

92.

Subcommittee on Sexual Orientation, Gender Identity, and Variations in Sex Characteristics (SOGI) Data, Subcommittee on Equitable Data. Federal evidence agenda on LGBTQI+ equity. National Science and Technology Council; 2023. Available from: https://www.whitehouse.gov/wp-content/uploads/2023/01/Federal-Evidence-Agenda-on-LGBTQI-Equity.pdf [Accessed 19 September, 2024].

93.

Office of the Commissioner. FDA guidance provides new details on diversity action plans required for certain clinical studies. US Food and Drug Administration; 2024. Available from: https://www.fda.gov/news-events/press-announcements/fda-guidance-provides-new-details-diversity-action-plans-required-certain-clinical-studies [Accessed 19 September, 2024].

94.

Watanabe

JH.

Enhancing drug evaluation in diverse populations and older adults: National Academies of Sciences, Engineering, and Medicine considerations. J Am Geriatr Soc. 2024;72(9):2912–20.

95.

Ehrhart

Aarons

Farahnak

LR.

Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014;9(1):157.

96.

Powell

Mettert

Dorsey

, et al. Measures of organizational culture, organizational climate, and implementation climate in behavioral health: a systematic review. Implement Res Pract. 2021;2:26334895211018862.

97.

Barnett

Dopp

Klein

, et al. Collaborating with health economists to advance implementation science: a qualitative study. Implement Sci Commun. 2020;1(1):82.

98.

Barnett

Stadnick

Proctor

, et al. Moving beyond Aim Three: a need for a transdisciplinary approach to build capacity for economic evaluations in implementation science. Implement Sci Commun. 2021;2(1):133.

99.

Andronis

Barton

PM.

Adjusting estimates of the expected value of information for implementation: theoretical framework and practical application. Med Decis Mak Int J Soc Med Decis Mak. 2016;36(3):296–307.

100.

Grimm

Dixon

Stevens

JW.

Assessing the expected value of research studies in reducing uncertainty and improving implementation dynamics. Med Decis Mak Int J Soc Med Decis Mak. 2017;37(5):523–33.

101.

Fenwick

Claxton

Sculpher

The value of implementation and the value of information: combined and uneven development. Med Decis Mak. 2008;28(1):21–32.

102.

Kunst

Wilson

ECF

Glynn

, et al. Computing the expected value of sample information efficiently: practical guidance and recommendations for four model-based methods. Value Health. 2020;23(6):734–42.

103.

Heath

Kunst

Jackson

, et al. Calculating the expected value of sample information in practice: considerations from 3 case studies. Med Decis Making. 2020;40(3):314–26.

104.

Johnson

Vermeer

Hills

, et al. Model-driven decision support: a community-based meta-implementation strategy to predict population impact. Ann Epidemiol. 2024;95:12–8.

105.

Epstein

Alper

Quill

TE.

Communicating evidence for participatory decision making. JAMA. 2004;291(19):2359–66.

106.

Zimmerman

Lounsbury

Rosen

, et al. Participatory system dynamics modeling: increasing stakeholder engagement and precision to improve implementation planning in systems. Adm Policy Ment Health Ment Health Serv Res. 2016;43(6):834–49.

107.

Hassmiller Lich

Urban

Frerichs

, et al. Extending systems thinking in planning and evaluation using group concept mapping and system dynamics to tackle complex problems. Eval Program Plann. 2017;60:254–64.

108.

Epstein

JM.

Why model?

J Artif Soc Soc Simul. 2008;11(4):12.

109.

Verguet

Kim

Jamison

DT.

Extended cost-effectiveness analysis for health policy assessment: a tutorial. PharmacoEconomics. 2016;34(9):913–23.

110.

Cookson

Griffin

Norheim

, et al. Distributional cost-effectiveness analysis comes of age. Value Health. 2021;24(1):118–20.

111.

Asaria

Griffin

Cookson

Distributional cost-effectiveness analysis: a tutorial. Med Decis Mak Int J Soc Med Decis Mak. 2016;36(1):8–19.

112.

Clayman

Elwy

Vassy

JL.

Reframing SDM using implementation science: SDM is the intervention. Med Decis Making. 2024;44:0272989X241285418.

113.

Matlock

Scherer

. Shared decision making “ought” to be done, but definitions need simplicity: response to “Reframing SDM Using Implementation Science: SDM Is the Intervention.” Med Decis Making. 2024;44(8):865–6.

114.

Bridges

Crossnohere

Schuster

, et al. A patient and community-centered approach selecting endpoints for a randomized trial of a novel advance care planning tool. Patient Prefer Adherence. 2018;12:241–9.

115.

Lancsar

Ride

Black

, et al. Social acceptability of standard and behavioral economic inspired policies designed to reduce and prevent obesity. Health Econ. 2022;31(1):197–214.

116.

Salloum

Shenkman

Louviere

, et al. Application of discrete choice experiments to enhance stakeholder engagement as a strategy for advancing implementation: a systematic review. Implement Sci. 2017;12(1):140.

117.

Pettersson

Millroth

Giannotta

, et al. Outcome preferences in fidelity-adaptation scenarios across evidence-based parenting programs: a discrete choice experiment. Implement Sci. 2025;20(1):10.

118.

Nosyk

Weiner

Krebs

, et al. Dissemination science to advance the use of simulation modeling: our obligation moving forward. Med Decis Mak Int J Soc Med Decis Mak. 2020;40(6):718–21.