Sage Journals: Discover world-class research

Abstract

Stakeholder involvement (SI) is essential for effective and sustainable intervention implementation, yet practical guidance is lacking. This study mapped SI use in implementation science studies, identified gaps, and proposed a practical framework for improved SI planning. Using an evidence gap map approach, this study built on Mielke et al.’s (2022) methodology, which identified implementation studies from 2015–2020. The search was updated to include studies from 2021–2023 from PubMed, using the same search strategy and inclusion criteria. Data extraction followed the Guidance for Reporting on Involvement of Patients and the Public reporting checklist. From 10,184 studies, a random sample of 2,005 was screened, adding 162 implementation science studies to Mielke et al.’s 110, totaling 272 studies for SI analysis. SI was reported in 89% of studies, but often lacked depth and strategic planning. Stakeholders were mainly engaged during the preparatory phase. Most studies involved micro- and meso-level stakeholders, rarely including macro-level stakeholders. Few described stakeholder selection or preparation. SI was mostly consultative, via interviews, surveys, and focus groups, with limited active collaboration. SI processes and costs were rarely evaluated. Our findings underscore the need for structured, comprehensive SI planning and offer practical recommendations to strengthen SI efforts in implementation research.

Keywords

implementation science stakeholder involvement evidence gap map research involvement involvement and engagement stakeholder engagement

Background

To address the challenges of translating research into real-world practice, implementation science relies on stakeholder involvement (SI) (Beidas et al., 2022; Neta et al., 2018). Stakeholder involvement can be defined as “an iterative process of actively soliciting the knowledge, experience, judgment and values of stakeholders selected to represent a broad range of direct interests in a particular issue, for the purposes of creating a shared understanding and making relevant, transparent and effective decisions” (Deverka et al., 2012, p. 5). Although the appropriateness of the term ‘stakeholder’ can be questioned (MacLeod, 2022; Rocheleau, nd), we used the definition “individuals, organizations or communities that have a direct interest in the process and outcomes of a project, research or policy endeavor” (Deverka et al., 2012, p. 5). Implementation Science takes a pragmatic approach to studying how best to overcome barriers to implementing interventions in practice (Eccles & Mittman, 2006). From analysing the context to explore and prepare for intervention development or adaptation, to implementation and evaluation of implementation and effectiveness outcomes (Aarons et al., 2011), SI is a key implementation strategy integral to the process of change and essential to study the implementation process.

Stakeholder involvement has become a standard expectation in funding proposals (Neta et al., 2018; Villalobos et al., 2023) due to its benefits across research stages and health contexts, from scaling-up mental health services (Eaton et al., 2011) to implementing prevention and control interventions during public health emergencies (Gilmore et al., 2020). It enhances the acceptability, feasibility, and relevance of research, supporting more sustainable, culturally relevant, and context-sensitive intervention implementation (Osobajo et al., 2023; Shrestha et al., 2022; Wallerstein & Duran, 2010). Engaging (non-research) stakeholders ensures that research addresses unmet needs and insights often missed by researchers alone (Pistollato et al., 2024; Schoemans et al., 2025). Their input can enhance study design (Brett et al., 2014; Maurer et al., 2022), improve recruitment and retention (Crocker et al., 2018), tailor data collection tools (Iezzoni et al., 2017), and clarify complex findings (Maurer et al., 2022). They can help overcome logistical, cultural, and regulatory barriers, ensuring appropriate and effective implementation (Gilmore et al., 2020; Nilsen et al., 2020). Stakeholder involvement helps support clear communication and dissemination, increasing public understanding, awareness, and acceptance of findings (Ball et al., 2019), and it promotes accountability and transparency, fostering trust in research (Esmail et al., 2015; LeClair et al., 2020). Furthermore, SI fosters mutual learning and empowerment by expanding knowledge and skills for both researchers and participants (Ball et al., 2019; Brett et al., 2014; Mathur et al., 2008).

Stakeholder involvement is not a binary concept: it ranges from “informing” to “co-creation”, with a continuum from limited power or decision making ability, up to an equal right and responsibility to impact the project for all participating organizations and/or individuals (Goodman & Sanders Thompson, 2017; Rowe & Frewer, 2005). “Informing” is the lowest level, where information only flows in one direction. Stakeholders have limited power and are only made aware of the project. “Consultation” seeks stakeholder input without making commitment that their voice will carry any weight (Goodman & Sanders Thompson, 2017). “Co-creation” involves active collaboration and shared power throughout the project lifecycle, where information sharing is bidirectional, including feedback on stakeholder contributions (Goodman & Sanders Thompson, 2017; Metz, 2015). The umbrella term “involvement” will be used here to cover these various levels.

Various organisations from different countries, including governments, non-profit and patient organisations, pharmaceutical industries and academia, have developed their own SI guidelines or theories, such as the Patient-Centered Outcomes Research Institute (PCORI) Engagement Rubric (Sheridan et al., 2017), the Patients Active in Research and Dialogues for an Improved Generation of Medicines (PARADIGM) patient engagement toolbox (PARADIGM, 2020), the principles for community-based participatory research (Israel et al., 1998), the INVOLVE briefing notes (NIHR, 2021), the ‘7Ps’ of stakeholders (Concannon et al., 2012), and the ‘ladder of participation’ (Arnstein, 1969), to name a few. Yet, the field is still missing an commonly accepted definition of SI so it remains an complex, unclear construct, leading to varied terms such as “patient and public involvement”, “community participation”, “participatory research”, “stakeholder engagement”, “co-creation”, “partnerships”, and “citizen science” (Ramanadhan et al., 2024; Ray Kristin & Miller Elizabeth, 2017). Moreover, while involving stakeholders has increasingly been emphasized, there is a lack of literature on how involvement costs and value are being calculated (Anggraeni et al., 2019).

While these existing theories offer valuable guidance, they lack practical steps to plan and initiate SI (Ray Kristin & Miller Elizabeth, 2017; Staniszewska et al., 2018). A structured and intentional approach to recruiting and involving stakeholders is essential (Boaz et al., 2018). Starting with clearly defining the purpose of SI, encouraging reflection on its scope (Boaz et al., 2018; Bombard et al., 2018; Potthoff et al., 2023). Planned stakeholder identification and recruitment enables research teams to engage diverse stakeholders to shape feasible, context-specific solutions (Leventon et al., 2016). Specifying the level of involvement and selecting appropriate methods (e.g., focus groups, individual feedback) ensures contributions are meaningful and that they match the input needs, stakeholder expertise and capacity for involvement (Concannon et al., 2019; NIHR, 2021). Finally, evaluating SI efforts allows for ongoing improvement, refining SI efforts to the dynamic real-world environments (Esmail et al., 2015). We propose here an overview of the current state of knowledge regarding SI in implementation science and propose a practical guidance to streamline further SI efforts within the field.

Objective

The overall objective of this work was to quantify the volume of SI and critically review and map current SI characteristics in implementation science studies, identifying gaps and challenges. The review was guided by the following key research questions: What proportion of papers address stakeholder involvement? What are the critical gaps in reporting on the Guidance for Reporting on Involvement of Patients and the Public (GRIPP2) reporting checklist recommended SI concepts (theoretical foundations of SI, stakeholder-related information, methodologies, and evaluation of SI)? For further details on the research questions, see Suppl. Table 4.

Methods

Study Design

Since our aim was to identify gaps that need further exploration, without providing an overview of all existing SI evidence, we used an evidence gap map (EGM), i.e., a visual or descriptive summary that presents an overview of relevant evidence to quickly understand the scope of the available evidence on a given topic (Campbell et al., 2023). Building on Mielke et al.’s (2022) methodology, who created an EGM for methodological approaches to contextual analysis (110 articles included), we conducted a sub-analysis of her data focusing on SI and supplemented it with an updated search to include more recent literature (162 articles included).

Study Selection and Screening

The articles identified from Mielke et al.’s (2022) study (search from 2015 to 2020) were used and, using the same search string, (Suppl. Table 1), were complemented by an updated search covering papers published in Pubmed between January 2021-2023. A random sample comprising 20% of the articles published in each year was selected using a random number generator. This selection was screened for intervention implementation science studies using the inclusion criteria from Mielke et al. (2022) for consistency; all criteria needed to apply (Suppl. Table 1): (a) peer-reviewed articles or study protocols (b) intervention implementation studies (c) using experimental or quasi-experimental designs (d) testing intervention effectiveness (e) in routine practice, (f) including at least one of Brown et al.’s (2017) interventions, and (g) reporting on the implementation pathway evaluation with qualitative or quantitative information on the process or at least one implementation outcome per Proctor et al. (2011) (Suppl. Table 2). Feasibility studies were included only if they assessed at least one additional implementation outcome. Studies had to be (h) written in English or German (i) with full texts available. For SI, we used the definition of Deverka et al. (2012), considering a broad range of stakeholders at the micro-, meso- and macro-level and a broad range of terms referring to the involvement of stakeholders (Suppl. Table 3). Studies were screened in two phases. In the first step, inclusion criteria from suppl. Table 1 were applied to identify intervention implementation science studies, based on review of titles and abstracts, and if relevant, full text. In step 2, the identified studies (n = 272) were further assessed regarding SI. Screening was independently done by the first (KA) and last author (SV) using the web-based tool “Rayyan” (Rayyan). After each step, screening of studies was unblinded and conflicts resolved.

Data Extraction to Explore Gaps in SI

Based on our research questions (Suppl. Table 4), a detailed data extraction table was drafted (this information can be obtained as an excel file from the authors). Data extraction was performed based on the GRIPP2 reporting checklist. It was developed to improve the quality, transparency, and consistency of reporting on patient and public involvement in research (Staniszewska et al., 2017). Although GRIPP2 was designed for patient and public involvement, it provides a strong foundation for reporting on broader SI. Since the GRIPP2 checklist is designed for reporting on involvement in a paper, we used its aspects but grouped them under SI categories rather than paper sections. Furthermore, to enrich our data extraction, we have customized the GRIPP2 checklist, focusing on capturing more specific details on SI by introducing essential aspects within existing categories to gather broader information. Nineteen dimensions of SI were coded: Theoretical gaps reported in SI, frameworks, aims; stakeholders, employing the 7Ps framework to delineate stakeholders across 7 stakeholder groups (Concannon et al., 2012; Deverka et al., 2012) and their identification (Concannon et al., 2019), preparation (Brett et al., 2010) what stakeholder had to do; the research phases in which stakeholders were involved (NIHR, 2021), the varying levels of SI (IAP2 Public Participation Spectrum), the methods and compensation for involvement; the aim for evaluating SI (Popay et al., 2014), the outcomes and impacts of SI (Ball et al., 2019; Staley, 2009), the methods for evaluation, reported robustness of SI methods, evaluation of frameworks, and context and process factors influencing SI.

Data Analysis and Reporting

General study characteristics were analysed descriptively, calculating frequencies and percentages. For the SI data, we initially conducted a descriptive analysis calculating frequencies and percentages to provide a visual representation of the gaps in SI practices.

Results

Screening

The updated PubMed search returned 10,184 records. After selecting the random sample, screening and removing duplicates, 2005 records relevant to implementation science remained. Applying implementation science selection criteria, we added 162 implementation studies to the initial 110 intervention implementation studies previously identified by Mielke et al. (2022), totalling 272 papers for SI data extraction (Suppl. Figure 1).

General Characteristics

Most of the extracted articles were study protocols (n = 195, 72%), primarily focusing on community services (n = 99, 36%). Most articles were from North America (n = 108, 39.7%) and used experimental study designs (n = 239, 88%). Among studies with hybrid designs (n = 193, 71%), most were described as Hybrid Type II (n = 36; 19%). SI was identified in 241 (89%) of 272 studies. (Suppl. Table 5)

Overview of EGM in SI

Figure 1 presents an EGM showcasing the reporting frequency of 241 studies across various dimensions of SI, as assessed using the GRIPP2 reporting checklist. Gaps are evident in aspects such as “Evaluation of SI Approaches” (see also Suppl. Table 6).

Figure 1

EGM Overview Reporting Frequency of Various Dimensions of SI.

Theoretical Underpinning for SI

Most studies (n = 196, 81%) reported an aim for their SI upfront (Figure 1). Only 27 studies (11%) reported underlying theories for SI efforts (Table 1), like community involvement (4%) or participatory research approaches (3%), which conceptualize the principles of empowerment, collaboration, and shared knowledge, emphasizing the active participation of stakeholders. Eight studies (3%) reported a rationale for why involvement is important (theoretical gaps).

Table 1

Applied Theories or SI Approaches to Inform SI and Information on Stakeholders Involved in Intervention Implementation Studies (N = 241)

Identified theories or SI approaches (n = 22)	Reported by studies (n, %)
Reported underlying theoretical foundation or SI approaches:	27 (11%)^a
Community involvement approaches (Banfield et al., 2020; Haynes et al., 2018; Johnson et al., 2018; Levison et al., 2023; Litchman et al., 2022; Liu et al., 2021; McGrath et al., 2021; Nahar et al., 2020; Richards et al., 2021; van Dongen et al., 2019)	10 (4%)
Participatory research approaches (Hartman et al., 2021; Kohrt et al., 2022; Kwan et al., 2020; Lakerveld et al., 2018; Lander et al., 2020; McKay et al., 2018; Seguin-Fowler et al., 2022; van Dongen et al., 2019)	8 (3%)
Citizen engagement approaches (Seguin-Fowler et al., 2022; Smit et al., 2022)	2 (1%)
Other theoretical foundation or methodological approach:	11 (5%)
(Experience-based) Co-design (Lander et al., 2020; Shanley et al., 2019)	2 (1%)
Collaborative intervention planning framework (Markle-Reid et al., 2020; McAiney et al., 2022)	2 (1%)
Team science approach (Smeltzer et al., 2018)	1 (0.4%)
Action cycle (Shanley et al., 2019)	1 (0.4%)
Collaborative participation (Apers et al., 2020)	1 (0.4%)
Darthmouth model for co-design and implementation (Perry et al., 2022)	1 (0.4%)
Patient-centered outcomes research institute engagement rubric (Kearney et al., 2021)	1 (0.4%)
Co-creation approach: Cooperative planning (Grüne et al., 2022)	1 (0.4%)
Normalisation process theory (Clark et al., 2022)	1 (0.4%)
Not reported/unclear	214 (89%)

Stakeholder type	Reported by studies (n, %)
Reported on stakeholders involved:	237 (98%)^b
Micro-level
Target population	176 (73%)
Family/caregivers	67 (28%)
Meso-level
Healthcare stakeholders (provider/institution)	185 (77%)
Public/community	73 (30%)
Other researchers	36 (15%)
Macro-level
Policy maker/(local) government or leadership	40 (17%)
Payers/funders	12 (5%)
Purchaser	1 (0.4%)
Other	51 (21%)
Not reported/unclear	4 (2%)

Stakeholder selection	Reported by studies (n, %)
Reported on methods for selection:	110 (46%)^c
Purposive sampling	77 (32%)
Convenience sampling	28 (12%)
Random sampling	25 (10%)
Snowball sampling	10 (4%)
Not reported/unclear	131 (54%)

Preparation for involvement	Reported by studies (n, %)
Reported on preparation:	7 (3%)^d
Provide project information to stakeholders	3 (1%)
Photovoice training for stakeholders	2 (1%)
Skills learning for stakeholders	1 (0.4%)
Developing cultural competence research team	1 (0.4%)
Not reported/unclear	234 (97%)

^aSeveral studies reported multiple theories, resulting in 22 theories identified in 27 studies.

^bSeveral studies reported several stakeholders.

^cSeveral studies reported several recruitment techniques.

^dSeveral studies reported several forms of preparing stakeholders, 1 study reported stakeholder preparation but did not specify.

Identifying Stakeholders and Preparing for Involvement

Authors described involving diverse stakeholders, predominantly micro- (e.g., target population, family/caregivers) and meso-level stakeholders (e.g., clinicians, healthcare administrators, community, and researchers). Macro-level stakeholders (e.g., payers, policymakers) were less reported. Purposive sampling was the predominant method for identifying stakeholders (n = 77, 32%). Only seven studies (3%) addressed stakeholder preparation before involvement (Hodgins et al., 2022; Hoskins et al., 2022; Kearney et al., 2021; Kohrt et al., 2022; Maxwell et al., 2016; Perry et al., 2022; Pevnick et al., 2021; Quintiliani et al., 2015; Shanley et al., 2019). Of these, three studies prepared stakeholders by providing information beforehand. Two studies prepared stakeholders for photovoice methods by familiarizing them with camera usage. One study provided skill-based training, such as public speaking. One study highlighted the research team’s preparation for involving Indigenous communities through cultural competence development (Table 1).

Methods for Conducting SI

Table 2 illustrates varying levels of SI across different research phases. Most studies involved stakeholders during the preparatory (52%), intervention (50%) or after trial phase (45%). There was little SI reported at baseline (n = 47, 17%) and during dissemination (n = 34, 14%).

Table 2

Methods for Conducting and Evaluating SI in Intervention Implementation Studies (N = 241)

Research phase	Reported by studies (n, % of 241)
Reported on research phase:	217 (90%)^a
Preparatory phase	126 (52%)
Baseline	47 (17%)
During the intervention trial	120 (50%)
After the intervention trial	110 (45%)
Process evaluation	69 (29%)
Dissemination of findings	34 (14%)
Not reported/unclear	24 (10%)

Level of involvement	Reported by studies (n, % of 241)
Reported on level:	238 (99%)^b
Consultation (asking for their views)	213 (88%)
Collaboration/co-creation (working together throughout the research process)	103 (43%)
Informing (outreach and education)	32 (13%)
Stakeholder-led (research initiative and decision-making lies with stakeholders)	3 (1%)
Not reported/unclear	3 (1%)

Methods for SI	Reported by studies (n, % of 241)
Reported on methods:	238 (99%)^c
Interviews	158 (65%)
Surveys	86 (36%)
Focus groups	73 (30%)
Meetings	36 (15%)
Steering/working committee	25 (10%)
Workshops/sessions	25 (10%)
(Community) advisory board	23 (10%)
Observations, notes/reports, logbooks	13 (5%)
Dissemination methods (social media, press release, presentations)	8 (3%)
Discussion groups	6 (3%)
Photovoice	5 (2%)
Jury panel	4 (2%)
Other	22 (9%)
Not reported/unclear	3 (1%)

SI evaluation	Reported by studies (n, % of 241)
Reported on aim SI evaluation:	10 (4%)
Formative assessment (to improve engagement processes)	1 (0.4%)
Process assessment (to identify factors affecting the engagement process)	1 (0.4%)
Summative assessment (to demonstrate the outcomes of the engagement process)	8 (3%)
Not reported/unclear	231 (96%)

Impact of SI	Reported by studies (n, % of 241)
Reported on SI impact:	154 (64%)^d
Impact on aspects of the research project	123 (51%)
Impact on the stakeholders engaged	5 (2%)
Impact on the research team	3 (1%)
Wider impact	38 (16%)
Economic impact (costs related to SI)	9 (4%)
Not reported/unclear	87 (36%)

Note. SI = stakeholder involvement.

^aSeveral studies reported involvement in multiple research phases.

^bSeveral studies reported several levels of involvement.

^cSeveral studies reported several kinds of methods.

^dSeveral studies reported several impacts.

The majority of studies (n = 213, 88%) involved stakeholders through consultation and only three authors (Kearney et al., 2021; Weinert et al., 2022; Wenzel et al., 2021) reported on stakeholder-led research. Tasks assigned to stakeholders were predominantly related to key informant data collection methods such as interviews (n = 158, 65%), focus groups (n = 73, 30%), and surveys (n = 86, 36%) to assess intervention experiences, barriers, facilitators, and implementation outcomes. Eleven percent (n = 27) of studies reported compensating their stakeholders for their input.

Methods for Evaluation of SI

Only ten studies (4%) reported evaluating SI, with few detailing their evaluation methods (Table 2). The main evaluation of SI reported related to its impact on the research project itself, such as its effect on intervention development or adaptation (De Geest et al., 2022). The impact of SI on the stakeholders themselves was noted in five studies (2%) (Kearney et al., 2021; Knight et al., 2016; Lander et al., 2020; Marcussen et al., 2022; Quintiliani et al., 2015) while three studies (1%) reported impact of SI on the research team (Fernandez Turienzo et al., 2023; Kearney et al., 2021; Marcussen et al., 2022). Broader impacts of SI were mentioned in 38 studies (16%). Economic impacts of SI were discussed in nine studies (4%), but only two studies described their evaluation methods (Bartakova et al., 2022; Seguin-Fowler et al., 2022). No study reflected on the theoretical underpinning or methods developed for conducting/evaluating SI. Two studies evaluated the external influence of (contextual, n = 1 and process, n = 1) factors on the conducted SI (Adu et al., 2021; Hoskins et al., 2022).

Discussion

Following the EGM methodology of Mielke et al. (2022), we sought to provide an overview to quickly understand the scope of the available evidence regarding SI in implementation studies. Across the 272 identified implementation science studies, 89% reported some form of SI, which was not surprising given the rising trend of SI in research (Villalobos et al., 2023). However, very few studies reported details regarding theoretical background, methodological aspects, and evaluation of SI, possibly due to a lack of skills and knowledge among scientists regarding SI (Shea et al., 2017), and the absence of SI principles in implementation theories (Villalobos et al., 2023).

Growing Field of Implementation Science

Compared to Mielke et al.’s (2022) study, encompassing 110 implementation science studies over five years, our updated analysis added 162 studies within a two-year timeframe, a 147% increase in publication rate. This growth underscores the rapid global expansion of this field (Rubin, 2023), and reflects advancements in research methodologies promoting real-world adoption of health innovations and ensuring their long-term sustainability (Blevins et al., 2010; Rubin, 2023).

SI in Implementation Science

Lack of Theory Application and SI Planning

While several theories and approaches for SI exist (Greenhalgh et al., 2019), only 11% of our included studies reported using a guiding theory for their SI efforts (Table 1). These theories broadly fell into three categories: (a) community-based approaches prioritizing partnerships with communities to address research questions (N. B. Wallerstein & Duran, 2006), (b) participatory research approaches focusing on engaging stakeholders to drive social change (Cornish et al., 2023), and (c) citizen involvement approaches that include the broader public in scientific research (Canadian Institute of Health Research, 2012). A few lesser-known models were identified, such as the Darthmouth model for co-design and implementation (Perry et al., 2022) and cooperative planning (Grüne et al., 2022). Despite lacking comprehensive step-by-step guidance, they still provide a structure to help ensure that key elements of SI are considered (Concannon et al., 2012).

Although 81% of studies outlined aims for SI, most focused narrowly on consulting stakeholders about implementation barriers and facilitators. Few studies articulated their specific goals for SI, or explained how stakeholder input would shape research decisions. For example, Grüne et al. (2022) involved stakeholders to identify relevant determinants of physical activity, aiming to enhance intervention’s acceptability. Clearly defining the objective of SI is essential, as it helps shape the scope of SI efforts (Boaz et al., 2018; Bombard et al., 2018; Potthoff et al., 2023). However, involving stakeholders can present challenges, such as power imbalances, language barriers, differing beliefs, and limited time, that should be addressed upfront (Shippee et al., 2015; Shrestha et al., 2022). Without thoughtful planning, SI effort risk resulting in misalignment in expectations, superficial involvement and ethical concerns about the transparency and value of stakeholder contributions (Hahn et al., 2017; Lindsay et al., 2015; Snape et al., 2014; Veisi et al., 2022; Wilkins, 2018).

Variability in Stakeholder Identification and Preparation

Our findings reveal a gap in reporting identification and selection of stakeholders. Some studies reported using techniques like convenience and purposive sampling, as described in the literature (Domecq et al., 2014). However, the majority of studies did not specify how stakeholders were identified, nor the criteria used for their selection and recruitment. This lack of detail contrasts with recommendations, which emphasize identifying stakeholders based on their relevance, influence, and interest in the research to ensure diverse perspectives and expertise are represented (Brugha & Varvasovszky, 2000; Concannon et al., 2019). Effective recruitment of stakeholders should involve clear communication to show the mutual benefits of their involvement. This would help build trust, encourage involvement and sustain it over time (Holifield & Williams, 2019).

In clinical research, stakeholders are typically patients and health care providers (Nilsen et al., 2020). However, implementation science requires involving a broader spectrum of stakeholders at the micro-, meso- and macro-level of the system (Brookman-Frazee et al., 2016; Grimshaw et al., 2012; Neta et al., 2018). While we identified substantial involvement of micro- and meso-level stakeholders, macro-level stakeholders, such as payers and policymakers, were rarely mentioned, consistent with findings in implementation grants (Villalobos et al., 2023). These stakeholders should not be overlooked as they play a crucial role in shaping the implementation environment and translating research findings to policy and practice, thereby impacting sustainability (Concannon et al., 2014).

Few studies explicitly addressed how stakeholders or research teams were prepared for involvement, suggesting that the importance of preparatory steps might be underrecognized and/or underreported in the literature. However, preparation is important for building and maintaining trust between stakeholders and researchers throughout the SI process (Boaz et al., 2018; Bombard et al., 2018). Only Shanley et al. (2019) highlighted the importance of preparing research teams, specifically by developing cultural competence and building relationships with stakeholders to increase trust and mutual understanding. Both stakeholders and researchers can benefit from support and training: stakeholders need guidance on the research process (Concannon et al., 2014; Staniszewska et al., 2018), while researchers should receive training on SI (Hubbard et al., 2007). Programs, such as those organized by EUPATI ((EUPATI) https://eupati.eu/eupati-fundamentals/) for instance, provide education and resources to empower patients and stakeholders to be involved in medicines research and development.

Lack of SI within Implementation Phases

Reporting on SI varied across research stages, with the highest involvement seen during the preparatory phase. However, SI should ideally be integrated across all phases of implementation projects (Metz & Boaz, 2016; Ramanadhan et al., 2024). In the preparatory phase, stakeholders can play a key role by helping identify and select relevant context factors for analysis, ensuring the work is grounded in practical needs and local insights (Landes et al., 2019; Moullin et al., 2020). They can also contribute to developing or adapting interventions by interpreting findings, and support implementation by helping to select appropriate strategies (Pérez Jolles et al., 2022; Powell et al., 2017; Ramanadhan et al., 2024). Involvement during evaluation phases is also important, as it helps extend outcomes and supports sustainable implementation (Curran et al., 2022; Ramanadhan et al., 2024; Villalobos et al., 2023). Limited reporting of SI in later stages may reflect a bias in literature, with studies focusing more on trial design and less on subsequent phases.

Interestingly, less than half of the studies reported active collaboration with stakeholders. While continuous collaboration or stakeholder-led research may not always be necessary or feasible, deeper and more frequent SI is preferred to fully benefit from stakeholder perspectives and enhance the research impact (Villalobos et al., 2023). Methods such as focus groups, interviews, surveys, are commonly used for SI, reflecting a reliance on traditional forms of consultation (Bombard et al., 2018; Concannon et al., 2014; Domecq et al., 2014). However, these approaches often position stakeholders more as information sources, serving as research participants rather than active collaborators (Villalobos et al., 2023). The limited level of active involvement may stem from the substantial time, resources flexibility required, demanding substantial commitment from researchers (Brett et al., 2014; Concannon et al., 2014).

Lack of SI Evaluation and Reporting

There is a notable lack of evaluation of SI, both within our EGM (4%) and in the broader literature (Brett et al., 2014; Price et al., 2018; Villalobos et al., 2023). Despite progress in SI, the evaluation of SI lags behind. Without proper evaluation and reporting, it is challenging to determine whether SI is simply a box-ticking exercise or making a meaningful contributor to the project’s success, and how future SI efforts can be improved (Staley, 2015). The limited availability of conceptual frameworks and validated measurement tools is a key barrier (Ball et al., 2019; Brett et al., 2010; Staniszewska et al., 2018). A few tools have been developed to support this process, such as the Public Involvement Impact Assessment Framework (PIIAF) (Popay et al., 2014) and the Public and Patient Engagement Evaluation Tool (PPEET) (Abelson et al., 2019). Combining these tools with a mixed methods approach can offer a comprehensive understanding of the value of SI (Marshall et al., 2024).

Terminology used in evaluating SI can be ambiguous, with terms like “impact” and “outcomes” often used interchangeably (Brett et al., 2010). Yet, they refer to different aspects of SI effectiveness. Outcomes are specific endpoints of a study resulting from SI efforts and were reported by 86% of our identified studies. Examples include identifying stakeholder needs, satisfaction levels, barriers and facilitators to implementation, and feedback on interventions or implementation strategies. However, many studies fail to report how these outcomes are used or what influence they have, with 63% of identified studies reporting the SI impact. Impact refers to the influence of SI. As noted by Brett et al. (2010), SI can impact various areas: the research process (e.g., refining interventions based on end-user needs (Steele Gray et al., 2016), improving implementation through tailored action plans (Hoskins et al., 2022), and disseminating research findings more broadly (Lander et al., 2020)); stakeholders involved (e.g., personal reflection (Quintiliani et al., 2015), knowledge and skill development (Lander et al., 2020); researcher teams (e.g., gaining insights into patient experiences (Marcussen et al., 2022)); and wider society (e.g., community trust in research, widespread integration of interventions, awareness, and influencing policy changes (Afolabi et al., 2022; Barnes et al., 2022; De Geest et al., 2022).

We observed minimal consideration of how context or process factors influence (<1%). Context is the environment and conditions under which SI occurs (e.g., policy, funding). Process factors relate to how involvement is carried out including the level of involvement, timing and design (Brett et al., 2010). Neglecting these factors can limit the potential impact of SI (Brett et al., 2010). This again highlights the importance of an well-planned, strategic approach to SI (Boaz et al., 2018).

Reporting of costs associated with SI was also typically missing, with only 4% reporting an economic outcome, and less than 1 % specifying methods for its evaluation (Bartakova et al., 2022; Seguin-Fowler et al., 2022). Economic evaluations of SI are generally lacking in the literature (Brett et al., 2010), despite potential impacts on project costs. This could stem from a lack of practices, for example compensating stakeholders for their time and expertise was reported in only 11% studies. Economic evaluations are also interesting for macro-level stakeholders such as funders and policymakers who need to assess cost against benefits (Brett et al., 2010).

Limitations

Despite our best efforts, we acknowledge several limitations to our work. First, our approach involved limited screening of the available literature, which could potentially result in missing important information from other studies. Additionally, we were limited by time and resources and therefore limited the scope to peer reviewed literature. Stakeholder involvement information was often scattered and poorly described, making it challenging to identify specific details. It is also possible that research projects involved stakeholders without reporting it, as this is not a standard reporting item for implementation studies. Moreover, most papers were protocol papers, which may only reflect aspirations and not actual practices. Further, our analysis does not clarify whether the absence of SI reporting in different research phases was due to the phases not being part of the study or simply SI being undocumented, preventing us from making this distinction. Due to restricted resources, we were unable to conduct any cross-gap analyses, however, this could be considered in future work. Finally, despite our thorough data extraction process, it remains subject to human judgment due to the current heterogeneity of the SI-terminology and definitions.

Recommendations

We recommend researchers to use SI resources that exist such as the GRIPP2 checklist and systematically reporting on key aspects and challenges linked to SI when publishing on implementation science projects. Systematically embedding SI requirements to implementation science research calls and funding schemes could further advance and normalize SI (Villalobos et al., 2023). Furthermore, there is a need to build an evidence base for evaluating SI, including defining appropriate outcomes and impacts, through systematic reviews on SI evaluation and mixed-methods studies on experiences and impacts.

Regarding conducting SI, researchers should prioritize integrating SI throughout the research process, and develop a comprehensive SI plan tailored to their research project, context, and resources (Boaz et al., 2018; Concannon et al., 2019). Drawing on previous literature and our EGM findings, we have developed recommendations to facilitate the planning of SI in implementation science studies (Table 3 and Suppl. Figure 2): we recommend engaging diverse stakeholders throughout the research project, using varied approaches such as workshops, advisory panels or committees, beyond focus groups or surveys. Researchers should evaluate SI by measuring the process and the impact using mixed methods and incorporate flexibility for continuous review and refinement (Boaz et al., 2018; Concannon et al., 2019; Domecq et al., 2014; Villalobos et al., 2023).

Table 3

Stakeholder Involvement Plan Encompassing Critical Aspects of SI

First steps at the beginning of an implementation science-SI project:
✓ Use a (reporting) theory/framework to ensure critical aspects are covered and reported: GRIPP2 checklist (Staniszewska et al., 2017)
✓ Conceptualize SI according to EPIS-framework: exploration, preparation, implementation, sustainment (see below) (Moullin et al., 2019)
✓ Use 7Ps framework: environmental scan to list all stakeholders and their interactions with the broader context or setting (Concannon et al., 2012; Deverka et al., 2012)
Implementation science phase A		Implementation science phase B
Exploration: plan involvement	Preparation: prepare SI With stakeholders	Implementation: conduct involvement	Sustainment: evaluate and reflect on involvement
✓ Define the purpose and desired outcomes by SI (Concannon et al., 2019)✓ Stakeholder analysis: Map stakeholders, narrowing down the list to stakeholders important for each research phase (reasons/importance) (Barkhordarian et al., 2015; Brugha & Varvasovszky, 2000; Parker et al., 2022; Smits et al., 2020)✓ Prepare the involvement plan (Concannon et al., 2019)○ Determine research activities○ Determine level of involvement for each research activity and each stakeholder (group): Informing (outreach and education), consulting, co-creation or stakeholder-led (Ball et al., 2019; Goodman & Sanders Thompson, 2017; NIHR, 2021; Vaughn & Jacquez, 2020)○ Define expectations and responsibilities, type of work for stakeholders and specify SI methods and approaches to be applied (Engage2020, 2015; Sheridan et al., 2017)✓ Consider resources for SI (budget, time, personnel, training opportunities, contracts) (NIHR, 2021; PARADIGM, 2020)	✓ Recruit stakeholders and prepare for involvement (introduction session or materials, share and discuss the contract, provide training if needed, …) (EUPATI, nd; NIHR, 2022)✓ Establish communication channels and build trust by continuous communication and feedback	✓ Gather, analyze, and incorporate stakeholder input into the research as described in the involvement plan✓ Document and monitor SI efforts✓ Provide feedback to stakeholders on the impact of their input on the project (Mathie et al., 2018)	✓ Evaluate SI efforts using validated tools or different mixed methods focusing on: (Abelson et al., 2019; Center of Excellence on Partnership with Patients and the public, nd; PARADIGM, 2020; Popay et al., 2014)○ Impact on the research, on the stakeholders themselves and on the research team○ SI process (e.g., what worked well/challenging, areas for improvement, appropriateness methods)✓ Document and report on the SI process, outcomes, and impact (Staniszewska et al., 2017)✓ Reflect and refine future involvement efforts if appropriate

Note. The table summarizes recommendations for planning and conducting SI in implementation science studies, providing researchers with a structured approach at each phase, drawing on established principles or tools. The table is designed as a practical reference for researchers, providing actions and resources that can be adapted to their context and needs. SI = stakeholder involvement.

Conclusion

Significant gaps exists in SI within intervention implementation science studies. Despite the increase in studies conducting SI, even in “engaged implementation research,” scientists typically lead projects, with stakeholders minimally and superficially involved, often turning the SI process into a box-ticking exercise (Blevins et al., 2010; Potthoff et al., 2023). Few studies adhere to a structured framework for SI, resulting in further gaps, such as overlooking methods to identify and prepare stakeholders. Furthermore, researchers should strive to improve transparency and clarity in SI, providing detailed descriptions rather than scattering information throughout the text to enhance the accessibility and usability of SI methods and findings (Mathie et al., 2018). Although summarizing all SI evidence is challenging due to its breadth and diversity, we offer recommendations to guide SI planning in implementation science studies.

Supplemental Material

Supplemental Material - Exploring Stakeholder Involvement in Intervention Implementation Studies: Systematic Evidence Synthesis With an Evidence Gap Map Approach

Supplemental Material for Exploring Stakeholder Involvement in Intervention Implementation Studies: Systematic Evidence Synthesis With an Evidence Gap Map Approach by Kristina Arnahoutova, Sabina De Geest, Juliane Mielke, Annette Boaz, Helene Schoemans, and Sabine Valenta in Evaluation & the Health Professions.

Footnotes

ORCID iDs

Juliane Mielke

Annette Boaz

Helene Schoemans

Sabine Valenta

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: KA reports the support of the Leerstoel Marc Boogaerts and the Michaël Van Waeyenberge Fund (King Baudouin Foundation Belgium) as part of the PhD. HS acknowledges the support of the UZ Leuven Klinische onderzoeks- en opleidingsraad (KOOR) of the University Hospitals of Leuven, Belgium.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KA reports a travel grant sponsored by Gilead not related to this work. HS reports having received personal fees from Incyte, Janssen, Novartis, Sanofi and from the Belgian Hematological Society (BHS), as well as research grants from Novartis and the BHS, all paid to her institution and not directly related to this work. She has also received non-financial support (travel grants) from Gilead, Pfizer, the EBMT (European Society for Blood and Marrow transplantation) and the CIBMTR (Center for International Bone Marrow Transplantation Research).

Supplemental Material

Supplemental material for this article is available online.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7

Abelson

Tripp

Kandasamy

Burrows

PPEET Implementation Study Team . (2019). Supporting the evaluation of public and patient engagement in health system organizations: Results from an implementation research study. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 22(5), 1132–1143. https://doi.org/10.1111/hex.12949

Adu

M. K.

Shalaby

Eboreime

Sapara

Nkire

Chawla

Chima

Achor

Osiogo

Chue

Greenshaw

A. J.

Agyapong

V. I.

(2021). Text messaging versus email messaging to support patients with major depressive disorder: Protocol for a randomized hybrid type II effectiveness-implementation trial. JMIR Research Protocols, 10(10), Article e29495. https://doi.org/10.2196/29495

Afolabi

B. B.

Babah

O. A.

Akinajo

O. R.

Adaramoye

V. O.

Adeyemo

T. A.

Balogun

Banke-Thomas

Quao

R. A.

Olorunfemi

Abioye

A. I.

Galadanci

H. S.

Sam-Agudu

N. A.

(2022a). Intravenous versus oral iron for iron deficiency anaemia in pregnant Nigerian women (IVON): Study protocol for a randomised hybrid effectiveness-implementation trial. Trials, 23(1), 763. https://doi.org/10.1186/s13063-022-06690-2

Anggraeni

Gupta

Verrest

H. J. L. M.

(2019). Cost and value of stakeholders participation: A systematic literature review. Environmental Science & Policy, 101, 364–373. https://doi.org/10.1016/j.envsci.2019.07.012

Apers

Vuylsteke

Loos

Smekens

Deblonde

Van Beckhoven

Nöstlinger

(2020). Development and evaluation of an HIV-testing intervention for primary care: Protocol for a mixed methods study. JMIR Research Protocols, 9(8), Article e16486. https://doi.org/10.2196/16486

Arnstein

S. R.

(1969). A ladder of citizen participation. Journal of the American Institute of Planners, 35(4), 216–224. https://doi.org/10.1080/01944366908977225

Ball

Harshfield

Carpenter

Bertscher

Marjanovic

(2019). Patient and public involvement in research: Enabling meaningful contributions. RAND Corporation. https://doi.org/10.7249/RR2678

Banfield

Morse

A. R.

Gulliver

(2020). Contextual influences on the impact of a peer worker-led self-stigma program for people with mental health issues: Protocol for an interventional implementation science study. Implementation Science Communications, 1, 26. https://doi.org/10.1186/s43058-020-00002-y

10.

Barkhordarian

Demerjian

Jan

Sama

Nguyen

Chiappelli

(2015). Stakeholder engagement analysis—a bioethics dilemma in patient-targeted intervention: Patients with temporomandibular joint disorders. Journal of Translational Medicine, 13(1), 15. https://doi.org/10.1186/s12967-014-0366-z

11.

Barnes

L. A. J.

Longman

Adams

Paul

Atkins

Bonevski

Cashmore

Twyman

Bailie

Pearce

Barker

Milat

A. J.

Dorling

Nicholl

Passey

(2022). The MOHMQuit (Midwives and Obstetricians Helping Mothers to Quit Smoking) Trial: Protocol for a stepped-wedge implementation trial to improve best practice smoking cessation support in public antenatal care services. Implementation Science: IS, 17(1), 79. https://doi.org/10.1186/s13012-022-01250-3

12.

Bartakova

Zúñiga

Guerbaai

R.-A.

Basinska

Brunkert

Simon

Denhaerynck

De Geest

Wellens

N. I. H.

Serdaly

Kressig

R. W.

Zeller

Popejoy

L. L.

Nicca

Desmedt

De Pietro

(2022). Health economic evaluation of a nurse-led care model from the nursing home perspective focusing on residents’ hospitalisations. BMC Geriatrics, 22(1), 496. https://doi.org/10.1186/s12877-022-03182-5

13.

Beidas

R. S.

Dorsey

Lewis

C. C.

Lyon

A. R.

Powell

B. J.

Purtle

Saldana

Shelton

R. C.

Stirman

S. W.

Lane-Fall

M. B.

(2022). Promises and pitfalls in implementation science from the perspective of US-based researchers: Learning from a pre-mortem. Implementation Science: IS, 17(1), 55. https://doi.org/10.1186/s13012-022-01226-3

14.

Blevins

Farmer

M. S.

Edlund

Sullivan

Kirchner

J. E.

(2010). Collaborative research between clinicians and researchers: A multiple case study of implementation. Implementation Science: IS, 5(1), 76. https://doi.org/10.1186/1748-5908-5-76

15.

Boaz

Hanney

Borst

O’Shea

Kok

(2018). How to engage stakeholders in research: Design principles to support improvement. Health Research Policy and Systems, 16(1), 60. https://doi.org/10.1186/s12961-018-0337-6

16.

Bombard

Baker

G. R.

Orlando

Fancott

Bhatia

Casalino

Onate

Denis

J.-L.

Pomey

M.-P.

(2018). Engaging patients to improve quality of care: A systematic review. Implementation Science: IS, 13(1), 98. https://doi.org/10.1186/s13012-018-0784-z

17.

Brett

Staniszewska

Mockford

Herron-Marx

Hughes

Tysall

Suleman

(2014). Mapping the impact of patient and public involvement on health and social care research: A systematic review. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 17(5), 637–650. https://doi.org/10.1111/j.1369-7625.2012.00795.x

18.

Brett

Staniszewska

Mockford

Seers

Herron-Marx

Bayliss

(2010). The PIRICOM study: A systematic review of the conceptualisation, measurement, impact and outcomes of patients and public involvement in health and social care research. University of Warwick.

19.

Brookman-Frazee

Stahmer

Stadnick

Chlebowski

Herschell

Garland

A. F.

(2016). Characterizing the use of research-community partnerships in studies of evidence-based interventions in children’s community services. Administration and Policy in Mental Health, 43(1), 93–104. https://doi.org/10.1007/s10488-014-0622-9

20.

Brown

C. H.

Curran

Palinkas

L. A.

Aarons

G. A.

Wells

K. B.

Jones

Collins

L. M.

Duan

Mittman

B. S.

Wallace

Tabak

R. G.

Ducharme

Chambers

D. A.

Neta

Wiley

Landsverk

Cheung

Cruden

(2017). An overview of research and evaluation designs for dissemination and implementation. Annual Review of Public Health, 38, 1–22. https://doi.org/10.1146/annurev-publhealth-031816-044215

21.

Brugha

Varvasovszky

(2000). Stakeholder analysis: A review. Health Policy and Planning, 15(3), 239–246. https://doi.org/10.1093/heapol/15.3.239

22.

Campbell

Tricco

A. C.

Munn

Pollock

Saran

Sutton

White

Khalil

(2023). Mapping reviews, scoping reviews, and evidence and gap maps (EGMs): The same but different—the “Big Picture” review family. Systematic Reviews, 12(1), 45. https://doi.org/10.1186/s13643-023-02178-5

23.

Canadian Institute of Health Research . (2012). Framework for citizen engagement. Government of Canada. https://cihr-irsc.gc.ca/e/41270.html

24.

Center of Excellence on Partnership with Patients and the public . (nd). Evaluation Toolkit. https://ceppp.ca/en/evaluation-toolkit/

25.

Clark

Copsey

Wright-Hughes

McNaught

Bijsterveld

McCormack

Foy

Wilkes

Dickson

J. M.

Meads

Farrin

Johnson

(2022). Cancer patients’ needs assessment in primary care: Study protocol for a cluster randomised controlled trial (cRCT), economic evaluation and normalisation process theory evaluation of the needs assessment tool cancer (CANAssess). BMJ Open, 12(5), Article e051394. https://doi.org/10.1136/bmjopen-2021-051394

26.

Concannon

T. W.

Fuster

Saunders

Patel

Wong

J. B.

Leslie

L. K.

Lau

(2014). A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. Journal of General Internal Medicine, 29(12), 1692–1701. https://doi.org/10.1007/s11606-014-2878-x

27.

Concannon

T. W.

Grant

Welch

Petkovic

Selby

Crowe

Synnot

Greer-Smith

Mayo-Wilson

Tambor

Tugwell

Multi Stakeholder Engagement MuSE Consortium . (2019). Practical guidance for involving stakeholders in health research. Journal of General Internal Medicine, 34(3), 458–463. https://doi.org/10.1007/s11606-018-4738-6

28.

Concannon

T. W.

Meissner

Grunbaum

J. A.

McElwee

Guise

J.-M.

Santa

Conway

P. H.

Daudelin

Morrato

E. H.

Leslie

L. K.

(2012). A new taxonomy for stakeholder engagement in patient-centered outcomes research. Journal of General Internal Medicine, 27(8), 985–991. https://doi.org/10.1007/s11606-012-2037-1

29.

Cornish

Breton

Moreno-Tabarez

Delgado

Rua

de-Graft Aikins

Hodgetts

(2023). Participatory action research. Nature Reviews Methods Primers, 3(1), 34. https://doi.org/10.1038/s43586-023-00214-1

30.

Crocker

J. C.

Ricci-Cabello

Parker

Hirst

J. A.

Chant

Petit-Zeman

Evans

Rees

(2018). Impact of patient and public involvement on enrolment and retention in clinical trials: Systematic review and meta-analysis. BMJ, 363, k4738. https://doi.org/10.1136/bmj.k4738

31.

Curran

G. M.

Landes

S. J.

McBain

S. A.

Pyne

J. M.

Smith

J. D.

Fernandez

M. E.

Chambers

D. A.

Mittman

B. S.

(2022). Reflections on 10 years of effectiveness-implementation hybrid studies. Frontiers in Health Services, 2, 1053496. https://doi.org/10.3389/frhs.2022.1053496

32.

De Geest

Valenta

Ribaut

Gerull

Mielke

Simon

Bartakova

Kaier

Eckstein

Leppla

Teynor

SMILe team . (2022). The SMILe integrated care model in allogeneic SteM cell TransplantatIon faciLitated by eHealth: A protocol for a hybrid effectiveness-implementation randomised controlled trial. BMC Health Services Research, 22(1), 1067. https://doi.org/10.1186/s12913-022-08293-8

33.

Deverka

P. A.

Lavallee

D. C.

Desai

P. J.

Esmail

L. C.

Ramsey

S. D.

Veenstra

D. L.

Tunis

S. R.

(2012). Stakeholder participation in comparative effectiveness research: Defining a framework for effective engagement. Journal of Comparative Effectiveness Research, 1(2), 181–194. https://doi.org/10.2217/cer.12.7

34.

Domecq

J. P.

Prutsky

Elraiyah

Wang

Nabhan

Shippee

Brito

J. P.

Boehmer

Hasan

Firwana

Erwin

Eton

Sloan

Montori

Asi

Dabrh

A. M. A.

Murad

M. H.

(2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14, 89. https://doi.org/10.1186/1472-6963-14-89

35.

Eaton

McCay

Semrau

Chatterjee

Baingana

Araya

Ntulo

Thornicroft

Saxena

(2011). Scale up of services for mental health in low-income and middle-income countries. Lancet (London, England), 378(9802), 1592–1603. https://doi.org/10.1016/S0140-6736(11)60891-X

36.

Eccles

M. P.

Mittman

B. S.

(2006). Welcome to implementation science. Implementation Science, 1(1), 1. https://doi.org/10.1186/1748-5908-1-1

37.

Engage2020 . (2015). Engage2020 action catalogue. https://actioncatalogue.eu/

38.

Esmail

Moore

Rein

(2015). Evaluating patient and stakeholder engagement in research: Moving from theory to practice. Journal of Comparative Effectiveness Research, 4(2), 133–145. https://doi.org/10.2217/cer.14.79

39.

EUPATI . (nd). EUPATI Fundamentals—learn the basics of patient engagement. https://eupati.eu/eupati-fundamentals/

40.

Fernandez Turienzo

Hull

L. H.

Coxon

Bollard

Cross

Seed

P. T.

Shennan

A. H.

Sandall

POPPIE Pilot Collaborative Group . (2023). A continuity of care programme for women at risk of preterm birth in the UK: Process evaluation of a hybrid randomised controlled pilot trial. PLoS One, 18(1), Article e0279695. https://doi.org/10.1371/journal.pone.0279695

41.

Gilmore

Ndejjo

Tchetchia

de Claro

Mago

Diallo

A. A.

Lopes

Bhattacharyya

(2020). Community engagement for COVID-19 prevention and control: A rapid evidence synthesis. BMJ Global Health, 5(10), Article e003188. https://doi.org/10.1136/bmjgh-2020-003188

42.

Goodman

M. S.

Sanders Thompson

V. L.

(2017). The science of stakeholder engagement in research: Classification, implementation, and evaluation. Translational Behavioral Medicine, 7(3), 486–491. https://doi.org/10.1007/s13142-017-0495-z

43.

Greenhalgh

Hinton

Finlay

Macfarlane

Fahy

Clyde

Chant

(2019). Frameworks for supporting patient and public involvement in research: Systematic review and co-design pilot. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 22(4), 785–801. https://doi.org/10.1111/hex.12888

44.

Grimshaw

J. M.

Eccles

M. P.

Lavis

J. N.

Hill

S. J.

Squires

J. E.

(2012). Knowledge translation of research findings. Implementation Science: IS, 7(1), 50. https://doi.org/10.1186/1748-5908-7-50

45.

Grüne

Popp

Carl

Semrau

Pfeifer

(2022). Examining the sustainability and effectiveness of co-created physical activity interventions in vocational education and training: A multimethod evaluation. BMC Public Health, 22(1), 765. https://doi.org/10.1186/s12889-022-13133-9

46.

Hahn

D. L.

Hoffmann

A. E.

Felzien

LeMaster

J. W.

Fagnan

L. J.

(2017). Tokenism in patient engagement. Family Practice, 34(3), 290–295. https://doi.org/10.1093/fampra/cmw097

47.

Hartman

E. A. R.

Groen

W. G.

Heltveit-Olsen

S. R.

Lindbaek

Hoye

Sundvall

P.-D.

Gunnarsson

Skoglund

Snaebjörnsson Arnljots

Godycki-Cwirko

Kowalczyk

Platteel

T. N.

Zuithoff

N. P. A.

Monnier

A. A.

Verheij

T. J. M.

Hertogh

C. M. P. M.

van de Pol

A. C.

(2021). Multifaceted antibiotic stewardship intervention using a participatory-action-research approach to improve antibiotic prescribing for urinary tract infections in frail elderly (ImpresU): Study protocol for a European qualitative study followed by a pragmatic cluster randomised controlled trial. BMJ Open, 11(10), Article e052552. https://doi.org/10.1136/bmjopen-2021-052552

48.

Haynes

Turner

Smith

Curran

Bryant-Moore

Ounpraseuth

S. T.

Kramer

Harris

Hutchins

Yeary

K. H.-C. K.

(2018). Reducing depressive symptoms through behavioral activation in churches: A hybrid-2 randomized effectiveness-implementation design. Contemporary Clinical Trials, 64, 22–29. https://doi.org/10.1016/j.cct.2017.11.010

49.

Hodgins

Ostojic

Lawson

K. D.

Samir

Webster

Rogers

Henry

Murphy

Lingam

Raman

Mendoza Diaz

Dadich

Eapen

Rimes

Woolfenden

(2022). Study protocol for a real-world evaluation of an integrated child and family health hub for migrant and refugee women. BMJ Open, 12(8), Article e061002. https://doi.org/10.1136/bmjopen-2022-061002

50.

Holifield

Williams

K. C.

(2019). Recruiting, integrating, and sustaining stakeholder participation in environmental management: A case study from the great lakes areas of concern. Journal of Environmental Management, 230, 422–433. https://doi.org/10.1016/j.jenvman.2018.09.081

51.

Hoskins

Sanchez

A. L.

Hoffacker

Momplaisir

Gross

Brady

K. A.

Pettit

A. R.

Zentgraf

Mills

Coley

Beidas

R. S.

(2022). Implementation mapping to plan for a hybrid trial testing the effectiveness and implementation of a behavioral intervention for HIV medication adherence and care retention. Frontiers in Public Health, 10, 872746. https://doi.org/10.3389/fpubh.2022.872746

52.

Hubbard

Kidd

Donaghy

McDonald

Kearney

(2007). A review of literature about involving people affected by cancer in research, policy and planning and practice. Patient Education and Counseling, 65(1), 21–33. https://doi.org/10.1016/j.pec.2006.02.009

53.

Iezzoni

L. I.

Matulewicz

Marsella

S. A.

Warsett

K. S.

Heaphy

Donelan

(2017). Collaborative design of a health care experience survey for persons with disability. Disability and Health Journal, 10(2), 231–239. https://doi.org/10.1016/j.dhjo.2016.12.022

54.

Israel

B. A.

Schulz

A. J.

Parker

E. A.

Becker

A. B.

(1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173

55.

Johnson

Gilbert

Hunt

Metsch

Goddard-Eckrich

Richards

Tibbetts

Rowe

J. C.

Wainberg

M. L.

El-Bassel

(2018). The effectiveness of a group-based computerized HIV/STI prevention intervention for black women who use drugs in the criminal justice system: Study protocol for E-WORTH (Empowering African-American Women on the Road to Health), a Hybrid Type 1 randomized controlled trial. Trials, 19(1), 486. https://doi.org/10.1186/s13063-018-2792-3

56.

Kearney

S. M.

Williams

Nikolajski

Park

M. J.

Kraemer

K. L.

Landsittel

Kang

Malito

Schuster

(2021). Stakeholder impact on the implementation of integrated care: Opportunities to consider for patient-centered outcomes research. Contemporary Clinical Trials, 101, 106256. https://doi.org/10.1016/j.cct.2020.106256

57.

Knight

D. K.

Belenko

Wiley

Robertson

A. A.

Arrigona

Dennis

Bartkowski

J. P.

McReynolds

L. S.

Becan

J. E.

Knudsen

H. K.

Wasserman

G. A.

Rose

DiClemente

Leukefeld

JJ-TRIALS Cooperative . (2016). Juvenile justice-translational research on interventions for adolescents in the legal system (JJ-TRIALS): A cluster randomized trial targeting system-wide improvement in substance use services. Implementation Science: IS, 11, 57. https://doi.org/10.1186/s13012-016-0423-5

58.

Kohrt

B. A.

Turner

E. L.

Gurung

Wang

Neupane

Luitel

N. P.

Kartha

M. R.

Poudyal

Singh

Rai

Baral

P. P.

McCutchan

Gronholm

P. C.

Hanlon

Lempp

Lund

Thornicroft

Gautam

Jordans

M. J. D.

(2022). Implementation strategy in collaboration with people with lived experience of mental illness to reduce stigma among primary care providers in Nepal (RESHAPE): Protocol for a type 3 hybrid implementation effectiveness cluster randomized controlled trial. Implementation Science: IS, 17(1), 39. https://doi.org/10.1186/s13012-022-01202-x

59.

Kwan

B. M.

Dickinson

L. M.

Glasgow

R. E.

Sajatovic

Gritz

Holtrop

J. S.

Nease

D. E. J.

Ritchie

Nederveld

Gurfinkel

Waxmonsky

J. A.

(2020). The invested in diabetes study protocol: A cluster randomized pragmatic trial comparing standardized and patient-driven diabetes shared medical appointments. Trials, 21(1), 65. https://doi.org/10.1186/s13063-019-3938-7

60.

Lakerveld

Mackenbach

J. D.

de Boer

Brandhorst

Broerse

J. E. W.

de Bruijn

G.-J.

Feunekes

Gillebaart

Harbers

Hoenink

Klein

Mensink

Middel

de Ridder

D. T. D.

Rutters

Sluijs

van der Schouw

Y. T.

Schuitmaker

T. J.

Te Velde

S. J.

Beulens

J. W. J.

(2018). Improving cardiometabolic health through nudging dietary behaviours and physical activity in low SES adults: Design of the Supreme Nudge project. BMC Public Health, 18(1), 899. https://doi.org/10.1186/s12889-018-5839-1

61.

Lander

Mazzoli

Cassar

Symington

Salmon

(2020). Embedding active pedagogies within pre-service teacher education: Implementation considerations and recommendations. Children, 7(11), 207. https://doi.org/10.3390/children7110207

62.

Landes

S. J.

McBain

S. A.

Curran

G. M.

(2019). An introduction to effectiveness-implementation hybrid designs. Psychiatry Research, 280, 112513. https://doi.org/10.1016/j.psychres.2019.112513

63.

LeClair

A. M.

Kotzias

Garlick

J. A.

Cole

A. M.

Kwon

Lightfoot

Concannon

T. W.

(2020). Facilitating stakeholder engagement in early stage translational research. PLoS One, 15(7), Article e0235400. https://doi.org/10.1371/journal.pone.0235400. https://api.semanticscholar.org/CorpusID:220327678

64.

Leventon

Fleskens

Claringbould

Schwilch

Hessel

(2016). An applied methodology for stakeholder identification in transdisciplinary research. Sustainability Science, 11(5), 763–775. https://doi.org/10.1007/s11625-016-0385-1

65.

Levison

J. H.

Krane

Donelan

Aschbrenner

Trieu

H. D.

Chau

Wilson

Oreskovic

N. M.

Irwin

Iezzoni

L. I.

Xie

Samuels

Silverman

Batson

Fathi

Gamse

Holland

Wolfe

Shellenberger

Bartels

(2023). Best practices to reduce COVID-19 in group homes for individuals with serious mental illness and intellectual and developmental disabilities: Protocol for a hybrid type 1 effectiveness-implementation cluster randomized trial. Contemporary Clinical Trials, 125, 107053. https://doi.org/10.1016/j.cct.2022.107053

66.

Lindsay

Kovacs Burns

Jackson

Surgeoner

Gallivan

(2015). Meaningful and effective patient engagement: What matters most to stakeholders. Patient Experience Journal, 2(1), 18–28. https://doi.org/10.35680/2372-0247.1069

67.

Litchman

M. L.

Kwan

B. M.

Zittleman

Simonetti

Iacob

Curcija

Neuberger

Latendress

Oser

T. K.

(2022). A telehealth diabetes intervention for rural populations: Protocol for a randomized controlled trial. JMIR Research Protocols, 11(6), Article e34255. https://doi.org/10.2196/34255

68.

Liu

Chen

Hao

Zhang

Bai

Song

Chen

Xie

Liu

Caine

E. D.

Hou

(2021). Sequential multiple assignment randomised trial of a brief contact intervention for suicide risk management among discharged psychiatric patients: An implementation study protocol. BMJ Open, 11(11), Article e054131. https://doi.org/10.1136/bmjopen-2021-054131

69.

MacLeod

(2022). ‘Stakeholder engagement’ needs A Rethink. Here’s what will replace it. Forbes. https://www.forbes.com/councils/theyec/2022/08/24/stakeholder-engagement-needs-a-rethink-heres-what-will-replace-it/

70.

Marcussen

Berring

Hørder

Søndergaard

Nørgaard

(2022). Development of a model for shared care between general practice and mental healthcare: A protocol for a co-production study. BMJ Open, 12(10), Article e061575. https://doi.org/10.1136/bmjopen-2022-061575

71.

Markle-Reid

McAiney

Ganann

Fisher

Gafni

Gauthier

A. P.

Heald-Taylor

McElhaney

Ploeg

Urajnik

D. J.

Valaitis

Whitmore

(2020). Study protocol for a hospital-to-home transitional care intervention for older adults with multiple chronic conditions and depressive symptoms: A pragmatic effectiveness-implementation trial. BMC Geriatrics, 20(1), 240. https://doi.org/10.1186/s12877-020-01638-0

72.

Marshall

D. A.

Suryaprakash

Lavallee

D. C.

McCarron

T. L.

Zelinsky

Barker

K. L.

MacKean

Santana

M. J.

Moayyedi

Bryan

(2024). Studying how patient engagement influences research: A mixed methods study. The Patient, 17(4), 379–395. https://doi.org/10.1007/s40271-024-00685-8

73.

Mathie

Wythe

Munday

Millac

Rhodes

Roberts

Smeeton

Poland

Jones

(2018). Reciprocal relationships and the importance of feedback in patient and public involvement: A mixed methods study. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 21(5), 899–908. https://doi.org/10.1111/hex.12684

74.

Mathur

V. N.

Price

A. D. F.

Austin

(2008). Conceptualizing stakeholder engagement in the context of sustainability and its assessment. Construction Management & Economics, 26(6), 601–609. https://doi.org/10.1080/01446190802061233

75.

Maurer

Mangrum

Hilliard-Boone

Amolegbe

Carman

K. L.

Forsythe

Mosbacher

Lesch

J. K.

Woodward

(2022). Understanding the influence and impact of stakeholder engagement in patient-centered outcomes research: A qualitative study. Journal of General Internal Medicine, 37(Suppl 1), 6–13. https://doi.org/10.1007/s11606-021-07104-w

76.

Maxwell

A. E.

Danao

L. L.

Cayetano

R. T.

Crespi

C. M.

Bastani

(2016). Implementation of an evidence-based intervention to promote colorectal cancer screening in community organizations: A cluster randomized trial. Translational Behavioral Medicine, 6(2), 295–305. https://doi.org/10.1007/s13142-015-0349-5

77.

McAiney

Markle-Reid

Ganann

Whitmore

Valaitis

Urajnik

D. J.

Fisher

Ploeg

Petrie

McMillan

McElhaney

J. E.

(2022). Implementation of the community assets supporting Transitions (CAST) transitional care intervention for older adults with multimorbidity and depressive symptoms: A qualitative descriptive study. PLoS One, 17(8), Article e0271500. https://doi.org/10.1371/journal.pone.0271500

78.

McGrath

Murphy

Richardson

(2021). Study protocol: Evaluation of sheds for life (SFL): A community-based men’s health initiative designed ‘for shedders by shedders’ in Irish men’s sheds using a hybrid effectiveness-implementation design. BMC Public Health, 21(1), 801. https://doi.org/10.1186/s12889-021-10823-8

79.

McKay

Nettlefold

Bauman

Hoy

Gray

S. M.

Lau

Sims-Gould

(2018). Implementation of a co-designed physical activity program for older adults: Positive impact when delivered at scale. BMC Public Health, 18(1), 1289. https://doi.org/10.1186/s12889-018-6210-2

80.

Metz

(2015). Implementation brief: The potential of co-creation in implementation science. https://nirn.fpg.unc.edu/resources/brief-2-potential-co-creation-implementation-science

81.

Metz

Boaz

(2016). Where are the stakeholders in implementation science? Integration and Implementation Insights. https://i2insights.org/2016/09/08/co-creation-and-implementation-science/

82.

Mielke

Brunkert

Zúñiga

Simon

Zullig

L. L.

De Geest

(2022). Methodological approaches to study context in intervention implementation studies: An evidence gap map. BMC Medical Research Methodology, 22(1), 320. https://doi.org/10.1186/s12874-022-01772-w

83.

Moullin

J. C.

Dickson

K. S.

Stadnick

N. A.

Albers

Nilsen

Broder-Fingert

Mukasa

Aarons

G. A.

(2020). Ten recommendations for using implementation frameworks in research and practice. Implementation Science Communications, 1(1), 42. https://doi.org/10.1186/s43058-020-00023-7

84.

Moullin

J. C.

Dickson

K. S.

Stadnick

N. A.

Rabin

Aarons

G. A.

(2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science: IS, 14(1), 1. https://doi.org/10.1186/s13012-018-0842-6

85.

Nahar

van Marwijk

Gibson

Musinguzi

Anthierens

Ford

Bremner

S. A.

Bowyer

Le Reste

J. Y.

Sodi

Bastiaens

(2020). A protocol paper: Community engagement interventions for cardiovascular disease prevention in socially disadvantaged populations in the UK: An implementation research study. Global Health Research and Policy, 5, 12. https://doi.org/10.1186/s41256-020-0131-1

86.

Neta

Brownson

R. C.

Chambers

D. A.

(2018). Opportunities for epidemiologists in implementation science: A primer. American Journal of Epidemiology, 187(5), 899–910. https://doi.org/10.1093/aje/kwx323

87.

NIHR . (2021). Briefing notes for researchers: Public involvement in NHS, health and social care research. NIHR. https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371?pr=

88.

NIHR . (2022). Starting out guide—why and how to get involved in research. https://www.nihr.ac.uk/documents/Starting-Out-Guide/30145

89.

Nilsen

E. R.

Stendal

Gullslett

M. K.

(2020). Implementation of eHealth technology in community health care: The complexity of stakeholder involvement. BMC Health Services Research, 20(1), 395. https://doi.org/10.1186/s12913-020-05287-2

90.

Osobajo

O. A.

Oke

Ajimmy

Otitoju

Adeyanju

G. C.

(2023). The role of culture in stakeholder engagement: Its implication for open innovation. Journal of Open Innovation: Technology, Market, and Complexity, 9(2), 100058. https://doi.org/10.1016/j.joitmc.2023.100058

91.

PARADIGM . (2020). Patient engagement toolbox—PARADIGM. https://imi-paradigm.eu/petoolbox/

92.

Parker

Tomlinson

Concannon

T. W.

Akl

Petkovic

Welch

V. A.

Crowe

Palm

Marusic

Ekanem

Akl

I. B.

Saginur

Moja

Kuchenmüller

Slingers

Teixeira

Dormer

Lang

Arayssi

Tugwell

(2022). Factors to consider during identification and invitation of individuals in a multi-stakeholder research partnership. Journal of General Internal Medicine, 37(16), 4047–4053. https://doi.org/10.1007/s11606-022-07411-w

93.

Pérez Jolles

Willging

C. E.

Stadnick

N. A.

Crable

E. L.

Lengnick-Hall

Hawkins

Aarons

G. A.

(2022). Understanding implementation research collaborations from a co-creation lens: Recommendations for a path forward. Frontiers in Health Services, 2, 942658. https://doi.org/10.3389/frhs.2022.942658

94.

Perry

L. M.

Morken

Peipert

J. D.

Yanez

Garcia

S. F.

Barnard

Hirschhorn

L. R.

Linder

J. A.

Jordan

Ackermann

R. T.

Harris

Kircher

Mohindra

Aggarwal

Frazier

Coughlin

Bedjeti

Weitzel

Nelson

E. C.

Cella

(2022). Patient-reported outcome dashboards within the electronic health record to support shared decision-making: Protocol for Co-design and clinical evaluation with patients with advanced cancer and chronic kidney disease. JMIR Research Protocols, 11(9), Article e38461. https://doi.org/10.2196/38461

95.

Pevnick

J. M.

Keller

M. S.

Kennelty

K. A.

Nuckols

T. K.

E. M.

Amer

Anderson

Armbruster

Conti

Fanikos

Guan

Knight

Leang

D. W.

Llamas-Sandoval

Matta

Moriarty

Murry

L. T.

Muske

A. M.

Nguyen

A. T.

Schnipper

J. L.

(2021). The pharmacist discharge care (PHARM-DC) study: A multicenter RCT of pharmacist-directed transitional care to reduce post-hospitalization utilization. Contemporary Clinical Trials, 106, 106419. https://doi.org/10.1016/j.cct.2021.106419

96.

Pistollato

Burkhart

Deceuninck

Bernasconi

Di Virgilio

Emili

Fauvel

A.-C.

Ferreira Bastos

Gastaldello

Gerardi

Habermann

J. K.

Hanes

Kyriakopoulou

Lanka

Lauriola

Laverty

Maisonneuve

B. G. C.

Mennecozzi

Pappalardo

Constantino

(2024). What public health challenges and unmet medical needs would benefit from interdisciplinary collaboration in the EU? A survey and multi-stakeholder debate. Frontiers in Public Health, 12, 1417684. https://doi.org/10.3389/fpubh.2024.1417684

97.

Popay

Collins

with the PiiAF Study Group . (2014). The public involvement impact assessment framework guidance. Universities of Lancaster. https://piiaf.org.uk/

98.

Potthoff

Finch

Bührmann

Etzelmüller

van Genugten

C. R.

Girling

May

C. R.

Perkins

Vis

Rapley

ImpleMentAll consortium . (2023). Towards an Implementation-STakeholder Engagement Model (I-STEM) for improving health and social care services. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 26(5), 1997–2012. https://doi.org/10.1111/hex.13808

99.

Powell

B. J.

Beidas

R. S.

Lewis

C. C.

Aarons

G. A.

McMillen

J. C.

Proctor

E. K.

Mandell

D. S.

(2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services & Research, 44(2), 177–194. https://doi.org/10.1007/s11414-015-9475-6

100.

Price

Schroter

Snow

Hicks

Harmston

Staniszewska

Parker

Richards

(2018). Frequency of reporting on patient and public involvement (PPI) in research studies published in a general medical journal: A descriptive study. BMJ Open, 8(3), Article e020452. https://doi.org/10.1136/bmjopen-2017-020452

101.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

102.

Quintiliani

L. M.

Russinova

Z. L.

Bloch

P. P.

Truong

Xuan

Pbert

Lasser

K. E.

(2015). Patient navigation and financial incentives to promote smoking cessation in an underserved primary care population: A randomized controlled trial protocol. Contemporary Clinical Trials, 45(Pt B), 449–457. https://doi.org/10.1016/j.cct.2015.09.005

103.

Ramanadhan

Alemán

Bradley

C. D.

Cruz

J. L.

Safaeinili

Simonds

Aveling

E.-L.

(2024). Using participatory implementation science to advance health equity. Annual Review of Public Health, 45(1), 47–67. https://doi.org/10.1146/annurev-publhealth-060722-024251

104.

Ray Kristin

Miller Elizabeth (2017). Strengthening stakeholder-engaged research and research on stakeholder engagement. Journal of Comparative Effectiveness Research, 6(4), 375–389. https://doi.org/10.2217/cer-2016-0096

105.

Rayyan . (nd). Rayyan: AI-powered systematic review management platform. [Computer software]. https://www.rayyan.ai/

106.

Richards

Begay

Chambers

R. A.

Patel

Mayhew

Allison-Burbank

Gishie

Tsingine

Badoni

Staley

Harvey

Tsosie

Begay

Mitchell

Tingey

(2021). Azhe’é Bidziil (strong fathers): Study protocol for the pilot evaluation of an American Indian fatherhood program to improve the Health and Wellbeing of Diné (Navajo) fathers. Frontiers in Public Health, 9, 790024. https://doi.org/10.3389/fpubh.2021.790024

107.

Rocheleau

(nd). The impact of language: Rethinking the term “stakeholder”. Laridae. https://laridaemc.com/the-impact-of-language-rethinking-the-term-stakeholder/

108.

Rowe

Frewer

L. J.

(2005). A typology of public engagement mechanisms. Science, Technology & Human Values, 30(2), 251–290. https://doi.org/10.1177/0162243904271724

109.

Rubin

(2023). It takes an average of 17 Years for evidence to change practice-the burgeoning field of implementation science seeks to speed things up. JAMA, 329(16), 1333–1336. https://doi.org/10.1001/jama.2023.4387

110.

Schoemans

Stienissen

Goris

Stewart

S. K.

Cowden

Arnahoutova

Pavletic

S. Z.

Greinix

Barata

Warwick

Bolanos

Barbosa

Bouguet

Lhenry-Yvon

Manninen

R.-L.

Pavukova

Tavori

Amerongen

van M.

Wolff

Lawitschka

Herrmann

(2025). Ask, do not tell: Consulting a patient advisory board to understand unmet needs of patients with GVHD in Europe. The Lancet. Haematology, 12(3), e214–e223. https://doi.org/10.1016/S2352-3026(24)00357-0

111.

Seguin-Fowler

R. A.

Hanson

K. L.

Villarreal

Rethorst

C. D.

Ayine

Folta

S. C.

Maddock

J. E.

Patterson

M. S.

Marshall

G. A.

Volpe

L. C.

Eldridge

G. D.

Kershaw

Luong

Wang

Kenkel

(2022). Evaluation of a civic engagement approach to catalyze built environment change and promote healthy eating and physical activity among rural residents: A cluster (community) randomized controlled trial. BMC Public Health, 22(1), 1674. https://doi.org/10.1186/s12889-022-13653-4

112.

Shanley

D. C.

Hawkins

Page

Shelton

Liu

Webster

Moritz

K. M.

Barry

Ziviani

Morrissey

O’Callaghan

Wood

Katsikitis

Reid

(2019). Protocol for the Yapatjarrathati project: A mixed-method implementation trial of a tiered assessment process for identifying fetal alcohol spectrum disorders in a remote Australian community. BMC Health Services Research, 19(1), 649. https://doi.org/10.1186/s12913-019-4378-5

113.

Shea

C. M.

Young

T. L.

Powell

B. J.

Rohweder

Enga

Z. K.

Scott

J. E.

Carter-Edwards

Corbie-Smith

(2017). Researcher readiness for participating in community-engaged dissemination and implementation research: A conceptual framework of core competencies. Translational Behavioral Medicine, 7(3), 393–404. https://doi.org/10.1007/s13142-017-0486-0

114.

Sheridan

Schrandt

Forsythe

Hilliard

T. S.

Paez

K. A.

Advisory Panel on Patient Engagement 2013 inaugural panel . (2017). The PCORI engagement rubric: Promising practices for partnering in research. The Annals of Family Medicine, 15(2), 165–170. https://doi.org/10.1370/afm.2042

115.

Shippee

N. D.

Domecq Garces

J. P.

Prutsky Lopez

G. J.

Wang

Elraiyah

T. A.

Nabhan

Brito

J. P.

Boehmer

Hasan

Firwana

Erwin

P. J.

Montori

V. M.

Murad

M. H.

(2015). Patient and service user engagement in research: A systematic review and synthesized framework. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 18(5), 1151–1166. https://doi.org/10.1111/hex.12090

116.

Shrestha

Tamrakar

Shrestha

Karmacharya

B. M.

Shrestha

Pyakurel

Spiegelman

(2022). Stakeholder engagement in a hypertension and diabetes prevention research program: Description and lessons learned. PLoS One, 17(10), Article e0276478. https://doi.org/10.1371/journal.pone.0276478. https://api.semanticscholar.org/CorpusID:253043535

117.

Smeltzer

M. P.

Rugless

F. E.

Jackson

B. M.

Berryman

C. L.

Faris

N. R.

Ray

M. A.

Meadows

Patel

A. A.

Roark

K. S.

Kedia

S. K.

DeBon

M. M.

Crossley

F. J.

Oliver

McHugh

L. M.

Hastings

Osborne

Ill

Osarogiagbon

R. U.

(2018). Pragmatic trial of a multidisciplinary lung cancer care model in a community healthcare setting: Study design, implementation evaluation, and baseline clinical results. Translational Lung Cancer Research, 7(1), 88–102. https://doi.org/10.21037/tlcr.2018.01.02

118.

Smit

D. J. M.

van Oostrom

S. H.

Engels

J. A.

van der Beek

A. J.

Proper

K. I.

(2022). A study protocol of the adaptation and evaluation by means of a cluster-RCT of an integrated workplace health promotion program based on a European good practice. BMC Public Health, 22(1), 1028. https://doi.org/10.1186/s12889-022-13352-0

119.

Smits

D.-W.

van Meeteren

Klem

Alsem

Ketelaar

(2020). Designing a tool to support patient and public involvement in research projects: The Involvement Matrix. Research Involvement and Engagement, 6, 30. https://doi.org/10.1186/s40900-020-00188-4

120.

Snape

Kirkham

Britten

Froggatt

Gradinger

Lobban

Popay

Wyatt

Jacoby

(2014). Exploring perceived barriers, drivers, impacts and the need for evaluation of public involvement in health and social care research: A modified Delphi study. BMJ Open, 4(6), Article e004943. https://doi.org/10.1136/bmjopen-2014-004943

121.

Staley

(2009). Exploring impact: Public involvement in NHS, public health and social care research. INVOLVE.

122.

Staley

(2015). ‘Is it worth doing?’ Measuring the impact of patient and public involvement in research. Research Involvement and Engagement, 1(1), 6. https://doi.org/10.1186/s40900-015-0008-5

123.

Staniszewska

Brett

Simera

Seers

Mockford

Goodlad

Altman

D. G.

Moher

Barber

Denegri

Entwistle

Littlejohns

Morris

Suleman

Thomas

Tysall

(2017). GRIPP2 reporting checklists: Tools to improve reporting of patient and public involvement in research. Research Involvement and Engagement, 3(1), 13. https://doi.org/10.1186/s40900-017-0062-2

124.

Staniszewska

Denegri

Matthews

Minogue

(2018). Reviewing progress in public involvement in NIHR research: Developing and implementing a new vision for the future. BMJ Open, 8(7), Article e017124. https://doi.org/10.1136/bmjopen-2017-017124

125.

Steele Gray

Wodchis

W. P.

Upshur

Cott

McKinstry

Mercer

Palen

T. E.

Ramsay

Thavorn

Project Collaborators And Technology Partner, QoC Health Inc . (2016). Supporting goal-oriented primary health care for seniors with complex care needs using mobile technology: Evaluation and implementation of the health system performance research network, bridgepoint electronic patient reported outcome tool. JMIR Research Protocols, 5(2), Article e126. https://doi.org/10.2196/resprot.5756

126.

van Dongen

B. M.

Ridder

M. A. M.

Steenhuis

I. H. M.

Renders

C. M.

(2019). Background and evaluation design of a community-based health-promoting school intervention: Fit Lifestyle at School and at Home (FLASH). BMC Public Health, 19(1), 784. https://doi.org/10.1186/s12889-019-7088-3

127.

Vaughn

Jacquez

(2020). Participatory research methods – choice points in the research process. Journal of Participatory Research Methods, 1(1). https://doi.org/10.35844/001c.13244

128.

Veisi

Jackson-Smith

Arrueta

(2022). Alignment of stakeholder and scientist understandings and expectations in a participatory modeling project. Environmental Science & Policy, 134, 57–66. https://doi.org/10.1016/j.envsci.2022.04.004

129.

Villalobos

Blachman-Demner

Percy-Laurry

Belis

Bhattacharya

(2023). Community and partner engagement in dissemination and implementation research at the National Institutes of Health: An analysis of recently funded studies and opportunities to advance the field. Implementation Science Communications, 4(1), 77. https://doi.org/10.1186/s43058-023-00462-y

130.

Wallerstein

Duran

(2010). Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(Suppl 1), S40-S46. https://doi.org/10.2105/AJPH.2009.184036

131.

Wallerstein

N. B.

Duran

(2006). Using community-based participatory research to address health disparities. Health Promotion Practice, 7(3), 312–323. https://doi.org/10.1177/1524839906289376

132.

Weinert

Listl

Dannewitz

Heinze

Mostovic

Kalmus

El Sayed

(2022). Engaging patients to develop a customized digital health companion for periodontitis: Study protocol. Frontiers in Oral Health, 3, 1004091. https://doi.org/10.3389/froh.2022.1004091

133.

Wenzel

Heesen

Scheiderbauer

van de Loo

Köpke

Rahn

A. C.

(2021). Evaluation of an interactive web-based programme on relapse management for people with multiple sclerosis (POWER@MS2): Study protocol for a process evaluation accompanying a randomised controlled trial. BMJ Open, 11(10), Article e046874. https://doi.org/10.1136/bmjopen-2020-046874

134.

Wilkins

C. H.

(2018). Effective engagement requires trust and being trustworthy. Medical Care, 56(10 Suppl 1), S6–S8. https://doi.org/10.1097/MLR.0000000000000953

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.43 MB