Abstract
Background
The field of Implementation science (IS) continues to evolve, and the number and type of IS capacity building Programs (ISCBPs) are in flux. These changes push the field to revisit the accepted IS competencies and to guide sustainment of ISCBPs. Our objectives were: (1) compare characteristics of current ISCBPs; (2) identify recommendations to support ISCBP sustainment; (3) measure how often ISCBPs address IS competencies; (4) identify novel and important IS competencies for the field.
Method
This multi-method study included ISCBPs delivering structured, longitudinal IS training, excluding single courses and brief workshops. We used three complementary methods to meet our objectives. First, we identified ISCBPs via an internet search and snowball sampling methods. Second, we surveyed these ISCBPs to identify areas of program focus, types of trainees, IS competencies addressed, and recommendations to sustain ISCBPs. Third, we conducted a modified Delphi process with IS researchers/leaders to reach consensus on the IS competencies that were both important and novel as compared to the IS competencies published to date.
Results
Among 74 eligible ISCBPs identified, 46 responded (62% response rate). Respondent ISCBPs represented diverse areas of focus (e.g., global health, cardiopulmonary disease) and trainee stages (e.g., graduate students, mid-career faculty). While most respondent ISCBPs addressed core IS methods, targeting IS competencies was less consistent (33% for nongraduate/non-fellowship ISCBPs; >90% for graduate/national ISCBPs). Our modified Delphi process identified eight novel and important IS competencies related to increasing health equity or the speed of translation. Recommendations to sustain ISCBPs included securing financial administrative support.
Conclusions
Current ISCBPs train learners across varying career stages in diverse focus areas. To promote rigor, we recommend ISCBPs address specific IS competencies, with consideration of these eight novel/emerging competencies. We also recommend ISCBPs report on their IS competencies, focus area(s), and trainee characteristics. ISCBP programs need administrative financial support.
Plain Language Summary
There is a limited workforce capacity to conduct implementation science (IS) research. To address this gap, the number and type of IS capacity building Programs (ISCBPs) focusing on training researchers and practitioners in IS methods continue to increase. Our efforts to comprehensively identify and describe ISCBPs for researchers and practitioners highlighted four implications for leaders of ISCBPs related to program sustainment and rigor. First, we identified a range of contextual characteristics of ISCBPs, including the research topics, methods, and IS competencies addressed, and the types of trainees accepted. Second, given the variability of trainee types and research, rigorous ISCBP programs should tailor the IS competencies and methods addressed to the skills needed by the types of trainees in their program. Third, the field of IS needs to periodically revisit the competencies needed with attention to the skills needed in the field. We used a consensus-building process with ISCBP leaders and other IS experts to expand existing IS competencies and identified eight important, novel IS competencies that broadly relate to promoting health equity and speeding the translation of research to practice. Finally, as more institutions consider developing ISCBPs, we identified factors needed to support ISCBP sustainment, including ongoing financial support. In addition to these implications for ISCBP leaders, there are also policy implications. For example, IS journals may enact policies to require manuscripts evaluating ISCBP performance to report on certain contextual characteristics, such as the IS competencies addressed and types of trainees accepted. The field may also consider developing an accreditation body to evaluate the rigor of ISCBP curricula.
Introduction
There are substantial delays in the implementation of diverse clinical and public health interventions (Balas & Boren, 2000; Green et al., 2009; Khan et al., 2021; Lee, 2007). Spurred on by this gap between research and practice, implementation science (IS) has emerged as a crosscutting, transdisciplinary, and rapidly developing area of science (Chambers, 2018; Eccles & Mittman, 2006; Khan et al., 2021; Kwan et al., 2022). The field of IS seeks to translate evidence-based programs and policies (EBPPs) into routine care and community practice in a timely fashion, in order to improve public health and healthcare delivery.
To leverage the benefit of IS for improving health, IS capacity building programs (ISCBPs) have been developed to train researchers to develop and test EBPPs with attention to context, and to train IS practitioners to competently implement and disseminate these EBPPs (Brownson, Jacob, et al., 2021; Davis & D'Lima, 2020; Leppin et al., 2021; Norton, 2014; Proctor & Chambers, 2017). However, due in part to the rapid evolution of IS as a scientific discipline, there has been a persistent under-supply of IS experts and ISCBPs, as compared to the high demand among researchers and practitioners to receive training in IS methods (Brownson et al., 2018). Unfortunately, many national ISCBPs can only accept ≤20% of applicants, due to limited slots for trainees (Davis & D'Lima, 2020; Kho et al., 2009; Vinson et al., 2019). In response to this need, there has been a burgeoning development of ISCBPs by academic institutions, national funding agencies, and other groups.
This creates a challenge for those leading current ISCBPs or seeking to develop new ISCBPs to meet this demand for training, both in terms of the rigor of the training delivered, and in terms of ensuring there are sufficient resources to sustain a program. For example, in terms of ensuring sufficient rigor of training, several reviews have recommended ISCBPs to address a gold standard set of IS competencies (Brownson, Jacob, et al., 2021; Padek et al., 2015), such as appropriate selection of IS frameworks, models, and theories. The initial set of 43 core IS competencies developed by Padek and colleagues are organized into three levels of expertise (i.e., beginner, intermediate, advanced), and four broad domains: 1. Definition, Background and Rationale; 2. Theory and Approaches; 3. Design and Analysis; 4. Practice-based considerations (Padek et al., 2015). To date, these are considered as the “gold standard” competencies for IS (Brownson, Jacob, et al., 2021; Padek et al., 2015). As reported previously, examples of these IS competencies include: appropriate selection of IS frameworks, models, and theories; designing and testing implementation and dissemination strategies (i.e., interventions at the setting level); determining appropriate pragmatic study designs, such as hybrid trials for testing effectiveness and implementation); processes for guiding iterative adaptation; and measurement of pragmatic outcomes, such as implementation and sustainability (Gonzales et al., 2012; Padek et al., 2015; Schultes et al., 2021).
Recent calls for ISCBPs to include IS competencies in their curriculum and evaluation process clarify the need for attention to rigor (Proctor & Chambers, 2017). In addition, there are general recommendations to tailor training to the specific needs of trainees, and to report on the types of IS competencies addressed and trainees served (Proctor & Chambers, 2017)—however, there have not been any consistent accreditation standards to follow. The recent influx of new ISCBPs and sunsetting of prior ISCBPs highlights a particular need to identify any gaps in the current programmatic focus and to consider program characteristics related to sustainment. Recommendations for the field of IS to expand its methodologic attention to health equity (Adsul et al., 2022; Brownson, Kumanyika, et al., 2021; Kwan et al., 2022; Leppin et al., 2021) and to differentiate training for implementation practitioners from implementation scientists (Leppin et al., 2021) are two examples of the need to periodically revisit competencies to ensure that they meet the needs of trainees. Informed by these needs in the literature and our collective experiences leading ISCBPs, we have identified several key questions to advance our understanding of ISCBPs and related IS competencies by conducting this multi-method study. This multi-method study included a scoping review that informed a survey of ISCBPs and a modified Delphi panel to build consensus on novel IS competencies.
Scoping reviews are defined as “exploratory projects that systematically map the literature available on a topic, identifying key concepts, theories, sources of evidence, and gaps in the research (Peters et al., 2021). Our rationale for use of scoping review methodology is the emerging nature of the data on ISCBPs. More specifically, few programs have published the competencies that they use or the resources that they require. Scoping reviews require identification of a target program (or population), a key focus, and the contextual influences. This review sought to identify contextual influences on the key foci of sustainment and rigor among the target programs of ISCBPs (Peters et al., 2021). Our multi-method study aims included: Aim 1. Contextual influences: what range of characteristics of ISCBPs exist in terms of trainee type, program resources/scope, research methods focus, research content area, and IS competencies addressed? Aim 2. Sustainment: what recommendations do ISCBPs have to promote their sustainment? Aim 3. Rigor: how often are ISCBPs addressing IS competencies in their curriculum, and what competencies do they currently address? Aim 4. Novel competencies/training needs: are there novel IS competencies that ISCBPs should address? These aims are relevant both to current leaders of ISCBP programs seeking to sustain a rigorous program, as well as to individuals seeking to find rigorous ISCBP programs with strong potential for sustainment.
Method
Brief Overview of Multi-Method Design
To accomplish the four key aims, this multi-method study included the following overarching approaches. Aims 1–3 sought to survey all current ISCBPs, not just those with published outcome data (Chambers et al., 2017; Davis & D'Lima, 2020; Proctor & Chambers, 2017). Aim 4 sought to identify novel/emerging IS competencies both from a scoping review of the published literature and from the needs identified by existing ISCBPs. The Aims 1–3 survey and Aim 4 competency revision processes received exempt status from the Colorado Multiple Institutional Review Board and the Washington University Institutional Review Board, respectively.
For Aims 1–3, we defined ISCBPs as a longitudinal, structured set of activities intended to provide training in IS methods to researchers, program evaluators, or implementation practitioners. A research assistant e-mailed a Qualtrics© survey link to program leaders of identified ISCBPs in July 2020 to obtain these data—two additional requests were e-mailed over 2 months to non-responding ISCBPs. The survey assessed several elements, including their training focus areas and any specific IS competencies addressed—see “Survey domains for Aims 1–3” below for details. Survey questions included both multiple-choice and open-ended answer response options, including an “Other” category with open-ended responses when multiple-choice responses were not applicable. For Aim 4, we sought to identify novel IS competencies as compared to the current gold standard for IS competencies (Padek et al., 2015), by extracting IS competencies through a three-step process of a literature review as well as the above-noted survey of IS competencies used by current ISCBP programs (Padek et al., 2015). We then used a modified Delphi approach to come to consensus on which of these novel competencies should be added to the current gold standard IS competency list. Based on these multiple methods, we synthesized the results of the Aims 1–3 survey of ISCBPs and the novel/emerging IS competencies from Aim 4 into recommendations for those seeking to develop and/or sustain an ISCBP. We provide further details on each aspect of these methods below.
Justification for Search Strategies for Aims 1–3 and Aim 4
According to current guidance for scoping reviews, we provide justification for the databases searched and the search strategy used (Peters et al., 2021). There were two separate search strategies undertaken for this study, each with its own database. For our search strategy for Aims 1–3 to identify ISCBPs, we utilized the internet (Google©). Our justification for conducting this search on the internet rather than in SCOPUS or National Institutes of Health (NIH) Reporter is that many ISCBPs are not funded by research dollars and publications do not cite them, but ISCBPs typically use a website for recruitment of trainees. The content obtained for Aims 1–3 came from the surveys of the ISCBP respondents rather than from the internet. The search strategy for Aim 4 to identify IS competencies for ISCBPs to address was conducted in the SCOPUS database with supplementation from the competencies reported in Aim 3 by the ISCBPs identified via internet search. The rationale for use of this database for the Aim 4 search is that it includes peer-reviewed journals, book series, and abstracts.
Implementation Science Capacity Building Program Identification for Aims 1–3
Three complementary approaches were used to identify ISCBPs: (1) internet search (Google©) on May 1, 2020 by a research assistant for current ISCBPs, using the following terms: (“implementation science” OR “implementation research” OR “D&I research”) AND (“training” OR “capacity building” OR “graduate”); (2) existing list of ISCBPs on the Society for Implementation Research Collaboration website; (3) snowball sampling process through referrals from leaders of the ISCBPs identified through the first two approaches for additional ISCBPs that were not yet included. This process identified potential ISCBPs for inclusion in the survey to ascertain data for Aims 1–3.
Inclusion Criteria for Intensive ISCBP
ISCBPs were included if they could be classified in one of the following categories of moderately intense IS training that would warrant attention to the use and evaluation of competencies: ISCBP Category 1: training institutes delivered to a national/international group of trainees by a national or international sponsor to build IS research capacity in that region; ISCBP Category 2: mentored fellowship training programs focused on IS science competencies, typically offered by a specific academic institutional department or organization; ISCBP Category 3: academic graduate-level training programs with a sequence of at least three complementary courses in IS that provide a strong foundation to build IS competencies; ISCBP Category 4: other training programs, typically short-term but at least 2 days in duration, and offered by an academic institutional department.
Survey Domains for Aims 1–3
For our survey assessment of Aims 1–3, the survey items were informed by the recommendations for ISCBP programs in recent reviews of IS capacity building and training (Davis & D'Lima, 2020; Proctor & Chambers, 2017). We use symbols to designate domains recommended by Davis & D’Lima (≠) and by Proctor and Chambers (*) in the following numbered list. Specifically, the survey assessed: (1) program type*≠; (2) target audience for the program*≠ (e.g., graduate student, postdoctoral fellow, early-career faculty); (3) specific areas of focus in terms of both research content area*≠ (e.g., cardiovascular disease, cancer, mental health) and research methods*≠; (4) average number of trainees accepted per year; (5) cost of the program to trainees, including whether trainee tuition is required ≠; (6) year when program was established≠; (7) any publications describing the program and/or its evaluation≠; (8) website for program≠; (9) the number of courses or credit hours required to complete (N/A—training institute)≠; (10) average length of time to complete the program≠; (11) IS competencies targeted by the program*≠; (12) status of the program (active/inactive/in development)*; (13) necessary program support for sustainability* (e.g., financial support for director and/or administrator, grant funding). The above items addressed Aims 1 and 3. To address Aim 2 regarding recommendations for sustainment, the survey included an open-ended question regarding lessons learned from ISCBP program development. Two coauthors (AGH and BAR) reviewed the program responses to the survey to confirm eligibility.
Aim 4 Approach to Revise IS Competencies and Identify Novel Competencies
We sought to replicate past approaches used to develop standards for IS competency-based education (Padek et al., 2015). Specifically, this included a consensus-building process to identify IS competencies that researchers need to develop and that ISCBPs need to provide, and to designate these competencies as attainable for a relative beginner, intermediate, or advanced trainee (Padek et al., 2015).
To identify novel competencies as compared to the gold standard IS competencies (Padek et al., 2015), we undertook a three-step process to extract novel IS competencies from our Aim 4 literature review and from the Aim 3 competencies submitted from our survey of ISCBPs. For the Aim 4 literature review, we used seven publications on ISCBPs as an initial set of “seed articles” for the identification of IS competencies—this included a seminal review on training needs for ISCBPs (Chambers et al., 2017) and publications on ISCBPs referenced therein as the primary articles for that review (Gonzales et al., 2012; Meissner et al., 2013; Norton, 2014; Padek et al., 2015; Proctor et al., 2013; Stamatakis et al., 2013). Using a snowball search method, we completed a reference review and citation review in SCOPUS to identify any supplementary articles either cited by one of the seven seed articles or referenced in these seed articles—this was Step 1 of the literature review.
We then completed the second step of reference review and citation review on the articles captured in Step 1 of the SCOPUS search that were published ISCBPs that cited the use of competencies in curriculum development. For the purposes of this project, we operationalized the definition of “competencies” to include specific skills, learning objectives, or broad domains related to IS. Once the search outcomes were consolidated and de-duplicated, one reviewer (EG) completed a title/abstract screen, and two reviewers (SJ, EG) completed the full-text review. Discrepancies between reviewers were resolved by consensus. Competencies were then extracted from the included articles. For the third and final step of the Aim 4 process to identify distinct IS competencies in use, we identified other novel IS competencies in prevailing use by other, non-published ISCBPs from the Aim 3 survey results—in addition to those extracted from the SCOPUS literature search.
Methods for Aim 4 Modified Delphi Process to Develop Consensus on Updated IS Competencies
Two members of the research team (SJ, AP) mapped the extracted competencies from the literature review and the environmental scan to the current gold standard IS competencies (Padek et al., 2015). If an extracted competency did not match a current IS competency (Padek et al., 2015), it was considered to be unique and novel. Discrepancies between reviewers were resolved by consensus. The list was edited by the core research team based on redundancy, clarity of meaning, and relevancy to IS research to a final list of 10 novel IS competencies.
We identified experts in the field of IS to determine if (1) the 10 novel competencies should be added to the current list (Padek et al., 2015) and (2) the hierarchical training levels of the novel competencies. To do this, we employed the use of a modified Delphi process and card sort approach in a web-based survey technology platform (Qualtrics©). Based upon our team's prior experience with IS competency development and refinement (Padek et al., 2015), we sought to include previous participants from ISCBPs and those with at least a base knowledge of the field, in order to obtain more accurate results. The experts identified (n = 258) consisted of faculty from the University of Colorado Adult and Child Center for Health Outcomes Research & Delivery Service (ACCORDS) program (n = 16), faculty and fellows from the 2010–2018 Implementation Research Institute (IRI, R25MH080916) training program (n = 96), the 2014–2017 Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC, R25CA171994) training program (n = 52), and the Washington University Dissemination and Implementation Research (WUNDIR) network (n = 94).
After consenting to complete the survey, we asked respondents to rate each novel IS competency on importance to add to the current IS competency list (i.e., How important do you feel it is to add the following competency to the current IS competency list [Padek et al., 2015], from 1 “not at all important” to 5 “extremely important”). Participants were also asked to place each competency statement into the column that best expressed the skill level needed to address that competency. The columns were labeled “Beginner,” “Intermediate,” and “Advanced.” These learning levels were not defined, in order to allow participants to self-define the categories and exclude any unintentional bias from the research team on these definitions. This step mimicked the card sort process used to develop the current IS competencies (Padek et al., 2015). After this activity, participants were asked to denote their (1) discipline (public health, health sciences, social work, etc.); (2) level of expertise in the field of IS research (beginner, intermediate, advanced); and (3) country of residence.
Data Analysis for IS Competency Update
We used descriptive statistics to analyze the results of the expert panel survey. Mean score and standard deviation were calculated for each competency based upon rated level of importance. Novel competencies were eliminated if <50% of experts rated the competency as “very” or “extremely important” to add to the current IS competency list. For the card sort portion of the survey, statements placed within the learning level groups (beginning, intermediate, or advanced) were coded as “1, 2, or 3,” respectively. Competencies were then sorted in ascending order and divided into tertiles based upon the distribution of scores. Competencies that fell within the first tertile were categorized as beginner, the second tertile as intermediate, and the third as advanced. Minor adjustments between the learning level groups were made if a particular competency had a higher distribution of scores in a different experience level than the placement of the competency by its mean score. For example, if a competency fell in the second tertile (i.e., intermediate) but had a higher frequency of scores in the beginner level, the team would shift the competency to the beginner level to reflect the majority of responses.
Results
Survey Participants
The Aim 1–3 internet search strategy for ISCBPs yielded 74 ISCBPs who were contacted thrice between July and August 2020. Among these, 46 eligible programs responded to the survey request (62% response rate).
Survey Findings for ISCBP Contextual Factors (Aim 1), Sustainment (Aim 2), and Rigor (Aim 3)
In terms of Aim 1 findings, the survey revealed programs within each of our four specified ISCBP categories: (1) academic graduate programs: graduate degree-granting programs (n = 6) and graduate certificate programs (n = 6), (2) mentored fellowships: K12 training programs (n = 9) or postdoctoral fellowships (n = 5), (3) national training programs (n = 8), and (4) other training (n = 12), as shown in Tables 1–4. Degree-granting programs often represented an IS focus or concentration as a part of a broader research training (e.g., master's in public health programs) rather than an IS degree specifically. Accordingly, the number of courses on IS methods tended to be similar across the graduate degree-granting programs and the graduate certificate programs. Degree- or certificate-granting programs typically ranged from 3 to 5 graduate-level courses expected to be completed over ∼2–3 years (Table 1). Among the n = 34 graduate programs, mentored fellowship programs, and national training programs, 100% reported IS researchers as a target trainee type; seven of these programs (21%) also reported enrolling implementers/implementation practitioners, program evaluators or scientists conducting quality improvement (QI) research. In contrast, among the n = 12 other IS programs (Table 4), n = 11 (92%) reported targeting implementers/evaluators/QI researchers as trainees. A distinguishing characteristic of graduate programs, K12 training programs, and fellowships versus national and other training programs concerned the need for the former to require a formal student and/or employee affiliation with the host institution.
High-Level Overview of ISCBPs Yielding Graduate Degrees or Certificates.
Note. IS = implementation science; ISCBP = Implementation Science Capacity Building Program; QI = quality improvement; USA = United States of America; Category of “Implementers/Evaluators” was defined for respondents as “Program implementers/evaluators”; Inclusion criteria: 3+ courses in IS methods for a graduate certificate or degree-granting program
High-Level Overview of ISCBP Postdoctoral Fellowships and K12 Programs.
Note. IS = implementation science; ISCBP = Implementation Science Capacity Building Program; USA = United States of America; Inclusion criteria: mentored training in IS methods.
aNHLBI K12 funding supported several IS training programs from 2017 to 2022.
High-Level Overview of ISCBP National Programs.
Note. AUS = Australia; IS = implementation science; ISCBP = Implementation Science Capacity Building Program; HIV = human immunodeficiency virus; QI = quality improvement; UK = United Kingdom; USA = United States of America; Category of “Implementers/Evaluators” was defined for respondents as “Program implementers/evaluators”; Inclusion criteria: multi-day intensive national training.
High-Level Overview of Other Intensive ISCBPs.
Note. IS = implementation science; ISCBP = Implementation Science Capacity Building Program; Inclusion criteria: multi-day intensive training.
In terms of Aim 2 findings related to sustainment, the vast majority of ISCBPs (79%) reported ongoing annual administrative costs to support the director(s) and/or program administrators. Some programs supported these administrative costs through grant funding, but others used alternative funding approaches, including institutional support, donor support, and student tuition/fees. Typically, student tuition/fees were not sufficient to fully cover an ISCBPs' administrative costs. A smaller number of ISCBPs also supported trainee fees (46%) and teaching faculty stipends (26%), respectively.
In terms of Aim 3 findings related to rigor and IS competencies used, 74% of all ISCBP programs (n = 34) reported training in general/core IS methods (e.g., IS frameworks, implementation outcomes, and measures), and often reported training in other specific IS methods (e.g., patient/public involvement in research; economic analysis, learning health systems). The research focus often aligned with the sponsor's priorities, such as the nine mentored K12 programs funded by the National Heart, Lung, and Blood Institute (NHLBI) from 2017 to 2022. Global health research was also common (n = 4 programs). According to program leader survey respondents, IS competencies guided the curriculum among the majority of national ISCBPs (100%), graduate programs (92%), and K12 training programs (78%). However, only 40% of the postdoctoral fellowships (Table 2) and 33% of the other trainings (Table 4) reported the use of IS competencies to inform their curriculum.
Aim 4 Revised IS Competencies
To develop the revised Aim 4 IS competencies, we identified 10 novel IS competencies from the 401 potential unique IS competencies identified in our three-step search approach. We derived these competencies in this way: the first two steps of our SCOPUS literature review search strategy yielded 5,029 articles, of which 16 articles met the eligibility criteria. The third step was to review the Aim 3 IS competencies submitted ISCBPs: this yielded 17 unique additional lists of competencies (or articles) for our consideration. From the combined 33 articles/lists, we extracted 401 discrete competencies. After comparing these 401 competencies to the current gold standard for IS competencies (Padek et al., 2015), we identified 10 novel competencies to be included in the modified Delphi process with our expert panel.
Among the 258 IS researchers and/or ISCBP leaders invited to participate in the modified Delphi process, 118 individuals completed the expert panel survey (response rate = 46%). The most common disciplines reported by respondents included public health (33%), health sciences (21%), health systems/health service administration (13%), and Psychology (12%). Just over half of the respondents (n = 61) described there IS expertise as “Intermediate.” Thirty-nine respondents described their expertise as “Advanced” (33%), and 19 respondents (17%) described their expertise as “Beginner.” Most respondents were from the United States (>95%).
The mean scores for importance for the 10 novel competencies ranged from 4.31 to 3.34 (Table 5). Eight out of 10 novel competencies were rated as “very” (score = 4) or “extremely important” (score = 5) by more than half of the experts and were incorporated into the updated IS competency list. Two out of those eight competencies (25%) fell into the “Advanced” category, with the remaining competencies rated as “Beginner” (38%) and “Intermediate” (38%; Table 6).
Novel/Emerging Competency Descriptions and Rankings.
Note. D&I = dissemination and implementation; SD = standard deviation.
Updated Competencies by Domain and Level of Trainee Expertise Needed to Attain.
Note. D&I = dissemination and implementation.
a Expertise levels: B = beginner; I = intermediate; A = advanced; b Updated competency; c Novel/emerging competency.
We categorized the eight novel/emerging IS competencies within the original four categories of IS competencies developed by Padek et al.: 1. Definition, Background, and Rationale; 2. Theory and Approaches; 3. Design and Analysis; 4. Practice-based considerations (Table 6). In terms of thematic commonalities of the 10 novel competencies, three explicitly address the intersection of health equity and IS: (a) Design strategies to address the multi-level influences of health inequities as it relates to the implementation of an evidence-based intervention; (b) Develop strategies to promote equity in resource distribution across all external research partners, including community partners or other external organizations and the researcher's institution; (c) Summarize the importance of ethically and culturally competent clinical and community-based research in Dissemination and Implementation science. Four of the novel competency statements relate to enhancing the impact of translational research (Peek et al., 2014) by making implementation more rapid and/or recursive, including the conduct of iterative cycles of implementation and shared data use approaches such as those found in Learning Health Systems (Bennett et al., 2020; Damschroder et al., 2021; Kilbourne et al., 2020). These novel competencies related to making implementation more rapid include: (a) Operationalize hybrid effectiveness-implementation designs when appropriate to accelerate the implementation of evidence-based interventions in real-world settings; (b) Develop and assess processes and outcomes that support iterative cycles of implementation and bidirectional flow of information (e.g., learning health systems); (c) Characterize process models that support iterative cycles of implementation and adaptation based on learning; (d) Examine the importance of rapid research to advance Dissemination and Implementation science concepts and directions. Overall, we would frame these novel competencies as efforts to make IS research more rapid and inclusive, by including diverse perspectives from those in the inner setting and outer setting across all study phases, and by attending to the inclusion of cultural norms and preferences in the approaches used.
Discussion
This multi-method study adds to the literature on ISCBPs in terms of their focus areas and characteristics, the rigor and IS competencies addressed, and factors related to their sustainment. We found that current ISCBPs train learners across varying career stages in diverse focus areas, in line with expert recommendations (Leppin et al., 2021; Proctor & Chambers, 2017). Related to sustainment of ISCBPs, we found that nearly 80% of ISCBPs reported ongoing annual administrative costs, and alternative grant funding or institutional funding sources in addition to student tuition/fees were recommended to maintain solvent programs. Related to the rigor of IS training, we found that while >70% of respondent ISCBPs addressed core/general IS methods, targeting IS competencies was less consistent. Specifically, ≤40% of non-graduate, non-fellowship, and postdoctoral fellowship ISCBPs reported addressing specific IS competencies, while over 90% of graduate and national ISCBPs targeted IS competencies. In addition, our expert panel came to consensus on a set of eight highly important IS competencies that are unique additions to the current gold standard for IS competencies (Padek et al., 2015). Thematically, these eight novel competencies represent a need to provide training at the intersection of IS and equity, as well as the need for more rapid translation of EBPPs.
The updated set of IS competencies is valuable for ISCBPs to develop and/or expand their curriculum, particularly for content related to the intersection of health equity and IS, and the rapidity of translating evidence into practice. These findings are also complementary with the domains of implementation competencies recently identified for implementation practitioners applying IS—as distinct from competencies for researchers (Leppin et al., 2021). Areas of overlap include the development of competencies for implementation practitioners to improve the rapidity of implementation by “diagnosing” contextual barriers and facilitators to implementation, and “treating” those contextual factors by developing and deploying appropriate implementation strategies (Leppin et al., 2021).
In terms of sustainment, we found that the vast majority of ISCBPs require ongoing funding from grants or institutional funds, in addition to trainee tuition/fees, in order to support their program administrators, and a smaller number also provide stipends to instructors. These findings relate to recommendations in a recent review by Proctor and Chambers to consider the sustainability of ISCBPs: to ensure funding sources and human capital are available to meet the training needs (Proctor & Chambers, 2017). The relevant staffing and administrative requirements for launching and sustaining an IS training program are important to consider at the outset. Most recently, Proctor and colleagues discussed the need for developing market viability considerations in the field of IS that are relevant to ISCBPs, a skill set less commonly obtained through traditional public health and health services research training (Proctor et al., 2021). One program specifically noted requiring 20% full-time equivalent (FTE) of effort for Graduate Certificate Program Directors and 25% FTE of effort for an administrative assistant, but noted this would be much higher—up to 50% FTE for the Director and 100% FTE for an administrative assistant in the absence of existing graduate school and NIH-funded Clinical Translational Science Agency (CTSA) resources leveraged to support that program. Another recommendation was to consider how to appropriately compensate IS research faculty for teaching, as compared to their compensation for serving as a co-investigator on grants. Based on the literature and the findings of this multi-method scoping review, survey, and modified Delphi process, factors relevant to ISCBP sustainment include:
Ability to differentiate from other ISCBPs to meet the needs of specific subsets of trainees, such as relevant local experts who can teach specific subsets of IS competencies or train individuals to apply IS methods to a specific content area (e.g., global health, cardiopulmonary disease, mental health); Adequate capacity for the proposed faculty/mentors, as the resources of faculty availability/time will dictate the number of courses taught and the number of trainees that the program can mentor—for this reason, some institutions may start by offering 1–2 IS courses rather than launching a full ISCBP Graduate Certificate or IS concentration, as the latter typically includes 3 or more IS-focused courses; Appropriate financial sponsorship/support—availability (or not) of external training grant support, or internal support in the form of administrative sponsorship by a Dean/Department chair, a local NIH-funded CTSA, and/or a local doctoral or master's level graduate training program that may wish to offer an IS concentration (or Certificate) as part of their existing program of training—these types of support are important as student fees will not typically cover the administrative costs of delivering a program; Sufficient infrastructure to provide distance learning (online platform) and/or in-person learning; Trainee demand for the type of training offered based on the above four elements
This review has several strengths and limitations. Its strengths include the robust, multi-method approach to identify ISCBP characteristics and competencies through the published academic literature and internet, as well as using snowball sampling and surveys of ISCBP leaders. Related to our inclusion criteria for graduate programs to offer at least three IS courses, this is not a comprehensive list of graduate programs with IS training concentrations, and also excludes individual graduate courses. Limitations of our multi-method literature search include the potential to miss publications of international ISCBPs by restricting our literature search to English-language publications, and the possibility to overlook publications of IS competencies that were not described in the title/abstract. Furthermore, given the large number of articles to screen, we only had capacity for one author to review the titles/abstracts, but did include two authors to review the full articles for possible IS competencies. The updated competency list was compiled based on a modified Delphi process with a snowball sample of international IS experts which may underrepresent the perspective of implementation practitioners. In addition, this process was undertaken prior to a recent publication by Schultes and colleagues on clusters of general IS competencies (e.g., knowledge and skills of IS, setting, program evaluation, and research methodology), so their recommended IS competencies were not incorporated into our modified Delphi process (Schultes et al., 2021). However, we do not find these clusters by Schultes et al. to be fundamentally distinct from the current Padek et al. competencies that we used (Schultes et al., 2021). Finally, Proctor and Chambers emphasized the need to provide support to alumni of IS training programs (Proctor & Chambers, 2017) and the need to provide mentored application to an IS research project in addition to didactic training; however, the scope of this review was limited to the training period itself, and did not specifically address the use of mentored research projects.
Conclusions and Recommendations
Our findings are a snapshot of the evolving field of IS training in a phase where the number and types of trainings continue to expand. The evolution of the field and expansion of programs warrants further attention to ISCBP rigor and sustainment, including our identification of eight novel, important IS competencies. Given our findings, we recommend the following: (1) To promote transparency: ISCBPs should report contextual characteristics of their program in line with recent recommendations (Davis & D'Lima, 2020; Proctor & Chambers, 2017) according to the examples provided in Tables 1–4, including the IS competencies addressed, program duration, course requirements, costs, and whether a program's focus is on developing IS researchers, implementation practitioners, program evaluators, QI researchers, or a combination of these; (2) To promote rigor: ISCBPs should include IS competencies consistently in the design of their curriculum, evaluate trainee self-assessment of progress toward the IS competencies targeted, and periodically revise the curriculum based on trainee gaps in progress; (3) It is not necessary for every program to address every competency. We encourage ISCBPs to select competencies with attention to the skill level of their trainees (e.g., beginner, intermediate, advanced), and with attention to the end goals of their trainees; (4) As there is a particular dearth of capacity building programs for implementation practitioners and those that exist are often siloed from current ISCBPs, we echo the call by Leppin et al. to continue to develop ISCBP models that train IS researchers and implementation practitioners together (Leppin et al., 2021), and we’ve identified several existing programs that do this (Tables 1–4); (5) We also recommend that those planning a new ISCBP anticipate the factors relevant to sustainment described above, including program differentiation from other ISCBPs, adequate faculty/mentor bandwidth, and sufficient financial support. In addition to these implications for ISCBP program leaders, there are also policy implications. As part of the program evaluation for new ISCBPs, IS journals may consider requiring reporting on as the ISCBP area of focus, trainee types accepted, and IS competencies addressed. The field may also consider developing an accreditation body to evaluate the rigor of the curriculum for ISCBPs. In sum, the future is bright for ISCBPs, but we need to continue to reflect and evolve our training goals as IS research methods also develop in this young field.
Footnotes
Acknowledgments
The authors would like thank Elise Robertson for her support in the review of IS capacity building programs and Emanuel Gyimah (EG) and Allison Phad (AP) for their support in the IS competency literature review and refinement process.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: EHG holds an educational grant from Viiv Healthcare. None reported for other coauthors.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Colorado Implementation Science Center for Cancer Control (ISC3; National Institutes of Health (NIH)/National Cancer Institute (NCI) award P50CA244688; Huebschmann, Rabin); Washington University ISC3 (NIH/NCI award P50CA244431; Haire-Joshu, Brownson); the NIH National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) award R25DK123008; Johnston, Haire-Joshu, Brownson) and NIH/NIDDK award P30DK092950 (Haire-Joshu, Brownson); the Wellcome Trust (award reference 219425/Z/19/Z; Davis); the National Institute for Health Research (NIHR) via the “Applied Research Collaboration South London” (NIHR ARC South London) at King’s College Hospital National Health Service (NHS) Foundation Trust, London, UK (Davis); the Colorado Clinical and Translational Sciences Institute (CCTSI) grant UL1 from the National Center for Advancing Translational Sciences (NCATS) of the NIH award TR002535 (Kwan, McNeal); the Washington University Institute of Clinical and Translational Sciences award from the NCATS of the NIH award UL1TR002345 (Geng, Sandler); the University of Colorado Adult and Child Center for Outcomes Research and Delivery Science (Huebschmann, Kwan, Rabin), the Washington University Institute for Public Health (Brownson, Geng), the Washington University Center for Dissemination and Implementation (Geng); the Centers for Disease Control and Prevention award U48DP006395 (Haire-Joshu, Brownson); the Foundation for Barnes-Jewish Hospital (Brownson); UC San Diego ACTRI Dissemination and Implementation Science Center (Rabin). The content is solely the responsibility of the authors and does not necessarily represent official views of the NIH or other funding agencies.
