Sage Journals: Discover world-class research

Abstract

Objective

Implementation outcomes are important in intervention research as a necessary precursor to achieving desired health outcomes. Considering the critical role of implementation outcomes, this study involved a comprehensive review of implementation outcome measures used in digital health interventions specifically targeting young adults.

Methods

This scoping review was reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews, and the search incorporated the elements of population, concept, and content framework in three electronic databases (PubMed, Embase, and CINAHL). A matrix was used for data extraction and integrative thematic synthesis for analysis. Implementation outcomes were reported based on the indicators in each study, totaling eight outcomes: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability.

Results

The search yielded 2441 articles, and 17 were finally identified. The intervention implementation techniques that were adopted were telephone calls (n = 1); social media (n = 2); web-based programs (n = 4); short message service (n = 5); wearable devices (n = 1); mobile applications (n = 3); and a combination of phone calls, emails, and text messaging (n = 1). The highest number of implementation outcomes that were assessed in all the studies were acceptability (n = 10), feasibility (n = 10), and fidelity (n = 8). Short message service (n = 14), web-based programs (n = 11), and mobile applications (n = 7) had the highest number of implementation outcomes.

Conclusions

Researchers have largely assessed the acceptability and feasibility outcomes. The need to integrate the implementation outcomes framework in intervention research design is underscored.

Keywords

Intervention implementation outcome digital-health young adults health promotion

Introduction

Implementation outcomes are a critical component that must be addressed by intervention science researchers. The translation of research into clinical or community practice has been slow because of the limited information regarding implementation outcomes.¹ Indeed, implementation outcomes narrow the research practice gap by identifying barriers and facilitators in the context of designing interventions to achieve desired outcomes, termed effectiveness.² Implementation outcomes must be distinguished from intervention effectiveness during research development and implementation.^3,4 The former refers to the intentional actions taken to deliver a program and implement new treatments, practices, and services by highlighting the processes that lead to effective interventions.⁴ Implementation outcomes are benchmarks for attaining service outcomes, which preceded client-level outcomes (satisfaction, function, and symptomatology) to measure treatment progress.⁵ In addition, differentiating implementation outcomes from service outcomes (efficiency, safety, equity, patient-centeredness, and timeliness); client-centered outcomes; and intervention context indicators (scalability and usability) are imperative for effectiveness.^3,4

Implementation outcomes are increasingly recognized as crucial in intervention research, as they provide insights into not only how interventions achieve their intended clinical outcomes but also the barriers and facilitators influencing their successful implementation. It is particularly noteworthy for such intervention with the increasing complexity of using digital health intervention strategies for health promotion among young adults.^6,7 Diverse health promotion interventions, including smoking and alcohol cessation, physical activity, improvement in health literacy, and stress management, focus on young adults by leveraging digital health interventions.⁷ Digital health interventions provide an avenue for health promotion among young adults because of their propensity to use healthcare technology and associated global trends.⁶ In essence, a digital health intervention refers to the secure use of information and communication technologies in support of health and related fields and encompasses telehealth, telemedicine, mobile health, electronic medical or health records, big data, wearables, artificial intelligence, and short message service.⁸ In terms of age, young adults between 20 and 24 years are in a stage when their technological use is at its peak.^8,9 Therefore, assessing implementation outcomes is important among young adults, because health interventions are likely to use digital health intervention strategies.⁹

The changing trend in assessing healthcare research effectiveness with the increasing use of digital health interventions makes it important to adopt standardized evaluation criteria.^4,10,11 This is critical for the quality evaluation of research to determine usability and scalability.¹² There has been an increasing trend in the use of implementation outcomes in the development of interventions, especially within the last decade.¹¹ This study adopted the implementation outcomes described by Proctor et al.⁴ because they are comprehensive, simple, easy to use, and robust.¹⁰ Other researchers Grimshaw et al.¹³ and Michie et al.¹⁴ have produced diverse taxonomies for implementation outcomes; however, Proctor et al.⁴ provided a clear definition with critical indicators. The implementation outcomes encompass eight measures (Supplemental File 1): acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability.

Previous evaluations of implementation outcomes have been criticized for focusing on mental health and behavioral healthcare,¹⁰ physical activity,¹⁵ and determining the trends of use of implementation outcomes.¹⁶ There is a lack of understanding regarding the use of implementation outcomes in digital health promotion interventions targeting young adults. To address this gap, the study aims to identify the inclusion of implementation outcomes in studies on digital health interventions among young adults and to analyze the tools used to assess these outcomes. This will contribute to our understanding of how these interventions can be effectively implemented and evaluated in this population, thereby facilitating the use of standardized and comprehensive evaluation frameworks.

Methods

Study design

This was a scoping review utilizing the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist. Before the conduct of this study, the study protocol had not yet been published.

Search strategy

This study focused on the elements of population, content, concept, and design framework to search three electronic databases: PubMed, Embase, and CINAHL (Supplemental File 2). The study was limited to these three electronic databases because additional searches in other databases did not yield any additional papers after screening. The population was young adults (18–25 years),¹⁷ the content was digital health interventions, and the concept was implementation outcomes and limited to randomized controlled trials (RCTs) or intervention studies. The keywords were developed by incorporating appropriate Boolean operators: MeSH in PubMed, CINAHL MeSH in CINAHL, and Entree in EMBASE. Wildcards and truncation were also used. The search was limited to the period from January 2018 to May 2023. The search was limited to the last 5 years because of the influence of fast-evolving technology, especially in the health sciences. Also, within the period, a grey search was done in Google Scholar, but no additional papers from this source were included after screening. The retrieved studies were transported to an endnote (24.3.1) for screening (by KDK and ZAI) and confirmed by the other researchers (JL and HL). In addition, one of the authors (HL), had several years of experience in conducting search terms for systematic reviews and had her experience guiding the process. Also, the Peer Review of Electronic Search Strategies (PRESS) checklist was used to ensure the adequacy and comprehensiveness of the search terms.

Study selection and eligibility criteria

Two researchers independently evaluated the retrieved titles and abstracts to identify articles that met the inclusion criteria. During the screening phase, studies were considered for inclusion if they specifically targeted young adults aged 18–25 or included this age group within a broader study population. Additionally, only studies focused on digital health interventions, published in peer-reviewed journals, and written in English were included. Studies that assessed digital health interventions in chronic diseases such as diabetes, hypertension, cardiovascular disease, depression, and dementia, and described any phase of testing a medication or disease-related technological product were excluded. During the screening process, discrepancies were discussed in meetings between the two screening researchers or with the other research team members. All discrepancies were resolved through consensus.

Data charting

Three researchers extracted, compared, and integrated data from each selected study using a predetermined matrix. First, a matrix was developed, discussed, and agreed upon by the research team. The data extraction matrix contained the study characteristics, implementation outcome reports, the type of implementation outcomes used based on the type of digital health intervention, and implementation outcomes based on health promotion. The integrative thematic synthesis analysis method¹⁸ was used. Data were transformed into descriptive qualitative statements and interventions were assessed for each implementation outcome. The implementation outcomes were mapped according to the type of digital health intervention and health promotion intervention. Consequently, the frequency of implementation outcomes was determined and recorded.

Results

The study results are presented as search results and characteristics, reported intervention outcome measures, implementation outcomes-based health promotions, and implementation outcomes-based digital health intervention methods.

Search results and study characteristics

The search in PubMed (1358), Embase (377), and CINAHL (706) identified 2441 titles. The titles were transferred to Endnote software (Version 24.3.1). Then, duplicates (77) and retractions (three) were identified electronically and manually and removed. Next, 2361 articles were screened and 2251 were excluded, resulting in 110 studies selected for full-text review. Eventually, 17 studies were selected as eligible (Figure 1). The studies were conducted in the United States of America (n = 6), the United Kingdom/England (n = 4), Australia (n = 2), Germany (n = 1), Taiwan (n = 1), Netherlands (n = 1), Japan (n = 1), and Singapore (n = 1). The studies were conducted at a university or college (n = 12), at the workplace (n = 1), using online crowdsourcing (n = 1), in the community (n = 1), or in a hospital setting (n = 2). Table 1 provides details on the characteristics of the studies.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Table 1.

Study characteristics and key findings.

Author (s)	Purpose	Design	Setting	Country	Randomization	Recruitment	Population
Allen et al., 2022	To examine the feasibility and acceptability of timing the quit date of tobacco smoking to the menstrual phase in women	Pilot RCT	Smoking quitline setting	USA	Randomized, two groups	Reactively via social media advertising and proactively through the quitline	18 treatment-seeking women aged 18–40 years
Bonar et al., 2023	To develop and pilot feasibility testing interventions on physical activity for emerging adults who use cannabis	Pilot RCT	Cannabis users on social media	USA	Randomized, three groups	Online via social media	54 cannabis users aged 18–25 years
Brown et al., 2018	To assess the acceptability and feasibility of digital health intervention to support an action plan for three sexual health behaviors	Pilot RCT	Integrated sexual health clinic	UK	Randomized, two groups	Posters, leaflets, and verbal introduction	88 young adults aged 14–24 years
Coleman et al., 2022	To test the efficacy of “MiQuit,” a tailored, self-help, text message stop-smoking program for pregnancy	Superiority RCT	24 English hospital antenatal clinics	UK	Randomized, two groups	During antenatal visits or consented by telephone	1002 females aged ≥ 16 years
Davies et al., 2020	To assess the feasibility, utility, and challenges of wearable camera physical activity measurement	One group post-test	Community	Australia	Feasibility none RCT	Facebook, Instagram, posters, invitation letters	53 young adults aged 18–30 years
Dietel et al., 2020	To investigate the acceptability and efficacy of a web-based cognitive bias modification for interpretation program for body dissatisfaction	RCT	University of Muenster	Germany	Randomized, three groups	Newspapers, internet, and participant pools	318 young adults aged 18–40 years
Godino et al., 2019	To examine the efficacy of a one-year intervention to reduce weight among overweight and obese English- and Spanish-speaking adults	RCT	Community	USA	Randomized, three groups	Advertisements, Listservs, and online community boards	252 overweight and obese participants aged 21–60 years
Hamilton et al., 2018	To determine the feasibility of undertaking a definitive trial comparing the effectiveness and cost-effectiveness of a digital health intervention with face-to-face treatment for alcohol misuse	Feasibility RCT	Emergency departments in North London (five hospitals)	UK	Randomized, two groups	Emergency department and invite online advertisement	19 persons with problematic alcohol use (Alcohol Use Disorders Identification Test score >8) aged ≥18 years
Hubert et al., 2022	To explore the feasibility of an online tailored message program for college women that aligns with changes in information, motivation, and confidence	One group post-test only	Multiple university campuses	UK	Observational none RCT	Social media by key stakeholders and interested groups	189 female students aged ≥17 years
Johnson and Mays, 2021	To test the effects of hookah tobacco risk messaging for delivery via Instagram	RCT	Instagram	USA	Randomized, 2 × 3 between-subjects	Internet-based crowdsourcing platform	601 hookah smokers aged 18–30 years
Liu and Willoughby, 2018	To assess the effectiveness of using goal-setting theory-based text message reminders to promote tracking activities on fitness apps	Pilot RCT	College-wide participant pool	USA	Randomized, two groups	Direct recruitment in a determined course	50 college students mean age 21 ± 1.8
Liu et al., 2020	To investigate the effectiveness and limitations of three food size measurement methods in a dietary intake reporting app	RCT	University cafeteria	Taiwan	Randomized, three groups	University students	124 college students aged 19–25 years
Merrill et al., 2018	To test the feasibility and acceptability of the preliminary efficacy of using text messages to promote pro-moderation descriptive and injunctive norms.	Pilot RCT	Residential 4-year college	USA	Randomized, two groups	Online and University's Office of Residential Life	68 persons with problematic alcohol use aged 18–20 years
Scholten et al., 2019	To examine the effects of HitnRun, a social mobile game, among young smokers who were motivated to quit smoking.	RCT	Behavioral Science Institute Laboratory of Radboud University	Netherlands	Randomized, two groups	Campus flyers, online on Facebook and Instagram	131 smokers aged 16–26 years
Song and Kanaoka, 2018	To investigate the effectiveness of the application from a health condition and labor productivity loss perspective, and its cost-effectiveness for women	RCT	Community	Japan	Randomized, two groups	Online panel	1526 employed women aged 20–45 years
Tay, 2022	To evaluate the online HOPE intervention on help-seeking attitudes and intentions among young adults	RCT	University, Singapore	Singapore	Randomized, two groups	Social media	174 young adults aged 18–24 years
Whatnall et al., 2019	To assess the feasibility and preliminary efficacy of the EATS brief web-based nutrition intervention	Pilot RCT	University of Newcastle	Australia	Randomized, two groups	Social media, posters, and digital signage	124 students aged 17–35 years

RCT: randomized controlled trial.

Reported intervention outcome measures

Among all the studies reviewed, eight implementation outcomes were reported, and the minimum number of outcomes reported was one (Table 2). No study assessed more than four implementation outcomes. Specifically, the number of implementation outcomes assessed in the studies was four (n = 3), three (n = 7), two (n = 6), and one (n = 1).

Acceptability: Acceptability was the most assessed implementation outcome (n = 10) and was directly measured in some of the studies, whereas in other studies, acceptability was assigned based on measurement indicators. The indicators included app usage, patient motivation and satisfaction with the intervention, complexity, and error^19–21; the Intrinsic Motivation Inventory Scale²²; and operability, complexity, and participant satisfaction.²¹

Adoption/uptake: None of the studies specifically reported on adoption or uptake as an implementation outcome. However, based on the reported indicators, adoption was assigned in three studies as an implementation outcome.^20,23,24 The studies where adoption was assigned reported on use preference based on participants’ preferred language,²³ applicability and social acceptance,²⁴ and intention to use.²⁰

Appropriateness: None of the studies reported appropriateness as an implementation outcome. However, based on the indicators that were reported, intervention appropriateness was assigned in seven studies. These studies assessed perceived fit^24,25; safety and suitability²⁶; usefulness²⁷; relevance and usefulness²⁰; interest and usefulness²²; and usefulness, relevance, and usability.²⁸

Cost: As an implementation outcome, cost was assessed in four of the studies.^29–32 Regarding assigned indicators of cost, the studies measured the total and recruitment costs.²⁹

Feasibility: Ten of the studies assessed the feasibility of the implementation outcomes. In four studies, the implementation outcome was assigned based on the indicators reported. These studies assessed recruitment and retention,^26,28,33,34 interest, and enjoyment.²²

Fidelity: Fidelity of the interventions was reported in eight of the studies, of which four assigned fidelities based on the indicators reported. These studies were based on a predetermined protocol,²⁴ ensured the quality of implementation intention plans,³³ adhered to previous camera protocols,²⁵ adhered to weekly topics with example text messages,²³ and ensured accuracy owing to predetermined protocol.²¹

Penetration/saturation: None of the studies specifically reported that they assessed penetration/saturation as an implementation outcome. However, in assessing each study using the proposed indicators, only one study had penetration/ saturation assigned as an implementation outcome.³⁵

Sustainability: None of the studies reported intervention sustainability as an implementation outcome. However, sustainability was assigned in three studies using the proposed indicators. These studies reported on the follow-up to evaluate the quit rate to show continuation into the study,³⁰ attitude based on a 7-point Likert-type motivation scale,²⁰ and willingness to recommend the intervention.²²

Table 2.

Intervention outcome measures.

Author(s)	Intervention	Strategy	Health promotion intervention	Intervention implementation outcome	Assigned indicator	Original measure
Allen et al., 2022	Telephone behavioral counseling	Telephone call	Smoking cessation	Fidelity	Protocol implementation	Counseling sessions were recorded and reviewed
				Feasibility	Recruitment, retention	Recruitment rate, retention rate, survey completion rate, dried blood sample received
				Acceptability	Participant satisfaction	Participant satisfaction, acceptability survey
Bonar et al., 2023	Eight-week social media (Facebook)	Social media	Physical activity	Feasibility	Recruitment, retention, engagement	Recruitment rate, retention rate, participants’ posts/comments, and reactions
				Cost*	Cost	Recruitment advertising, total costs
				Fidelity	Protocol implementation	Psychologists’ weekly supervision, collaboration
				Acceptability	Content	Acceptability survey (helpful, enjoyable, motivated)
Brown et al., 2018	Tailored web-based program	Web-based	Contraceptive use	Feasibility*	Recruitment, retention, suitability	Recruitment, completion of measures, attrition completeness of implementation intention plans
Brown et al., 2018	Tailored web-based program	Web-based	Contraceptive use	Fidelity*	Protocol implementation	Quality of implementation intention plans
Coleman et al., 2022	MiQuit	Short message service	Smoking cessation	Cost	Cost	Cost per text message and participant, the monthly cost
				Cost	Cost-utility	Cost per participant and quit rates
				Sustainability*	Continuation	Quit rate
Davies et al., 2020	Wearable cameras	Wearable	Physical activity	Appropriateness*	Perceived fit	Reliability testing (inter-rater agreement testing)
Davies et al., 2020	Wearable cameras	Wearable	Physical activity	Fidelity*	Protocol implementation	Adherence to previous camera protocols (Wellard-Cole et al., 2018)
Dietel et al., 2020	Appearance-related cognitive bias modification for interpretation program	Web-based	Cognitive behavioral models	Acceptability	Credibility, expectancy, participant satisfaction	Credibility and expectancy, client satisfaction, and feedback
				Appropriateness*	Safety/suitability	Adverse reactions
				Feasibility*	Retention	Session compliance, dropout, and engagement
Godino et al., 2019	Personalized, tailored, and interactive short message service	Short message service	Weight loss	Feasibility	Actual fit	Daily engagement (likes/dislikes, response)
				Fidelity*	Protocol implementation	Weekly topics and example messages
				Adoption*	Preference	Language preference
Hamilton et al., 2018	HeLP-Alcohol	Phone calls, emails, and short message service	Alcohol cessation	Feasibility	Recruitment, retention	Recruitment numbers, retention numbers,
Hamilton et al., 2018	HeLP-Alcohol	Phone calls, emails, and short message service	Alcohol cessation	Cost	Cost-effectiveness	Cost-effectiveness
Hubert et al., 2022	Theory-based, online tailored message	Short message service	Healthy behavior	Acceptability	Agreement	Answer quickly, recommend to a friend
				Appropriateness*	Usefulness	Helpfulness, question relevance
				Feasibility	##	##
Johnson and Mays, 2021	Hookah tobacco social media risk communication	Social media	Smoking cessation	Appropriateness*	Perceived fit	Attitudes, emotional response
				Fidelity*	Protocol implementation	##
				Adoption*	Applicability and social acceptance	Social media use, engagement, and positive
Liu and Willoughby, 2018	Goal-setting theory-based text messages in fitness apps	Short message service	Physical activity and body fitness	Sustainability*	Attitude	Motivation (7-point Likert scale)
				Adoption*	Intention to use	Intention (7-point Likert scale)
				Appropriateness*	Relevance and usefulness	Attitude (7-point Likert scale)
				Acceptability*	Participant satisfaction	Message satisfaction (7-point Likert scale)
Liu et al., 2020	DigiDiet (Digital Diet)	Mobile application	Healthy diet	Fidelity*	Accuracy, protocol implementation	Accuracy, task duration, perception (System Usability Scale)
Liu et al., 2020	DigiDiet (Digital Diet)	Mobile application	Healthy diet	Acceptability*	Operability, complexity, participant satisfaction	Portion input is easy to operate, estimation error, takes time to complete, greater complexity
Merrill et al., 2018	Normative feedback via text messaging	Short message service	Alcohol cessation	Acceptability	Acceptable and convenient	Message delivery times and frequency, overall satisfaction with a 5-point Likert scale
Merrill et al., 2018	Normative feedback via text messaging	Short message service	Alcohol cessation	Feasibility	Recruitment	Response rate
Scholten et al., 2019	Mobile game HitnRun	Mobile application	Smoking cessation	Acceptability*	Intrinsic motivation	Spend time to use, like the game, attractive, useful, concentration, reality, Intrinsic Motivation Inventory (subjective experience)
				Feasibility*	Interest/enjoyment
				Sustainability*	Recommendable
				Appropriateness*	Interest and usefulness
Song and Kanaoka, 2018	“Karada-no-Kimochi”	Mobile application	Menstrual health	Cost	Application cost and medical expense	The sum of the application fee, medical expense, productivity loss, sensitivity analysis
Tay, 2022	Online HOPE intervention	Web-based	Mental health literacy	Acceptability	Barriers and motivations	Thoughts, experiences, and feelings of the participants; barriers and motivations for recommendation
				Penetration*	Access
				Fidelity*	Protocol implementation	Prior evaluation with two feasibility studies (Tay, 2020)
Whatnall et al., 2019	EATS (Eating Advice to Students) brief	Web-based	Healthy diet	Acceptability*	Participant satisfaction, appropriateness, retention	Satisfaction, usability(met)
				Appropriateness*	Usefulness, relevance, usability	Expectation, appropriateness
				Feasibility*	Recruitment, retention	Recruitment, retention

*Indicates cases where the implementation outcome was originally specified using a different measurement method but was reassigned by the authors based on the reported indicators. ## Indicates cases where the specific measurement method was not detailed, but the authors assessed that it had been measured.

Implementation outcomes based on health promotion intervention

The health promotion outcomes targeted in the various interventions included smoking cessation (n = 4), physical activity (n = 2), contraceptive use (n = 1), cognitive behavioral models (n = 1), weight loss (n = 1), alcohol cessation (n = 2), positive health behavior (n = 1), menstrual health (n = 1), a healthy diet (n = 2), and mental health literacy (n = 1). Regarding the implementation outcomes reported based on the specific health promotion intervention in each study, the highest single reported interventions were smoking cessation (n = 12) and physical activity (n = 10), while menstrual health was the least reported intervention. The health promotion interventions that reported at least all the implementation outcomes were physical activity and smoking cessation, except for penetration. In addition, for smoking cessation, acceptability, appropriateness, feasibility, fidelity, and sustainability were the most assessed (n = 2), while cost was the least assessed (n = 1). For physical activity, acceptability, appropriateness, feasibility, and fidelity were the most assessed (n = 2), while adoption, cost, feasibility, and sustainability were the least assessed. Figure 2 visualizes these findings, with each axis representing a specific health promotion domain (e.g. physical activity, smoking cessation, and healthy diet). The values increased outward from the center, indicating a higher frequency of reported implementation outcomes.

Figure 2.

Implementation outcomes based on health promotion interventions.

Implementation outcomes based on digital health intervention methods

The digital implementation techniques used included telephone calls (n = 1); social media (n = 2); web-based programs (n = 4); short message service (n = 5); wearable devices (n = 1); a mobile application (n = 3); and a combination of telephone calls, emails, and short message service (n = 1). The implementation outcomes that were reported under each method showed that short message service (n = 14), web-based programs (n = 11), and mobile applications (n = 7) were the most frequently reported, whereas telephone calls (n = 3); wearables (n = 2); and the combination of phone calls, emails, and short message service (n = 2) were the least frequently reported. The short message service-based techniques assessed all implementation outcomes except for penetration. Web-based interventions also reported all the implementation outcomes except for adoption, cost, and penetration, whereas mobile applications reported all except for adoption and penetration. Social media methods assessed all the implementation outcomes except for penetration and sustainability (Figure 3).

Figure 3.

Implementation outcomes based on digital health intervention.

Discussion

This review contributes to the scientific literature by examining the types of implementation outcomes in digital health interventions targeting young adults, focusing on a specific healthy population and intervention field. Considering that digital health interventions are becoming an increasingly important strategy among young people, the contribution of this study is even more meaningful. A key strength of the review lies in its systematic approach, which includes a focused research scope, stringent screening criteria, and the thorough mapping of implementation outcomes to specific health promotion and digital health interventions. Furthermore, unlike previous reviews that assigned indicators only when explicitly mentioned in studies, this review provides a more comprehensive synthesis by incorporating a diverse range of reported indicators.

Implementation outcomes are essential for the generation of scientific knowledge for the advancement of implementation science,⁴ particularly in ensuring the successful translation of interventions from controlled, laboratory-type settings to real-world clinical environments.¹¹ Both implementation outcomes and intervention outcomes (effectiveness) are crucial for creating scientific knowledge that is beneficial for the entire population,³⁶ but implementation outcomes are critical for determining the effectiveness of intervention on a target population.^37,38 Implementation outcomes serve as prerequisites for achieving service-level outcomes, which in turn influence client outcomes.^4,36,38–40 They play a fundamental role in facilitating the successful delivery of complex interventions owing to the mitigating influence they possess over the sociopolitical context.⁴¹ Therefore, considering implementation outcomes during program development will promote decision-making and optimizing service and client outcomes.⁴⁰

Nevertheless, this review found that although at least one implementation outcome, a crucial factor for ensuring scientific rigor,¹⁵ was reported in each study, none of the studies assessed all implementation outcomes comprehensively. Additionally, there was an uneven distribution in the evaluation of implementation outcomes across studies, depending on intervention type, duration, participants, and resource availability. This review aligns with a previous study that also reported acceptability and feasibility as the most assessed implementation outcome.^15,42 However, unlike other fields where appropriateness was considered as a key outcome,^16,43 our review found that fidelity was more commonly prioritized in digital health interventions. Implementation outcomes are valuable in the decision-making process, as they are adaptable to research settings, easy to implement, and highly acceptable.⁴⁴ Therefore, it is crucial for researchers to select implementation outcomes that are appropriate for the target population and intervention context.

Furthermore, this review identified that the definition of implementation outcomes tends to be ambiguous and often overlaps with other indicators. In some cases, implementation outcomes were applied inconsistently or inappropriately.^11,45 For instance, one study reported consistently assessing feasibility,²⁷ yet failed to specify concrete measurement criteria. Additionally, terms such as recruitment and retention, which are often classified under feasibility, and attitude, categorized under sustainability, were not originally included in the standard implementation outcome indicators.¹⁵ These inconsistencies and overlaps complicate cross-study comparisons, underscoring the need for clearer definitions and standardized measurement criteria for implementation outcomes.^10,43,46

Given that implementation outcomes often focus on specific aspects and do not comprehensively address all relevant indicators,^15,43 there is a clear need for conceptual refinement and a systematic approach to their assessment. With researchers increasingly recognizing the importance of implementation outcomes and incorporating them in the early stages of intervention development, it is essential to reach a consensus on the most critical implementation outcomes to be evaluated. In conclusion, researchers should strive to establish clearer definitions of implementation outcomes and integrate them with theoretical frameworks to effectively analyze intervention effectiveness and service outcomes.^15,16 Additionally, strategies should be developed to engage stakeholders in the decision-making process, facilitating policy development and practical application.

We mapped the implementation outcomes to digital health interventions and health promotion interventions among young adults. Understanding and evaluating implementation outcomes in digital health interventions, regardless of the intervention focus, target disease/problem, setting, and population, can be a panacea for attaining intervention effectiveness. Implementation outcomes were mostly used among web-based interventions compared to telephone call interventions, indicating that with the increasing complexity of intervention methods, researchers have assessed implementation outcomes. Regardless of the implementation method, implementation outcomes are important to consider, and implementation research should not focus only on effectiveness.⁴³ We showed that smoking cessation and physical activity were the most prevalent types of health promotion intervention and had the most implementation outcomes. Additionally, implementation outcomes are important measures for health promotion intervention suitability, scalability, usability, policy, and program outcomes.^47–49 These are critical indices for translating research into real-world clinical practice.⁴⁸

However, the limitation of this study is the exclusion of non-English studies, which limits the generalizability of the findings. Notably, the majority of the interventions analyzed in this review were conducted in high-resource settings, highlighting the existing disparity in the advancement of implementation science within low-resource contexts.^1,2 Researchers must address this gap by ensuring the efficient utilization of implementation outcomes to facilitate the development and application of digital health interventions in resource-limited settings.^10,11,15 Such efforts are essential to promote equitable access to evidence-based interventions across diverse populations. Moreover, as implementation science is still evolving, there is a likelihood that studies that used different terminologies to refer to or describe the implementation outcomes might not have been indexed appropriately and were excluded from the search or screening. In addition, this scoping review focused on implementation outcomes instead of other implementation outcomes. Hence, it is difficult to establish a direct relationship between the influence of implementation outcomes assessment and intervention effectiveness. Therefore, future systematic reviews should determine the influence of assessing implementation outcomes and its subsequent influence on intervention effectiveness and client-level outcome measures. Furthermore, due to the limited number of studies available in this field, this review conceptually included both implementation and intervention studies and expanded the age range to encompass all age groups that include young adults. Given that the objective of this scoping review is to assess the utilization of implementation outcomes rather than to evaluate intervention effectiveness, the inclusion of such studies was deemed appropriate. Therefore, future reviews should consider refining study designs to more precisely identify the utilization of implementation outcomes within specific research frameworks.

Conclusions

Researchers studying digital health interventions that target young adults in health promotion have largely assessed acceptability and feasibility, whereas there was no significant emphasis on other outcomes. There is a need to focus on the process of generating knowledge and intervening in implementation outcomes, a critical aspect. This study found that diverse and multiple indicators were used to describe implementation outcomes. Therefore, researchers in implementation science must develop and use standardized terminology to describe the different indicators for implementation outcomes. Developing and utilizing standard concepts, terminology, and indicators for implementation outcomes is critical for advancing implementation science. In this review, it was noted that none of the studies assessed all the implementation outcomes. For intervention scientists to achieve the desired impact, implementation planning is warranted, including identifying barriers in advance, using theoretical frameworks, tailoring interventions, engaging stakeholders, and co-designing processes by prioritizing implementation outcomes.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251330194 - Supplemental material for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review

Supplemental material, sj-docx-1-dhj-10.1177_20552076251330194 for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review by Kennedy Diema Konlan, Zainab Auwalu Ibrahim, Jisu Lee and Hyeonkyeong Lee in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076251330194 - Supplemental material for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review

Supplemental material, sj-docx-2-dhj-10.1177_20552076251330194 for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review by Kennedy Diema Konlan, Zainab Auwalu Ibrahim, Jisu Lee and Hyeonkyeong Lee in DIGITAL HEALTH

Supplemental Material

sj-docx-3-dhj-10.1177_20552076251330194 - Supplemental material for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review

Supplemental material, sj-docx-3-dhj-10.1177_20552076251330194 for The inclusion of implementation outcomes in digital health interventions for young adults: A scoping review by Kennedy Diema Konlan, Zainab Auwalu Ibrahim, Jisu Lee and Hyeonkyeong Lee in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors ZAI and JL received a scholarship from Brain Korea 21 FOUR Project funded by NRF of Korea, Yonsei University College of Nursing.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Contributorship

KDK and HKL: conceptualization; KDK, HKL, ZAI, and JL: methodology; KDK, ZAI, and JL: investigation; KDK, HKL, ZAI, and JL: data curation; KDK: writing–original draft preparation; KDK, HKL, ZAI, and JL: writing–review and editing; HKL: supervision; HKL: project administration.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Research Foundation of Korea (NRF) grant, funded by the Ministry of Education under Grant [NRF-2020R11A2069894] and the Ministry of Science and ICT [RS-2023-00279916].

Ethical approval

Not applicable.

ORCID iDs

Kennedy Diema Konlan

Zainab Auwalu Ibrahim

Jisu Lee

Hyeonkyeong Lee

Supplemental material

Supplemental material for this article is available online.

References

Bansal

Bertels

Ewart

, et al. Bridging the research–practice gap. New York: Academy of Management Briarcliff Manor, 2012.

Schreiweis

Pobiruchin

Strotbaum

, et al. Barriers and facilitators to the implementation of eHealth services: systematic literature analysis. J Med Internet Res 2019; 21: e14197.

Churruca

Ludlow

Taylor

, et al. The time has come: embedded implementation research for health care improvement. J Eval Clin Pract 2019; 25: 373–380.

Proctor

Silmere

Raghavan

, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38: 65–76.

Durlak

DuPre

. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008; 41: 327–350.

Arigo

Jake-Schoffman

Wolin

, et al. The history and future of digital health in the field of behavioral medicine. J Behav Med 2019; 42: 67–83.

Kemp

Trigg

Beatty

, et al. Health literacy, digital health literacy and the implementation of digital health technologies in cancer care: the need for a strategic approach. Health Promot J Austr 2021; 32: 104–114.

World Health Organization. Using e-health and information technology to improve health. https://www.who.int/westernpacific/activities/guiding-optimal-development-and-use-of-digital-health-towards-improved-health-outcomes (2022, accessed 10 10 2023).

World Health Organization. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Report. 2016.

10.

Mettert

Lewis

Dorsey

, et al. Measuring implementation outcomes: an updated systematic review of measures’ psychometric properties. Implement Res Pract 2020; 1: 2633489520936644.

11.

Proctor

Bunger

Lengnick-Hall

, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci 2023; 18: 31.

12.

Fagan

Bumbarger

Barth

, et al. Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: challenges and opportunities. Prev Sci 2019; 20: 1147–1168.

13.

Grimshaw

Eccles

Thomas

, et al. Toward evidence-based quality improvement: evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med 2006; 21: S14–S20.

14.

Michie

Fixsen

Grimshaw

, et al. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci 2009; 4: 1–6.

15.

Khadjesari

Boufkhed

Vitoratou

, et al. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implement Sci 2020; 15: 66.

16.

Peters

Sukumar

Blanchard

, et al. Trends in guideline implementation: an updated scoping review. Implement Sci 2022; 17: 50.

17.

Arnett

. Emerging adulthood. A theory of development from the late teens through the twenties. Am Psychol 2000; 55: 469–480.

18.

Miles

Huberman

. Qualitative data analysis: an expanded sourcebook. Thousand Oaks, CA: Sage Publications, 1994.

19.

Allen

Skobic

Bell

, et al. Feasibility and acceptability of testing a menstrual-cycle timed smoking cessation intervention for women of reproductive age (Project Phase): results of a pilot randomized control trial. Addictive Behaviors 2022; 125: 107153.

20.

Liu

Willoughby

. Do fitness apps need text reminders? An experiment testing goal-setting text message reminders to promote self-monitoring. J Health Commun 2018; 23: 379–386.

21.

Liu

Lin

, et al. Usability of food size aids in mobile dietary reporting apps for young adults: randomized controlled trial. JMIR Mhealth Uhealth 2020; 8: e14543.

22.

Scholten

Luijten

Granic

. A randomized controlled trial to test the effectiveness of a peer-based social mobile game intervention to reduce smoking in youth. Dev Psychopathol 2019; 31: 1923–1943.

23.

Godino

Golaszewski

Norman

, et al. Text messaging and brief phone calls for weight loss in overweight and obese English- and Spanish-speaking adults: a 1-year, parallel-group, randomized controlled trial. PLoS Med 2019; 16: e1002917.

24.

Johnson

Mays

. Testing the effects of hookah tobacco social media risk communication messages among young adults. Health Educ Behav 2021; 48: 627–636.

25.

Davies

Allman-Farinelli

Owen

, et al. Feasibility study comparing physical activity classifications from accelerometers with wearable camera data. Int J Environ Res Public Health 2020; 17: 9323.

26.

Dietel

Zache

Burkner

, et al. Internet-based interpretation bias modification for body dissatisfaction: a three-armed randomized controlled trial. Int J Eat Disord 2020; 53: 972–986.

27.

Hubert

Fiorenti

Duffy

. Feasibility of a theory-based, online tailored message program to motivate healthier behaviors in college women. Nutrients 2022; 14: 4012.

28.

Whatnall

Patterson

Chiu

, et al. Feasibility and preliminary efficacy of the eating advice to students (eats) brief web-based nutrition intervention for young adult university students: a pilot randomized controlled trial. Nutrients 2019; 11: 905.

29.

Bonar

Chapman

Pagoto

, et al. Social media interventions addressing physical activity among emerging adults who use cannabis: a pilot trial of feasibility and acceptability. Drug Alcohol Depend 2023; 242: 109693.

30.

Coleman

Clark

Welch

, et al. Effectiveness of offering tailored text message, self-help smoking cessation support to pregnant women who want information on stopping smoking: miQuit3 randomised controlled trial and meta-analysis. Addiction 2022; 117: 1079–1094.

31.

Hamilton

Hornby

Sheringham

, et al. DIAMOND (DIgital Alcohol Management ON Demand): a feasibility RCT and embedded process evaluation of a digital health intervention to reduce hazardous and harmful alcohol use recruiting in hospital emergency departments and online. Pilot Feasibility Stud 2018; 4: 114.

32.

Song

Kanaoka

. Effectiveness of mobile application for menstrual management of working women in Japan: randomized controlled trial and medical economic evaluation. J Med Econ 2018; 21: 1131–1138.

33.

Brown

Beasley

Das

. Self-control, plan quality, and digital delivery of action planning for condom and contraceptive pill use of 14–24-year-olds: findings from a clinic-based online pilot randomised controlled trial. Appl Psychol Health Well Being 2018; 10: 391–413.

34.

Merrill

Boyle

Barnett

, et al. Delivering normative feedback to heavy drinking college students via text messaging: a pilot feasibility study. Addictive Behaviors 2018; 83: 175–181.

35.

Tay

. Online HOPE intervention on help-seeking attitudes and intentions among young adults in Singapore: a randomized controlled trial and process evaluation. Arch Psychiatr Nurs 2022; 41: 286–294.

36.

Clarke

Conti

Wolters

, et al. Evaluating the impact of healthcare interventions using routine data. Br Med J 2019: 365: l2239.

37.

Davis

Guterman

Collins

, et al. Starting on the path to a high performance health system: analysis of the payment and system reform provisions in the Patient Protection and Affordable Care Act of 2010. Report. The Commonwealth Fund, New York, 2010.

38.

Djulbegovic

. A framework to bridge the gaps between evidence-based medicine, health outcomes, and improvement and implementation science. J Oncol Pract 2014; 10: 200–202.

39.

Proctor

Landsverk

Aarons

, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009; 36: 24–34.

40.

Zamboni

Schellenberg

Hanson

, et al.

Assessing scalability of an intervention: why, how and who?

Health Policy Plan 2019; 34: 544–552.

41.

Garner

Hunter

Funk

, et al. Toward evidence-based measures of implementation: examining the relationship between implementation outcomes and client outcomes. J Subst Abuse Treat 2016; 67: 15–21.

42.

Lewis

Fischer

Weiner

, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci 2015; 10: 155.

43.

Weiner

Lewis

Stanick

, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12: 1–12.

44.

Powell

Weiner

Stanick

, et al. Stakeholders’ Perceptions of Criteria for Pragmatic Measurement in Implementation: A Concept Mapping Approach. In: 9th Annual Conference on the Science of Dissemination and Implementation, 2016. AcademyHealth.

45.

Lengnick-Hall

Gerke

Proctor

, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci 2022; 17: 16.

46.

Martinez

Lewis

Weiner

. Instrumentation issues in implementation science. Implement Science 2014; 9: 1–9.

47.

Willmeroth

Wesselborg

Kuske

. Implementation outcomes and indicators as a new challenge in health services research: a systematic scoping review. Inquiry 2019; 56: 46958019861257.

48.

Milat

King

Bauman

, et al. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int 2013; 28: 285–298.

49.

Shah

Allison

Schoueri-Mychasiw

, et al. A review of implementation outcome measures of school-based physical activity interventions. J Sch Health 2017; 87: 474–486.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB

0.11 MB