Development of an Implementation Science Telehealth Toolkit to Promote Research Capacity in Evaluation of Telehealth Programs

Abstract

Background:

The field of telehealth is rapidly growing and expanding access to quality health care, although there have been varied implementation outcomes in telehealth modalities. Dissemination and implementation (D&I) research can provide a systematic approach to identifying barriers and facilitators to telehealth implementation processes and outcomes.

Methods:

An interdisciplinary research and clinical team developed an implementation science telehealth toolkit to guide D&I evaluations of new and existing telehealth innovations.

Results:

The toolkit includes a separate section to correspond to each step in the D&I evaluation process. Each section includes resources to guide evaluation steps, telehealth specific considerations, and case study examples based on three completed telehealth evaluations.

Discussion:

The field of telehealth is forecasted to continue to expand, with potential to increase health care access to populations in need. This toolkit can help guide health care stakeholders to develop and carry out evaluations to improve understanding of telehealth processes and outcomes to maximize implementation and sustainability of these valuable innovations.

Background

In recent years, utilization of telehealth modalities for health care delivery has expanded, both due to the COVID-19 pandemic and documented literature of the efficacy, acceptability, and value in these innovations.^1–7 Despite the rapid expansion of telehealth, implementation has varied across care settings and specialties and among different patient populations, raising questions about equity and the resources required for quality telehealth programs.^8–11 Thus, it is vital to understand implementation processes, characterize determinants of successful telehealth implementation, and define strategies for quality telehealth care across settings.

The field of dissemination and implementation (D&I) research has been showcased as key to accelerating the dissemination, implementation, and sustainability of evidence-based interventions into health care practice.^12,13 D&I research is typically guided by frameworks, theories, and models with interdisciplinary research, clinical, and administrative teams using multiple methods to describe a variety of implementation processes and outcomes related to health care innovations. The multitude of approaches to D&I research can pose barriers to engagement in this field, but educational resources such as toolkits have been leveraged to increase engagement. ¹⁴ The Agency for Healthcare Research and Quality (AHRQ) defines a “toolkit” as a collection of resources that can guide users in implementation of evidence-based practices or development of plans to reach evidence-based recommendations. ¹⁵

While there are numerous existing D&I toolkits in the literature to support D&I evaluations of evidence-based interventions to health care practice, no D&I toolkits specifically focus on telehealth interventions. ¹⁶ The field of D&I research offers many benefits to telehealth including providing a systematic approach to understanding telehealth outcomes and processes, demonstrating the value of telehealth programs, identifying challenges and success in telehealth delivery, facilitating utilization of data to overcome barriers and challenges, and guiding implementation strategies to improve patient and system outcomes.

Due to recent rapid telehealth expansion and benefits of the D&I field to contribute to efficient and effective telehealth implementation processes and outcomes, a team within the Telehealth Center of Excellence (COE) created a D&I telehealth toolkit to guide interprofessional telehealth teams to design and conduct D&I telehealth evaluations. One of the goals of the COE is to develop resources for telehealth providers, staff, and researchers to improve quality of telehealth innovations. This toolkit seeks to share best practices in D&I research steps and utilizes three completed telehealth evaluations to demonstrate D&I methodologies for telehealth innovations.^17–23 The telehealth toolkit is intended to be a dynamic adaptable resource that can be utilized in a variety of telehealth settings by a broad group of telehealth stakeholders.

The objective of this manuscript is to describe the telehealth toolkit development process, content included in the toolkit, and applications to telehealth D&I evaluation. The toolkit and other COE resources can be accessed publicly at no cost online via the Telehealth Center of Excellence website.

Methods

The development of the telehealth toolkit originated with an interdisciplinary team review of literature related to the evolving resources and tools in the D&I field. This literature review included an inventory of existing published D&I toolkits and examination of key components of D&I evaluation including common study designs, existing frameworks and models, implementation outcomes, and measures to assess those outcomes. Our team then framed the conceptualization of the telehealth toolkit with a series of interdisciplinary team (physicians, health services researchers, nurses, telehealth research coordinators) discussions to outline the format and content, after which two researchers (E.E.J. and K.R.S.) developed the content for each toolkit section. Throughout the process of content development, we followed the AHRQ toolkit checklist for content and usability to ensure sufficient information was presented in a clear format. ¹⁵ The process of toolkit development was considered exempt from the university IRB.

After the initial toolkit draft was completed, our internal interdisciplinary telehealth team (N = 5) provided comprehensive review and feedback, which was used to update the toolkit content and formatting. Next, we completed an external review to assess toolkit acceptability and usability by telehealth experts, including three reviewers knowledgeable in the telehealth field. Participants reviewed the toolkit and completed an electronic survey. The survey included three items to rate usefulness of the toolkit to different audiences (scale of 1–4), seven questions assessing usability of the toolkit (scale of 1–100) and two questions rating content and clarity for each section of the toolkit (scale of 1–4). Open-ended feedback about each toolkit section was also collected.

Results

Toolkit content

Figure 1 illustrates the sections of the D&I telehealth toolkit and Table 1 describes the content within each section. Each toolkit section corresponds to a step in the D&I evaluation process: refine research question, determine study design, define implementation outcomes, select implementation framework, develop data collection tools and measures, and select implementation strategies. Each section includes consistent content to demonstrate the corresponding D&I evaluation step with introductory materials, practical questions for the research team, telehealth specific considerations, case study examples, and external articles and resources. The section on selecting implementation strategies is unique in that the steps in this section can be utilized throughout the D&I evaluation process to overcome identified barriers to implementation. Three case study examples from South Carolina telehealth programs were included throughout the toolkit, with one illustrative example in each section to further elucidate that specific D&I evaluation step.

FIG. 1.

D&I telehealth toolkit sections; D&I evaluation process. D&I, dissemination and implementation.

Table 1.

Sections of Telehealth Toolkit

Telehealth Toolkit content in each section	Description of content
Introductory Materials	Background information to define each D&I research step
Practical Questions to Guide Team	Key questions that a research team should consider during the current phase of research
Telehealth Specific Considerations	Relates the section content to the field of telehealth with examples
Case Study Examples	Provides a specific example of the D&I research step from one of three completed telehealth team projects to operationalize and illustrate the step
Key Articles and Resources	A reference list of external articles and websites to correspond with section materials

D&I, dissemination and implementation.

The first case study was an evaluation of a school-based telehealth asthma program. These types of telehealth programs have previously been demonstrated as cost-effective in improving asthma patient outcomes yet are challenging to implement.^24–26 The D&I objective for this study was to characterize school-based telehealth asthma program delivery experiences and examine barriers and facilitators to telehealth program implementation.^20–23

The second case study was an evaluation of a remote patient monitoring program to support diabetes management. While remote patient monitoring programs have exhibited success in improving patient diabetes outcomes, a program in South Carolina demonstrated variable implementation success across clinics. ²⁷ The objective for this D&I case study was to characterize clinic delivery strategies for a South Carolina remote patient monitoring program and to examine barriers and facilitators to program implementation in underserved and/or low-income community settings. ¹⁹

The third case study was an evaluation of a program to treat perinatal and postpartum mental health and substance use disorders. These disorders are common across many populations, with high resulting morbidity and mortality.^28–31 Although evidence-based treatment is available for these conditions, many barriers to care exist. Thus, the objective for this D&I study was to characterize barriers and facilitators to implementing a mental health and substance-use disorder telehealth program in community obstetric and pediatric clinics.^17,18

Internal and external review

The internal toolkit review guided changes to content, the order of sections, and overall formatting. Table 2 illustrates the feedback from the toolkit external review, which was also utilized to update the toolkit content and format. The toolkit content, clarity, and flow received high rankings from all three reviewers. In addition, the toolkit received high rankings in usability and recommendations for use by others. However, the ability of beginner D&I researchers to independently utilize the toolkit without insight from mentors or other resources received lower endorsement scores. These scores on usability for beginner researchers were considered acceptable as our team intends for this toolkit to serve as a supplemental guide to D&I telehealth evaluation and not replace expert mentorship.

Table 2.

External Review of Telehealth Toolkit

Item	Scale response set average (median, range); N = 3
Extent to which toolkit will be useful as a resource for	1–4 (1 = not very useful; 4 = extremely useful)
D&I Beginners	3.7 (4, 3–4)
D&I Intermediates	3.7 (4, 3–4)
D&I Experts	2.7 (3, 2–3)

Toolkit acceptability/usability ratings	1–100 (1 = completely disagree; 100 = completely agree)
After reviewing the toolkit, I understand its content	90 (100, 70–100)
This toolkit is easy to understand	88 (90, 78–95)
The flow of the toolkit topics was appropriate	94 (100, 81–100)
A person could use this toolkit and apply to the telehealth field without the help of a D&I expert	60 (60, 40–80)
This toolkit provides all the resources/information that I need to know about D&I evaluations related to telehealth	73 (71, 57–90)
I think this toolkit would be very useful for someone completing a D&I telehealth evaluation	93 (100, 78–100)
I would recommend this toolkit to others who are going to carry out a D&I telehealth evaluation	97 (100, 90–100)

Content and clarity ratings for toolkit sections	1–4 (1 = not very comprehensive/clear; 4 = extremely comprehensive/clear)
Section	Content	Clarity
1. Introduction	3.3 (3, 3–4)	3.3 (4, 2–4)
2. Refining Research Question	4.0 (4, 4)	3.7 (4, 3–4)
3. Determining Study Design	3.3 (3, 3–4)	3.7 (4, 3–4)
4. Defining Implementation Outcome(s)	3.0 (3, 2–4)	4.0 (4, 4)
5. Selecting a Guiding Implementation Framework	3.7 (4, 3–4)	3.0 (3, 3)
6. Developing Data Collection Tools and Measures	4.0 (4, 4)	3.7 (4, 3–4)
7. Selecting Implementation Strategies	3.7 (4, 3–4)	3.3 (3, 3–4)
8. D&I Toolkit Resources	3.3 (4, 2–4)	3.3 (4, 2–4)

Based on the qualitative suggestions for improvement, we reordered the introduction section of the toolkit and reformatted all tables within the toolkit for consistency. We also added additional clarification in the implementation framework and implementation strategy sections to define terms and concepts more clearly. Lastly, after incorporation of external reviewer feedback, our internal telehealth team reviewed the toolkit again, revisions were finalized, we created a dissemination plan, a final draft was sent to the formatting team, the formatted version was reviewed by our team, and the toolkit was released as an interactive PDF to the HRSA Telehealth Center of Excellence Website. ³²

Discussion

The expansion of telehealth in recent years has led to a proliferation of research demonstrating the potential for telehealth to increase health care access, improve patient outcomes, and lower costs.^33–35 Moreover, patients and providers appreciate the convenience telehealth affords.^3,36 These benefits, however, have neither been universal nor experienced equitably. Research has shown variation among practices in terms of their readiness and the resources available to them to implement telehealth. Similarly, despite telehealth being an effective tool to support health care access in rural communities, rural patients were less likely to engage in telehealth during the COVID-19 pandemic compared with urban patients.^9,37 There also have been reported disparities in use of telehealth among different patient populations due to various demographic characteristics and social determinants of health.^8,9,11,34

With the Centers for Medicare and Medicaid Services making permanent many of the telehealth flexibilities authorized during the pandemic and many state Medicaid and private payers following suit, it is evident telehealth will continue to play an important role in the US health care system and the evolving landscape of evidence-based interventions.^38,39 D&I research has great promise to help stakeholders understand global processes and complexities related to telehealth clinical practice and support more equitable access to novel telehealth programs across care settings and among diverse populations.

Telehealth researchers and administrators have the opportunity to leverage D&I research to better understand the barriers and facilitators to telehealth implementation and sustainment and how to best adapt telehealth programs to meet their specific contexts. Despite the utility of D&I, many have noted barriers to engagement in D&I research due to lack of training and the diversity of frameworks, approaches, and tools within the field. ¹⁴ Resources are needed to support uptake of D&I research within different scientific communities. Thus, a primary strength of this telehealth toolkit is that it meets a gap in the literature to provide D&I guidance specifically to the field of telehealth. The toolkit can be utilized in a variety of settings as a template that will inform development, implementation, and sustainment of telehealth innovations. The toolkit can also identify resource and logistical strategies that may be utilized to mitigate telehealth disparities among rural communities. With tools like this to support the development of D&I research questions and rigorous methods to evaluate telehealth outcomes, ultimately additional telehealth-specific D&I research can advance the development and implementation of appropriate and efficacious strategies to reach and provide quality care through telehealth.

Future directions for this telehealth toolkit include continued modification of the toolkit content and format by our interdisciplinary team to meet the dynamic D&I research field. Based on the external review, we can add introductory content to the toolkit in the future to make it more functional for a beginner D&I researcher. Our team is currently tracking ongoing dissemination of the toolkit by monitoring counts of toolkit downloads from the COE website and will plan future evaluations of usability and outcomes generated from toolkit use.

Conclusion

With telehealth becoming a mainstay in the US health care system, it is vital to identify barriers and facilitators to implementation and sustainment of telehealth innovations. The field of D&I offers structured methods to assess implementation processes and outcomes. Developed by a multidisciplinary team of D&I scientists and telehealth experts, the toolkit described in this manuscript provides stepwise resources to guide telehealth teams and stakeholders in D&I evaluation.

Authorship Contribution Statements

E.E.J.: Toolkit conceptualization, toolkit development and editing, manuscript composition and editing; R.K., R.V., E.W., and D.F.: Toolkit conceptualization, toolkit editing, and manuscript composition and editing; K.R.S.: Toolkit conceptualization, toolkit development and editing, and manuscript composition and editing.