A comprehensive evaluation tool to assess community capacity and readiness for virtual care implementation

Abstract

Introduction

The rapid evolution and implementation of virtual care technologies for clinical use often exceeds the development of standardized implementation protocols, leading to gaps in the equitable and sustainable adoption of virtual care services, particularly in rural and remote areas. This paper introduces a comprehensive evaluation tool designed to assess community capacity and readiness for virtual care.

Methods

The development of the tool was informed by experiences from the Virtual Care and Robotics Program at the University of Saskatchewan. It involved a collaborative, multi-stakeholder approach that engaged healthcare leaders, IT experts and community healthcare workers. This iterative process included defining evaluation categories, mapping evaluative domains and refining the tool into a user-friendly checklist manual.

Results

The tool identifies three key domains for assessing readiness: clinical needs, technology infrastructure and human resources/workflows. It was piloted in the remote community of Fond du Lac, Saskatchewan, where it successfully qualified the community for implementing telerobotic ultrasound services. The tool facilitated local engagement and highlighted the community's specific needs and readiness, enhancing the implementation process.

Conclusion

This evaluation tool contributes to bridging the gap between the rapid deployment of virtual care technologies and the need for comprehensive, standardized implementation protocols. It offers a structured, practical approach to assessing and enhancing community readiness for virtual care while promoting successful clinical implementation and equitable access to healthcare.

Introduction

Despite virtual care's increasing prevalence in healthcare delivery, the rapid evolution of technology often outpaces the development of standardized implementation protocols necessary to direct safe and effective use.^1,2 As virtual care solutions become increasingly integral to efficient healthcare delivery, this gap between technologies and their implementation protocols can impede equitable and sustainable health system adoption and integration.^3,4

The complexity of implementing virtual care programs in existing healthcare systems is difficult to overestimate. In some settings, virtual care might be deployed efficiently with seamless integration into existing healthcare structures. ⁴ In contrast, in rural and remote communities with unique challenges and characteristics including geographic remoteness, limited workforce and variable expertise and experience in digital health, the lack of guidelines can lead to inefficiencies, underutilization or even mismanagement of these technologies.^2,5 As a result, there is a growing recognition of the need for tools and frameworks that can standardize the implementation of virtual care, ensuring that all communities, regardless of their geographic location or socio-economic status, can benefit from these technological advancements.^6,7

The purpose of this paper is to describe a new comprehensive evaluation tool designed to assess community capacity and readiness for virtual care implementation, including its development process and functionality. The tool evaluates several key areas including community clinical needs, technological infrastructure, human resources and clinical workflows, to identify and mitigate potential barriers to deploying virtual care technologies. We assess the utility of the tool in informing the implementation of a telerobotic ultrasound system in an isolated northern Canadian community restricted to fly-in only accessibility. Informed by experiences from the Virtual Care and Robotics Program at the University of Saskatchewan as well as the literature surrounding eHealth and digital health readiness, this tool can play an important role in qualifying remote sites as appropriate candidates for virtual care services. We make this tool available to virtual care and information technology (IT) practitioners, leaders and researchers to use or adapt for their local context.^8,9,10

Methods

Experience from Saskatchewan informing development of the tool

In developing the tool, the firsthand experiences and challenges encountered by the Virtual Care and Robotics Program at the University of Saskatchewan played a foundational role. For the past decade, this program has been at the forefront of collaborating with rural and remote communities in underserved areas of the province, introducing virtual care solutions and enhancing healthcare delivery through advanced technologies. ¹¹ The program's accomplishments include the successful implementation of medical robotics, such as remote presence robots and tele-robotic diagnostic systems, to improve access to healthcare services for under-resourced populations in remote areas.^12–14 The program's efforts include research and advancing real-world applications of these technologies, supported by collaborations with academic and industry partners.

While working with remote communities in Saskatchewan over the last decade, the Virtual Care and Robotics Program has faced unique challenges that have provided important insights surrounding implementation. For example, the program has navigated issues such as limited internet connectivity, varying levels of digital literacy among clinicians and community members, and logistical hurdles in delivering healthcare equipment to remote locations. Furthermore, the program has encountered barriers to implementations of newly deployed virtual care systems due to incompatibilities of workflow with existing clinical structures. These practical experiences were instrumental in shaping the design and functionality of the tool and provided insights into forces that drive sustainable change.

Process for development

The comprehensive evaluation tool was developed using an iterative and multi-stakeholder collaborative approach, engaging stakeholders from diverse backgrounds in a cyclical process of feedback and refinement. ¹⁵ The development process acknowledged that a co-creation engagement approach with diverse partners is needed to fully understand key contextual characteristics. ¹⁵ Highlighting the importance of individual and group agency in implementation design, ¹⁶ an environment was fostered where stakeholders were encouraged to make meaningful and purposeful contributions, influencing specific aspects of the tool while responding to the diverse needs and insights of all parties involved. The development of the tool drew upon insights and recommendations from representatives from the Virtual Care and Robotics Program, the Saskatchewan Ministry of Health, eHealth Saskatchewan, Northern Medical Services, and clinicians with experience working in Saskatchewan's remote communities. Collectively, this multidisciplinary group brought together experts and leaders in the fields of IT, healthcare and project management, while facilitating collaboration across multiple levels of government. This co-creation process allowed for the incorporation of local insights and needs into the tool, ensuring it was adaptable and responsive to the unique challenges of each community.

The first interactions between stakeholders to inform the development of the framework occurred organically through in-person and virtual meetings, emails and telephone calls from March to July 2018. This initial phase of development involved defining the primary categories for evaluation, followed by a delineation of specific areas of assessment within each category. Multiple cycles of stakeholder feedback and revisions aimed to capture all critical aspects necessary to comprehensively assess the compatibility of virtual care systems within the distinctive fabric of any community. Subsequently, the evaluative domains were visually mapped using user-friendly flowcharts, completed by January 2019 before later simplification into a checklist manual by May 2023 (Figure 1), with the goal of enabling communities to drive their own evaluations in collaboration with the Virtual Care and Robotics Program, enhancing local engagement and implementation.

Figure 1.

Developmental process and timeline.

Principles considered for the development

In collaboration with the stakeholders, the development of the evaluation framework was guided by four key principles: equity, practicality, sustainability and flexibility. Equity was a foundational aspect, acknowledging disparities in resources and conditions across different communities, and the impact these gaps have on successful virtual care uptake. It was critical for the tool to define the requisite conditions for successful implementation, and to support communities in accurately assessing whether these criteria are met. By identifying and highlighting discrepancies, the tool can support targeted investments into key areas, ensuring equitable opportunities for all communities to benefit from virtual care. Emphasizing practicality ensures the tool's usability, making it straightforward for communities and healthcare providers to employ. ¹⁷ Sustainability was another priority, with strategies embedded within the tool to ensure the longevity of virtual care initiatives to enhance health and well-being. Flexibility within the tool serves a dual purpose: to ensure the tool is technologically agnostic, allowing for adaptability in the face of virtual care's rapid evolution, and to ensure the tool can be used or easily adapted in any community. This flexibility ensures the framework can be effectively implemented in any setting, regardless of community size, location or unique healthcare needs.

Piloting the tool: case study with Fond du Lac

The tool was piloted for the deployment of telerobotic ultrasound services in the community of Fond du Lac. Fond du Lac is a remote Dene First Nation community of 980 people located in northern Saskatchewan and is part of the Athabasca Denesuline Territory. The community is situated on the east side of Lake Athabasca, near the border with the Northwest Territories and 686 km north of the closest tertiary care centre. Fond du Lac is accessible primarily by air and seasonal ice roads, reflecting its remote and isolated geographic position. The population of Fond du Lac is predominantly Indigenous, with most residents belonging to the Fond du Lac First Nation. The community has retained a strong connection to its traditional Dene culture and language. Remote communities in Saskatchewan often face pronounced healthcare disparities with respect to access and utilization of ultrasound services.^18,19 These disparities are particularly acute in remote Indigenous communities, where isolation and resource scarcity are exacerbated by a complex history of marginalization leading to institutional distrust.^20,21

Individuals from the Athabasca Health Authority, the community's healthcare system, and eHealth Saskatchewan, the network administrators and healthcare application supporters for the jurisdiction, collaborated to complete the evaluation tool to assess site compatibility for a telerobotic ultrasound system. Telerobotic ultrasound is an advanced technology that allows for the remote operation of an ultrasound system by a sonographer from any location, thereby enhancing access to diagnostic imaging services.^13,22

Results

Components of the tool

Three domains were identified as critical to assess community capacity and readiness for virtual care implementation and were included in the evaluation tool: (1) clinical needs assessment, (2) technology readiness assessment and (3) human resources and workflow assessment (Figure 2). The complete tool is available in Supplemental Information for virtual care and IT practitioners, leaders and researchers to use or adapt for their local context.

Figure 2.

Three domains of the tool – clinical needs assessment, technology readiness assessment, and human resources and workflow assessment.

(1) Clinical needs assessment

A foundational component of the tool is the facilitation of a clinical needs assessment within remote communities. This assessment is designed not to impose a predetermined set of criteria but to assist communities in identifying and articulating their specific healthcare needs. By doing so, the tool places the power of decision-making into the hands of the community members themselves, recognizing that they are best positioned to understand their unique healthcare challenges and priorities.

The tool guides communities through a comprehensive evaluation of their healthcare landscape, including identifying prevalent health issues, gaps in current healthcare services and potential areas where virtual care could be most beneficial. It offers guidance on how to conduct this analysis, suggesting critical focus areas and recommendations for collaborating with healthcare authorities and service providers to gather necessary health information to support the assessment's conclusions. Ultimately, the community leads the discovery and decision-making process, ensuring that the implementation of virtual care is tailored to address their most pressing clinical needs, and fostering a sense of ownership and commitment among its members.

(2) Technology readiness assessment

Virtual care technologies have baseline system requirements that must be met to ensure the consistency, quality and reliability of their functions. Before communities can be considered for implementation, their healthcare facilities must have the requisite IT infrastructure and support systems in place to meet or exceed these standards. The tool assesses facilities’ IT environments with a focus on essential elements including communication bandwidth, Wi-Fi access point distribution and Quality of Service (QoS). QoS is the use of mechanisms to manage and prioritize network traffic, ensuring performance levels for specific technologies in environments where bandwidth is limited. In instances where a community's IT framework exhibits deficiencies, our team offers guidance and mentorship to help bridge these gaps, thereby facilitating the pathway towards technological readiness.

Our experience in dealing with technical challenges has highlighted the critical role of designating a specialized technical liaison within each community. This role is critical for streamlining the management of technical issues related to virtual care systems. The liaison acts as a primary point of contact and first responder to any technical challenges, liaising with our technical team for efficient testing and troubleshooting. This approach provides a reliable and responsive first-line of defence to maintain smooth operations by addressing technical issues as they arise.

(3) Human resources and workflow assessment

The integration of a human resources and workflow analysis component into the tool reflects an understanding of the practical side of virtual care implementation. Recognizing that technology is just one facet of successful virtual care delivery, this component focuses on a team evaluation of the available human resources and existing healthcare workflows within the healthcare facility and ensuring alignment with new operational protocols. This includes analyzing the skill sets of local healthcare workers, their capacity to adapt to virtual care technologies, and the need for further training or support.

Additionally, the tool evaluates the potential impact on the workload of healthcare providers, the adaptability of current practices to incorporate virtual care, and the overall readiness of the healthcare system to embrace and sustain this change. The tool maps out a step-by-step workflow standard for the newly operating technology, ensuring that duties and responsibilities are assigned to and accepted by specific individuals. By including this human resources and workflow evaluation, the tool ensures that the implementation of virtual care is not only technologically feasible but also practically viable and compatible with the community's existing healthcare structure.

Case study: results from the Fond du Lac pilot

The evaluation tool was successful in qualifying the community of Fond du Lac for the implementation of telerobotic ultrasound services. In addition to confirming this compatibility, the tool facilitated a nuanced understanding of the community's unique needs and capacities. Highlights of the evaluation and implementation process are summarized in Figure 3.

Figure 3.

Highlights of Fond du Lac’s evaluation and implementation of telerobotic ultrasound.

Clinical needs and technological infrastructure

The evaluation identified a pressing need for enhanced diagnostic services, with abdominal ultrasound services identified as a critical gap. Fond du Lac's existing IT infrastructure, featuring over 10 Mbps of communication bandwidth, comfortably exceeded the baseline requirements for the efficient operation of the telerobotic system. This technological readiness was a key enabler, ensuring high-quality, real-time ultrasound imaging could be delivered without any technical issues.

Operational adjustments and space modification

Operational adjustments were necessary to integrate the new system within the existing clinical workflow seamlessly. The designated room underwent modifications to support the telerobotic system's operational requirements, including the installation of four additional Ethernet jacks and power outlets. These adjustments ensured that the system's three main components could communicate effectively and operate without disruptions, integrating the technology smoothly into the healthcare facility.

Human resources and training

A vital component of the implementation process was human resources evaluation and training. The initial two weeks post-deployment saw nine individuals trained in the patient site assistant role, a testament to the community's commitment to building local capacity for sustainable operation of the telerobotic ultrasound system. This training ensured that staff were well-equipped to support the technology, enhancing the efficacy of the service and fostering a sense of ownership within the healthcare team.

Clinical impact and service launch

The launch of five clinics following the training period marked the inauguration of telerobotic ultrasound services for Fond du Lac. Anecdotally, the early stages of deployment demonstrated the tangible benefits of the new service by enabling patients to have an ultrasound exam in their home community without the need to travel long distances. This service not only filled a critical gap in local healthcare provision but also showcased the potential for such technologies to revolutionize care delivery in remote communities.

Discussion

The comprehensive evaluation tool presented in this paper serves as a practical guide to assessing the capacity and readiness for virtual care implementation in rural and remote communities with variable resources, expertise and experience. The tool addresses the critical gap between the rapid evolution of virtual care technologies and standardized protocols necessary for their effective, equitable and sustainable deployment in remote communities. By assessing community capacity and readiness across multiple domains, including clinical needs, technological infrastructure, human resources and workflows, the tool provides a systematic approach to identifying and mitigating potential barriers to virtual care implementation.

Key readiness types commonly identified in the eHealth and digital health readiness literature include organisational readiness, technological/infrastructural readiness, healthcare provider readiness, engagement readiness, societal readiness, core readiness, government readiness and public/patient readiness. ²³ However, Mauco et al. found that none of these readiness types are comprehensively addressed by current e-health readiness assessment frameworks. ²³ The authors emphasize the need for context-specific tools that cater to the unique challenges faced in specific regions, including infrastructure limitations, socio-cultural dynamics and varying levels of ICT literacy. ²³ There is a notable absence of standardized and reliable tools for assessment of e-health readiness and consistent application across diverse healthcare settings, contributing to variable results and high failure rates. ²⁴ In contrast to prior research on eHealth and digital health readiness, one of the key distinctions of the tool which we present in this paper is its algorithmic approach. Unlike most prior studies which focus on identifying readiness types and constructs for evaluation, this tool provides a finite, structured set of instructions to solve specific problems surrounding various stages of readiness. As a result, it offers a more actionable, reproducible and transparent process, making the steps required for implementation clearer and more defined. The tool supports the domains included by Katz et al. in their modified Delphi study evaluating constructs of a telemedicine framework for designing and implementing telemedicine interventions. ²⁴

Although telehealth adoption surged during the COVID-19 pandemic, many vulnerable populations, such as those in lower socioeconomic areas, people with disabilities, and individuals with low digital literacy, face significant barriers to accessing these services. ²⁵ Previous research into identifying and assessing these barriers has highlighted that co-designing telehealth-enabled models of care with consumer involvement to ensure culturally appropriate solutions is essential to bridging gaps to equitable access. ²⁵ Furthermore, a multi-stakeholder approach that includes a focus on workforce training and improving digital health literacy is deemed crucial for success. ²⁵ These identified themes are congruent with not only the multi-stakeholder approach for designing the tool, but the collaborative processes within, emphasizing the practical side of workflow planning and workforce training. The tool expands on the existing literature by offering organizations an actionable, step-by-step approach, helping to parameterize and navigate the complexities of virtual care adoption. Prior work has highlighted the challenges faced by remote and rural communities in adopting virtual care solutions due to a lack of standardized implementation protocols. ²⁶ Additionally, with the underrepresentation of well-defined frameworks for interacting with Indigenous communities to deliver virtual healthcare, previous studies have indicated a heightened potential for causing harm.^25–28 The tool builds upon this literature by offering a practical framework that draws upon firsthand experiences and insights gained from implementing virtual care solutions in remote settings. By engaging stakeholders from various sectors including healthcare, IT and project management, the tool is both comprehensive and user-friendly, guided by principles of equity, practicality, sustainability and flexibility. ¹⁶

By engaging a diverse group of representatives, the tool was not only customized to address the unique needs and challenges of communities, but also responded to the priorities of multiple stakeholders. For example, the Ministry of Health was motivated to produce a tool that was transparent and politically defendable in the context of resource investment into virtual care, whereas members of the Virtual Care and Robotics Program needed an actionable framework to scale up virtual care operations in a practical and standardized fashion.

The pilot study in Fond du Lac demonstrated the utility and effectiveness of the evaluation tool in qualifying a remote community for the implementation of telerobotic ultrasound services. By systematically assessing clinical needs, technological infrastructure and human resources, the tool identified and addressed key considerations for successful implementation. The incorporation of virtual care technologies such as telerobotic ultrasound through the application of the evaluation tool has the potential to increase service capacity, reduce costs, promote interprofessional collaboration, enhance patient/provider experience and ultimately improve the health of more underserved populations such as Fond du Lac.

Engaging closely with Fond du Lac's team members to complete all sections of the comprehensive evaluation tool highlighted the critical importance of collaborative processes in securing the buy-in necessary for successful virtual care implementation. By mentoring the clinical needs assessment, walking through workflows, helping to assign tasks and responsibilities, working with IT personnel and enlisting a local technical liaison, the research team effectively built relationships, garnered commitment and gained the necessary influence for adoption and uptake. Tools that promote interdisciplinary collaboration and co-development of implementation processes between service administrators and sites, like the method taken in the comprehensive evaluation tool, may be an effective approach towards achieving influence and fostering a sense of ownership from impactful stakeholders.

Despite these promising findings, this study has several limitations that warrant discussion. First, the pilot study was conducted in a single remote community, limiting the generalizability of the findings. Future research should aim to validate the tool in diverse settings to ensure its applicability across different contexts. Second, while the tool assesses community capacity and readiness for virtual care implementation, it assumes the broader system is ready and does not address systemic issues such as funding models or policy frameworks, which may impact the sustainability of virtual care initiatives. Third, to validate the selection of telerobotic ultrasound for Fond du Lac as the most appropriate clinical solution to address the community's healthcare needs, an outcomes assessment will need to be conducted. Finally, ongoing evaluation and refinement of the tool will be necessary to keep up with the evolving landscape of virtual care technologies and best practices.

In conclusion, this comprehensive evaluation tool may contribute to the building of a battery of standardized evaluation and implementation guides for the deployment of virtual care technologies in remote and underserved communities. By systematically assessing community capacity and readiness, the tool provides a practical framework for identifying and addressing barriers to virtual care adoption. Moving forward, further research and collaboration will be essential to refine and validate the tool, ensuring its effectiveness in promoting equitable access to virtual care services across diverse communities.