Evaluating the sector-wide implementation of virtual child and youth mental health services in response to the COVID-19 pandemic: Perspectives from service providers,agency leaders and clients

Abstract

Background

The COVID-19 pandemic prompted an abrupt shift in the delivery of community-based child and youth mental health services as virtual care was rapidly adopted. The objective of this study was to evaluate the sector-wide transition to virtual care across Ontario, with a focus on implementation facilitators and barriers.

Methods

We used a multi-level mixed-methods design where agency leaders, service providers, and clients shared their experiences planning, implementing, and accessing virtual care. In total, 97 agency leaders, and 192 youth and family members responded to the surveys; 13 agency leaders, and 11 service providers participated in interviews or focus groups.

Results

Most agencies undertook a similar journey to implement virtual care. Stakeholders described common facilitators such as staff engagement, leadership support, and training activities. Barriers included internet connection issues, lack of resources, and privacy concerns. Service providers innovated as they implemented by partnering with agencies to meet clients’ needs, using multiple platforms to engage clients, and altering session duration to reduce fatigue. Clients found virtual care easy to use, felt confident using it, and intend to continue accessing virtual care.

Conclusion

Implementation of virtual care during the pandemic was complex and the evaluation involved obtaining perspectives at multiple levels. This research provides a blueprint for evaluations of the implementation of virtual mental health services, particularly in a child/youth context. Virtual care is a viable way to deliver mental health services, however, equity, accessibility, and appropriateness need to be addressed to ensure services are effective for children, youth, and their families.

Plain language abstract

Academic literature suggests that using technology to deliver child and youth mental health services is a promising way to enhance access to care and improve engagement for many children and youth. Despite this, the provision of virtual child and youth mental health services in Ontario prior to the COVID-19 pandemic was limited. Efforts that did exist were largely focused on providing care to those in rural and remote areas. The COVID-19 pandemic prompted a rapid shift to virtual care, as most in-person mental health services were suspended. This paper presents new insight into how virtual mental health services were quickly established and used across Ontario from the perspectives of senior leaders, service providers, and clients. Results from this evaluation showed that agencies followed similar steps to prepare to use virtual services. Staff engagement, support from leadership, and opportunities for staff training supported the implementation of virtual care while internet connections issues, lack of resources (like computers or phones), and privacy and safety concerns hindered the implementation. Most youth and family members found virtual services easy to use and intend to continue using them. Most agencies intend to continue to offer virtual services post-pandemic but noted that it was not appropriate or accessible for all clients. This study provides a foundation for additional research to examine situations and conditions that are most conducive to virtual care delivery to address child and youth mental health concerns. These results may encourage agencies to rely more confidently on virtual services as another means to meet clients’ needs and preferences.

Introduction

Virtual care, the use of internet and related technology to deliver mental health services, has the potential to improve access (Comer & Myers, 2016), efficacy and empowerment, and reduce costs (Leblanc et al., 2019). In recent years, service providers in Ontario's child and youth mental health sector have been exploring how best to integrate virtual care options into service delivery. The emergence of the COVID-19 pandemic accelerated these efforts. To continue to meet the needs of children, youth, and families during the pandemic, most service-providing agencies rapidly shifted to deliver virtual care.

Prior to the pandemic, efforts were already underway to support the implementation of virtual care across Canada (Lal, 2019). The Mental Health Commission of Canada (MHCC) developed a virtual care/e-mental health toolkit which included frameworks for staff training and evaluation of various digital technologies (McGrath et al., 2018). In Ontario, psychiatric services, specialist consultations and training events are delivered to those in remote locations through the Ontario Telemedicine Network (OTN) (Serhal et al., 2017). Similarly, the Project Extension for Community Healthcare Outcomes (Project ECHO) focusing on child and youth mental health (CYMH) in Ontario promotes knowledge-sharing between primary care physicians and clinicians using a virtual platform (Serhal et al., 2018). Both services are funded by the government of Ontario. Online psychotherapy based on cognitive behavior therapy videos combined with coaching supports has been recently launched and funded by the government for youth ages 15 to 18 years (https://bouncebackontario.ca/).

Hence, in the publicly funded child and youth mental health sector in Ontario, the use of digital technologies has primarily been limited to participation in specialist consultations and training for providers. Challenges in the widespread adoption of virtual platforms for service delivery include low clinician uptake, technological challenges, low availability of the infrastructure across Ontario, privacy concerns, and limited government policies to guide this work (Auditor General of Ontario [AGO], 2020; Lal, 2019).

The Ontario Centre of Excellence for Child and Youth Mental Health (the Centre) and Children's Mental Health Ontario (CMHO) have been working closely together to support community-based agencies in providing child and youth mental health services across Ontario. The Centre is a government-funded intermediary organization working to bridge the research-practice gap (Ontario Centre of Excellence for Child and Youth Mental Health, 2020a). The Centre identified digital mental health as a strategic priority and is currently developing recommendations for implementing virtual walk-in mental health in the CYMH sector (Ontario Centre of Excellence for Child and Youth Mental Health, 2021). CMHO is a member-based organization focused on advocacy for issues related to system capacity, access, and improvement of child and youth mental health services in Ontario.

In the early days of the pandemic in Ontario, only essential services were open; schools were closed, and public health directives were to stay at home. This emergency lockdown from March to July 2020, prompted a sector-wide shift to virtual care. Our two organizations recognized the opportunity to study the implementation process, to support virtual mental health service delivery in the longer-term. We conducted an evaluation on the implementation of virtual care in Ontario's child and youth mental health sector during the COVID-19 pandemic, including the perspectives of clients, service providers, and agency leaders. We set out to understand how Ontario's publicly funded agencies shifted to deliver virtual synchronous mental health services, using telecommunication technology, including videoconference, phone, texting, and apps. At the time of this study, asynchronous mental health services (e.g. internet-delivered cognitive behavioural therapy) for children and youth were not commonly available nor funded.

Our evaluation framework was informed by implementation science and quality improvement approaches (Danseco, Brown, et al., 2020) and included the complexity and range of mental health conditions; the knowledge, attitudes, and readiness of staff delivering virtual care; the organizational contexts such as size, culture, leadership engagement; policy considerations, and the impact of social determinants of health.

The Consolidated Framework for Implementation Research [CFIR] (Damschroder et al., 2009) proposed five categories for implementation: the outer setting, the inner setting, innovation characteristics, and characteristics of individuals and processes. The Nonadoption, Abandonment, Scale-up, Spread and Sustainability [NASSS] framework (Greenhalgh et al., 2017) identified several elements to examine at the client, provider, organizational, and larger environmental levels. We engaged with partners in the province (i.e., senior management from agencies, researchers, youth, and families) to guide areas of focus based on these frameworks and to inform recommendations for longer-term system-wide implementation.

Our goals for this evaluation study were to: (1) learn about agencies’ experiences with the rapid transition to virtual care, including key facilitators and barriers, and how challenges were managed; (2) understand how service providers have experienced the transition to virtual care; (3) understand how clients have experienced the transition to virtual care; and (4) identify potential areas of knowledge and resource needs to support the ongoing implementation of virtual care.

Methods

Setting, context and procedures

Publicly funded child and youth mental health services in Ontario, Canada are delivered across 33 service areas in five geographical regions (central, eastern, northern, western, south, and Toronto). At the time of the study, there were 189 government-funded agencies delivering child and youth mental health services, including prevention programs, brief counseling, counseling and therapy sessions, family support, crisis support, day treatment programs, and live-in treatment services. Since 2014, a lead agency in each service area has been identified to deliver and coordinate these services across multiple agencies, known as core service providers (CSP), for children and youth up to 18 years of age (Government of Ontario, 2015).

Agency leaders in the child and youth mental health sector had professional relationships with the Centre and CMHO staff, as part of our roles in providing support to agencies in our sector. Agency leaders met regularly to discuss emerging issues resulting from the pandemic and agreed that an evaluation of their shift to virtual care would be valuable.

This evaluation study has been approved by the Children's Hospital of Eastern Ontario (CHEO) Research Ethics Board, (the Centre is housed at CHEO). This evaluation involved participation from agency leaders from an initial list of 189 agencies. Agencies providing only residential or live-in programs were excluded since staff were physically present and did not use virtual services. Some agencies were undergoing mergers, resulting in a final pool of 166 agencies.

We used a mixed methods approach to evaluate the shift to virtual care at the organizational, service provider and the youth/family levels using surveys, interviews and focus groups. The participants, measures, and procedures for each of these levels are described below. More detailed descriptions are available in the technical report (Danseco, Kurzawa, et al., 2020). The first page of the online survey included the information letter and consent form. Respondents who indicated consent continued to the survey questions.

The first and second authors conducted the English interviews and focus groups, and the second author conducted the French interview together with another bilingual Centre staff. The first author is experienced in qualitative research and is a credentialled evaluator in Canada. The second author is a research coordinator and previously led focus groups and interviews in other studies. All interviewers were female and were working at the Centre at the time of the study. All participants were briefed on the study and provided their written consent via email. All interviews and focus groups were conducted online and took place from mid-June to mid-July 2020, when we reached data saturation. Interviews and focus groups lasted approximately an hour.

Organizational perspective

Organizational participants

Agency leaders were invited to respond to an online survey and participate in interviews via email. Agency leaders were those with formal leadership roles who were responsible for the implementation of virtual care in their organizations. We received completed unduplicated responses (i.e., single response per agency) from 97 of 166 agencies (58% response rate). Response rates across the five regions were similar, with the Toronto region having the lowest response rate with 17 out of 33 agencies (52% agencies) and the central region having the highest response rate with 19 out of 28 agencies (68% agencies).

Twenty-one of the 97 agencies (22%) had an organizational budget under two million dollars representing small-sized organizations, 22 (23%) agencies had a budget between two and five million dollars, and 51 (53%) had budgets over five million dollars.

Organizational survey and interviews

Agency leaders responded to an online survey which was open for seven weeks from June 2020. The survey included five close-ended questions about the virtual care services offered to their clients (e.g., virtual platforms used, what core services were delivered virtually before and during the lockdown). The survey included four open-ended questions on the perceived facilitators, barriers, and costs for implementing virtual care during the pandemic.

Agency leaders indicated their interest in participating in interviews through the survey or via email. Criteria for selection included: (1) at least one from a large agency or in a large service area (e.g., lead agency with more than 10 core service providers) to identify unique challenges for a larger population or service area; (2) at least one from the northern region to identify unique challenges and experiences in a geographically dispersed service area; (3) at least one from a CSP (i.e., not a lead agency) to identify any unique challenges from a CSP perspective; (4) at least one agency providing services primarily to children below 12 years to obtain potential unique challenges when delivering services to younger children and their families; and (5) at least one Francophone serving agency to identify any unique challenges for Francophone staff and agencies.

Thirteen agency leaders were interviewed and represented agencies from the central, northern, and western regions of Ontario. All leaders who indicated interest in participating were interviewed. Interview questions focused on the factors that facilitated virtual care delivery, challenges, and implementation considerations, including any policies, procedures, or processes that were put in place to enable virtual care. They were also asked about their plans and recommendations for implementing virtual care in the longer-term.

Service provider perspective

Service provider participants

Thirteen clinicians took part in focus groups or interviews in three regions and one provincial session for those who spoke French. Criteria for participation include being employed in a core service provider or lead agency, currently providing virtual care to children and/or youth, and available during the scheduled time for their region. In cases where there were less than three participants who registered, the focus groups were conducted as interviews. Focus groups took place in the western region (with eight service providers) and central region (with three service providers). We interviewed two service providers in the eastern region, with one conducted in French. While we were not able to have clinicians in the Toronto or northern regions due to scheduling conflicts, data saturation was reached.

Service provider focus groups and interviews

We asked service providers about their steps for preparing for delivering virtual care, perceived facilitators, and challenges. We also asked about their experiences with virtual care with younger children, families, and groups; ethical dilemmas and safety considerations; challenges in establishing therapeutic alliance; challenges in delivering evidence-based protocols from programs; and their strategies to address these.

Youth and family perspective

Agency leaders forwarded an email survey invitation to youth and families who had received virtual mental health services over the previous three months. Clients were defined as youth ages 12 to 25 years, or parents or caregivers of children under age 12. The survey was also promoted in our social media channels and circulated through our youth and family advisory councils. The survey was open from June 18, 2020 to July 31, 2020. Participants were offered the option of entering their name in a weekly draw for a $25 gift card.

One hundred and ninety-two youth and parents/caregivers completed the survey. Thirty-four responses were excluded from the analysis since they indicated that they did not receive services in the past three months, resulting in a final total of 158 responses. Respondents were primarily female (73% for those who indicated their gender) and of European heritage (42%). There were 85 (54%) respondents who were parents or caregivers, 63 youth (40%), and 10 (6%) who indicated they were siblings or grandparents of a child or youth who sought services.

The survey consisted of five close-ended questions about the ease of use, usefulness of the virtual mental health services and intention to continue to use virtual care on a five-point scale (from 1  =  strongly disagree to 5  =  strongly agree) (Baumel, 2018; Castañeda et al., 2007). We also included two open-ended questions asking participants what they liked about virtual care and their suggestions for improvements.

Data analysis

Responses to the close-ended questions in the organizational and client surveys were analyzed using IBM SPSS software (Version 27). Interviews and focus groups were recorded, transcribed, then analyzed for themes using NVivo (Version 12). No additional field notes were collected. Transcripts were anonymized and were not returned to participants. The CFIR codebook was used to code the qualitative data (https://cfirguide.org/tools/tools-and-templates/). Coding for each of the data sources (key informant interviews, focus groups, open-ended responses from surveys) were led by three team members. Team meetings were held to ensure consistency in coding, clarify constructs, identify any new codes or revise existing codes. We conducted a thematic analysis of the common CFIR constructs through team discussions and iterative review of the qualitative coding. Major themes were then named using the language used by participants (Patton, 2015).

Initial findings of themes and preliminary recommendations were presented to key stakeholders, then iteratively revised to incorporate their perspectives, as part of a utilization-focused approach in evaluation (Patton, 2012). The stakeholders included six members of our advisory group, eight members of the advisory group for our virtual walk-in guidelines (Ontario Centre of Excellence for Child and Youth Mental Health, 2021), and executive members of the lead agencies who had been engaged from the beginning of the evaluation. Major themes on their implementation journey, facilitators and challenges based on the surveys, interviews and focus groups are presented in the next section, while minor themes and more detailed findings are available in the technical report.

Results

Planning to deliver virtual care

Prior to the pandemic, 45 of 97 organizations (46%) surveyed provided virtual mental health services. By July 2020, this number rose to 94 (97%). Some agencies (3/97 or 3%) continued in-person services. Survey respondents indicated that the shift to virtual care was necessary to continue to meet the mental health needs of children, youth and families while adhering to public health guidelines/directives. This abrupt shift to virtual care meant that agencies had limited time to prepare. Instead, organizations implemented and learned along the way, or as one agency lead put it, they “built the airplane while flying” through a process of ongoing adaptation. Initially, most organizations focused on establishing core services such as counselling and therapy services (94%) and brief counselling services (80%). Some organizations offered more comprehensive virtual services such as specialized consultations and assessments (60%) and day treatment programs (40%).

Agency leaders described a similar implementation journey. These activities were iterative in nature: (a) assess the needs and preferences of clients, clinicians and the organization, (b) select virtual platform(s) that met privacy and security criteria, (c) revise or develop policies, (d) revise workflows, (e) provide training, and (f) reflect, evaluate and improve.

Service providers also followed a similar implementation journey: (a) learn about the organizational policies, procedures and guidelines from their professional associations, (b) obtain the right equipment and tools, (c) receive training on the tools or platforms that they would use, (d) coach clients on how to use the new tools or technologies and ensure there is a safe, private and confidential space. Together, service providers and clients established new routines and protocols that fit with the virtual setting and addressed concerns such as privacy and safety. For example, service providers confirmed the client's location as a safety measure and engaged in a conversation about how to address potential issues such as what to do in the event of a technical issue and how to get back in touch with one another.

Facilitators of virtual care

Agency leaders told us that factors that facilitated successful virtual care included: staff engagement and motivation, leadership support, availability of training materials, platforms and software, devices and accessories (i.e., laptops, phones, headsets, internet sticks) and, in general, maintaining a commitment to a collaborative approach when responding to challenges. Regular check-ins and opportunities for problem-solving helped to ensure successful implementation. Service providers expanded on the theme of collaboration and emphasized the value of having an established relationship with clients and a focus on providing client-centered care. They also spoke to the importance of having a designated champion or community of colleagues with whom they could learn together.

Barriers to virtual care

One of the primary challenges to the successful implementation of virtual mental health services was the absence of a reliable and accessible internet connection, particularly in rural and remote areas, and among clients with low socioeconomic status. This was exacerbated by the closure of public spaces with free access to the internet. A related challenge was a lack of resources such as laptops, tablets, or phones needed to engage in virtual sessions.

Privacy and safety concerns were frequently mentioned by service providers and agency leaders as barriers to the successful implementation of virtual care. Holding virtual sessions at home opened the possibility that family members or roommates may overhear clients’ sessions with a therapist, which can threaten confidentiality. Service providers were also concerned about their ability to effectively assess safety risks. For example, if a client suddenly dropped off a call, while this could be the result of connection issues, it might also indicate a safety risk. Agency leaders and service providers had to navigate the new work-from-home reality where privacy may be limited, and the lines between work and personal lives are blurred. They had to adjust to new workflows, technology and protocols while still working to provide evidence-based services.

For some clients, virtual care was not as engaging as in-person sessions. This was particularly the case with young children and those with attention difficulties (explored in our discussion of equity considerations). Service providers spoke to the inherent difficulties in establishing therapeutic rapport with new clients, which was made more difficult with internet connection issues and lags in video/audio feed. They also expressed that non-verbal communication was harder to read over the virtual modality and especially so without video. Service providers expressed fatigue from engaging in intense, online sessions and a feeling of isolation from their colleagues.

Mitigation strategies

Agencies expressed dedication and innovation in meeting clients’ needs. They engaged children, youth and families with access issues in new ways such as porch drop-offs (i.e., physically dropping off materials and worksheets to clients’ homes) and offering phone sessions. They partnered with community-based agencies to provide resources like tablets and internet sticks to clients when needed. They also upgraded software and hardware for staff and offered flexible hours of service to accommodate clients’ and providers’ other responsibilities.

Text-based support through the chat function in video calls or through text messages was offered when privacy was an issue. The chat function was frequently used in group sessions, with some clients reporting that it increased their comfort to express themselves. Establishing clear safety protocols (such as the practice of stating their location at the beginning of each session and having an alternative method to reach a client in the event of a technical issue) was an important step to mitigate safety risks.

To increase client engagement, service providers would sometimes break up sessions into smaller segments to keep the time manageable. As well, some provided focused workshops for parents to help them understand how to play a supportive role for their children, particularly when children were very young, or when their needs were not being adequately met virtually.

To counter fatigue from virtual sessions, staff were encouraged to engage in self-care activities and take frequent breaks. Supervisors checked-in frequently with service providers to provide support and/or problem solve issues (typically once a day at the onset and then moving to weekly). Gradually, staff and clients became more familiar and comfortable with new routines and technology, easing some of the pressure. For instance, an agency lead indicated, “Initially, it was more of a challenge for both staff and [clients], but once we were able to get some comfort with the technology aspects, it ended up being very [client]-focused and easy to access for [clients]”.

Equity considerations

Service providers felt that virtual care had enhanced access to care for many children, youth and families. From clients’ perspectives, many indicated that virtual care had reduced barriers to care such as transportation and childcare. Some clients voiced that they felt more comfortable accessing care virtually than in-person. Despite the potential benefits, service providers felt that virtual care was not appropriate for all clients. Agency leaders and service providers cited challenges with the use of virtual care with young children, individuals with attention difficulties, high acuity clients, clients in rural and remote areas, and clients with low socioeconomic status. This was due to a variety of factors including engagement level, privacy and safety risks, access to internet (i.e., quality, availability, price), and availability of resources. It was clear that virtual care could not effectively serve all populations and that alternative strategies would need to be used. For example, an agency leader stated:

“But, when we talk about who we aren’t …. effectively providing service for, I would say it's the high-risk kids who of course have lower socioeconomic status, less stability with the parents, less support from the parents to engage in this kind of counselling.”

At the time of the evaluation, agency leaders were just beginning to examine equity issues and discuss solutions. Most organizations prioritized high-risk clients and clients with access issues for in-person services once lockdown was lifted.

Clients’ experiences

Clients demonstrated openness to participating in virtual care. Of 192 youth and family members who completed the survey, 130 or 82%, opted to use virtual services. These findings were consistent with qualitative data, as service providers and agency leaders reflected on the higher than anticipated uptake of virtual care. The most common ways in which clients accessed virtual services were via videoconference (76%), telephone (68%), and texting (31%).

Clients’ experiences with virtual care were largely favorable. They felt that the technology was easy to use (83%), felt confident in using this technology (84%), and intended to continue to use virtual services (68%). Clients felt confident about what to do after the sessions (70%) and believed they had a better plan for how to handle their concerns (58%). They cited many benefits to virtual care such as time and cost savings. Some clients expressed a preference for virtual care over in-person care, citing enhanced flexibility, and enhanced comfort expressing themselves. Clients and family members also spoke to challenges with technical issues, internet connection, and privacy. Some clients preferred in-person sessions but were happy to have an alternative method to access care while this was not possible. Clients also brought up the importance of improving communication and consistency between providers for those with multiple service providers. For example, one respondent suggested it would be helpful “knowing what other access points for service might also be available or having a group conference with multiple service providers at once to maximize the session”. Another respondent commented that “there are a variety of different platforms used by different providers. Sometimes the differences between the platforms can be a little confusing/time-wasting.”

Organizational plans for evaluating and integrating virtual care

At the time of this study, agencies were in the early stages of conceptualizing and implementing evaluation strategies to assess ease of use, clinical implications, and equity issues for virtual care. Among agencies that had developed an evaluation strategy, they commonly used client and staff surveys to gather feedback. Some agencies had a central repository to log issues and resolutions, reviewed recorded virtual sessions for quality assurance purposes, and analyzed no-show data.

Most agencies planned to continue offering virtual care as an ongoing service. Work was ongoing to determine what this model of care would look like and how best to match modality to client need and preference.

Discussion

Our evaluation study found that many agencies had similar experiences as they implemented virtual care in response to the public health restrictions from the pandemic. Common facilitators to implementation included staff engagement, leadership support, and training activities. Barriers to implementation included internet connection issues, lack of resources, and privacy concerns. A key factor to success and an important component of ongoing evaluation is understanding the experience for both service providers and clients to adapt and meet the needs of service providers and clients. With the appropriate support, infrastructure, and resources, many service providers and clients found virtual services to be easy to use and would like to continue using it or having it as an option.

Our assessment of the facilitators and barriers to successful implementation of virtual mental health care reflect those that have been studied in the literature. The Consolidated Framework for Implementation Research [CFIR] (Damschroder et al., 2009) proposed five categories for implementation: the outer setting, the inner setting, innovation characteristics, and characteristics of individuals and processes. In our study, the implementation climate, especially the tension for change, was a strong impetus for the shift to virtual care. Innovation characteristics did not figure prominently in the themes except for costs for equipment or subscriptions to the virtual platforms. Implementation activities focused on new policies, procedures, infrastructure, and resources to deliver virtual care. Ensuring privacy and security of virtual care involved the organizational policies, training of providers, and creating safety protocols during the clinical encounter. Addressing the service preferences and mental health needs of children, youth and families were core values in this rapid shift to virtual care. Our study of the organizational, provider and client level perspectives highlighted implementation factors that need to be considered for virtual care to remain viable and effective.

Using technology to provide mental health services to children and youth has been a promising practice for many years now (Boydell et al., 2013; Comer & Myers, 2016). Virtual modalities can help to address issues related to access and efficiency, and indeed some children/youth prefer to receive services in this way (Lal, 2019; Leblanc et al., 2019; McGrath et al., 2018; Ontario Centre of Excellence for Child and Youth Mental Health, 2020b). While a number of CYMH agencies have been delivering some form of virtual mental health care for many years (particularly in geographically expansive areas of the province), prior to the arrival of COVID-19, the real and perceived implementation barriers have prevented the broad uptake of virtual care across Ontario (AGO, 2020).

Throughout the pandemic, there has been a significant increase in the demand for mental health support as children, youth, and families have had to deal with uncertainties and stress related to the pandemic and drastic changes in their life and regular routines (e.g., Radomski et al., 2020; Singh et al., 2020; Wagner, 2020). With public health measures that have encouraged people to stay at home as much as possible, children and youth have had limited access to in-person informal and formal mental health supports. Consequently, community-based agencies have struggled to keep pace with the growing number of children and youth presenting with new mental health challenges, as well as those young people on existing caseloads for whom the sweeping lockdown measures exacerbated pre-existing issues.

Through the spring of 2020, in-person counseling and therapy had been either paused or significantly reduced. Agencies had to quickly pivot and implement policies and processes to provide services virtually while agency staff had to quickly learn how to deliver care virtually. For many agencies this was an iterative process. Similarly, children and families also had to adapt to a new approach to receiving services, particularly for those who had not used or accessed services virtually prior to the pandemic. While organizations have returned to in-person service delivery, many are offering a mix of both virtual and face-to-face care given the positive feedback from staff and clients and ongoing lockdowns. Virtual care is clearly here to stay; however, to ensure that this service modality produces positive outcomes for children, youth and families, there are several steps that need to take place.

First, a critical driver for the success of any implementation effort is adequate resourcing (Asmundson et al., 2020; Damschroder et al., 2009; Greenhalgh et al., 2017). Funds are needed for agencies to be set-up to provide virtual care. From the technology side, this includes the purchase of devices for staff, and in some cases for clients, and set-up and implementation of required infrastructure. From the operations side, this includes the development of organizational policies and procedures to support the delivery of virtual care, revision of workflows, and the selection of virtual care platforms. There are also costs to sustain the delivery of virtual care, including ongoing enhancements to keep pace with evolving technology, and staff and client preferences. On a related note, continued investments in training and professional development (another key implementation driver) would be necessary for service providers to feel equipped to deliver services in this way.

Second, while growing, the evidence base to guide implementation of virtual care services is limited. At the present time, opportunities to generate new knowledge in this area are plentiful – and we should take advantage of this by investing in research, knowledge mobilization, and capacity building. This includes opportunities for agencies to partner with academics and other experts in the sector and across the health system to implement and evaluate promising models. In addition to providing guidance for implementation, this new knowledge will contribute to the development of standards/guidelines for service delivery, which in turn will help agencies undertake quality improvement and evaluation activities on virtual care within their organizations. Agency-specific information about the experiences and outcomes of clients who receive care virtually is essential for informed decision-making at both the system- and provider-level regarding delivery and quality of care.

Further research is needed on the efficacy and effectiveness of virtual care for different clinical profiles, types of therapy, and population groups, particularly as it complements or supplements existing services. This research will inform the development of standards and guidelines for virtual mental health care, but also help to develop the training that service providers will need to deliver virtual care effectively. For example, we have integrated facilitators, barriers, and steps to deliver virtual care into a provincial guideline on implementing virtual walk-in services (Ontario Centre of Excellence for Child and Youth Mental Health, 2021), with considerations for leadership, organizational, and competency drivers (Fixsen et al., 2005; Bertram et al., 2015).

Finally, the COVID-19 pandemic has revealed the disparities that exist across our communities (Asmundson et al., 2020; Crawford & Serhal, 2020). Similar equity considerations have been raised in accessing virtual children's mental health services in the United States (Martinelli et al., 2020). To ensure equitable, accessible care for all, we must keep a laser focus on the unique needs and barriers experienced by priority populations across our communities, and endeavor to address these at both system and local levels (Asmundson et al., 2020). Data on who is not accessing services, as well as race-based data need to be collected, followed by engagement and ongoing dialogues to identify and address their needs.

Limitations

In response to the shift to virtual care, this evaluation study was rapidly designed and deployed. As a result, there was limited opportunity to engage a broad group of stakeholders to co-develop tools and methods to ensure appropriateness, clarity, and comprehension. Despite this, an advisory committee (including youth and family voice, agency leaders, and service providers from across the province) was formed to provide feedback throughout the study period.

At the time of this study, there were several ongoing evaluation efforts aimed at understanding the impact of COVID-19 on youth and families’ mental health. This may have caused confusion and/or survey fatigue for clients who received invitations to participate in more than one survey. This is reflected in the lower than anticipated responses. Family and youth survey responses could not be directly linked to services provided. Additionally, the results may be biased by our sample as they do not necessarily represent all clients and service providers who received or delivered virtual services (e.g., it is possible that only respondents who were satisfied with virtual services participated in the study).

This evaluation study did not investigate Indigenous children and youths’ experience of virtual mental health services, which requires co-design and co-development of the evaluation. A study on the experiences of racialized youth on in-person and virtual mental health services is currently underway, led by Ontario's The New Mentality, a youth advocacy group.

Findings from this study may be generalized to other geographic areas that deliver virtual care services in a similar context (e.g., availability and access to internet and technological devices, privacy legislations). However, even within Canada, the delivery of mental health services and the infrastructure available for virtual care vary greatly (e.g., between rural and urban areas). To this end, survey tools may need to be adapted for use outside of Ontario.