Implementing YouthCHAT – evaluation and development of an implementation framework

Abstract

Purpose

A high proportion of indigenous Māori, a generally socioeconomically deprived population, live in Northland, New Zealand. Māori youth suffer a high rate of mental ill-health, substance misuse, and other risky behaviours. While evidence demonstrates that early detection and management of these issues leads to long-term positive health outcomes, implementation of systematic screening and intervention is challenging. YouthCHAT is a self-administered digital tool screening young people for mental health concerns and risky health behaviours. Our purpose was to implement YouthCHAT into Northland youth services using an iterative process of implementation, evaluation, and modification, and create a framework for national-level rollout.

Approach

A bicultural approach combined Kaupapa Māori research and co-design principles to inform implementation and evaluation. Informed by field notes, informal communications plus focus groups, and interviews with end-users, we identified and addressed barriers to uptake of YouthCHAT. Consolidated framework for implementation research and normalisation process theory was theoretical underpinnings of the project. We measured uptake by the number of YouthCHAT screens completed.

Discussion

It took three years to successfully embed YouthCHAT and implement it into usual practice. A change was needed at all levels, from the individual practitioner to the service team through to the over-arching health system. A bicultural co-design approach and programming remote functionality during coronavirus disease 2019 (COVID-19) lockdown contributed to YouthCHAT's ultimate acceptability and implementation. The cultural context in which an intervention is introduced is often overlooked. This partnership approach may be time-consuming and challenging. However, an intervention will only be adopted where there is local ownership and it is seen as fit for purpose. Our framework addresses the importance of communication and coordination in the acceptance and uptake of an innovation, with ongoing evaluation as an integral part of implementation. It is generalisable to other settings and augments other models such as the consolidated framework for implementation research currently available.

Plain language summary

The Northland region of New Zealand has a particularly socioeconomically deprived population with a high proportion of indigenous Māori. Māori youth suffer a high rate of mental ill-health, substance misuse, and other risky behaviours. Early detection of these issues means they can be addressed and youth are less likely to have future health and social problems. YouthCHAT is a self-administered digital tool screening young people for mental health concerns and risky health behaviours. We implemented YouthCHAT into Northland youth services using an iterative process of implementation, evaluation, and modification, and created a framework for national-level rollout. A bicultural approach combined Māori research and co-design principles, using information from many sources including focus groups, staff surveys, field notes, and informal communications to inform implementation and evaluation. It took three years to successfully embed YouthCHAT into usual practice. A change was needed at all levels, from the individual practitioner to the service team through to the over-arching health system. Programming remote functionality during COVID-19 lockdown contributed to YouthCHAT's ultimate acceptability and implementation. This bicultural partnership approach may be time-consuming and challenging. However, ultimately an intervention will only be enthusiastically adopted if there is local ownership and it is seen as fit for purpose. Our success is measured by the number of YouthCHAT screens now being completed. Our framework addresses the importance of communication and coordination in the acceptance and uptake of an innovation, with ongoing evaluation as an integral part of implementation. It is generalisable to other settings and augments other models currently available.

Background to YouthCHAT

YouthCHAT is a self-administered digital screening tool that assesses young people for mental health concerns and risky health behaviour, providing clinicians with an immediate overview of the young person's psychosocial status (Goodyear-Smith et al., 2017). The domains screened are tobacco, alcohol, and recreational drug misuse, problematic gambling and gaming, depression, anxiety, stress, eating and conduct disorders, sexual issues, exposure to abuse, difficulty controlling anger, and physical inactivity. It is clinically well-established, backed by a large body of literature, that early detection and management of mental health and psychosocial issues has many subsequent significant health and social benefits (Gibb et al., 2010; Kieling et al., 2011; Patel et al., 2007).

Recent research indicates that YouthCHAT is a valid, acceptable, and feasible tool that has the potential to offer benefit as both a time- and cost-effective alternative to the standard home, education/employment, eating, activities, drugs, sexuality, suicide, safety (HEEADSSS) youth assessment in New Zealand (NZ) (Thabrew et al., 2017). However, research to date has focused on small and for the most part, single-site implementations, such as a school or an individual youth clinic (Clare et al., 2021; Goodyear-Smith et al., 2016; Thabrew et al., 2017). Rolling out a health care innovation to multiple sites across a larger area, such as a specific geographic region, is more complex.

Northland (Te Tai Tokerau) is a relatively large region in the upper North Island of NZ, with a scattered and generally socioeconomically deprived population. The overall population is about 193,000, of whom roughly 38,600 are aged 10 to 25 years (Northland District Health Board, 2017). Northland has a high proportion of Māori, NZ's indigenous population, and its young people are particularly vulnerable to mental health and risky health behaviours (Government Inquiry into Mental Health & Addiction, 2018) due, in part, to barriers to primary care (Crengle et al., 2013). The issues for which YouthCHAT screens all have higher prevalence in Māori youth (rangatahi) than the overall population, hence YouthCHAT is particularly applicable to the Māori population. This was recognised by the youth health leaders in Northland, who approached us asking for YouthCHAT to be implemented in this region. Screening at-risk young people enables early intervention and the possibility of preventing the development of problems into adulthood (Taliaferro & Borowsky, 2012), and the implementation of a resource-efficient psychosocial screening tool would offer considerable benefit in this locality.

This paper describes the progressive rollout of YouthCHAT into Northland youth care services with an iterative process of implementation, evaluation, and modification using an implementation science approach. We describe the strategies used to implement systematic practice and organisational changes in a practical implementation report, and how this informed development of an implementation framework and guide for a wider-scale rollout of YouthCHAT.

Theoretical underpinnings to our approach

Implementation science is the translation of evidence-based complex interventions into routine clinical practice (Malterud et al., 2018). Such translation necessitates individual action and behaviour change which is not achieved in one single event, but is influenced by the dynamic interplay between the intervention, characteristics of the intended users, contextual factors, and wider organisational aspects (May et al., 2009, 2011). The tailoring of existing validated interventions to suit the needs of different populations is considered to be the most cost and time-efficient way to increase an intervention's acceptability and reach (Kumpfer et al., 2017). Thus, interventions with both a non-modifiable core (to maintain integrity and validity), and a softer ‘outer’ layer (to allow tailoring to fit into the implementing organisation) are the most successful (Barrera Jr. et al., 2013; Damschroder & Hagedorn, 2011). Such innovations have the ability to adapt to meet context-specific needs, while at the same time maintaining the integrity and validity of the tool (Murray et al., 2010).

Co-design and Kaupapa Māori

For an intervention to be successfully implemented, users must find it both acceptable and feasible, and for the implementation of innovation to become sustained and normalised, the process must be ongoing, not linear (May et al., 2009). Co-designed iterative processes allow reflection and action and ensure research is carried out with the end-users, rather than on them (Goodyear-Smith, 2016; Greenhalgh et al., 2016). Co-design can take place throughout a research process, from the generation of research ideas to co-creation of the research protocol, conducting of a study, through to dissemination of the findings through a variety of networks. The co-design approach facilitates a sense of ownership for the end-users. Participatory action research emphasises the participation and action in conducting the research by community members who will be affected by study outcomes. This approach seeks to enable and empower participants to modify an intervention so that it overcomes context-specific barriers and suits their needs, thus being appropriate and responsive for the community it is supposed to serve (Jagosh et al., 2012).

Interventions may require cultural adaptations when implementing them into real-life settings, which was the case when YouthCHAT was introduced into Northland (Marsiglia & Booth, 2015). Our work was undertaken in a predominantly Māori area of NZ and brought Māori and non-Māori researchers to work together sharing research principles, processes, and skills in bicultural research (Hudson & Russell, 2009). Kaupapa Māori research (by Māori, for Māori, of direct benefit to Māori) asserts the Māori worldview, cultural practices, and language as the status quo (Barnes, 2000). Engaging and partnering with stakeholders and users involved participation in Māori customs and rituals and validation of Māori culture, values, and beliefs by the research team, with the development of respectful relationships. Establishing these relationships took place prior to, and throughout this study, and was integral to its success (Martel et al., 2020). Using the Māori metaphor of he awa whiria, a braided river, a framework for bicultural research was developed by blending the principles of co-design research and those of Kaupapa Māori (Martel et al., 2021). Figure 1 shows the alignment of the Kaupapa Māori stream with the principles and processes of the co-design stream. Bicultural research can take place where these overlap.

Figure 1.

He awa whiria – braiding the rivers.

Theoretical constructs

Two theoretical concepts and approaches informed our implementation strategy and evaluation. The consolidated framework for implementation research combines a number of constructs derived from earlier work, such as the diffusion of innovations theory (Rogers, 1962) and the work of Greenhalgh et al. (2016) into a single framework. This serves as a practical guide to identify potential facilitators and barriers across five domains: the specific characteristics of the intervention; the outer setting (such as patient needs and resources); the inner setting (such as culture and leadership engagement); characteristics of individuals including their knowledge and beliefs; and the processes required such as planning, evaluating, and reflecting (Damschroder & Hagedorn, 2011).

Having identified potential implementation barriers using colidated framework for implementation research (CFIR), we used normalisation process theory (NPT) to further examine process issues potentially hindering implementation, and structure problems needing to be overcome to integrate the use of YouthCHAT into existing professional and organisational settings. NPT looks at the procedural translation of evidence into an intervention in the real world, until it becomes embedded and integrated into routine practice (May & Finch, 2009). Implementation requires collective action and collaborative work, with changes to behaviour, rules, resources, and organising logics and restructured relationships (May et al., 2018). Using NPT, we specifically looked at processes relating to the properties of the intervention (micro-level), the actions required to integrate it into practice (meso-level), and how they work together in context (macro-level) (May et al., 2018).

NPT has four constructs required for people to take up an intervention:

Coherence: How do end-users understand and make sense YouthCHAT?

Cognitive participation: How do end-users engage and commit to using YouthCHAT?

Collective action: How do clinical staff integrate YouthCHAT with their usual practice?

Reflexive monitoring: How can the benefits of using YouthCHAT and the challenges to its implementation be evaluated? What effect might YouthCHAT have on clinical outcomes? How can its implementation inform new ways of working? (May et al., 2009)

Implementation processes require iterative feedback loops to become sustained and are impacted by organisational structures, the complexity of clinical practice, and the group dynamics within any given setting (May et al., 2016, 2018). This necessitates a participatory research co-design approach. These NPT principles informed the pragmatic implementation of YouthCHAT into clinical settings in Northland, with iterative cycles of implementation, evaluation, and modification of not only YouthCHAT itself, but also the procedures around its use and the implementation process.

Approach

The YouthCHAT Northland project took place in youth, centres, school-based primary care clinics, and outreach services in Northland, NZ between October 2017 to October 2020. Following ethical and locality approval, consultation with key informants, local health providers, and other cultural and community agencies took place through a series of co-design hui (meetings) in 2018. This served to introduce YouthCHAT and kōrero (have conversations) about the study.

These hui enabled the Māori and Pākehā (non-Māori) researchers to whakawhanaunga (get to know each other) using the principles of ata (proceeding slowly and with thoughtful deliberation) and manaakitanga (reciprocal hospitality between different people), treating each other with mana (respect). The hui followed tikanga (traditional customs) including whakatau (welcome ceremony) in te reo (Māori language), karakia (prayer), waiata (song), and giving of koha (gifts). These meetings enabled the introduction of the values of tino rangatiratanga (Māori self-determination) and ako (the reciprocal non-hierarchical relationship between those learning and those teaching). They were also a rich source of feedback about the context-specific needs of the community with respect to YouthCHAT.

The hui took place over the first year of the project. Consenting clinical and administrative staff who had used YouthCHAT also participated in focus groups, face-to-face semi-structured interviews, and online surveys in 2019 and 2020 on acceptability and feasibility of YouthCHAT use, but the numbers were small. However, field notes and informal communications with a number of stakeholders contributed greatly to our understanding of what was working and what needed to be addressed and further informed understanding of the challenges to implementation in an ongoing process. While it proved difficult to recruit participants for focus groups and interviews, we were in regular communication with a variety of end-users, and identifying and addressing barriers to implementation was largely in response to these many conversations and emails.

YouthCHAT was implemented into one youth clinic in late 2018, operational feedback data were gathered and then used to modify and improve the YouthCHAT instrument and processes (Figure 2). After these changes were incorporated, and following further consultation, YouthCHAT 2.0 was implemented in October 2018. In collaboration with staff and students from the local Māori cultural immersion school, Māori co-investigators, and iwi (tribe) representatives from a local Māori health care trust, the screen was translated into the Māori language. Further operational data were gathered, and YouthCHAT 3.0 launched in March 2019. After further consultations, the updated implementation processes were used to roll out the final iteration, YouthCHAT PRO, into Northland youth health services in February 2020. Students at the local kura (Māori emersion school) provided the YouthCHAT translation into a local dialect. A separate YouthCHAT version was created for the kura called KuraKōrero. This was completely in te reo (Māori language) and featured a video introduction by students and a logo designed through a student competition.

Figure 2.

Number of YouthCHAT screens completed over time in relation to key events.

Throughout the project, informal feedback on many factors at the NPT micro, meso, and macro levels informed us about what was impeding or facilitating YouthCHAT use, leading to iterative changes.

Qualities of the intervention (NPT micro-level)

On the micro-level, ongoing feedback from staff and young people led to changes in YouthCHAT 1.0 to the 2.0, 3.0, and PRO versions. Changes in response to feedback included adding more youth-friendly language, an animated video introduction, a progress bar, and a text-to-voice option to hear rather than read the questions for youth with literacy issues. This addresses the CFIR characteristics of the intervention domain. The lack of integration with the e-health record was a major issue with staff, who were cutting and pasting text from YouthCHAT answers into their HEEADSSS form. Providing an information technology (IT) solution to this made using YouthCHAT much more efficient (CFIR domain of helping individuals in their work).

Actions of participants (NPT meso-level)

On the meso-level, staff feedback from interviews and online surveys revealed that they had not received clear communication from management as to how and why they should use YouthCHAT, and hence were unmotivated to do so. Lack of engagement with leadership maps to a CFIR inner setting domain. Facilitating the communication from management about YouthCHAT use, providing accessible training, and responding to their request for IT integration assisted engagement and uptake.

Co-design approaches are aimed at working with and for local partners, not conducting research on them, which will facilitate engagement and uptake. In the final evaluation, one staff member clearly identified that the goal of the project was to find out how best YouthCHAT might help her in her work to improve youth health when she said “The actual research process and data gathering aspect of it, I mean is this about us and youth? or is like getting data and more data from our youth? and so I mean it is actually going to benefit us.”

Working in context (macro-level)

Figure 2 shows that for the first two and a half years of the project, there was very little uptake of YouthCHAT. On the macro-level, a number of contextual factors that impacted on implementation, which were mostly external in origin. The bicultural co-design model required extensive engagement with Māori before releasing YouthCHAT, with hui (community-based meetings) observing Māori protocols and took up most of 2018. A meningococcal outbreak in Northland in 2019, followed by a measles outbreak, led to many nurses who would normally be seeing youth redeployed into rapid mass vaccination programmes. Because YouthCHAT screens were predominantly delivered within a school-based context, little YouthCHAT activity was expected during school holidays, especially during the summer break from December through February.

Some macro-level factors, such as appropriate cultural engagement (an inner setting construct), had modifiable components. Over time the local team was able to facilitate effective and clear two-way communication between managers and service providers around YouthCHAT use.

There were major health system restructures. Although the implementation of YouthCHAT had been at the invitation of the Northland Youth Governance Group, the intention for YouthCHAT to be routinely used had not been communicated to staff on the ground. Local services were fragmented with inequitable funding. In December 2018 the service underwent a restructure and the Northland Youth Health Service was launched. In June 2019 YouthCHAT was formally approved as part of the HEEADSSS assessment and this was communicated to youth services. In February 2020 guardianship of Northland YouthCHAT formally passed to the Youth Health Service.

Another major restructure was the two Northland Primary Health Organisations, with which the research team worked closely, underwent a merger to form a single primary health entity in July 2019. During this process, many staff left or changed jobs, and others had uncertainty around their new roles and responsibilities. This included the loss of the YouthCHAT champion from a key clinic and the redeployment of our local research team staff into new roles within the health system. By the end of the project, staff were settling into their new roles. A new manager supported training, resource provision such as e-tablets for nurses to deliver YouthCHAT in remote locations, and new champions were encouraged and emerged.

The advent of the coronavirus disease 2019 (COVID-19) pandemic was another unexpected macro-level factor that impacted the project. In March 2020, NZ went into lockdown. Processes needed to be changed. Youth providers expressed concern about rising mental health issues in isolated youth. In response, remote functionality was programmed, allowing youth to be emailed or texted invites to complete YouthCHAT from their phone or other devices at home. The provider could then access the summary report and arrange a consultation with the young person by phone or video.

The first lockdown ended in April 2020. By May 2020 most barriers had been overcome and YouthCHAT started to be routinely implemented (Figure 2). Since completion of the project, use continues to grow and screens now number in the thousands.

Full ownership of their version of YouthCHAT by the Youth Health Service, including the passing over of computer code and local hosting, took place at the end of 2020. YouthCHAT is now viewed as a toanga (precious gift and treasure).

Development of the implementation framework

Findings from this, as well as other studies, indicate that successful YouthCHAT implementation has two key aspects: tool acceptance, and tool uptake, but also relies on communication, coordination, and evaluation activities as outlined below.

The first step in the successful rollout of YouthCHAT was to create a tool that is acceptable to its stakeholders. We identified four key activities:

Establish cultural partnerships: We needed to establish cultural partnerships to ensure that YouthCHAT itself, as well as the way it is used, is appropriate for the Māori community it serves.

Endorsement from leaders: End-user staff needed a clear endorsement of YouthCHAT by leaders, managers, and governance before they accepted the tool.

End-user and community acceptance: Consultation with end-users and the community led to a number of changes to the modifiable aspects of YouthCHAT to meet their requirements. These included word changes, an animated video option, text-to-voice, translation into local Māori dialect, a progress bar, and integration into the existing electronic health record.

Encourage emerging champions: For YouthCHAT to be accepted, champions were needed. Once there were staff who understood their need and believed that YouthCHAT offered real benefit, they promoted, encouraged, and supported its use. Once we had a champion experienced enough to train other users and help with basic troubleshooting there was growing acceptance and uptake of the tool.

Successful uptake of YouthCHAT requires an iterative implementation and review process:

1. Assess: We needed to assess the current screening or assessment procedures of the youth workers and understand their workflow. Different settings had different screening criteria and targets, but all needed to conduct HEEADSSS assessments on many of their clients.

2. Appoint: Once we had an organisational administrator to champion YouthCHAT, the potential barriers to uptake (e.g. shortage in staff, infrastructure, technology, or resistance to change) were able to be identified, and then work could take place to resolve these.

3. Assist: We offered end-users practical support and a detailed user manual to help them determine the best way to integrate YouthCHAT into their daily workflow and establish a set screening protocol.

4. Adapt: We discovered that the process needs to be iterative. We needed to adapt the procedures as necessary to overcome barriers as well as update goals, targets, and criteria for use. Rapidly programming remote functionality in response to COVID-19 lockdown is an example of this.

A vital part of implementation is the communication and coordination of key messages and activities:

1. Communicate: While the youth health governance group had approached us to implement YouthCHAT use in all their youth services, this message had not been clearly articulated to the end-users, who resented being burdened with another tool to use without being given clear direction. Two-way communication was required so that both top-down expectations and goals were explained, and bottom-up information flow enabling end-users able to communicate their needs.

2. Clarify: We needed the roles and responsibilities of all involved to be clarified. Users needed to know whom to contact with both operational (such as ongoing training in YouthCHAT use) and IT issues (such as log-in difficulties).

3. Check: We needed to check the availably of sufficient resources. When the tool was being used by mobile nurses, e-tablets needed to be sourced, including data plans to access the internet. When we moved to remote functionality we needed to establish that YouthCHAT could be completed on any mobile device, including phones that youth might be able to access at home.

4. Coordinate: Ongoing coordination of support needed to be provided, including practical hands-on YouthCHAT training, and ensuring the necessary support materials are in place (such as the YouthCHAT manual and stepped care resource guide).

Ideally, evaluation of an implementation takes place in parallel to the actual implementation. Tasks and metrics might include:

1. Review goals, targets, criteria, and procedures to implement YouthCHAT (process evaluation).

2. Record barriers and facilitators to YouthCHAT uptake (process evaluation).

3. Record outputs such as the number of screens conducted or the number of users/clinics set up over a set time (outcome evaluation).

4. Review outcomes such as changes in mental health and/or risky health behaviours over time (outcome evaluation).

From these lessons, we have developed a four-by-four factor implementation framework (Figure 3). This is accompanied by a practical ten-step implementation guide (Supplemental material). While designed for YouthCHAT, these principles are generic. Our framework provides a pragmatic guide to gaining acceptance of a proposed new intervention by all relevant stakeholders; ensuring clear lines of communication and coordinating the relevant processes and people; and promoting uptake through iterative feedback processes. The model highlights the importance of a culturally sensitive approach, with the early establishment of bicultural partnerships, a co-design approach with clear communication and respectful relationships, and the use of Māori principles and protocols to promote uptake.

Figure 3.

YouthCHAT implementation framework.

Discussion

For YouthCHAT to become part of routine care, change was needed at all levels, from the individual practitioner to the service team through to the over-arching health system. A study similarly looking at embedding effective depression management in primary care found a team approach and organisational and system-level change, as well as practitioner education, were required (Gunn et al., 2010).

Murray et al. (2010) advise abandoning a trial if it becomes apparent that there is little likelihood that the intervention will become normalised into routine care. It took nearly three years to successfully embed YouthCHAT and implement it into usual practice. We persevered because we knew that YouthCHAT had the potential to reduce workload if properly integrated, rather than add an extra burden (May et al., 2018). We also were able to make changes in response to the views of actual users in the real world, which led to their engagement and eventual intervention uptake.

Future directions

Not all evidence-based interventions will become normalised into routine primary care. There needs to be an ongoing examination of how things are progressing, what is working, and where the barriers are. Failure to launch may be due to the interplay between the intervention, its users, contextual factors, and wider organisational aspects. The intervention may need to be tailored to a specific context to meet the needs of users, and organisational and system barriers addressed. Ultimately there will only be uptake where providers see this as worthwhile. The perception that its effective use will reduce their workload serves as a valuable incentive.

The strengths of this project rest in its co-design participatory research and bicultural Kaupapa Māori approaches, which eventually led to a successful implementation of YouthCHAT in Northland, where YouthCHAT is now viewed as a treasured gift. The cultural context in which an intervention is introduced is often overlooked. This partnership approach may be time-consuming and challenging. However, ultimately an intervention will only be enthusiastically adopted if there is local ownership and it is seen as fit for purpose.

The importance of early detection and management of mental health issues, substance misuse, and other risky behaviours to improve long-term positive health and social outcomes is well-established. The challenge is how to implement a systematic process of screening and intervention. While our project focuses on a particular population, the principles are generic. Our framework addresses the importance of communication and coordination in the acceptance and uptake of an innovation, with ongoing evaluation as an integral part of the implementation. It is therefore generalisable to other settings and augments other models such as CFIR currently available.