Recommendations for the development of digital conversational agents for youth with mental health conditions: A position paper

Abstract

Digital conversational agents (“chatbots”) hold interesting potential as a technology that can be used to deliver mental health interventions and supports to people in need. Youth are high users of digital technology and are widely affected by mental health conditions; youth might therefore be particularly interested in using digital conversational agents to support their mental health. We reviewed the literature on digital conversational agents for mental health, then conducted a two-pronged qualitative study of these tools for youth with mental health conditions. While the evidence base is limited, there is some preliminary evidence of utility and acceptability to users of this technology. In our qualitative study, youth were cautiously optimistic about using a digital conversational agent for mental health and identified potential benefits, but also substantial concerns. In terms of content and format, they wanted reliable, accurate, validated information, a flexible format, and a friendly interaction style, with attention to confidentiality and security. From these findings, we propose eight distinct recommendations to guide the rigorous development of digital conversational agents for youth mental health. By following these recommendations, it may be possible to build tools that leverage the strengths and potential of modern conversational agents while mitigating the potential harms to youth.

Generative digital conversational agents (“chatbots”) available today can hold conversations with users by employing artificial intelligence and natural language processing to support human-like conversational experiences. They can respond intelligently to user inputs in order to provide support and information. Conversational agents have been used to support youth in the mental health sector, focusing on emotional health, sometimes providing direct mental health-related supports such as psychoeducation, information about coping skills, cognitive-behavioral therapy exercises, and support with making a diagnosis.^1–4 Indeed, the first ever chatbot to be developed was in the field of mental health. ⁵

The evidence on conversational agents for mental healthcare today is limited.^3,4 However, the existing research has suggested that this technology can simulate empathy ⁶ and is capable of building something like a therapeutic alliance, ⁷ showing promise for mental health care. Other potential benefits include flexibility, accessibility, cost effectiveness, and providing a stigma-free service.^8–10 However, a range of ethical concerns have been raised with regard to conversational agents for mental health, such as ensuring confidentiality and transparency, as well as safety in crisis situations. ¹¹ Also of ethical importance are testing efficacy, encouraging human services, not purporting to treat severe mental illness, and protecting against technology addiction or dependent use. ¹¹

Digital technologies are omnipresent among youth, including those with mental health concerns. In Canada, a national report indicates that 67.9% of Canadian youth have gone online to find health information. ¹² A 2024 report cited that more than half of youth had used generative artificial intelligence, ¹³ a number that continues to grow as publicly available conversational agents and other forms of generative artificial intelligence expand in implementation and scale. Anecdotally, we hear from youth in our networks that many youth are already using publicly available conversational agents to support their mental and emotional health. Considering that youth are intensive users of digital technologies, conversational agents need to be developed with youth in mind. Indeed, conversational agents might be a form of e-health innovation that is particularly relevant to young people. ¹⁴

Since about 50% of youth have had one or more lifetime mental health or substance use health condition, ¹⁵ providing accessible, stigma-free care is a priority. Mental health concerns among youth can have long-term impacts on a broad range of spheres of life, including education, vocational development, and quality of life, as they interfere with developmental milestones.^16–20 However, rates of unmet need for mental healthcare services are high. ²¹ It is possible that conversational agents may fill in a void in the support needed for youth mental health, if the evidence points in this direction. We therefore conducted a set of studies to explore this possibility.

Our research to date

First, we conducted a scoping review of the literature to understand the current state of the research on conversational agents. ²² Ten published articles were reviewed, focusing on conversational agents for treatment-seeking youth with mental health conditions in clinical contexts. Most studies reported high acceptability and positive perceptions of digital conversational agents among youth populations. However, research examining conversational agents for youth mental health in clinical contexts is in the preliminary stages, with minimal established outcomes in terms of efficacy or effectiveness. Given the fast pace of technological development and the substantial time required to generate rigorous evidence, research on the most recent generative conversational agent technology used today is limited.

We then discussed conversational agents with a diverse sample of 28 youth with multiple mental health conditions.^{23, 24} We qualitatively asked them about their perceptions of such a tool, their willingness to use it, desirable content and use cases, advantages and disadvantages, and any concerns they may have. Participants described feelings of cautious optimism around the use of a conversational agent. While they recognized potential benefits, they expressed a noted preference for in-person services, as well as substantial concerns around ethics, confidentiality, and privacy. The majority of participants were in favor of the development of a conversational agent for youth mental health, but not all. From a content standpoint, they expressed that a conversational agent should offer reliable, accurate information that has been user tested and professionally validated. They wanted it to be a highly flexible and customizable tool with a friendly, human-like interaction style. They clearly expressed that careful attention should be paid to confidentiality and security features.

The research conducted to date provides sufficient evidence to cautiously pursue the development of conversational agents for youth with mental health conditions. However, this readiness comes with a number of caveats. We therefore make a number of recommendations to guide the development of conversational agents for youth with mental health conditions. The recommendations are directly based on the literature and our qualitative findings. From a pragmatism paradigm and lived experience engagement framework,^25,26 we further consulted with young people on the recommendations and refined them accordingly.

Recommendations for the development of digital conversational agents for youth with mental health conditions

1. Continually confirm acceptability . The acceptability of conversational agents to youth with mental health conditions has not been definitively confirmed, as youth in our study expressed only mitigated willingness to “try it out”. ²³ Based on this finding, the acceptability of any emerging conversational agent should be continually explored directly with young people at each step of its design and development. Design preferences, application features, and use should be interactively validated with youth as initial designs and prototypes are developed. It is important to return to the acceptability stage of research continually as the work progresses, consulting with youth to inquire about acceptability, usability, and utility of the new format.

2. Adhere to privacy ethical standards. The literature, our qualitative findings, and our youth consultation converge to highlight the importance of adhering the highest standards guiding the ethical development of such e-health tools. ¹¹ This includes maintaining confidentiality, privacy, transparency, and the ethical use of any data collected from users. It is important to ensure that appropriate confidentiality and privacy safeguards are put into place and that these are acceptable to the youth who use the conversational agent. Cybersecurity must also be ensured to the strongest degree. Ethical safeguards also include ensuring strong pro-health usage protocols, as well as protections against dependence, over-use, and the erosion of in-person communication skills. Established ethical standards should be reviewed at each stage of development, including through ethical review boards trained in the ethics of digital technology and youth advisory panels, to ensure ongoing adherence.

3. Use an integrated knowledge translation and user-centered design. In alignment with patient-oriented research principles and our experience conducting highly productive and essential collaborative activities, any prototyping, pilot testing, efficacy, and effectiveness research must be co-designed in an integrated knowledge translation framework.^27,28 This involves engaging with a range of relevant members in the design of a tool and a project, notably including engaging with young people. Any iterative design and evolution of the tool should be co-designed together with youth, clinicians, industry partners, and researchers. Youth have many preferences regarding design and functionality and must be involved in the design/prototyping stage, as well as the ongoing development and evaluation of the tool. Collaborations with industry partners with expertise in conversational agents can help the team understand technical aspects of the tool and thereby design it in applicable, technologically sound manners. In doing so, it is important to carefully manage any real or perceived conflicts of interest to ensure the legitimacy of the research and the resulting conversational agent. ²⁹ While recognizing the value that each partner can bring to the table, it is also important to disclose any conflicts of interest, manage equitable decision-making processes, report transparently, and monitor impacts. ³⁰

4. Ensure safety protocols are in place. Ethics discussions in the literature ¹¹ and youth perspectives in our work highlight that the careful programming of safety protocols is essential. Safety protocols should be co-designed with youth and clinicians to ensure acceptability and to address high-risk situations and utterances by conversational agent users, including suicidality and self-harm. It may be necessary to include a function to escalate a conversation to a human contact when self-harm, suicidality, or other high-risk activities are detected. Escalation might be to a widely available crisis line, for example. This intersects with the requirement to adhere to ethical standards, as escalation would require identified users and possibly user location awareness, which increases confidentiality concerns. Careful co-design of safety protocols is therefore essential.

5. The conversational agent should augment human supports. As suggested in the literature and as expressed by youth, the conversational agent should be designed to augment human supports, not to replace them. ³¹ It is important that the conversational agent be designed specifically for use in an integrated manner with human-delivered supports, including clinical and community-based support services. The content should align with the therapeutic modality, and the human-delivered therapy should take precedence. Augmentation might take the form of a conversational agent designed to accompany psychotherapy, for use outside of sessions. For example, they could be designed to help youth review new skills or track their symptoms between sessions. Other formats of augmentation might also be possible. Any clinician oversight of application use would require information sharing, which should be co-developed with attention to privacy and ethical considerations.

6. Provide informative, up-to-date content, as well as conversational support. Youth in our qualitative study strongly expressed that the content provided by the conversational agent should be accurate, practical, informative, and well justified. ²⁴ They also wanted the conversational agent should to be able to engage in friendly, human-like, supportive conversations. ²⁴ Concrete content might include psychoeducation about mental health, information about sources of support available to youth, and multiple coping skills for any given challenge. The content should be reviewed regularly for continued accuracy and kept up to date as needed.

7. Develop as an engaging application. It is strongly recommended that any conversational agent be developed through a collaboration between application developers specialized in producing flexible, customizable, engaging digital applications and a conversational intelligence developer specialized in creating strong content and a friendly, human-like conversational style. Youth in our study clearly expressed their wish for conversational agents to offer multiple customizable features and enable multi-modal interaction. ²⁴ While this is a strong youth preference, we understand that it may extend beyond the expertise of conversational agent developers. Collaboration with mobile application developers would therefore be required to fill this gap.

8. Conduct a progressive series of projects, with ongoing evaluation. Based on the sum of the recommendations, we recommend that any conversational agent project be conducted in a stepped manner with reflection at each stage, using an appropriate, developmentally informed intervention research framework. ³² The work should begin with the co-design of the content areas, a storyboard, and a format with the input of all relevant clinical and community members, including youth, following the principles for patient engagement in digital health interventions. ²⁸ This stage should be followed by user testing by youth and clinicians. Subsequent stages include pilot testing for refinement and rigorous efficacy testing in a clinical trial to determine outcomes, followed by implementation, scale, and ongoing evaluation and optimization of acceptability, usability, and performance. All considerations about the possible ethical and safety concerns need to be kept in mind during all stages of testing and evaluation, ensuring that the youth have appropriate supports in place. Each stage should progress only if the previous stages warrant it.

Conclusions

Conversational agents appear to have the potential to contribute to mental health support for vulnerable youth seeking care. However, the potential is mitigated by substantial ethical and usability concerns. Given that some youth are already using publicly available conversational agents for mental health support, this justifies conducting thoughtful, rigorous e-health research to develop mental health conversational agents that are ethically sound, effective, and designed specifically for youth mental health applications. For some youth with limited access to traditional care, such e-health tools could serve as low-threshold, easily accessible supportive tools.

By following these recommendations, which were derived from the literature, research, and youth consultation, it may be possible to build tools that leverage the strengths and potential of modern conversational agents while mitigating the potential harms to youth. However, the concurrent research in this area must continue, specifically involving treatment-seeking youth with mental health concerns and in a manner that keeps pace with the rapidly advancing digital technology and conversational agents. Collaborative partnerships and supportive policy environments are also essential to ensure responsible development and implementation.