Sage Journals: Discover world-class research

Abstract

The integration of artificially intelligent conversational agents (AICAs), variously referred to as chatbots and virtual humans (VHs), is transforming health care delivery and education. This article explores our perspective on best practices for the evolution, potential, and ethical considerations of AICAs in clinical and educational contexts. Early applications of simulation technology in health care focused on productivity improvements, teletherapy, and virtual reality therapy applications. Recent technological advancements have enabled the development of high-fidelity extended reality systems and AICAs capable of engaging users in credible interactions. These systems leverage natural language processing, machine learning, large language models, and advanced VH authoring software to create interactive, personalized, and engaging experiences. Recent efforts in the creation of AICAs suggest significant potential benefits, including enhanced patient engagement, improved access to self-care resources, and low-stigma interaction environments. They have demonstrated promise in mental health support, providing a sense of safety and encouraging open disclosure. However, the rapid adoption of AICAs raises critical challenges, including safeguarding user privacy, ensuring system reliability, and addressing ethical concerns. Incidents of harm, such as inappropriate interactions and psychological distress, highlight the need for rigorous design and implementation best practices. This article outlines key principles for developing safe, effective, and equitable AICAs, emphasizing transparency in artificial intelligence (AI) identity, accountability, cultural sensitivity, and informed consent. Additionally, the authors advocate for robust privacy measures, adaptive learning capabilities, and evidence-based content validation to optimize user experience and maintain trust. To mitigate risks, a “human-in-the-loop” approach is recommended, ensuring health care professionals oversee AI-supported decisions. By adhering to these best practices, AICAs can enhance health care accessibility, support clinical training, and complement human professionals. This work aims to provide a foundation for the ethical and effective integration of AICAs, maximizing their potential while minimizing risks, ultimately advancing patient care and education in the digital age.

Introduction

A virtual revolution is ongoing in the use of simulation technology for clinical and educational purposes. When discussion of the potential use of extended reality (XR) applications for human research and clinical intervention first emerged in the early 1990s, the technology required further development for feasibility in both clinical and educational spheres. Consequently, during these early years, XR suffered from a somewhat imbalanced “expectation-to-delivery” ratio. Technologically driven innovations in health care were considered and prototyped during the “computer revolution” in the 1990s. Early advances from this period focused on research and development (R&D) that aimed at enhancing productivity in patient documentation and recordkeeping,^1–3 improving access to clinical care via internet-based teletherapy,^4–6 and using virtual reality (VR) simulations to deliver exposure therapy for treating specific phobias and post-traumatic stress disorder (PTSD)^7,8 and for neurocognitive assessment and rehabilitation.^9–11

However, since those early days, the technology required to deliver health care applications has significantly matured. This can be observed in the continuing advances of underlying technologies (e.g., computational speed, 3D graphics rendering, audio/visual/haptic displays, user interfaces/tracking, voice recognition, wearable sensors, artificial intelligence [AI], and authoring software) that support the creation of low-cost, yet sophisticated XR systems capable of running on commodity-level personal computers, mobile devices, and stand-alone head-mounted display systems. Driven largely by advancements in the digital gaming and entertainment industries, these technological developments have provided the hardware and software platforms needed for practical, high-fidelity XR experiences in human research and clinical interventions. Thus, evolving mental and physical health care applications can now usefully leverage the interactive and immersive assets that XR affords as the technology continues to get faster, better, and cheaper.^12,13

While such advances have now enabled the creation of more believable context-relevant “structural” VR environments^7–9,11 (e.g., combat scenes, homes, hospital settings, classrooms, offices, markets, and natural and imaginative surreal worlds with the focus on representing places and spatial context rather than interactive agents), the next step in the evolution of medical XR involves the creation of virtual human (VH) representations that can engage real human users in credible and meaningful interactions. The stage is set for a transformative leap in the use of VH agents, leveraging advanced AI technologies such as natural language processing (NLP), machine learning, and deep neural networks to serve as virtual interactors across a wide range of clinical applications. These specific kinds of VHs (presented in the literature as chatbot, conversational agents, conversational AI, virtual assistant, intelligent personal assistant, conversational interface, natural language interface, embodied conversational agent, cognitive agent, or interactive AI) can be used to populate immersive VR/augmented reality environments^13–18 or be delivered on non-immersive displays via home computer screens and mobile phones for user interaction.^19–23 For the purpose of this article, we refer to these systems collectively as artificially intelligent conversational agents (AICAs). AICAs now operate across a wide range of digital contexts, opening new possibilities for patient-facing health care delivery and clinical training. These applications can support user self-awareness, enhance access to engaging self-care content, and provide emotional support and guidance—all through low-stigma, user-friendly interactions.²⁴

Imagine a world where users have private access and interact with an AICA coach/mentor/guide who is capable of answering questions about clinical issues, provide access and guidance on tailored self-assessment questionnaires, can discuss evidence-based treatment options, help user to locate accessible online psychoeducation or a physical treatment center suited to their needs, and generally provide a face-to-face expert humanlike interaction to support health care awareness and access. Users could also engage with an AICA conveniently through their mobile phones, enabling on-the-go interactions for guidance, support, and psychoeducational resources. Moreover, an AICA could be waiting for a patient to appear at a clinic and conduct an initial clinical interview to gather information that may be useful to start the early steps for creating a treatment plan that would be followed on by a live clinician. At the same time, this kiosk-based AICA can leverage advances in sensing technology (e.g., cameras, microphones, and passive biosensors) to “observe” facial/body gestures and vocal features and use those sensed signals to infer psychological state from these observable behaviors to enhance the interaction or document mental status over time.

These types of AICA interaction possibilities are not merely distant visions awaiting future research to assess their value and explore ethical boundaries. In fact, commercial use of AICAs has already moved beyond the laboratory and is being implemented on current-generation computing devices with end users capable of designing and deploying their own AICAs specific to their needs. However, as the promise of AICAs expands, so too do the ethical concerns and potential risks that arise from their widespread use. In certain cases, there have already been disputes over harm potentially stemming from interactions with AICAs. A recent investigation brought to light the potential psychological risks associated with AICAs when a 14-year-old adolescent male engaged in extensive interactions with an AI conversational agent on a public AICA creation platform.²⁵ The subject subsequently died by suicide, prompting a legal challenge against the software company. The lawsuit alleges that the AI, modeled after a fictional character, engaged in inappropriate sexual discourse and failed to respond adequately to expressions of suicidal ideation. Of note, over 18 million AICAs have been developed on this single platform, highlighting a meteoric rise in the utilization of this technology by the masses.^26,27 Thus, while these tools leveraging AI hold revolutionary possibilities, they also possess considerable potential to confer harm if not designed and implemented in a rigorous, systematic way. There is an urgent need for robust safeguards and ethical guidelines in the development and deployment of AI conversational agents, particularly when accessible to vulnerable populations and especially if being utilized in clinical and educational contexts. Here, we put forward a series of best practice recommendations to inform the design and implementation of AICAs for medical applications.

Methodology

Development of expert consensus best practices

The development of these recommendations for the safe, ethical, and effective design and implementation of AICA health care applications followed a structured expert consensus methodology. This approach was selected due to the rapidly evolving nature of AICA technology in health care, where high-quality empirical evidence is still emerging, yet guidance is urgently needed to ensure thoughtful design, patient safety, and ethical implementation.

Expert panel formation

A multidisciplinary expert panel was assembled, comprising specialists from XR technology, AI, clinical medicine, bioethics, and health care education. Panel members were selected through the invitation of the Journal of Medical Extended Reality editorial board based on their demonstrated expertise in AICA development, implementation, or evaluation in health care settings, as evidenced by their publication record, clinical experience, or technological contributions. The panel included representation from multiple academic institutions, clinical practice, and technology development to ensure comprehensive coverage of relevant perspectives.

Initial expert consensus development

The initial draft best practices were developed by two lead authors (R.J. and J.E.R.) who conducted a comprehensive review of existing literature on AICAs in health care, including published research, case studies, ethical frameworks, and regulatory considerations. This preliminary work established the foundational structure, organizing recommendations into five key domains: (1) AICA Manifestations and User Engagement, (2) Privacy, Safety, and Security, (3) Optimizing User Experience, (4) Systems Improvements, and (5) Integration of External Data. Within each domain, specific statements were formulated based on available evidence and preliminary expert opinion.

Consensus process

The consensus development process followed a modified Delphi approach, consisting of two structured rounds of review:

Round 1

The initial draft consensus statements were distributed to all panel members for independent review. Experts evaluated each statement for clarity, clinical relevance, ethical soundness, technical feasibility, and overall importance. Panel members provided written feedback, suggested modifications, and identified gaps requiring additional guidance. Feedback was collected using a standardized form that included both quantitative ratings (5-point Likert scale) and qualitative comments for each statement.

Round 2

The lead authors synthesized all feedback from Round 1 and revised the statement accordingly. Areas of disagreement or uncertainty were explicitly highlighted for focused discussion. The revised statements were redistributed to all panel members for a second review. Experts indicated their level of agreement with each revised statement and provided final recommendations for refinement. Particular attention was given to areas where significant disagreement remained after the first round.

Consensus definition and finalization

Consensus was defined a priori as complete agreement across all panel members on the inclusion and wording of each statement. Statements not meeting this threshold were either revised further based on expert feedback or excluded from the final statements. The final document was compiled by the lead authors and reviewed by all panel members to ensure accurate representation of the consensus process and outcomes.

Limitations

The panel acknowledges several limitations of this methodology. The rapidly evolving nature of AICA technology means that some recommendations may require updating as new evidence emerges. While the panel included diverse expertise, additional perspectives from patient advocates and experts from low-resource settings would strengthen future iterations. This expert consensus represents opinion rather than empirically validated standards, and implementation should be accompanied by ongoing evaluation of effectiveness and safety.

The panel recommends that these best practices be reviewed and updated regularly as the field advances and more empirical evidence becomes available regarding the impact of AICAs in health care settings.

Best practices for the design, development, implementation, and evaluation of AICAs in clinical or health care education applications

The next section presents our non-exhaustive recommendations for creating and using autonomous AICAs to support patient interactions within a health care setting. These AICAs, equipped with varying levels of AI, are intended to complement—not replace—well-trained health care professionals. They are designed to address gaps where economic constraints make human providers less accessible or in situations where individuals may hesitate to discuss their experiences due to perceived stigma. To guide the development of clinically oriented AICA systems, we present a set of best practices aimed at supporting the design, development, implementation, and evaluation of safe, ethical, and effective systems for clinical or educational use [summarized in Table 1]. These best practices emphasize informed design, continuous evaluation, necessary collaborations, user experience and empowerment, and ethical considerations in the development and deployment of these systems for health care.

Table 1.
Best Practice

Best practice Risks mitigated

1. AICA Manifestations and User Engagement

a. Virtual Human Transparency in Identity Prevent unwitting participants from interacting with AICA without consent

b. Authenticity of Virtual Human Representation Prevent propagation of negative stereotypes, biases, and unjust cultural portrayals

c. User Expectations and Consent Prevent improper user expectations including exposure to triggering or harmful outputs

d. Accountability Prevent unsafe and unreliable AICA systems

e. Supervision and Oversight Ensure human-in-the-loop engineering to facilitate AI as a supportive tool

2. Privacy, Safety, and Security

a. Peer-Review and Collaboration Ensure appropriate capabilities, limitations and proper use of AICAs

b. Privacy, Data Security, Compliance, and Ethical AI Principles Ensure privacy and compliance of data, particularly confidential healthcare data

c. Data Use Protections Users must opt-in for data collection

d. Reliability and Monitoring Ensure proper functioning of AICAs in “edge” case situations

e. Clinical Decisions Support Versus Decision Making Ensure generic advice given to patients; clinical diagnoses provided only by human after considering whole picture

f. Emergency Protocols for AICAs Ensure facilitation of human intervention during crises

3. Optimizing User Experience

a. User Autonomy Prevent lack of user autonomy

b. AICA Design for Health Equity Ensure cultural sensitivity so they are fair and equitable for all

c. Empathy and Emotional Support Better enable humans to realize that AICAs are not human and are capable of empathy

d. Adaptive Learning Facilitate long-term credibility between the user and AICA

e. Evidence-Based Information Prevent AICA from providing advice from non-reputable sources

4. Systems Improvement

a. AICA Validation Ensure AICA continues to function as designed

b. Design Process Incorporates Relevant User Feedback Ensure AICA adapts over time based on user feedback

c. Long-Term Monitoring and Evaluation Ensure AICA continues to function as designed

5. Integration of External Data

a. Physiological Data Ensure data is collected and incorporated with user consent

b. Limitations of Use Ensure user expectations regarding the benefits and limitations of AICA capabilities

Best practice	Risks mitigated
1. AICA Manifestations and User Engagement
a. Virtual Human Transparency in Identity	Prevent unwitting participants from interacting with AICA without consent
b. Authenticity of Virtual Human Representation	Prevent propagation of negative stereotypes, biases, and unjust cultural portrayals
c. User Expectations and Consent	Prevent improper user expectations including exposure to triggering or harmful outputs
d. Accountability	Prevent unsafe and unreliable AICA systems
e. Supervision and Oversight	Ensure human-in-the-loop engineering to facilitate AI as a supportive tool
2. Privacy, Safety, and Security
a. Peer-Review and Collaboration	Ensure appropriate capabilities, limitations and proper use of AICAs
b. Privacy, Data Security, Compliance, and Ethical AI Principles	Ensure privacy and compliance of data, particularly confidential healthcare data
c. Data Use Protections	Users must opt-in for data collection
d. Reliability and Monitoring	Ensure proper functioning of AICAs in “edge” case situations
e. Clinical Decisions Support Versus Decision Making	Ensure generic advice given to patients; clinical diagnoses provided only by human after considering whole picture
f. Emergency Protocols for AICAs	Ensure facilitation of human intervention during crises
3. Optimizing User Experience
a. User Autonomy	Prevent lack of user autonomy
b. AICA Design for Health Equity	Ensure cultural sensitivity so they are fair and equitable for all
c. Empathy and Emotional Support	Better enable humans to realize that AICAs are not human and are capable of empathy
d. Adaptive Learning	Facilitate long-term credibility between the user and AICA
e. Evidence-Based Information	Prevent AICA from providing advice from non-reputable sources
4. Systems Improvement
a. AICA Validation	Ensure AICA continues to function as designed
b. Design Process Incorporates Relevant User Feedback	Ensure AICA adapts over time based on user feedback
c. Long-Term Monitoring and Evaluation	Ensure AICA continues to function as designed
5. Integration of External Data
a. Physiological Data	Ensure data is collected and incorporated with user consent
b. Limitations of Use	Ensure user expectations regarding the benefits and limitations of AICA capabilities

1. AICA Manifestations and User Engagement.

a. VH Transparency in Identity

A key challenge in incorporating AICAs is the ability of AI to emulate human emotion and potentially pass as human to unsuspecting or unwilling participants. This capability may lead to unethical or unjust situations in clinical care or medical education. Therefore, protecting the autonomy of patients and learners is paramount as this technology rapidly evolves to imitate human intelligence. It is important to recognize that no single “Turing test” can fully account for the ethical risks posed across a diverse population. Clear disclosure that a user is interacting with an AICA is a foundational ethical responsibility—especially in health care contexts. Hospitals, clinics, and care platforms must communicate this clearly from the outset. Without transparent identity cues, users may overestimate the AI’s authority, empathy, or level of human oversight. This can result in therapeutic misconception or misplaced trust, which is particularly concerning in mental health and other sensitive care environments where users may be especially vulnerable.

We propose that all AICAs should be transparent in their identity, clearly disclosing to users—whether solicited or unsolicited depending on the context—that the interaction is with AI, not a human. In clinical care and educational settings, developers must also recognize that some patients or learners may object to engaging with AICAs on moral or philosophical grounds. In such cases, alternative methodologies should be available. This recommendation aligns with ethical practices in computer science regarding the disclosure of AI systems to human users.²⁸

There may be rare educational scenarios, such as simulation-based evaluations, where an AICA is required to maintain its presented identity despite being asked. However, even in these cases, participants should be informed and consented ahead of time regarding the nature of the interaction.

While health care institutions bear the frontline responsibility, other stakeholders—including developers, platform providers, and vendors—also play a critical role. AICAs should be designed with identity transparency in mind, using visual markers, verbal disclosures, or interface prompts. As these systems grow increasingly humanlike, transparency must remain a nonnegotiable principle—not only to uphold ethical standards but to protect users and ensure informed engagement.

b. Authenticity of VH Representation

AICAs may have humanlike characteristics. Care should be taken when creating these representations such that they do not present a bias or unjust portrayal toward any particular identity or culture.^29–31 When possible, it may be advisable to have an agreement between a diverse committee of representatives across identities, class, ethnicity, ability, gender, and sexuality in the development of an AICA representation, specifically to include voices and narratives of that which is being represented. There is a high rate of variability and risk when allowing the AICA to determine representation of human qualities and experiences, as this runs the risk of incorporating mainstream bias/harmful ideology.³²

c. User Expectations and Consent

Users should be informed about the capabilities and limitations of AICA output prior to engagement. Furthermore, users should provide informed consent prior to use. Continuous consent mechanisms should also be implemented, ensuring that users can regularly reassess and reaffirm their consent as interactions progress, particularly in contexts involving sensitive or personal information. Users should be advised on the risks and benefits of AICA engagement, including the risk of interacting with outputs that may be triggering or harmful. Provide clear disclaimers about the limitations of the chatbot in delivering mental health care and the importance of seeking professional help for serious issues.

d. Accountability

Clear designation of responsibility for the design, operation, and outcomes of VHs in clinical settings is essential to ensure accountability and trust. Accountability should encompass all stages, from the initial design to ongoing operation, addressing who is ultimately responsible for the AICAs’ responses. Additionally, care must be taken to avoid overstating the AICAs’ capabilities or effectiveness, as exaggerating these claims can mislead users and diminish trust in these technologies. To mitigate this, developers and health care providers should define and transparently communicate any limitations, including the specific contexts and with what populations the AICA is intended to be effective. Furthermore, AICA effectiveness should be evaluated by an independent standardized set of metrics. This evaluation process should include both qualitative and quantitative metrics that reflect real-world outcomes, such as patient engagement, satisfaction, accuracy in providing health information, and alignment with evidence-based practices. Accountability frameworks should also incorporate regular audits, performance reviews, and updates to ensure the VHs remain reliable, safe, and compliant with evolving health care standards and ethical best practices.

e. Supervision and Oversight

The use of AI systems in clinical settings requires careful supervision and oversight by qualified health care professionals to ensure patient safety, maintain quality of care, and address potential ethical concerns. At the time of writing these best practices, the authors are unable to locate any consensus within the literature that AICAs are capable of independent clinical work that does not require human oversight.^33,34 The proposed model for safe and practical implementation is described as a “human-in-the-loop” approach to maintain the balance between AI assistance and human expertise. This approach ensures that health care professionals remain the primary decision-makers, with AI serving as a supportive tool. This should also apply to health care educational use cases for AICAs to ensure an appropriate environment for clinical learners.

2. Privacy, Safety, and Security.

a. Peer-Review and Collaboration

Like the peer-review process in academic medicine, AICA developers should remain collaborative and subject their work to review by other developers to ensure alignment with evidence-based practices and ethical guidelines. Developers should be open about which open-source large language models (LLMs) or compound LLMs are being utilized in the development of their AICA. If a novel LLM was created in the development of a VH, then the authors should disclose what datasets were utilized to train and test the models. This review should also rigorously evaluate capabilities for AICA privacy and safety.

On an institutional implementation side, health care professionals supervising AI systems should receive specialized training on the capabilities, limitations, and proper use of these technologies. Institutions should develop competency assessment programs to ensure that staff members are adequately prepared to oversee AI applications in clinical settings.

b. Institutional Standards

As with all areas of human research, institutional review boards (IRBs) are essential in overseeing research involving AICA applications in health care and wellness, ensuring human subject protection. IRBs evaluate the ethical implications, focusing on informed consent, especially for vulnerable populations (e.g., patients with mental health conditions). They assess AI’s risks and benefits, ensuring participant safety and proper AI usage, while also reviewing privacy and data protection measures for compliance with Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR). They also address concerns regarding bias in AI algorithms and ensure that the research includes adequate safeguards against potential harm, such as misdiagnosis or exacerbation of health conditions. By overseeing these areas, IRBs ensure ethical and scientifically sound development and testing of AICA applications.

Similarly, AICA applications face regulatory challenges under Food and Drug Administration (FDA) oversight. If classified as medical devices or Software as a Medical Device, they must demonstrate safety, efficacy, and validation through clinical trials and post-market surveillance. Key concerns include compliance with privacy regulations, algorithmic transparency, and the potential for bias and depersonalization of care. The FDA mandates that AI applications provide clear decision-making processes, maintain strong security to protect sensitive health data, and undergo continuous monitoring to ensure ongoing safety and effectiveness. Given the emerging nature of AIAC technologies and implementation, it is expected that core principles governing human research and commercial applications will remain steadfast. However, continuous vigilance is necessary to identify and address unanticipated consequences and events, as this technology evolves and new use cases inevitably arise.

c. Privacy, Data Security, Compliance, and Ethical AI Principles

Robust data protection measures should be implemented to safeguard user information. As it relates to sensitive health care information but not limited only to protected health information, such as for AICAs engaging in patient intake or mental health care, relevant data should be held to the highest level of privacy standard, including the HIPAA³⁵ in the United States, the Personal Health Information Protection Act³⁶ in Canada, and the European Union’s GDPR³⁷ standards. Key elements to privacy, data security, and compliance are to ensure this sensitive health care information is not transmitted outside the secure infrastructure, making it accessible to third-party users outside of the intended use of the health care institution and the specific service it provides such as training, retraining, fine-tuning, or improving any other foundational model, product, or service offered by a third party. One way to achieve this in the United States is to deploy an open-source LLM in a HIPAA Health Information Trust Alliance (HITRUST) compliant environment, whether on premises or in the cloud, ensuring that information does not leave the predetermined infrastructure. When the LLM desired is not available through that format, a business associate agreement with a technology provider that can offer HIPAA and HITRUST compliance certifications and provide assurances that the data transmitted to their server is not available to anyone else, and the data are not used in any other way other than the agreed intended purpose. In addition, several providers are offering zero data retention methods, ensuring that the information is processed instantly and deleted right after.

Finally, all AICAs should comply with relevant regulations governing the use of AI, ensuring legal protections for both users and developers. In the United States alone, additional guidance is available on the Common Rule,³⁸ the 21st Century Cures Act,³⁹ and has become a prominent topic in political and legislative discourse via the White House Executive Order,⁴⁰ and the White House Blueprint for an AI Bill of Rights,⁴¹ introduced by the previous administration, and more recently through the introduction of the bipartisan bill “No Adversarial AI Act.”⁴² At a world level, World Health Organization’s ethics and governance on AI for health with LLMs⁴³ is an excellent resource on overall guidance pertaining to LLMs and their use in health care. Furthermore, following suit to the United States, Europe has introduced its own AI Act⁴⁴ and AI Pact,⁴⁵ and other countries such as Australia have proposed voluntary AI safety standards.⁴⁶ Finally, any AICAs developed or utilized should adhere to ethical frameworks in health care, such as the Belmont Report⁴⁷ and the Declaration of Helsinki.⁴⁸

d. Data Use Protections

User data may be regularly used as part of a quality improvement initiative to improve user experience and AICA functionality. However, research requiring user data should follow rigorous standards. First, user consent should be obtained using easy-to-understand language in an opt-in process. Similar to an opt-in process for a research study, the user should have a clear understanding of the research question for which their data will be investigated, regardless of anonymity. It should never be acceptable for an AICA to share identified data.

e. Reliability and Monitoring for Emergency Support

AICAs should be rigorously tested to ensure consistency and reliability across a variety of topics and interactions to ensure responses are generated by best practices. These interactions should be continuously monitored for quality improvement to prevent misuse and detect the generation of harmful content. Users should also have the capability to easily provide feedback on their interactions to reinforce effective conversations and deter harmful interactions. Optimally, for users interacting with an AICA for mental health purposes, the chatbot should have defined protocols to detect patients in crisis and have the capability to direct them to easily accessible human resources or emergency medical services for immediate help.

Implementing clear protocols for identifying and responding to user distress or emergency situations is crucial, especially in mental health applications. Previous studies have shown that AI systems are capable of identifying suicidal ideation and crisis-type behaviors.⁴⁹ In health care environments, whether an AICA is being used in mental health situations or not, patients or users may present information to AICAs that could be warnings for self-harm. Thus, AICAs being used in clinical contexts should uniformly have protocols designed to ensure timely and appropriate intervention when users express signs of acute distress, suicidal ideation, or other crisis situations. AICAs should implement NLP algorithms trained to recognize linguistic patterns and keywords associated with acute distress, suicidal ideation, or crisis situations. Furthermore, they should develop a tiered risk assessment system to categorize the severity of detected distress signals, and their developers should establish clear escalation pathways based on the assessed risk level, including immediate handover to human professionals for high-risk situations. Finally, programs should provide users with easily accessible emergency contact information and crisis resources within the AI interface, implement procedures for following up with users who have experienced a crisis event, ensure continuity of care, and maintain detailed records of crisis events and interventions for quality improvement and legal compliance.

Proper documentation of AI system usage, including the rationale for AI-assisted decisions and any overrides by human professionals, is crucial for maintaining transparency and facilitating quality improvement efforts. This documentation also supports legal and ethical compliance.

f. Clinical Decision Support Versus Decision Making

Establishing clear lines of accountability is essential when integrating AI into clinical practice. Health care organizations should develop frameworks that delineate responsibilities for AI-assisted decisions and actions, ensuring that human professionals remain ultimately accountable for patient care outcomes.

For patients engaging an AICA for support, the AICA should avoid generating a specific diagnosis of a medical condition. Instead, it should parse through and analyze relevant data to provide clinical information and support for the user, perhaps suggesting a differential diagnosis when specifically prompted by the user. However, the AICA should specifically encourage the user to seek out human professional consultation when a specific medical diagnosis is required. Similar to interpreting results from diagnostic tools such as a blood test or a neuropsychological test, information provided by an AICA should be viewed as one component of a holistic evaluation of the patient, ensuring that clinical decisions consider the full context of the patient’s health and circumstances. In addition, the AICA should be designed to defer to professionals when needed to prevent users from developing a dependency on the AICA for diagnosis and treatment of serious health conditions.

3. Optimizing User Experience.

a. User Autonomy

Users should have the capability to control their interactions with an AICA, including the ability to pause, adjust, or terminate the interaction. Clear instructions on these functions need to be presented to the user at the onset of their participation.

b. AICA Design for Health Equity

AICAs should be specifically designed to optimize cultural sensitivity, ensuring that they are aware of the diverse set of values and norms one might encounter across cultures. In addition, they should be designed to enable fair and equitable interactions among patients using different languages, slang, and people of different education levels. AICAs should also be easily accessible for users with disabilities, including those with visual, hearing, or motor impairments.

c. Empathy and Emotional Support

AICAs should be designed to facilitate empathetic responses and supportive language while specifically acknowledging that, as an AICA, it cannot specifically empathize with a human. Users should be provided specific tools and resources to enhance their health literacy and self-care practices beyond chatbot interactions. Content provided to patients should be rigorously evaluated by a panel of experts to ensure it is accurate, evidence-based, and of the highest quality. These resources should be updated and improved over time in response to user feedback.

d. Adaptive Learning

Adaptive learning mechanisms should be incorporated into AICA responses to facilitate a therapeutic relationship. Maintaining a strong memory of prior user interactions will better enable credible long-term interactions between the user and AICA. Moreover, by utilizing the content of previous user-AICA dialogues, a more comprehensive picture of the user state can be mined to enhance the personalization and quality of future dialogues.

e. Evidence-Based Information

Ensuring that AI systems are trained on and provide information from reputable, evidence-based sources is crucial for maintaining the quality, reliability, and safety of interactions, particularly in clinical and educational environments. This best practice aims to prevent the dissemination of misinformation, enhance the credibility of AI-generated responses, and align AI systems with best practices in health care and education. AICAs should be rigorously evaluated and select reputable, peer-reviewed sources for training data and knowledge bases. Furthermore, AICAs should be capable of providing users with access to the sources of information used by the AI system. Developers creating AICAs should involve subject matter experts in the development and validation of AI knowledge bases and, when possible, implement systems to grade the strength of evidence behind AI-provided information based on such as the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach for evidence-based medicine or similar accepted criteria.⁵⁰

For situations that present uncertainty or evidence-based answers are not available, developer teams train AI systems to express uncertainty rather than hallucinate or project unfounded confidence in inaccurate data. Finally, AICAs should, when possible, educate users on the importance of evidence-based information and how to interpret AI-provided information critically rather than simply presenting data without the ability to contextualize or frame such information.

4. Systems Improvement.

a. AICA Validation

As part of a continuous process for quality improvement, studies should be conducted to validate the effectiveness and safety of the chatbot for its specific goals. Validation of AICAs should be conducted by comparing user–chatbot interactions to scientifically validated processes. For research beyond this purpose, user consent must be obtained, and rigorous safeguards must be in place in line with the section “Data Use Protections.”

b. Design Process Incorporates Relevant User Feedback

User feedback should generate a preliminary analysis for improvement. Iterative feedback cycles should be utilized to determine needed updates to the chatbot. Health care institutions should establish protocols for continuous monitoring and evaluation of AI system performance. This includes regular audits of AI-generated recommendations and decisions to ensure accuracy and relevance.

Such oversight helps identify potential biases or errors in the AI system and allows for timely interventions.

c. Long-Term Monitoring and Evaluation

Based on an opt-in system, user outcomes should be tracked to assess the specific impact of AICA interactions. These outcomes should be de-identified and based on the same standards laid out in the section “Data Use Protections.”

5. Integration of External Data.

a. Physiological and Behavioral Data

Data from wearable devices and other sensors (e.g., sleep, activity, biosignals) may be incorporated into AICA systems to leverage deep learning to better help users as they seek out care. Integration of device data should take place after informed consent and clear explanation about the privacy and security of that data. All data incorporated into an AICA system should be limited to specific encounters, unless otherwise approved by the user (24/7 tracking of activity or sleep). No data should be accessible for sharing with external users or for research purposes, unless the user opts in via an informed consent process.

Integrating data from wearable devices and biosensors—such as heart rate variability, sleep quality, physical activity, and galvanic skin response—alongside facial expressions, gesture patterns, and voice analytics into an AICA offers a powerful avenue for enhancing user engagement and emotional health support. These continuous, real-time physiological and behavioral data streams can be processed using deep learning models to detect subtle patterns indicative of stress, anxiety, depression, or fatigue. Facial microexpressions, posture shifts, and changes in vocal tone or speech cadence can provide additional layers of insight into emotional state, especially when combined with biosignals. By training neural networks on large, anonymized multimodal datasets, AI systems can learn to make increasingly accurate inferences about a user’s emotional and physiological state, even when traditional self-report cues are absent. Inferences from these signals have been found to improve the detection of depression and PTSD,⁵¹ suicidal ideation,⁵² and general emotional distress.⁵³ This allows AIACs to respond with greater empathy, timing, and contextual relevance—prompting interventions such as guided breathing, mood journaling, or connecting the user to human care when elevated risk is detected.

However, for all their potential, collecting and using data from wearables and biosensors comes with significant ethical responsibilities. Users must give explicit, informed consent for the collection and use, with clear options to opt out at any time. The data must be anonymized and encrypted, both in storage and during transit, to prevent unauthorized access or misuse. Sensing systems should be designed with minimal data collection principles, gathering only what is necessary for the intended function. Additionally, users should be made aware of what is being sensed, how frequently, and whether any third parties (e.g., cloud platforms or health partners) will access or process that data. Finally, real-time feedback or alerts triggered by biometric/behavioral inferences should be designed to avoid causing unnecessary alarm or psychological distress, ensuring that sensor-based insights are communicated ethically and responsibly.

b. Limitations of Use

When physiological data are incorporated into the AIAC for diagnostic purposes, users should be counseled on the benefits and limitations of that technology integration. Users should have a clear understanding that the outputs generated by an AICA are limited both in terms of the quality of the input data, which may itself be flawed, and the interpretation of the data from the AICA. The probabilistic nature of inferences made from such biosignals and behaviors should be an introductory element of discussions that incorporate this information in user-AICA dialogues.

Discussion

Clinical interest in artificially intelligent conversational agents designed for interaction with humans can trace its roots to the work of MIT AI researcher, Joe Weizenbaum. In 1966, he wrote a language analysis program called ELIZA that was designed to imitate a Rogerian therapist. The system allowed a computer user to interact with a virtual therapist by typing simple sentence responses to the computerized therapist’s questions. Weizenbaum reasoned that simulating a nondirective psychotherapist was one of the easiest ways of simulating human verbal interactions, and it was a compelling simulation that worked well on teletype computers. Despite the fact that the illusion of ELIZA's intelligence soon disappears due to its inability to handle complexity or nuance, Weizenbaum was reportedly shocked upon learning how seriously people took the ELIZA program.⁵⁴ This led him to conclude that it would be immoral to “substitute” a computer for human functions that “… involves interpersonal respect, understanding, and love.”⁵⁵ While Weizenbaum’s perspective is understandable given the era in which he worked, modern approaches to fostering interactions with AICAs for practical purposes—especially when no human is available—do not align with the “substitute” concept he described.

More recently, seminal R&D has appeared in the creation of highly interactive, artificially intelligent and natural language capable VH agents.^53–57 No longer at the level of a prop to add context or minimal faux interaction in a virtual world, these agents are designed to perceive and act in a 3D digital world, engage in face-to-face spoken dialogues with real users, and in some cases, they are capable of exhibiting humanlike emotional reactions. Such fully embodied conversational characters have been around for nearly 30 years,⁵⁸ and there has been much work on full systems that have been designed and used for training,^20,59–63 intelligent kiosks,⁶⁴ and virtual receptionists.⁶⁵ Previous classical work on AICAs in the computer graphics community focused on perception and action in 3D worlds but largely ignored dialogue and emotions. This has now changed. AICAs are now being created that control computer-generated bodies and can interact with users through speech and gesture in virtual environments.^66–69 AICAs can engage in rich conversations,⁷⁰ recognize nonverbal cues,^71–73 reason about social and emotional factors,⁶⁷ and synthesize human communication and nonverbal expressions.⁷⁴

A common observation from more recent AICA research is that individuals have reported a heightened perception of safety and a lower stigma when disclosing personal issues and information to AI-driven characters compared with human counterparts.^75–77 This phenomenon can be attributed to the nonjudgmental, consistent, and emotionally neutral nature of virtual interactions, which may reduce the fear of negative social evaluation and concerns regarding impression management. The anonymity and control provided by AI interfaces allow users to engage in self-disclosure without concerns about bias or negative evaluation. Consequently, VH characters have demonstrated potential as effective tools in mental health support and therapeutic interventions, fostering an environment conducive to open dialogue and honest communication. Understanding the psychological factors underpinning this perception could further inform the design of AICAs that optimize trust, empathy, and user engagement in sensitive contexts.

However, although the use of AICAs in clinical settings holds significant promise, it also presents a range of challenges that must be carefully addressed to maximize their effectiveness and integration within health care environments. The integration of LLMs and related NLP tools into AICAs further raises safety concerns. While these tools enable sophisticated, context-aware dialogues, they require rigorous safeguards to ensure accuracy and appropriateness, as errors or biases in responses could pose significant risks in high-stakes clinical scenarios.^78,79 Data privacy and security add further layers of complexity to deploying AICAs in clinical environments. With health care data being one of the most sensitive types, AI systems must comply with strict regulations, such as HIPAA, which mandates robust encryption, access controls, and logging mechanisms to protect patient information. Any breach or perceived vulnerability could lead to a loss of patient trust and pose significant legal risks. Moreover, AICAs must be able to securely access and process these data in real time, which involves not only technical infrastructure but also the implementation of advanced encryption methods and secure data pathways that meet regulatory standards.^80,81

Here, we have created best practices for the design and implementation of AICAs for health care utilization. Our best practices center around five domains: (1) AICA Manifestations and User Engagement, (2) Privacy, Safety, and Security, (3) Optimizing User Experience, (4) Systems Improvements, and (5) Integration of External Data. These domains aim to optimize user experience by synergizing patient autonomy, emotional support, data security, realistic user expectations, and the capability for continuous systems improvement. With the best practices set forth in this document, we hope to improve health equity and access to important health care resources for those in need of support. By adhering to these best practices and other important guardrails, we hope that future AICAs will maintain rigorous standards that add to traditional human support models and do not violate the norms of Weizenbaum’s “substitute” concept.

Addressing these challenges is vital for AICAs to contribute meaningfully to patient care and to gain acceptance from both patients and health care providers. Each of these areas requires careful consideration and further research to optimize the design, ethical standards, and practical implementation of AICAs in health care contexts. Considering the rich and complex history of the research underpinning the technology and concepts behind AICAs, we hope these best practices will serve as a foundation for facilitating the ethical and professional development of systems in this clinical field.

Footnotes

Acknowledgment

The University of South California component of this work was sponsored by U.S. Army Research Laboratory contract W911NF-14-D-0005 and the Department of Veterans Affairs’ Mission Daybreak Challenge.

Authors’ Contributions

All authors contributed equally in the conceptualization, writing, reviewing, and editing of this article.

Disclaimer

The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Author Disclosure Statement

The authors of this article declare potential conflicts of interest in relation to their academic and commercial work in the field of AICAs and XR. These connections were considered both unavoidable and beneficial to the creation of this article. The standard blinded peer-review process, as outlined by the journal, was rigorously followed for all stages of article submission. Reviewer identities were kept confidential, and while the authors cited their own research, this was done within the broader context of the field and did not exclude the work of other relevant scholars. Specific disclosures include the following: A.R. is a Senior Associate Editor for the Journal of Medical Extended Reality. He has conducted research on the clinical use of VHs since 2007 and serves as an advisor to Sidekick Platform Inc., a company developing technologies related to the content of this article. S.M. has researched the clinical use of VHs since 2013 and is an advisor to Sidekick Platform Inc., which is involved in technologies discussed in this article. A.S. and J.E.R.: No conflicts of interest to disclose. R.J. is the Chief Editor of the Journal of Medical Extended Reality, serves as a consultant for editorial services with Mary Ann Liebert Publications Inc., and as a speaker with Abbott Laboratories Inc., outside the submitted work. He also serves as a volunteer board advisor to 3D Organon/Medis Media, outside the submitted work. No direct mention or citation of the authors’ commercial activities or products was made in the article. The content of this article addresses broader conceptual frameworks within the field, with no specific focus on commercial endeavors.

Funding Information

This is an author-initiated project, not associated with any specific funding source.

Abbreviations Used

References

McDonald

, Tierney

. Computer-stored medical records: Their future role in medical practice. JAMA, 1988; 259(23):3433–3440.

Pryor

, Gardner

, Clayton

, et al. The HELP system. In: Proceedings of the Annual Symposium on Computer Application in Medical Care. American Medical Informatics Association; 1982. p. 19.

Rector

, Nowlan

, Kay

. Foundations for an electronic medical record. Methods Inf Med, 1991; 30(3):179–186.

Andersson

, Bergström

, Holländare

, et al. Internet-based self-help for depression: Randomised controlled trial. Br J Psychiatry, 2005; 187(5):456–461.

Taylor

, Luce

. Computer-and internet-based psychotherapy interventions. Curr Dir Psychol Sci, 2003; 12(1):18–22.

Ybarra

, Eaton

. Internet-based mental health interventions. Ment Health Serv Res, 2005; 7(2):75–87.

Rothbaum

, Hodges

, Kooper

, et al. Virtual reality graded exposure in the treatment of acrophobia: A case report. Behav Ther, 1995; 26(3):547–554.

Rothbaum

, Hodges

, Alarcon

, et al. Virtual reality exposure therapy for PTSD Vietnam veterans: A case study. J Trauma Stress, 1999; 12(2):263–271.

Brown

, Kerr

, Bayon

. The development of the Virtual City: A user centred approach. In: Sharkey

, Rose

, Lindstrom

, editors. Proceedings of the 2nd European Conference on Disability, Virtual Reality and Associated Techniques. Reading: University of Reading; 1998. pp. 11–6.

10.

Pugnetti

, Mendozzi

, Motta

, et al. Evaluation and retraining of adults’ cognitive impairments: Which role for virtual reality technology? Comput Biol Med, 1995; 25(2):213–227.

11.

Rizzo

, Buckwalter

, Neumann

. Virtual reality and cognitive rehabilitation: A brief review of the future. J Head Trauma Rehabil, 1997; 12(6):1–15.

12.

, Lau

, Simoff

, et al. Review of innovative immersive technologies for healthcare applications. Innov Digit Health Diagn Biomark, 2022; 2(2022):27–39.

13.

Spiegel

BMR

, Rizzo

, Persky

, et al. What is medical extended reality? A taxonomy defining the current breadth and depth of an evolving field. J Med Ext Real, 2024; 1(1):4–12.

14.

Freeman

, Haselton

, Freeman

, et al. Automated psychological therapy using immersive virtual reality for treatment of fear of heights: A single-blind, parallel-group, randomised controlled trial. Lancet Psychiatry, 2018; 5(8):625–632.

15.

Freeman

, Lambe

, Kabir

, et al.; gameChange Trial Group. Automated virtual reality therapy to treat agoraphobic avoidance and distress in patients with psychosis (gameChange): A multicentre, parallel-group, single-blind, randomised, controlled trial in England with mediation and moderation analyses. Lancet Psychiatry, 2022; 9(5):375–388.

16.

Hartholt

, Fast

, Reilly

, et al. Ubiquitous virtual humans: A multi-platform framework for embodied AI agents in XR. In: 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE; 2019. pp. 308–384.

17.

Torous

, Bucci

, Bell

, et al. The growing field of digital psychiatry: Current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry, 2021; 20(3):318–335.

18.

Yeo

, Samaan

, Ng

, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol, 2023; 29(3):721–732.

19.

Mozgai

, Rizzo

, Hartholt

. Human-centered design for a virtual human led mHealth intervention for suicide prevention. In: Human Systems Engineering and Design (IHSED2023). 2023; 112:98–106.

20.

Reger

, Norr

, Rizzo

, et al. Comparison of virtual standardized patients to controls to teach motivational interviewing to healthcare professionals in the Department of Veterans Affairs and the U.S. Military: A randomized controlled trial. JAMA Netw Open. 2020. Available from: https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2771733

21.

Rizzo

, Lange

, Buckwalter

, et al. An intelligent virtual human system for providing healthcare information and support. Stud Health Technol Inform, 2011; 163:503–509.

22.

Rizzo

, Shilling

, Forbell

, et al. Autonomous virtual human agents for healthcare information support and clinical interviewing. In: Luxton

, editor. Artificial Intelligence in Mental Healthcare Practice. Oxford: Academic Press; 2015. pp. 53–80.

23.

Talbot

, Rizzo

. Virtual human standardized patients for clinical training. In: Rizzo

, Bouchard

, editors. Virtual Reality for Psychological and Neurocognitive Interventions. Springer; 2019. pp. 387–405.

24.

Chattopadhyay

, Ma

, Sharifi

, et al. Computer-controlled virtual humans in patient-facing systems: Systematic review and meta-analysis. J Med Internet Res, 2020; 22(7):e18839.

25.

Allan

. Can A.I. be blamed for a teen’s suicide? The New York Times. 2024. Available from: https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html [Last accessed: November 1, 2024].

26.

Tidy

. Character.ai: Young people turning to AI therapist bots. BBC News. 2024. Available from: https://www.bbc.com/news/technology-67872693. [Last accessed: January 15, 2025].

27.

Kumar

. Character AI statistics (2025) — 20 million active users. DemandSage. 2025. Available from: https://www.demandsage.com/character-ai-statistics/ [Last accessed: August 9, 2025].

28.

Mozafari

, Weiger

, Hammerschmidt

. The chatbot disclosure dilemma: Desirable and undesirable effects of disclosing the non-human identity of chatbots. In: ICIS Proceedings. 2020. pp. 1–18.

29.

Ali

, Tang

, Connolly

, et al. Demographic representation in 3 leading artificial intelligence text-to-image generators. JAMA Surg, 2024; 159(1):87–95.

30.

Kenig

, Echeverria

, Vives

. Human beauty according to artificial intelligence. Plast Reconstr Surg Glob Open, 2023; 11(7):e5153.

31.

Morcos

, Duggan

, Young

, et al. Artificial intelligence portrayals in orthopaedic surgery: An analysis of gender and racial diversity using text-to-image generators. J Bone Joint Surg Am, 2024; 106(23):2278–2285.

32.

Cuadra

, Wang

, Stein

, et al. The illusion of empathy? Notes on displays of emotion in human-computer interaction. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. 2024. pp. 1–18.

33.

Asan

, Bayrak

, Choudhury

. Artificial intelligence and human trust in healthcare: Focus on clinicians. J Med Internet Res, 2020; 22(6):e15154.

34.

Reddy

, Allan

, Coghlan

, et al. A governance model for the application of AI in health care. J Am Med Inform Assoc, 2020; 27(3):491–497.

35.

Act

. Health insurance portability and accountability act of 1996. Public Law, 1996; 104:191.

36.

Personal Health Information Protection Act, 2004, S.O. 2004, c. 3, Ontario, Canada. Available from: https://www.ontario.ca/laws/statute/04p03 [Last accessed: July 20, 2025].

37.

General Data Protection Regulation (GDPR), Regulation (EU) 2016/679 of the European Parliament and of the Council. Official Journal of the European Union, 2016. Available from: https://eur-lex.europa.eu/eli/reg/2016/679/oj [August 1, 2025].

38.

Department of Health and Human Services. Federal Policy for the Protection of Human Subjects (‘Common Rule’). 45 CFR 46. Washington, D.C.: Department of Health and Human Services; 2018. Available from: https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html [Last accessed: August 1, 2025].

39.

21st Century Cures Act, Public Law 114-255, 114th Congress, United States. 2016. Available from: https://www.congress.gov/bill/114th-congress/house-bill/34 [Last accessed: August 1, 2025].

40.

The White House. Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The White House: Washington, D.C.; 2023. Available from: https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/ [Last accessed: November 1, 2024].

41.

The White House Office of Science and Technology Policy. Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. Washington, D.C.: The White House; 2022. Available from: https://bidenwhitehouse.archives.gov/ostp/ai-bill-of-rights/ [Last accessed: November 1, 2024].

42.

Scott

. No Adversarial AI Act [Internet]. 119th Congress (2025–2026). S. 2177, Senate; introduced 25 Jun 2025 [cited 2025 Jul 6]. Available from: https://www.congress.gov/bill/119th-congress/senate-bill/2177 [Last accessed: July 5, 2025].

43.

World Health Organization. Ethics and Governance of Artificial Intelligence for Health. Geneva: World Health Organization; 2021. Available from: https://www.who.int/publications/i/item/9789240029200 [Last accessed: August 9, 2025].

44.

2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act), OJ:L_202401689. 2024. Available from: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202401689

45.

European Commission. AI Pact [Internet]. Brussels: European Commission; [cited 2025 Jul 6]. Available from: https://digital-strategy.ec.europa.eu/en/policies/ai-pact [Last accessed: July 15, 2025].

46.

Australian Government Department of Industry, Science and Resources. Voluntary AI safety standard [Internet]. Canberra: DISR; 2024 [cited 2025 Jul 6]. Available from: https://www.industry.gov.au/publications/voluntary-ai-safety-standard [Last accessed: July 15, 2025].

47.

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Washington, D.C.: U.S. Department of Health, Education, and Welfare; 1979. Available from: https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html [Last accessed: November 20, 2024].

48.

Wthoircladl-MpreindcicipaleAs-sfsoorc-imateidoinc.aWl-roersledarMche-diincvaol lAvisnsgo-chiuatmioann-Dsuecbljaercattsi/on of Helsinki: Ethical.

49.

Bernert

, Hilberg

, Melia

, et al. Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations. Int. J. Environ. Res. Public Health, 2020; 17(16):5929; doi: 10.3390/ijerph17165929

50.

Schünemann

, Oxman

, Brozek

JAN

, et al. GRADE: assessing the quality of evidence for diagnostic recommendations. Evid Based Med, 2008; 13(6):162–163; doi: 10.1136/ebm.13.6.162-a

51.

Scherer

, Lucas

, Gratch

, et al. Self-reported symptoms of depression and PTSD are associated with reduced vowel space in screening interviews. IEEE Trans Affective Comput, 2016; 7(1):59–73.

52.

Venek

, Scherer

, Morency

, et al. Adolescent suicidal risk assessment in clinician-patient interaction. IEEE Trans Affective Comput, 2017; 8(2):204–215.

53.

Rizzo

, Scherer

, DeVault

, et al. Detection and computational analysis of psychological signals using a virtual human interviewing agent. J Pain Manag, 2016; 9(3):311–321.

54.

Howell

, Muller

. Computers in psychotherapy: A new prescription [Internet]. 2000 [cited 2024 Dec 28]. Available from: http://www.psychology.mcmaster.ca/beckerlab/showell/ComputerTherapy.PDF

55.

Weizenbaum

. Eliza—A computer program for the study of natural language communication between man and machine. Commun ACM, 1966; 9(1):36–45; doi: 10.1145/365153.365168

56.

Burden

, Savin-Baden

. Virtual Humans: Today and Tomorrow. Chapman and Hall/CRC; 2019.

57.

Spiegel

, Liran

, Clark

, et al. Feasibility of combining spatial computing and AI for mental health support in anxiety and depression. npj Digital Medicine, 2024 Jan 26; 7(1):22; doi: 10.1038/s41746-024-01011-0

58.

Bickmore

, Cassell

. Social dialogue with embodied conversational agents. In: van Kuppevelt

, Dybkjaer

, Bernsen

, editors. Advances in Natural, Multimodal Dialogue Systems. New York, NY: Kluwer Academic; 2005; doi: 10.1007/1-4020-3933-6_2

59.

Kenny

, Rizzo

, Parsons

, et al. A virtual human agent for training clinical interviewing skills to novice therapists. Ann Rev Cybertherapy Telemed, 2007; 5:81–89.

60.

Prendinger

, Ishizuka

. Life-Like Characters—Tools, Affective Functions, and Applications. Berlin, Germany: Springer; 2004; doi: 10.1007/978-3-662-08373-4

61.

Rickel

, Gratch

, Hill

, et al. Steve goes to Bosnia: Towards a new generation of virtual humans for interactive experiences. In: Proceedings of the AAAI Spring Symposium on AI and Interactive Entertainment. Stanford University: CA; 2001.

62.

Talbot

, Sagae

, John

, et al. Sorting out the virtual patient: How to exploit artificial intelligence, game technology and sound educational practices to create engaging role-playing simulations. Int J Gaming Comput-Mediated Simul, 2012; 4(3):1–19.

63.

Zaizar

, Gramlich

, Rizzo

, et al. Exploration of the impact of baseline clinician learner characteristics on motivational interviewing skill improvement following training with a virtual standardized patient. J Train Educ Prof Psychol, 2024; 18(4):378–385; doi: 10.1037/tep0000490

64.

McCauley

, D’Mello

. A speech-enabled intelligent kiosk. In: Proceedings of the 6th Annual Intelligent Virtual Agents Conference. Marina del Ray, CA; 2006. pp. 132–44.

65.

Babu

, Schmugge

, Barnes

, et al. What would you like to talk about? An evaluation of social conversations with a virtual receptionist. In: Gratch

, editors. IVA, 2006. Berlin, Germany: Springer-Verlag; 2006. pp. 169–180; doi: 10.1007/11821830_14

66.

DeVault

, Georgila

, Artstein

, et al. Verbal indicators of psychological distress in interactive dialogue with a virtual human. In: Proceedings of the SIGDIAL 2013 Conference. 2013. p. 193–202.

67.

Gratch

, Rickel

, Andre

, et al. Creating interactive virtual humans: Some assembly required. IEEE Intell Syst, 2002; 17(4):54–63; doi: 10.1109/MIS.2002.1024753

68.

Gratch

, Marsella

. A domain-independent framework for modeling emotion. J Cogn Syst Res, 2004; 5(4):269–306; doi: 10.1016/j.cogsys.2004.02.002

69.

Gratch

, Morency

, Scherer

, et al. User-state sensing for virtual health agents and telehealth applications. In: Proceedings of the 20th Annual Medicine Meets Virtual Reality Conference. 2013.

70.

Traum

, Marsella

, Gratch

, et al. Multi-party, multi-issue, multi-strategy negotiation for multi-modal virtual agents. In: Proceedings of the Conference on Intelligent Virtual Agents. 2008. pp. 117–130.

71.

Morency

, de Kok

, Gratch

. Context-based recognition during human interactions: Automatic feature selection and encoding dictionary. In: Proceedings of the 10th International Conference on Multimodal Interfaces. Chania, Greece; 2008.

72.

Rizzo

, Scherer

, DeVault

, et al. Detection and computational analysis of psychological signals using a virtual human interviewing agent. In: Proceedings of the International Conference on Disability, Virtual Reality and Associated Technologies. 2014. Available at: https://www.icdvrat.com/2014/P/2014_S03N3_Rizzo_etal.pdf

73.

Scherer

, Stratou

, Lucas

, et al. Automatic audiovisual behavior descriptors for psychological disorder analysis. Image Vis Comput, 2014; 32(10):648–658.

74.

Thiebaux

, Marshall

, Marsella

, et al. SmartBody: Behavior realization for embodied conversational agents. In: Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). Estoril, Portugal; 2008.

75.

Lucas

, Gratch

, King

, et al. It’s only a computer: Virtual humans increase willingness to disclose. Comput Human Behav, 2014; 37:94–100.

76.

Lucas

, Rizzo

, Gratch

, et al. Reporting mental health symptoms: Breaking down barriers to care with virtual human interviewers. Front Robot AI, 2017; 4:51; doi: 10.3389/frobt.2017.00051

77.

Mozgai

, Lucas

, Gratch

. To tell the truth: Virtual agents and morning morality. In: Proceedings of the 17th International Conference on Intelligent Virtual Agents (IVA). Stockholm, Sweden; 2017. pp. 283–286.

78.

Llanes-Jurado

, Gómez-Zaragozá

, Minissi

, et al. Developing conversational virtual humans for social emotion elicitation based on large language models. Expert Syst Appl, 2024; 246:123261.

79.

Haltaufderheide

, Ranisch

. The ethics of ChatGPT in medicine and healthcare: A systematic review on large language models (LLMs). NPJ Digit Med, 2024; 7(1):183.

80.

Marengo

. Navigating the nexus of AI and IoT: A comprehensive review of data analytics and privacy paradigms. Internet Things, 2024; 27:101318.

81.

Zhou

, Piramuthu

. IoT security perspective of a flexible healthcare supply chain. Inf Technol Manag, 2018; 19(3):141–153.

Expert Consensus Best Practices for the Safe,Ethical,and Effective Design and Implementation of Artificially Intelligent Conversational Agent (i.e.,Chatbot/Virtual Human) Systems in Health Care Applications

Abstract

Introduction

Methodology

Development of expert consensus best practices

Expert panel formation

Initial expert consensus development

Consensus process

Round 1

Round 2

Consensus definition and finalization

Limitations

Best practices for the design, development, implementation, and evaluation of AICAs in clinical or health care education applications

a. VH Transparency in Identity

b. Authenticity of VH Representation

c. User Expectations and Consent

d. Accountability

e. Supervision and Oversight

2. Privacy, Safety, and Security.

a. Peer-Review and Collaboration

b. Institutional Standards

c. Privacy, Data Security, Compliance, and Ethical AI Principles

d. Data Use Protections

e. Reliability and Monitoring for Emergency Support

f. Clinical Decision Support Versus Decision Making

3. Optimizing User Experience.

a. User Autonomy

b. AICA Design for Health Equity

c. Empathy and Emotional Support

d. Adaptive Learning

e. Evidence-Based Information

4. Systems Improvement.

a. AICA Validation

b. Design Process Incorporates Relevant User Feedback

c. Long-Term Monitoring and Evaluation

5. Integration of External Data.

a. Physiological and Behavioral Data

b. Limitations of Use

Discussion

Footnotes

Acknowledgment

Authors’ Contributions

Disclaimer

Author Disclosure Statement

Funding Information

Abbreviations Used

References