Sage Journals: Discover world-class research

Abstract

Objectives

This study examined the perceptions of caregivers of young childhood cancer survivors (YCCS) regarding the use of virtual assistant (VA) technology for health information seeking and care management. The study aim was to understand how VAs can support caregivers, especially those from underserved communities, in navigating health information related to cancer survivorship.

Methods

A qualitative study design was employed, involving semi-structured interviews and focus groups with 10 caregivers of YCCS from metropolitan, rural, and Appalachian regions, recruited from a large pediatric academic medical center in the Midwest. A web-based VA prototype was tested with caregivers, who provided feedback on its usability, utility, and feasibility. Data were analyzed using thematic analysis to identify key themes related to caregivers’ interactions with and perceptions of the VA technology.

Results

We identified four major themes: Interface and Interaction, User Experience, Content Relevance, and Trust. Caregivers expressed preferences for multimodal interactions (voice and text), particularly valuing flexibility based on context. They emphasized the need for accurate, relevant, and easily retrievable health information tailored to their child's needs. Trust and confidentiality were critical, with caregivers favoring VAs integrated with trusted healthcare systems. While VAs were perceived as valuable tools for reducing search fatigue and cognitive burden, caregivers highlighted the need for improved conversational depth, personalization, and empathetic response.

Conclusions

VAs hold promise as support tools for caregivers of YCCS, particularly in underserved communities, by offering personalized, credible, and accessible health information. To maximize their potential, research and development efforts should focus on building trust-building, integrated, and personalized VAs. These enhancements can help VAs further ease caregiving tasks and support caregivers in managing complex health needs.

Keywords

Caregivers health information seeking pediatric cancer survivors rural health underserved communities virtual assistants

Introduction

There are over half a million survivors of pediatric cancer in the United States.^1,2 The complexity of pediatric cancer treatment necessitates that caregivers, often parents, close family members, or legal guardians, navigate an intricate web of information related to diagnosis, treatment options, and side effects.³ Caregivers receive cancer-related information during the acute treatment phase, as well as in survivorship, when over two-thirds of survivors are at risk of developing one or more chronic health problem, known as late effects.⁴ The high cognitive and emotional burden of navigating this process makes it imperative to investigate new avenues of tailored informational support for these caregivers.^5,6

Young childhood cancer survivors (YCCS), who are diagnosed and treated in early childhood (i.e., less than age 7 years), face the highest risk for late effects given their treatment during a highly sensitive developmental period, particularly for neurodevelopmental late effects.^7–9 Risks are elevated in the context of invasive treatments such as intrathecal chemotherapy and cranial directed radiation.¹⁰ Neurodevelopmental late effects after cancer may include developmental delays (e.g., speech, motor, and cognitive), neurocognitive deficits (e.g., memory, attention, and executive functioning, and processing speed), and psychosocial difficulties.¹¹ Thus, caregivers of YCCS must navigate the additional complexity of locating resources related to the diagnosis and treatment of neurodevelopmental conditions and identifying providers with expertise in treating children with medical complexity.^7,12

Resources to mitigate these neurodevelopmental late effects, such as state-based early intervention and home visiting programs, school liaison support, and rehabilitative therapies (e.g., speech, physical, occupational therapies) may be beneficial, especially when delays are identified and treated early.¹³ Still, the availability and quality of these services may vary by geographic location. Specifically, healthcare disparities are known to be more pronounced in rural and Appalachian communities due to factors like geographical isolation, widespread poverty, lack of educational resources, and limited access to specialized medical care.^14–16 Additionally, the extent to which caregivers seek these resources and use digital tools to identify quality providers in their area remains under-investigated.¹⁷

Existing studies on online health information-seeking behavior reveal that caregivers often prefer personalized, easily accessible, and reliable sources of information.^18,19 Today, virtual assistants (VAs) such as Amazon's Alexa, Google Assistant, and Apple's Siri have become ubiquitous platforms that facilitate various forms of user–technology interaction, from simple tasks like setting reminders to complex ones like accessing health information and care management.^20–24 These VAs have become an integral part of modern information-seeking behavior, with studies indicating a growing preference for voice-based interfaces over traditional textual search in various demographics.^25–27 Notably, with recent advancements in large language models (LLMs), the capabilities of VAs show improved comprehension and responses.^28,29 However, generative AI models carry a risk of producing misinformation or “hallucinations,” particularly in medical contexts where accuracy is critical. LLMs are highly sensitive to the quality, scope, and relevance of their training data and the way queries are framed, which can influence the reliability of their responses.³⁰

Given the capabilities of VAs in providing information that is not only quick but also personalized through machine learning algorithms,^31–34 they may offer a promising solution to the challenges faced by caregivers of YCCS. Caregivers are likely to benefit most if they hold one or more marginalized identities (e.g., racial/ethnic identities, low socioeconomic status) or live in underserved areas (e.g., rural or Appalachian), where resources may be scarce or nonexistent. Yet, we lack empirical data that directly address the adoption, feasibility, and utility of VAs across each of these unique contexts. In particular, the rural and Appalachian communities have been underrepresented in existing research, leading to an incomplete understanding of how these technological tools might be leveraged to meet the unique needs of more isolated populations.³⁵ Therefore, there is a need to bridge this gap in the literature by investigating perceptions about how VAs could be effectively utilized as an information-sharing tool among caregivers of YCCS, especially amongst those in underserved communities.

The primary aim of this study is to examine the perception of caregivers of YCCS, including those from underserved communities, about VA technology use for health information seeking and care management. Given the potential of VAs to deliver complex, multidimensional information, including that related to neurodevelopmental conditions,³⁶ this study seeks to fill a vital gap in our understanding of how VAs could support caregivers facing these multifaceted challenges and aid in identification and access to early intervention. Furthermore, this study intends to contribute to our broader understanding of how emerging technologies can be adapted to serve marginalized populations, thereby addressing healthcare disparities through targeted, accessible, and high-quality information dissemination.

Methods

This study used a qualitative approach to understand the perception of caregivers, further elaborating challenges and opportunities associated with the use of VAs for health information-seeking and care management. The research methodology was informed by constructivist paradigms that acknowledge the subjective experiences of caregivers.³⁷ Please see the Consolidated Criteria for Reporting Qualitative Research (COREQ)³⁸ Checklist in Appendix A to further review reported elements in our study.

Ethics

The study was approved by the Institutional Review Board (IRB) of Nationwide Children's Hospital (Ethics Approval Number: 00002939). This study was considered as minimal risk, and a waiver of written documentation of informed consent was granted. Participants reviewed an IRB approved information sheet (similar to a consent form outlining the potential risks and benefits) prior to the study and verbally agreed to participate.

Virtual assistant prototype

A web-based VA was created using a conversational agent platform, Voiceflow.³⁹ The VA was designed with three components; Request, Delivery, and Continuation to simulate naturalistic conversation. (1) Request: The prototype initiated the conversation by requesting a description or question about a topic of interest. The VA was designed to use button responses, voice commands, or free text. This feature was to demonstrate VA capabilities and provided options to caregivers. (2) Delivery: The VA would respond with relevant information using the knowledge base. This knowledge base included a number of documents referencing from reports by cancer advocacy organizations,^40,41 government policy documents,^42–44 information sheets from children's hospitals,^45,46 and the Family Handbook™, a general guide to pediatric oncology.⁴⁷ These knowledge base documents were curated by searching government and advocacy organization repositories for cancer and survivorship. All documents were manually reviewed and approved by a licensed clinical psychologist with pediatric oncology experience. The knowledge base was necessary to ensure the accuracy and relevance of responses with evidence-based materials in-line with testing scenarios. (3) Continuation: The user had an option to follow up on their initial inquiry, such as asking for more information about complications or ask for further clarifications. At this stage, the VA stored a copy of the conversation to regenerate a specific answer or expand with a comprehensive statement. Figure 1 outlines the VA prototype and interaction flow.

Figure 1.

Flow diagram of the prototype web-based virtual assistant on a mobile smartphone.

VA knowledge base

The knowledge base was structured using a built-in parser and retriever service in Voiceflow³⁹ (a platform used to build our VA), leveraging a Retrieval-Augmented Generation (RAG) approach.⁴⁸ Voiceflow's parser service processes uploaded documents into smaller semantic chunks, converts them into vector embeddings, and integrates them into the knowledge base for efficient retrieval.⁴⁹ The retriever service converts user queries into vectors, compares them with the knowledge base, and returns the most relevant chunks for accurate, context-aware responses.⁴⁸ Eventually, an LLM (GPT-3.5 Turbo) synthesized retrieved chunks into cohesive, conversational responses, enabling the VA to deliver intuitive, engaging, and human-like interactions tailored to the informal tone users expect in text-based communication.

Sample selection

Purposive sampling was employed at a large pediatric academic medical center to recruit caregivers of YCCS who had formerly participated in a stakeholder analysis (n = 29) and/or were currently involved in a Community Advisory Board (n = 13; ∼40% rural or Appalachian) from a larger study. These groups involved caregivers of multiple types (biological mothers and fathers, grandparents, adoptive parents, and legal guardians) who had YCCS currently between the ages of 3 and 12 years, at least 6 months post-treatment or were on maintenance therapies, and without evidence of recurrent disease. We aimed to enroll 10 caregivers, which was in line with user experience studies in the literature.^50,51 Twenty caregivers were contacted to participate in the study, with half agreeing to participate. Three declined, while the other 7 were unreachable prior to us meeting our goal of 10 participants. Eligible caregivers were defined as: (1) legal guardians, (2) living with their children ≥50% of the time, (3) residing in the surrounding states, and (4) English-speaking. Caregivers who participated in interviews were compensated with a $50 gift card.

Data collection

Caregivers were recruited by trained research coordinators via phone. They viewed an online information sheet and proceeded with surveys to indicate their consent to participate. We used a semi-structured qualitative interview methodology designed to capture insights from caregivers (see Appendix B for Interview Protocol). Ten participants were given the choice to participate in a single-session focus group (n = 7) or one-on-one interviews (n = 3) with research staff via Microsoft Teams. The procedure began with an introduction to VA technology currently in use in daily life. Concepts related to popular market products (i.e., Siri from Apple, Google Assistant, and Alexa from Amazon) were briefly described based on their similar functionalities and potential. Study staff then asked caregivers to describe how they use VA-based technologies in their daily life.

Following the introduction, each caregiver interacted with a prototype VA for a set duration within the interview session. This phase included previewing four scenarios about late effects of cancer treatment, returning to school, reviewing cancer history, and handling the emotional impact on the child. This process guided caregivers’ engagement and helped them explore the VA's features and functions thoroughly. Caregivers were also given the opportunity to ask their own questions indirectly (queries monitored by study staff) to the prototype VA using voice or text. Finally, caregivers were prompted to express their thoughts about how a VA similar to the prototype could assist in their role as a caregiver to a YCCS.

After the engagement period, study staff solicited caregiver feedback on VA usability, utility, and feasibility. Focus groups and interviews lasted between 30 and 60 minutes (M = 42, SD = 13). All interactions were audio-recorded on Microsoft Teams with the caregivers’ consent and imported to NVivo. Research staff took observational notes during the interview sessions. Additionally, qualitative work incorporated Manning's In Vivo⁵² coding of transcription data through NVivo on release version 14.23.2. Transcripts and thematic analysis were not shared with them for review; and thus, accurate interpretation of comments was primarily confirmed through a brief verbal summary of the content of each focus group/interview. Recruitment and data collection for the study concluded after achieving data saturation. The research team determined that the current sample size provided rich and meaningful data for analysis. This approach aligns with the qualitative research in the literature.⁵³

Data analysis

Qualitative data was analyzed through a thematic analysis following Braun and Clarke's⁵⁴ six-step process. Each dialogue file was cleaned, and transcription data was organized into an NVivo file. The conversations were divided into sentence segments, summarizing groups of sentences with code words derived from the dialogue. Co-authors (ES, DIJ) reviewed and coded the transcripts (Cohen's kappa, κ = 0.862). Conflicts were resolved through consensus discussion among authors. Subsequently, codes were organized into overarching themes. Quantitative data was analyzed with descriptive statistics using R version 4.3.0 (2023-04-21 Universal C Runtime)⁵⁵ through RStudio 2023.12.1 Ocean Storm Build 402.⁵⁶ Additionally, we conducted post study analysis on VA engagement logs to assess quality of responses overall (see Appendix C for details).

Results

Demographics

The study sample consisted of caregivers of YCCS (n = 10) who were predominantly female (n = 9) with one male participant (n = 1). In terms of relationship status, most were married (n = 8), with one single caregiver (n = 1) and another cohabitating (n = 1). Educational attainment was high, with caregivers having at least a technical/trade school education (n = 1), college degree (n = 7), or graduate school education (n = 1). One participant had a high school degree only (n = 1). Annually, family income varied, with the highest earners making over $100,000 (n = 3), followed by a few in the $75,001–$100,000 (n = 2) and $50,001–$75,000 (n = 3) ranges, and a couple earning between $25,000 and $50,000 (n = 1) or less than $25,001 (n = 1). Most caregivers self-identified as White (n = 9), with one participant identifying as Black (n = 1). Additionally, one participant endorsed having a Hispanic/Latino ethnicity (n = 1). Child diagnoses were evenly distributed among leukemia/lymphoma (n = 3), non-central nervous system solid tumors (n = 4), and brain tumors (n = 3). Child patients were distributed evenly since time of diagnosis (n = 10 children, min = 2.17 years, max = 9.75 years, M = 5.97 years, SD = 2.80 years). Over half of children received one or more central nervous system-directed treatment (n = 6), while the rest did not (n = 4). Residential areas varied, with caregivers residing in metropolitan (n = 4), rural (n = 3), Appalachian (n = 2), and rural-Appalachian (n = 1) regions.

Themes

Thematic analysis resulted in 22 codes and four themes: Interface and Interaction, User Experience, Content Relevance, and Trust (Table 1). Interface and Interaction describes the different ways users preferred to access a VA, such as, through speech-only, text, or both. User Experience encompasses experiences and feelings about caregiving and seeking medical advice. Content Relevance describes the topics of value to caregivers and the accuracy of VA responses, such as, neurodevelopmental therapies, medication side effects, and educational resources. Trust describes security measures and the preference for confidentiality when handling VA data.

Table 1.

Themes and codes.

Theme	Definition	Code	References	Frequency (%)
Interface and Interaction	This theme describes the ways, preferences, and modalities that users prefer to interact with or access to a VA.	Interaction modes	7	4.7
		Verbal interaction	5	3.4
		Vocal preferences	4	2.7
		Linguistic support	4	2.7
		Feature set	12	8.1
		Personalization	9	6.1
User Experience	This theme describes personal experience about caregiving and seeking medical advice via the technology.	Caregiving experience	8	5.4
		Search frustration	4	2.7
		Caregiving challenges	8	5.4
		Guidance support	4	2.7
		Alternative sources	5	3.4
Content Relevance	This theme describes the content that is relevant, valuable, and needed by caregivers from VAs.	Response specificity	19	12.8
		Inquiry types	16	10.8
		Oncology focus	7	4.7
		Educational queries	4	2.7
		Therapeutic queries	4	2.7
		Medical treatments	5	3.4
Trust	This theme describes the perception of trust, security, and ethics and the preferences for confidentiality expected from VAs.	Data security concerns	4	2.7
		Source credibility	4	2.7
		Credibility assessment	6	4.1
		Ethical applications	5	3.4
		Usage contexts	4	2.7
		Total	148	100.0

Interface and interaction

Caregivers explained the technologies that they used in daily life, and their preferences for accessing and using VAs. They outlined that they have already used mHealth apps, search engines, and smart technologies to navigate the landscape of healthcare information, from connecting with their oncology team to managing a treatment plan long term. For some caregivers, VAs were a way to stay on top of daily life.

Smart phones, you know the [smart speaker brand], the smart watches, all of that. I think helps [us] to keep a little organized. Caregiver #10

Caregivers also compared voice versus text-only interactions and highlighted that the modality of conversations mattered based on their location and situation. One caregiver described a scenario where voice interaction may not be appropriate for after-hours and text-based interaction could be more appropriate.

When your child is ill and they’re finally asleep, you don’t wanna do anything to wake them up [so] you’re not gonna use voice. Caregiver #7

Caregivers acknowledged how VAs may handle administrative tasks well, such as reminders and scheduling, but struggle with extended conversations about medical information and questions. Caregivers desired to use VAs with a memory component (i.e., a knowledge base) for recalling past topics or exploring options at their healthcare center.

What if I forgot what I had asked it, and I’m like, ‘Oh, there was some really good information there?’ Caregiver #1

There was something there that I wanted to revisit. So, maybe a search history within the [VA] app itself might be something valuable. Caregiver #3

Caregivers indicated that VAs could be integrated to the health system and referred by their clinicians for personalized oncological questions and responses. Many caregivers recognized how VAs for healthcare are typically on the landing page of a pediatric oncology clinic's website to support navigating treatment services. Instead of using generic website VAs, several caregivers wanted to download an application to engage in further personal care questions, which was perceived to increase their access to information and convenience. Similarly, most caregivers noted the value of multi-language personal health education options. They indicated that most informational materials were printed and distributed in English-only. One caregiver also mentioned having a VA feature that could educate users on common medical terminology.

When you have a doctor's visit, and they say, ‘oh, we’ve got this VA. Download the app, and you can ask your questions right there’. Caregiver #3

Could I ask it like a dictionary? Would you be able to ask things like, ‘what does hemoglobin mean?’ It would be great if I could. Caregiver #5

Unfortunately, there's a very large portion of families that like, their ability to read is limited. So, if you can read back to them, you’re catching a whole other demographic that you’re missing when you provide pamphlets and stuff. Caregiver #6

User experience

Most caregivers expressed that seeking health information has been challenging. Caregivers mentioned concern about “going down the rabbit hole” where they became stuck endlessly searching online. Many caregivers agreed that a major obstacle to accessing quality health information online was their difficulty finding useful medical advice specific to their child's condition. Regardless of their backgrounds, caregivers were appreciative of the wealth of information that exists about oncology, yet acknowledged the limitations of navigating to that information and understanding it based on health literacy.

We are given so much stuff and you just can’t keep track of everything. Caregiver #1

[Do] you know what I mean? The medical terms that sometimes you’re given, and you have no clue, you know, what they’re talking about. Caregiver #5

Furthermore, caregivers recalled moments where they had to use alternative information sources (i.e., healthcare team); they would either take their concerns to their peer groups, or go through search engine results, acknowledging the risk of misinformation. Caregivers identified this cycle as a high contributor to caregiver burden and exhaustion. Alternatively, some caregivers were proactive with their network and contacted other healthcare providers for extra advice that is often not included in informational handouts and flyers.

If Googling [an oncology symptom and diagnosis], there's so much stuff out there… You don’t know what's gonna come up! Caregiver #2

As an oncology parent, if we have a quick question or we kind of need that ‘college of knowledge,’ we go to the [social media] page and get 50 answers right away. Caregiver #10

I would try to find a couple nurses on the oncology floor saying, ‘Hey, you got a minute?’ Caregiver #1

It's not just the doctors, but a lot of times the nurses have a lot more information to share about the day-to-day life. Caregiver #4

Caregivers summarized their needs from VA to access reliable answers quickly while filtering out less credible resources. Caregivers indicated VAs may reduce or eliminate both extended searching sessions and the burden paired with it in the future.

This [the VA] would be a good resource for that tired, exhausted parent who's at their wit's end and just needs something. Caregiver #7

Content relevance

Caregivers expect VAs and other conversational technologies to deliver niche and highly accurate information. After interacting with the VA prototype, caregivers listed common topics of interest, such as educational and therapeutic information (e.g., learning more about speech therapy or special education options) to address neurodevelopmental delays and late effects, as well as treatment-based instructions. While testing the prototype VA, caregivers reacted to the breadth of information it could cover, such as the use of therapeutics and mental health resources.

Like, that's a start. That is something where- you could take ‘that’ and go back to your physician and say… ‘Do we need occupational therapy to help them hold a pencil?’ Caregiver #1

Services for mental health of my pediatric cancer patient could be catered to provide information that [my] hospital would offer. Caregiver #10

You could ask it like, I’m driving from out of town, [are] there resources for me to stay [nearby]… while my child receives treatment. Caregiver #6

Caregivers expressed their use of VA with clarifying questions and scenario-based explanations. Some caregivers needed summaries, while others explored alternative interpretations of their doctor's orders.

…instead of having to cycle through the various pages to my messages or to my lab results, if I [could] just ask the question; ‘Hey, what was the summary of [patient's] last [diagnostic procedure]?’ Caregiver #4

Caregivers also referred to the need for evidence-based content and the VA's ability to access credible resources, such as safe dosage instructions. They indicated that this information could be useful to determine when they should reach out to their oncology team:

I would just say one of the first things that popped in my head when it comes to questions was definitely medication. I just think that would be, like very helpful to have that information here. Caregiver #9

[The VA] made me think about how sometimes we get packets of information, and we don’t have the time to read through it. It would be great to have something like this to ask a question and get an answer right away. Caregiver #8

It would be a great tool that people could use when it's not necessarily worthy of calling a doctor, but they just need that additional resource. Caregiver #7

Trust

Caregivers expressed that they expected their VAs to access credible sources and maintain confidentiality. Caregivers identified that it could be helpful to have a healthcare provider connected to the VA to monitor questions asked and provide urgent intervention when needed, rather than receiving strictly VA-based feedback. Caregivers also expressed that connecting a VA to a trusted provider or hospital system could also help them to trust the VA system and place more value on its answers. Finally, they expressed a potential point of frustration if the VA's response or health advice comes from advertised or unreliable webpages:

If you scroll at the very top… [reads opening message of prototype VA app], that helped me realize that they know this hospital and they’re aware of the resources that are available there… Caregiver #6

…let them know like these are from trusted places; websites or facilities, studies… I think that will be very helpful because then it will put you guys separate from a general place we will go for other questions. Caregiver #9

As long as they [VA answers] were [using] reliable sources, you know, like cancer organizations or making sure it's like an org or something, or like the hospital websites versus… not like a [blog] answer. Caregiver #7

In line with that, endorsement or ownership of a VA by known health institutions was perceived to be a trust building indicator toward adopting, building trust, and using VA for personal health. Specifically, caregivers indicated the importance of displaying affiliations, logos, and titles from established organizations and healthcare systems.

When you’re using this type of medium and you’re putting [hospital logo] on here, I’m going to immediately trust this as a reputable resource for my family's medical inquiries. Caregiver #4

If I have a dedicated resource that I know I trust by putting that name [points at hospital name & logo] on there, I think it [the VA] can go a long way. Caregiver #8

In terms of security, caregivers would rather log into an app or service than use publicly available VAs. Caregivers wanted the flexibility to discuss health-related problems without having to worry about how the conversational data is handled, highlighting the need for a trustworthy VA system.

This [could] be like a secure kind of app that you could just like, log into once and then [use], versus [having] to log into [patient's electronic health record portal]. Caregiver #10

Caregivers also expressed that VAs need to use ethical practices, providing disclosures on its capabilities and knowledge expertise at the start of a conversation, and also indicating when a question warrants medical, rather than VA, consultation. Beyond low-risk question, such as benign symptoms, VAs were expected to clearly direct users when noticed as urgent cases. Additionally, some caregivers indicated that VAs could build trust via conversations, particularly in how it responds and guides high-risk cases:

[My child] had bad cramps in [their] feet and legs. My first thought was, ‘Oh, is the cancer back?’ because it started from a limb. [sighs] I googled it and found it was actually very normal; they’re growing pains, so I just monitored it and know when to reach out to my physician. Caregiver #1

‘When should I consult my doctor about a fever?’ And then it could respond, ‘if the fever is 100.4 or more, you should contact your physician immediately’. Caregiver #10

Minor themes

We observed slight differences in responses based on geographic residence (metropolitan, rural/Appalachian, rural, Appalachian). For instance, caregivers from metropolitan areas focused mostly on access to information, documentation, and tracking treatment symptoms via technology. Caregivers from rural or Appalachian regions focused mostly on the convenience and accessibility side of technology. We also observed that caregivers from rural and Appalachian caregivers expressed more reliance on technology as a learning tool as compared to those from Metropolitan areas to facilitate communication with their healthcare team.

Overall, the qualitative findings suggest that the strengths of VAs include convenience, accessibility, and delivering relevant, niche information. Caregivers appreciate tools like reminders, quick retrieval of credible answers, multi-modal interaction (voice and text), and features like search history. Personalized responses, multi-language support, and affiliation with trusted healthcare institutions build trust and reduce search fatigue. However, the weaknesses include limited ability to handle complex conversations, potential reliance on unverified sources, and privacy concerns. Furthermore, difficulty simplifying medical terms, impracticality in certain contexts, like quiet environments, and the lack of the empathetic responses were also noted as weaknesses.

VA engagement analysis

We analyzed participant interaction and engagement data (VA responses) to evaluate response readability and correctness. The VA responses (14 interactions or session, and in total 24 responses) were found readable at a high school level and contained minimal complex terminology.VA interactions were brief, with users rarely asking follow-up questions. Most responses were accurate (87.5%, 21/24), with a low hallucination rate (12.5%, 3/24). Notably, inaccuracies occurred in mathematical reasoning and legal guidance, reflecting limitations of LLMs. Full details, including measures and examples, are provided in Appendix C.

Discussion

Limited access to early intervention and related information for caregivers, particularly in rural and Appalachian areas, exacerbates recovery processes that could otherwise be mitigated with timely treatment. We investigated VA engagement with caregivers to explore the technology's potential to bridge the knowledge accessibility gap while supporting marginalized communities. More specifically, this study provides critical insights into the perceptions of caregivers of YCCS regarding the use of VAs for health information seeking and care management. Overall, the findings further emphasize the potential of VAs to meet the information needs of these caregivers, particularly in underserved communities, by offering a convenient, personalized, and reliable source of support.

A common challenge expressed by caregivers was difficulty seeking and accessing reliable health information online, often experiencing frustration and exhaustion from extensive searching. Such an experience has other caregivers and patients using digital health technologies (e.g., mHealth apps, patient portals).⁵⁷ Furthermore, difficulties navigating online health information are well-documented.⁵⁸ In our study, caregivers valued the potential of VAs to provide quick, and ideally reliable answers, thereby reducing their cognitive and emotional burden. Still, in the current state of commercial VAs, the literature reports mixed evidence on their utility and reliability.⁵⁹ Misinformation has been a risk factor, especially for online health information seeking.^60,61 Therefore, the role of VAs in alleviating these challenges by filtering out less credible sources and providing immediate responses, provides an efficient means to access accurate and evidence-based information.⁶² With a healthcare information database for pediatric oncology, which we developed as a proof-of-concept for this study, specialized VAs may have a role in the digital health ecosystem to improve the efficiency and effectiveness of health information seeking for caregivers.^63,64

Different caregiver preferences regarding interaction with VAs (including speech-only, text-only, and a combination of both modalities) highlighted the expected flexibility from VAs to adapt to different contexts, such as using voice commands when hands-free interaction is needed or text-based interaction during quiet times (e.g., when the patient is resting). Adaptive modality to engage with digital health tools has been a common theme among caregivers of children with special healthcare needs.^65,66 The ability of VAs to switch between modalities based on user context may further contribute to user experience and satisfaction.^67,68 Therefore, the capabilities of VAs outlined in our study suggest the need for intelligent systems that support continuity in information seeking. To fit within the digital ecosystem of caregivers, such preferences should be integrated into VAs for care support. This finding is consistent with other populations with healthcare needs, who used conversational agents in patient care and remote care.⁶⁹

In line with caregiver preferences, the importance of personalization was highlighted, such as receiving specific, evidence-based content from VAs, particularly related to educational, therapeutic, neurodevelopmental, and treatment-based information. Earlier studies show alignment with our findings with other digital health technologies, such as mobile apps for cancer education, therapeutics, and neurodevelopmental interventions.^70,71 We found that these needs remain similar to those of other VAs. However, more personalization is required given the desire for highly specific and reliable health information, similar to studies on digital health interventions.^72,73

Recent advancements in artificial intelligence and LLMs indicate an increasing ability for VAs to provide tailored information based on individual queries, which might be instrumental to address core needs for personalization.⁷⁴ This technology also introduces risks to information delivery, including misinformation, misguidance, and hallucinations, which can reduce the accuracy and perceived trust to the technology (please see Appendix C for post-study VA response analysis). To mitigate such cases, we may further employ: (1) ensemble models,⁷⁵ which combine outputs from multiple specialized models or retrieval methods to ensure diverse perspectives and consensus-based accuracy; (2) context validation⁷⁶ approaches to cross-check retrieved information against the user query to confirm relevance before synthesis; (3) fallback mechanisms⁷⁷ to handle low-confidence scenarios by prompting clarification or acknowledging knowledge gaps; (4) a user feedback loop⁷⁸ allows users to flag inaccuracies, and training on these flagged cases enables the system to iteratively improve its performance over time.⁷⁹

Similar to user perceptions toward other digital health tools, trust and confidentiality has been an emerging concern, and therefore, it has been crucial to caregivers’ adoption of VAs.^80,81 Caregivers expressed a preference for secure and confidential interactions, ideally through a dedicated system, which is consistent with literature on the use of patient portals^82,83 and health information technology.⁸⁴ Moreover, the trust factor emerged as a multifaceted construct that identified expectations including data security, source credibility, and ethical implementations.^85,86 Therefore, trust in VA may require further investigation.

Considering the residential differences, the literature suggests a potential disparity in digital health technology access and use among rural and Appalachian populations.^87–89 Specifically, caregivers that live closer to major healthcare systems in metropolitan areas may have more opportunities to learn and organize general knowledge about oncology directly from their oncology specialists than those in rural and Appalachian regions who may have less access to specialized healthcare.⁹⁰ In our study with a limited sample size, we observed minimal differences among groups in their health-seeking attitudes and preferences for interacting with a VA. This suggests the need for further investigation of both groups to understand the needs and expectations of VA in patient care and how those might differ based on access to care.

Clinical implications

Our study suggests that caregivers of YCCS are largely open-minded and optimistic about the potential utility of VAs as they seek information and resources about their child's oncology care, including treatment options and management of emerging late effects. Caregivers suggested in retrospect that VAs may have been useful when their child was in active treatment to identify common side effects of medications or to track symptoms over time. Therefore, VAs might be an effective clinical communication tool for providers.²⁹ With future refinement and clinical validation, VAs might also facilitate early detection of late effects, such as prompting caregivers to track developmental milestones, which could then be flagged based on guidelines for further clinical assessment by a child's healthcare provider. If medical or developmental needs are identified, VAs could also assist in streamlining referrals to relevant local resources based on where the family lives and waitlists, which may reduce provider and family burden. These advances could be particularly important in the context of families with limited access to integrated psychosocial and developmental care in less resourced areas, yet ongoing barriers to implementation, as previously identified, include liability, regulation, security, and privacy for VA and other technology solutions.

Practical and ethical implications

Underserved communities’ access to essential technology and infrastructure (e.g., Wi-Fi, cellular hotspots, and other information technologies) remains inequitable.⁹¹ To ensure VAs benefit all users, it is crucial to address these disparities. Integrating the knowledge and familiarity of underserved communities with VA functionality will present challenges during implementation. This requires a strategic approach to ensure adoption, accessibility, and maintenance, regardless of personal or educational backgrounds. For instance, involving community partners in the design process and incorporating culturally relevant features can help foster trust and encourage sustained use.⁹²

Ethical considerations for VAs as digital health tools demand careful attention. Beyond building trust, caregivers may place excessive trust in VA guidance without consulting medical professionals, amplifying the risk of misinformation, particularly in underserved communities. As aforementioned, the potential for “hallucinations” or inaccuracies in VAs could worsen healthcare disparities and erode trust. Encountering such issues may deter users, especially in communities with a history of distrust in healthcare systems. Promoting transparency about how VAs process information, highlighting their limitations, and providing user education are essential strategies for building trust and mitigating these risks.⁹³

The responsibility to design and deploy digital health tools equitably lies with developers, policymakers, care teams, and healthcare providers. They must verify health guidance accuracy and address community-specific concerns to build trust.²⁴ Caregivers may inadvertently share personal health information with VAs, so conversations must be securely stored with explicit user consent. Robust data protection measures, including end-to-end encryption and privacy-by-design principles, are necessary to safeguard user information. Developers should also offer tools for users to manage their data, such as deleting or anonymizing information as needed. Furthermore, caregivers may over-rely on VAs in critical healthcare scenarios, underscoring the need for fallback systems. These systems should include clear escalation protocols, such as connecting users to emergency services or human healthcare providers, to address the VA's limitations in handling urgent or complex situations.⁹⁴

Limitations

While this study provides insights into the perceptions of caregivers, several limitations must be acknowledged. One notable limitation is the potential bias in our sample. The caregivers who agreed to participate were part of an ongoing virtual Community Advisory Board (CAB) and may already be technologically inclined, which could skew the results toward a more favorable view of VAs. None of the caregivers expressed strong negative perceptions of VAs, which might not fully represent the broader population of caregivers who may be less familiar or uncomfortable with technology. Additionally, the sample included few caregivers from racially and ethnically marginalized backgrounds, limiting the generalizability of our findings across diverse populations. The study also focused solely on caregivers. Future work should gather healthcare provider perspectives regarding the utilization of VAs and directly test the clinical accuracy of the information provided with VAs in the pediatric oncology context. While qualitative methods are effective for exploring in-depth perceptions and experiences, they do not provide the same level of generalizability as quantitative or mixed-method approaches. Furthermore, the potential for social desirability bias exists, as participants may have provided responses they believed were expected or favorable rather than their true feelings and experiences. The interaction period with the VA prototype was relatively short, which may have limited the depth of insights gained regarding long-term use and utility. Future work with extended interactions over a longer period could provide more detailed information on the practicality and effectiveness of VAs in supporting caregivers.

We presented preliminary evidence of the utility of VAs in supporting caregivers, and a preliminary understanding of the behaviors and needs of caregivers. These findings are relevant to policymakers and healthcare practitioners, enabling them to make informed decisions while incorporating VAs into health communication strategies, and understanding how perceptions might differ in these communities. This effort can lead to the effective tailoring of existing public health programs to integrate digital platforms. The preliminary findings from this study will further serve as a critical basis for future research, development, and pilot testing of VA-based systems designed to meet the specialized needs of families of pediatric cancer patients and other clinical and chronic conditions.

Conclusions

This study highlights the potential of VAs to support caregivers of YCCS, particularly in underserved communities, by providing personalized and accessible health information. To improve VA systems, developers should focus on addressing known limitations, such as vague responses and limited capacity for handling critical scenarios, through enhancements like fallback mechanisms, user feedback loops, and adaptive multimodal engagement. Strengthening privacy and security measures, including end-to-end encryption and robust consent processes, is crucial for safeguarding user information. Future research should involve large-scale validation and long-term studies to assess the sustained impact of VAs on caregiver well-being and patient outcomes. Expanding research to include diverse populations and evaluating VA integration into clinical workflows will further ensure these tools effectively address the needs of all users equitably.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076251326160 - Supplemental material for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors

Supplemental material, sj-pdf-1-dhj-10.1177_20552076251326160 for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors by Emre Sezgin, Daniel I. Jackson, Kate Kaufman, Micah A. Skeens, Cynthia A. Gerhardt and Emily Moscato in DIGITAL HEALTH

Supplemental Material

sj-pdf-2-dhj-10.1177_20552076251326160 - Supplemental material for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors

Supplemental material, sj-pdf-2-dhj-10.1177_20552076251326160 for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors by Emre Sezgin, Daniel I. Jackson, Kate Kaufman, Micah A. Skeens, Cynthia A. Gerhardt and Emily Moscato in DIGITAL HEALTH

Supplemental Material

sj-pdf-3-dhj-10.1177_20552076251326160 - Supplemental material for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors

Supplemental material, sj-pdf-3-dhj-10.1177_20552076251326160 for Perceptions about the use of virtual assistants for seeking health information among caregivers of young childhood cancer survivors by Emre Sezgin, Daniel I. Jackson, Kate Kaufman, Micah A. Skeens, Cynthia A. Gerhardt and Emily Moscato in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors acknowledge Eva Darow for her support in study recruitment and data collection.

Author contributions

Emre Sezgin and Emily Moscato contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Emre Sezgin, Emily Moscato, Kate Kaufman, and Daniel I. Jackson. Micah A. Skeens and Cynthia A. Gerhardt supervised the study and contributed to manuscript. The first draft of the manuscript was written by Emre Sezgin and Daniel I. Jackson and all authors significantly contributed to the manuscript. All authors read and approved the final manuscript.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This study was funded by the intramural funding program of the Neurodevelopmental Research Affinity Group (NRAG) at Nationwide Children's Hospital. The Caregiver Advisory Board formation was supported by the American Psychological Association Division 54 Drotar-Crawford Postdoctoral Research Grant (Emily L. Moscato, Mentor: Cynthia A. Gerhardt).

ORCID iDs

Emre Sezgin

Micah A. Skeens

Supplemental material

Supplemental material for this article is available online.

References

Ostrom

Gittleman

, et al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2009–2013. Neuro Oncol 2016; 18: v1–v75.

Phillips

Padgett

Leisenring

, et al. Survivors of childhood cancer in the United States: prevalence and burden of morbidity. Cancer Epidemiol Biomarkers Prev 2015; 24: 653–663.

Racine

Smith

Pelletier

, et al. Evaluation of a support group for parents of children hospitalized for cancer and hematopoietic stem cell transplantation. Soc Work Groups 2018; 41: 276–290.

Sun

C-L

Francisco

Kawashima

, et al. Prevalence and predictors of chronic health conditions after hematopoietic cell transplantation: a report from the Bone Marrow Transplant Survivor Study. Blood 2010; 116: 3129–3139. quiz 3377.

Wiener

Kazak

Noll

, et al. Standards for the psychosocial care of children with cancer and their families: an Introduction to the special issue. Pediatr Blood Cancer 2015; 62: S419–S424.

National Academies of Sciences, Engineering, and Medicine, Institute of Medicine, Board on Health Care Services, et al. Comprehensive cancer care for children and their families: summary of a joint workshop by the Institute of Medicine and the American Cancer Society. National Academies Press. Available at: https://play.google.com/store/books/details?id=9hmbCgAAQBAJ (2015).

Marusak

Iadipaolo

Harper

, et al. Neurodevelopmental consequences of pediatric cancer and its treatment: applying an early adversity framework to understanding cognitive, behavioral, and emotional outcomes. Neuropsychol Rev 2018; 28: 123–175.

Willard

Cox

Russell

, et al. Cognitive and psychosocial functioning of preschool-aged children with cancer. J Dev Behav Pediatr 2017; 38: 638–645.

Grieco

Evans

Yock

, et al. Early psychosocial and executive functioning outcomes in pediatric brain tumor survivors after proton radiation. Authorea . DOI: 10.22541/au.165570670.02539759/v1. Epub ahead of print June 20, 2022.

10.

Jacola

Partanen

Lemiere

, et al. Assessment and monitoring of neurocognitive function in pediatric cancer. J Clin Oncol 2021; 39: 1696–1704.

11.

Cox

Kenney

Harman

, et al. Psychosocial functioning of young children treated for cancer: findings from a clinical sample. J Pediatr Oncol Nurs 2019; 36: 17–23.

12.

National Academies of Sciences, Engineering, and Medicine, Health and Medicine Division, Board on Health Care Services, et al. Long-term survivorship care after cancer treatment: proceedings of a workshop. National Academies Press. Available at: https://play.google.com/store/books/details?id=u-BjDwAAQBAJ (2018).

13.

Harman

Wise

Willard

. Early intervention for infants and toddlers: applications for pediatric oncology. Pediatr Blood Cancer 2018; 65. DOI: 10.1002/pbc.26921. Epub ahead of print May 2018.

14.

Yao

Alcalá

Anderson

, et al. Cancer disparities in rural Appalachia: incidence, early detection, and survivorship. J Rural Health 2017; 33: 375–381.

15.

Fish

Pinkerman

. Language skills in low-SES rural Appalachian children: normative development and individual differences, infancy to preschool. J Appl Dev Psychol 2003; 23: 539–565.

16.

Rhoad-Drogalis

Justice

. Absenteeism in Appalachian preschool classrooms and children’s academic achievement. J Appl Dev Psychol 2018; 58: 1–8.

17.

Shorey

Lau

LST

Tan

, et al. Families with children with neurodevelopmental disorders during COVID-19: a scoping review. J Pediatr Psychol 2021; 46: 514–525.

18.

Lambert

Loiselle

. Health information seeking behavior. Qual Health Res 2007; 17: 1006–1019.

19.

Bright

Fleisher

Thomsen

, et al. Exploring e-Health usage and interest among cancer information service users: the need for personalized interactions and multiple channels remains. J Health Commun 2005; 10: 35–52.

20.

Luger

Moran

Rodden

. Consent for all: revealing the hidden complexity of terms and conditions. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 2687–2696. New York, NY, USA: ACM.

21.

Myers

Furqan

Nebolsky

, et al. Patterns for how users overcome obstacles in voice user interfaces. In: Proceedings of the 2018 CHI conference on human factors in computing systems. New York, NY: ACM. Epub ahead of print April 19, 2018. DOI: 10.1145/3173574.3173580.

22.

Yang

Lee

Sezgin

, et al. Clinical advice by voice assistants on postpartum depression: cross-sectional investigation using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana. JMIR Mhealth Uhealth 2021; 9: e24045.

23.

Sezgin

Militello

Huang

, et al. A scoping review of patient-facing, behavioral health interventions with voice assistant technology targeting self-management and healthy lifestyle behaviors. Transl Behav Med 2020; 10: 606–628.

24.

Sezgin

Huang

Ramtekkar

, et al. Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic. NPJ Digit Med 2020; 3: 122.

25.

Amershi

Cakmak

Knox

, et al. Power to the people: the role of humans in interactive machine learning. AIMag 2014; 35: 105–120.

26.

Brewer

Findlater

Kaye

, et al. Accessible voice interfaces. In: Companion of the 2018 ACM conference on computer supported cooperative work and social computing. New York, NY, USA: ACM. Epub ahead of print October 30, 2018. DOI: 10.1145/3272973.3273006.

27.

Nobles

Leas

Caputi

, et al. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. NPJ Digit Med 2020; 3: 11.

28.

Wagner

Churchill

Sigtia

, et al. A multimodal approach to device-directed speech detection with large language models. In: ICASSP 2024–2024 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.10451–10455. IEEE.

29.

Sezgin

. Redefining virtual assistants in health care: the future with large language models. J Med Internet Res 2024; 26: e53225.

30.

Han

Nebelung

Khader

, et al. Medical large language models are susceptible to targeted misinformation attacks. NPJ Digit Med 2024; 7: 288.

31.

Vaidyam

Wisniewski

Halamka

, et al. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry 2019; 64: 456–464.

32.

Sanders

, et al. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR Cancer 2021; 7: e27850.

33.

Chaix

Bibault

J-E

Pienkowski

, et al. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019; 5: e12856.

34.

Sezgin

Hussain

S-A

Rust

, et al. Extracting medical information from free-text and unstructured patient-generated health data using natural language processing methods: feasibility study with real-world data. JMIR Form Res 2023; 7: e43014.

35.

Anderson

Rainie

. The internet of things will thrive by 2025. Pew Research Center [Internet & American Life Project]. Available at: https://play.google.com/store/books/details?id=l46toAEACAAJ (2014).

36.

Catania

Spitale

Garzotto

. Conversational agents in therapeutic interventions for neurodevelopmental disorders: a survey. ACM Comput Surv 2023; 55: 1–34.

37.

Creswell

. Research design. 6th ed. Christchurch, New Zealand: Sage Publications, 2022.

38.

Tong

Sainsbury

Craig

. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357.

39.

Voiceflow, Inc. Build and deploy AI customer experiences. Available at: https://www.voiceflow.com/ (2018, accessed 24 July, 2024).

40.

Lisa Hickman

. OCECD publications. Ohio coalition for the education of children with disabilities, https://www.ocecd.org/Publications1.aspx (2024, accessed 24 July, 2024).

41.

Buff

Paciente

Zengel

, et al. Peer support matters. Momcology. Available at: https://momcology.org-stg.gmheventweb.com/peer-support-matters/ (2018, accessed 24 July, 2024).

42.

The Ohio Department of Education & Workforce. Sitemap. The Ohio Department of Education & Workforce. Available at: https://education.ohio.gov/Miscellaneous/Sitemap (2023, accessed 24 July 2024).

43.

The Ohio Developmental Disabilities Council. Resources and publications. The Ohio Developmental Disabilities Council. Available at: https://ddc.ohio.gov/resources-and-publications (2024, accessed 24 July 2024).

44.

U.S. Department of Education. Individuals with Disabilities Education Act (IDEA). U.S. Department of Education. Available at: https://sites.ed.gov/idea/ (2017, accessed 24 July 2024).

45.

St. Jude Hospital. Medicines used to treat pediatric cancer. St. Jude together. Available at: https://together.stjude.org/en-us/diagnosis-treatment/medicines-list.html (2023, accessed 24 July 2024).

46.

Cancer Resources from OncoLink, Treatment, Research, Coping, Clinical Trials, Prevention. OncoLink Tools in PennChart. Available at: https://www.oncolink.org/oncolink-tools-in-pennchart (2023, accessed 24 July 2024).

47.

Children’s Oncology Group. Children’s Oncology Group Family Handbook for children with cancer. 2nd ed. St. Baldrick’s Foundation. Available at: https://childrensoncologygroup.org/docs/default-source/pdf/english_cog_family_handbook.pdf?sfvrsn=abd6b5f5_2 (2023).

48.

Voiceflow, Inc. How does the knowledge base work. Voiceflow, Inc. Available at: https://docs.voiceflow.com/docs/how-does-the-knowledge-base-work (2024, accessed 16 December 2024).

49.

Lewis

Perez

Piktus

, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv [cs.CL]. Available at: http://arxiv.org/abs/2005.11401 (2020, accessed 16 December 2024).

50.

Motta

Groves

Schneider

, et al. Designing self-tracking experiences: a qualitative study of the perceptions of barriers and facilitators to adopting digital health technology for automatic urine analysis at home. PLoS Digit Health 2023; 2: e0000319.

51.

Tilley

McKeon

Ibrahim

, et al. A snapshot on a journey from frustration to readiness—a qualitative pre-implementation exploration of readiness for technology adoption in public health protection in Ireland. PLoS Digit Health 2024; 3: e0000453.

52.

Manning

. Vivo coding. In: Matthes

Davis

Potter

(eds) The international encyclopedia of communication research methods. Hoboken, NJ: Wiley, 2017, pp.1–2.

53.

Braun

Clarke

. To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qual Res Sport Exerc Health 2021; 13: 201–216.

54.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

55.

R Foundation. Index of/src/base/R-4. Available at: https://cran.r-project.org/src/base/R-4/ (2023, accessed 29 October 2024).

56.

Posit Software. RStudio releases. Available at: https://dailies.rstudio.com/release/ (2023, accessed 29 October 2024).

57.

Choukou

M-A

Olatoye

Urbanowski

, et al. Digital health technology to support health care professionals and family caregivers caring for patients with cognitive impairment: scoping review. JMIR Ment Health 2023; 10: e40330.

58.

Strand

Westergren

. A meta-ethnography on parents’ experiences of the Internet as a source of health information. Glob Qual Nurs Res 2024; 11. DOI: 10.1177/23333936241259246. Epub ahead of print January 2024.

59.

Alagha

Helbing

. Evaluating the quality of voice assistants’ responses to consumer health questions about vaccines: an exploratory comparison of Alexa, Google Assistant and Siri. BMJ Health Care Inform 2019; 26: e100075.

60.

Fridman

Johnson

Elston Lafata

. Health information and misinformation: a framework to guide research and practice. JMIR Med Educ 2023; 9: e38687.

61.

Wang

McKee

Torbica

, et al. Systematic literature review on the spread of health-related misinformation on social media. Soc Sci Med 2019; 240: 112552.

62.

Dutsinma

FLI

Pal

Funilkul

, et al. A systematic review of voice assistant usability: an ISO 9241–11 approach. SN Comput Sci 2022; 3. DOI: 10.1007/s42979-022-01172-3. Epub ahead of print July 2022.

63.

Sezgin

Noritz

Lin

, et al. Feasibility of a voice-enabled medical diary app (SpeakHealth) for caregivers of children with special health care needs and health care providers: mixed methods study. JMIR Form Res 2021; 5: e25503.

64.

Sezgin

Weiler

, et al. Proposing an ecosystem of digital health solutions for teens with chronic conditions transitioning to self-management and independence: exploratory qualitative study. J Med Internet Res 2018; 20: e10285.

65.

Kadam

. Crafting multi-modal interactions on voice assistants. J Eng Appl Sci Technol 2022; 4: 1–5.

66.

Lee

Wang

Brown

, et al. GazePointAR: a context-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality. In: Proceedings of the CHI conference on human factors in computing systems. New York, NY, USA: ACM. Epub ahead of print May 11, 2024. doi: 10.1145/3613904.3642230.

67.

Tennant

Allana

Mercer

, et al. Caregiver expectations of interfacing with voice assistants to support complex home care: mixed methods study. JMIR Hum Factors 2022; 9: e37688.

68.

Denis

Maurier

Carillo

, et al. Early detection of neurodevelopmental disorders of toddlers and postnatal depression by mobile health app: observational cross-sectional study. JMIR MHealth UHealth 2022; 10: e38181.

69.

Khan

Hall

Davies

, et al. The effectiveness of web-based interventions delivered to children and young people with neurodevelopmental disorders: systematic review and meta-analysis. J Med Internet Res 2019; 21: e13478.

70.

Chen

Liu

Huang

, et al. When large language models meet personalization: perspectives of challenges and opportunities. World Wide Web 2024; 27: 1–45.

71.

Whang

Lee

, et al. CUE-M: contextual understanding and enhanced search with multimodal large language model. arXiv [cs.CL]. http://arxiv.org/abs/2411.12287 (2024, accessed 16 December 2024).

72.

Gallo

Paterno

Malizia

. Conversational interfaces in IoT ecosystems: where we are, what is still missing. In: Proceedings of the 22nd international conference on mobile and ubiquitous multimedia, pp.279–293. New York, NY, USA: ACM.

73.

Sánchez Cuadrado

Pérez-Soler

Guerra

, et al. Automating the development of task-oriented LLM-based chatbots. In: ACM conversational user interfaces 2024, pp.1–10. New York, NY, USA: ACM.

74.

Hancock

Bordes

Mazaré

P-E

, et al. Learning from dialogue after deployment: feed yourself, chatbot! arXiv [cs.CL], http://arxiv.org/abs/1901.05415 (2019).

75.

Bolton

Dargahi

Belguith

, et al. On the security and privacy challenges of virtual assistants. Sensors (Basel) 2021; 21: 2312.

76.

Okpomu

George

. Security and privacy concerns in the implementation of virtual assistants in multinational companies. Am J Curr Tendency Innovation 2024; 1: 17–31.

77.

Latulipe

Mazumder

Wilson

RKW

, et al. Security and privacy risks associated with adult patient portal accounts in US hospitals. JAMA Intern Med 2020; 180: 845–849.

78.

Sethness

Golub

Evans

. Adolescent patient portals and concerns about confidentiality. Curr Opin Pediatr 2023; 35: 430–435.

79.

Egan

Clark

Deen

, et al. Understanding current needs and future expectations of informal caregivers for technology to support health and well-being: national survey study. JMIR Aging 2022; 5: e15413.

80.

Okobi

Adigun

Ozobokeme

O-E

, et al. Examining disparities in ownership and use of digital health technology between rural and urban adults in the US: an analysis of the 2019 Health Information National Trends Survey. Cureus 2023; 15: e38417.

81.

Tudor

Ingram

Wackerbarth

. Patient engagement in patient portals in Appalachia v. surrounding U.S. census regions: an analysis of HINTS (health information national trends survey) data, 2017-2020. J Appalach Health 2023; 5: 50–65.

82.

Vanderpool

Stradtman

Gaysynsky

, et al. Access to and use of technology for health: comparisons between Appalachian Kentuckains and the general U.S. Population. J Appalach Health 2021; 3: 60–73.

83.

Kent

Lee

Asad

, et al. “If I wasn’t in a rural area, I would definitely have more support”: social needs identified by rural cancer caregivers and hospital staff. J Psychosoc Oncol 2023; 41: 393–410.

84.

Ragnedda

. The digital divide: the internet and social inequality in international perspective. 1st edn. London, England: Routledge. 2013. DOI: 10.4324/9780203069769.

85.

Zafar

Parthasarathy

Van

, et al. Building trust in conversational AI: a comprehensive review and solution architecture for explainable, privacy-aware systems using LLMs and knowledge graph. arXiv [cs.CL]. Epub ahead of print August 13, 2023. DOI: 10.3390/bdcc8060070.

86.

Piñeiro-Martín

García-Mateo

Docío-Fernández

, et al. Ethical challenges in the development of virtual assistants powered by large language models. Electronics (Basel) 2023; 12: 3170.

87.

Kumah-Crystal

Pirtle

Whyte

, et al. Electronic health record interactions through voice: a review. Appl Clin Inform 2018; 09: 541–552.

88.

Hong

Folcarelli

Less

, et al. Voice assistants and cancer screening: a comparison of Alexa, Siri, Google assistant, and Cortana. Ann Fam Med 2021; 19: 447–449.

89.

Erku

Khatri

Endalamaw

, et al. Digital health interventions to improve access to and quality of primary health care services: a scoping review. Int J Environ Res Public Health 2023; 20. DOI: 10.3390/ijerph20196854. Epub ahead of print September 28, 2023.

90.

Wienert

Jahnel

Maaß

. What are digital public health interventions? First steps toward a definition and an intervention classification framework. J Med Internet Res 2022; 24: e31921.

91.

Tekin

Ilhan

Huang

, et al. LLM-TOPLA: efficient LLM ensemble by maximising diversity. arXiv [cs.CL], https://aclanthology.org/2024.findings-emnlp.698.pdf (2024, accessed 16 December 2024).

92.

Manzini

Keeling

Marchal

, et al. Should users trust advanced AI assistants? Justified trust as a function of competence and alignment. In: The 2024 ACM conference on fairness, accountability, and transparency. New York, NY, USA: ACM. Epub ahead of print June 3, 2024. DOI: 10.1145/3630106.3658964.

93.

Wienrich

Reitelbach

Carolus

. The trustworthiness of voice assistants in the context of healthcare investigating the effect of perceived expertise on the trustworthiness of voice assistants, providers, data receivers, and automatic speech recognition. Front Comput Sci 2021; 3. DOI: 10.3389/fcomp.2021.685250. Epub ahead of print June 17, 2021.

94.

Easton

Potter

Bec

, et al. A virtual agent to support individuals living with physical and mental comorbidities: co-design and acceptability testing. J Med Internet Res 2019; 21: e12996.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.41 MB

0.07 MB

0.25 MB