Sage Journals: Discover world-class research

Abstract

Palliative care aims to improve the quality of life for seriously ill individuals and their caregivers by addressing their holistic care needs through a person- and family-centered approach. While there have been growing efforts to integrate Artificial Intelligence (AI) into palliative care practice and research, it remains unclear whether the use of AI can facilitate the goals of palliative care. In this paper, we present three hypothetical case examples of using AI in the palliative care context, covering machine learning algorithms that predict patient mortality, natural language processing models that detect psychological symptoms, and AI chatbots addressing caregivers’ unmet needs. Using these cases, we examine the ethical dimensions of utilizing AI in palliative care by applying five widely accepted moral principles that guide ethical deliberations in AI: beneficence, nonmaleficence, autonomy, justice, and explicability. We address key ethical questions arising from these five core moral principles and analyze the potential impact the use of AI can have on palliative care stakeholders. Applying a critical lens, we assess whether AI can facilitate the primary aim of palliative care to support seriously ill individuals and their families. We conclude by discussing the gaps that need to be further addressed in order to promote ethical and responsible AI usage in palliative care.

Keywords

Artificial intelligence palliative care principle-based ethics

Introduction

The integration of artificial intelligence (AI) into healthcare has seen great advances, especially in supporting disease detection, clinical decision-making, health services management, and prognosis prediction.¹ Studies have shown that state-of-the-art AI may demonstrate comparable performance to clinicians in image recognition and even occasionally outperform medical students in open-ended medical examinations.^2,3 Such advancements imply that AI has the potential to provoke revolutionary changes to the structure of how patient care is provided.

A healthcare field that has not yet been largely affected by the advancements of AI is palliative care.^4,5 Palliative care is specialized care that aims to reduce the suffering of patients with life-threatening illnesses and their families.⁶ It often focuses on symptom management but also on other aspects of person-centered care and supports decision-making, regardless of the diagnosis and disease stage.⁷ In this context, AI could be integrated into practice or research to promote timely palliative care delivery and personalize support to care recipients. For instance, providers could identify patients who would benefit from advanced care planning (ACP) using algorithms that predict patients’ prognoses.⁸ AI can also be used to review clinical notes and detect signals of patients’ psychological distress or other symptoms.^9,10 Furthermore, considering the current popularity and accessibility of generative AI (e.g., OpenAI’s ChatGPT), caregivers could utilize these tools to address their informational needs.

As AI is becoming increasingly pervasive within society, use of AI in palliative care is not simply a technical innovation but rather a socio-technological phenomenon, comprising both technological (e.g., information structure of the AI model) and sociological (e.g., how AI is comprehended by users and how it changes their workflow) integrations.¹¹ Therefore, an examination of its ethical dimensions is needed to understand how the use of AI may impact seriously ill individuals and other stakeholders, while also determining whether AI can strengthen the person-centered intent of palliative care. However, recent reviews exploring AI in palliative care have focused more on technical aspects, assessing the purpose for which the tool was developed, its training data, methodologies, and model performance.^4,12,13 This leaves a crucial void surrounding whether the use of AI is ethically sound in the context of palliative care.

Therefore, in this paper, we present three case examples of AI application in the field of palliative care and analyze the ethical dimensions based on five moral principles that are widely accepted to guide ethical decision-making and are directly relevant to AI (i.e., beneficence, nonmaleficence, autonomy, justice, and explicability).¹⁴ By addressing key questions based on these core moral principles, we identify opportunities and concerns associated with the use of AI in palliative care and introduce insights on how to develop and deploy AI in a way that facilitates the goal of palliative care.

What is Artificial Intelligence?

AI is a field of computer science that develops systems to imitate human-like behaviors, such as learning, reasoning, and self-correction.¹⁵ Alan Turing’s work on the Turing Test laid the theoretical foundation of AI, considering a machine to be “intelligent” if a human judge cannot distinguish its responses from those of a human.¹⁶

In this article, we specifically focus on machine learning and natural language processing, which are both computer-based technologies under the scope of AI. Machine learning is a set of techniques and tools used to classify, make predictions, or cluster groups based on similar characteristics,¹⁷ enabling data-driven decisions or automations. On the other hand, natural language processing transforms unstructured data (e.g., human narrative) into structured data, allowing computers to derive meaning from human language.¹⁸ This forms the basis for computers to perform various language-related tasks, including chatbot development, language translation, information extraction and summarization, and spam detection.¹⁹

Principles for ethical AI

As the development and deployment of AI has had major impacts on society, there is a growing need to establish ethical principles to adopt AI in a way that can benefit society. Based on a review of high-profile initiatives, Floridi and Cowls put together a unified framework on what constitutes “ethical AI.”¹⁴ This framework, built upon principlism,²⁰ includes core ethical principles widely accepted in bioethics: beneficence, nonmaleficence, autonomy, and justice. In the context of AI, Floridi and Cowls illustrated beneficence as underscoring the role of AI in promoting the well-being of people and nonmaleficence as focusing on preventing breaches of personal privacy and AI misuse. Autonomy emphasizes human agency in choosing their own decisions, while justice involves eliminating discrimination and unfairness and also preserving unity. However, recognizing that these four principles alone do not fully translate all ethical challenges associated with the use of AI, Floridi and Cowls also proposed a fifth principle: explicability. Explicability encompasses both the epistemological aspect of understanding how AI works and the ethical aspect of accountability. These five principles altogether encapsulate the guidance provided by expert-driven documents for adopting ethical and socially beneficial AI.¹⁴

Case examples of utilizing AI in palliative care

To examine the ethical dimensions of utilizing AI in palliative care, we illustrate three hypothetical case examples (Tables 1–3) and address key questions built upon the core principles for ethical AI (Table 4).

Table 1.

Case 1.

AI technology

Machine learning algorithm that predicts patient mortality within a 3- to 12-month time frame

Application within palliative care

Dr. Lena Park is a board-certified palliative care specialist. She works at a hospital that has recently decided to incorporate machine learning algorithms into the electronic health records to identify candidate patients for palliative care at admission. Due to the shortage of palliative care specialists, the hospital administrators believe it would be more beneficial and efficient to use machine learning algorithms instead of having specialists perform time-consuming manual chart views of all admissions.The machine learning algorithm takes into account various factors, including patients’ age, gender, and race/ethnicity, along with diagnostic, prescription, and billing codes, and predicts patient mortality within a 3- to 12-month time frame. The hospital policy encourages providers to refer patients to Dr. Park’s palliative care team based on these predictions.

Note. AI: artificial intelligence, EHR: electronic health records.

Table 2.

Case 2.

AI technology

Natural language processing tools that detect signals for psychological distress within clinical notes

Application within palliative care

Dr. Theo Smith is a nurse scientist whose research interest lies in promoting early integration of palliative care into routine care for seriously ill individuals. He is currently using EHR to develop a natural language processing system that detects severe depressive and anxiety symptoms in hospitalized patients with cancer. He is extracting unstructured text data from narrative records, such as reports filled in by physicians, nurses, and other staff, as well as medication orders. To identify search terms for his model, he conducted literature reviews, consulted experts, and utilized an open-source library. Subsequently, the model was trained to extract phrases with concepts that represent severe depressive and anxiety symptoms. Dr. Smith aims to integrate this algorithm into the EHR to implement automatic symptom screening and to help identify patients who would benefit from being referred to a specialized palliative care team for further symptom management.

Note. AI: artificial intelligence; EHR: electronic health records.

Table 3.

Case 3.

AI technology

Chatbots based on advanced natural language processing models

Application within palliative care

Allen Brown’s father has been diagnosed with advanced dementia and heart failure. Living with dementia for over 5 years, his father now experiences severe cognitive decline, and Allen is his surrogate decision-maker. As his father’s condition heavily deteriorated over the past month, Allen and his father have recently transitioned to home hospice care so their family can spend his father’s last phase of life together. Although the hospice team visits to regularly check on his father, Allen experiences high anxiety symptoms due to predeath grief, his fathers’ symptom burden, and concerns about what to do before and after his father’s death. He has many questions about hospice care and decides to use a publicly available AI chatbot to freely ask questions both about symptom management and overall coping whenever he wants and gets personalized support.

Note. AI: artificial intelligence.

Table 4.

Key questions for examining the ethical dimensions of AI integration examples in palliative care.

	Predicting mortality using machine learning	Detecting psychological symptoms using natural language processing	AI chatbots
Beneficence	Can mortality prediction based on machine learning improve patients’ well-being at the end of life?	Will using natural language processing for automated symptom screening promote the well-being of seriously ill individuals?	Will AI chatbots improve caregivers’ access to healthcare information?
Nonmaleficence	Will palliative care consultations initiated by machine learning algorithms provoke harm to patients and their families?	Will using natural language processing to review clinical notes infringe on patients’ personal privacy?	What harm could arise from the outputs generated by AI chatbots?
Autonomy	To what extent will the computed mortality shape providers’ decisions about whom to recommend for palliative care?	Does automated symptom screening respect patients’ autonomy to determine their own emotional state?	Would the use and advice of a chatbot augment one’s decisions?
Justice	Will the use of these machine learning algorithms contribute to equitable distribution of palliative care resources?	How can we use natural language processing algorithms to reduce health disparities within palliative care?	Will AI chatbots ensure equal distribution of information for all?
Explicability	Is the decision-making process of the algorithm transparent, and who is responsible for algorithm errors and biases?	Is the dataset representative of the population of interest?	Where does the data for the AI chatbot come from? How does AI handle errors and bias?

Note. AI: artificial intelligence.

Case 1. Mortality-predicting machine learning algorithms used for palliative care referrals

To date, mortality prediction has been the most common purpose of applying AI in palliative care.^4,12 Regarding beneficence, mortality-predicting algorithms may promote patient well-being by prompting opportunities for goals-of-care discussions and ACP. Previous research underscores the benefits of discussing goals of care and preferred treatment throughout the course of a serious illness. These include improved quality of life, reduced use of life-sustaining treatments, and increased utilization of hospice and palliative care.^21,22 Nonetheless, providers find it difficult to initiate ACP in real-world practice, often due to time constraints, uncertain prognosis, patient and family disagreements, and lack of knowledge and skills in managing ACP.^23,24 Case 1’s algorithm could act as a nudge to prompt ACP specifically tailored to the patients’ illness trajectory and ensure that no high-risk patients are being missed in the system.²⁵ This would be particularly beneficial for high-risk patients who lack access to or awareness of palliative care resources.

However, these AI applications are not free from potential patient harm, because computed predictions may cause distress to seriously ill individuals and families. Some patient-caregiver dyads may feel uneasy about palliative care consultations being initiated based on an algorithm rather than a provider’s assessment, especially if they struggle with accepting terminality. Discomfort caused by the perceived absence of personal interaction in the patient-provider relationship could later escalate to heightened perceptions of mistrust in the healthcare system. For historically marginalized populations, mistrust is already identified as a barrier to palliative care,²⁶ and when combined with a lack of confidence in AI, it may worsen their sense of institutional and provider-related biases. Conversely, there may be other cases where patients hear about or read the specific numbers produced from the AI model (e.g., 180-day mortality of 30%) and decide that it is too great a risk and refuse possibly beneficial treatments. While patients can, and sometimes do, also refuse clinical care without AI involvement, the influence of AI is unique and one where the ethical, technological, and pragmatic day-to-day challenges have not been fully addressed. The quality of the output depends on the quality of the data used to train and test the model. If the data itself represent certain biases, then these algorithms will reintroduce them.²⁷ Deploying a biased algorithm in practice may perpetuate harm, especially to the patients whose characteristics differ from the training data. False positives can unnecessarily lead to distress and/or impact care decisions, while false negatives contribute to a systematic oversight of high-risk patients.

Another crucial consideration is the extent to which these mortality-predicting algorithms can shape providers’ decision-making. The algorithms could complement providers’ prognostic intuition or offer additional insights.^25,28 However, solely relying on this metric for identifying potential palliative care recipients can subtly sway providers’ focus toward patient survival as the primary outcome of interest. Palliative care is inherently holistic, taking into account the patients’ psychosocial well-being, spirituality, and culture,⁶ and high mortality risk may not always correspond with palliative care needs. The diversion from the goal of palliative care can be more pronounced when involving automation bias (i.e., overreliance on automated context and ignoring other information, including one’s own judgment) or confirmation bias (i.e., tendency to accept information that aligns with one’s previous hypothesis or beliefs). These cognitive biases triggered by the use of AI can further interfere with providers’ agency in decision-making.

As palliative care is not solely for those facing imminent death, allocating palliative care resources primarily based on mortality jeopardizes its inclusive scope and raises ethical concerns on who should benefit from this resource and who makes that determination. For instance, the National Comprehensive Cancer Network identified several problems that might indicate a patient’s need for palliative care, including uncontrolled symptoms, patient or family distress related to the diagnosis and treatment, and challenges in goal-setting or decision-making.²⁹ Offering palliative care based on mortality and training algorithms accordingly can imply a prioritization of those at end of life and disregard others who may not be at end of life but nonetheless have greater needs for palliative care. Therefore, while integrating mortality-predicting algorithms may increase the volume of referrals to palliative care, it does not guarantee that patients with high palliative care needs will receive adequate access. Instead, given the great shortage of palliative care specialists,³⁰ without clear alternatives, these tools may inadvertently burden the few specialists available by directing a greater number of referrals for critically ill patients facing imminent death.

Although machine learning algorithms have become sophisticated in predicting outcomes, the major obstacle of deploying such tools in practice is that we cannot understand why the model made a certain prediction. This makes the algorithm seem like a “black box”—it is difficult to see what is happening inside. When providers cannot understand the logic behind the algorithm’s decision-making process, they are unable to judge whether or not the results are trustworthy.³¹ This lack of transparency becomes especially problematic when end-users nevertheless over-rely on the technology. It leaves them vulnerable to the harm associated with potential algorithmic errors and biases, without any clear entity to take responsibility for these mistakes.

In summary, Case 1’s AI-based mortality prediction may help facilitate goals-of-care discussions and ACP for high-risk patients, personalized to their illness trajectory. Nonetheless, if mortality were to be the standard parameter for offering palliative care, it conflicts with the ethical principle of justice due to unbalanced resource allocation and the equitable treatment of patients. It may also increase the burden on the limited number of palliative care providers, create discomfort for patients and families due to AI-initiated referrals, and affect providers’ decision-making processes. Moreover, the potential lack of transparency in the algorithms’ decision-making process further exacerbates these concerns.

Case 2. Natural language processing systems detecting psychological distress

Symptom management is an essential component of palliative care. Automated symptom screening based on AI may, to some degree, correspond with the principle of beneficence by improving patient well-being through timely symptom management. Given the descriptive nature of affective symptoms, related data are often documented as free-text in the electronic health records (EHR). Even when patients do not directly report the presence of psychological distress, its indicators may be embedded in the text documented by the medical staff. Thus, Case 2’s tool can help capture signals of emotional distress and save time and costs by not having to manually review volumes of clinical notes accumulated throughout patient care.¹⁸ However, the algorithm in Case 2 focuses only on a limited set of predefined symptoms. Palliative care addresses the holistic needs of patients (e.g., physical, emotional, financial, spiritual),⁶ but the algorithm’s narrow focus might fail to capture the complex nature of patients’ experience with a serious illness. While the use of this algorithm may benefit certain patients with emotional distress, in the palliative care context, this approach does not fully support its comprehensive nature.

Case 2 also raises privacy concerns as it is unclear whether patients are sufficiently informed that third-party researchers can access their clinical notes. Although de-identified, narrative documentations reviewed for potential palliative care referrals may still include personal information about patients and their families, especially when discussing distressful life events or sharing raw emotions. Concerns about serious illness and one’s end of life is particularly sensitive with expressions of fears, loneliness, grief, and insecurities about death.³² Despite this vulnerability, researchers working with secondary EHR data are typically not required by institutional review boards to re-contact patients for informed consent. Although patients have the right to opt out of having their data used for secondary purposes, there are also cases where patients are not aware of such options or that their inclusion in the EHR automatically makes them eligible for a future study, as described by Ulrich et al.³³ Therefore, patients and families remain unaware of the specific research objectives and potential algorithms that could be built using their health information. These issues raise concerns about the original consent for the secondary use of health data, as patients may not fully comprehend the implications, let alone be aware of such use.

Regarding patient autonomy, automated symptom screening based on an AI algorithm may not fully embrace the individualized nature of psychological distress experienced by each person. Unlike structured questionnaires that allow respondents to choose the severity of their depressive or anxiety symptoms, natural language processing systems detect terms or context learned from the training data.¹⁰ This approach may not consider the subjective nature of certain symptoms, as patients may describe a stressful event without necessarily feeling that they are experiencing stress themselves. Each person possesses a unique threshold for resilience to distress, and thus they retain the power to determine their own emotional state. Although some patients may indeed be distressed without realizing it, an algorithm making this indication on their behalf mitigates their right to self-determination and conflicts with the person-centered approach of palliative care.

Nonetheless, natural language processing systems designed to detect symptoms can contribute to identifying systematic injustice in symptom management. Using these algorithms, researchers can more readily analyze symptom data to identify patient subgroups experiencing unequal access to symptom management and palliative care. These findings could then inform targeted interventions to address health disparities within the field. Striving for equitable distribution of palliative care resources and ensuring individuals, regardless of their background or demographics, receive the support they need aligns with the principle of justice.

However, it is important to note that if the data are not representative of the population of interest, employing these algorithms will not rectify health disparity in palliative care. It will rather exacerbate it. If the medical center in Case 2 primarily represents a specific demographic or socioeconomic group, the model’s prediction will be skewed toward that group. This unintended prioritization, combined with the black-box issue, can reinforce existing biases.

Case 2’s use of natural language processing to detect signals of anxiety and depression in clinical notes may contribute to timely management of these particular symptoms and help researchers identify health disparity gaps within palliative care. However, these benefits rely on the assumption that the data accurately represent the population of interest. Furthermore, the signals identified by these algorithms may not align with patients’ actual perceptions or experiences. To support the holistic and person-centered approach of palliative care, the full scope of one’s symptom experience has to be considered, while also safeguarding privacy concerning patient distress.

Case 3. AI chatbots addressing unmet information needs in palliative care

AI chatbots that generate human-like responses have been receiving great attention, especially after the public release of OpenAI’s ChatGPT. Chatbots such as ChatGPT not only respond to user queries at any time, but they also customize the responses based on user preferences and can perform tasks such as summarizing or comparing lengthy literature. Mirza et al.³⁴ recently introduced the potential of utilizing ChatGPT to adjust the reading level of medical documentations, thereby improving readers’ comprehension. This additional layer of support for information is especially important in the field of palliative care, given the lack of awareness and knowledge regarding the specialty among the general public.³⁵

However, the responses of AI chatbots are not always accurate, which may create more harm than benefit. For instance, Kim et al. asked ChatGPT, Microsoft Bing Chat, and Google Bard to differentiate palliative, supportive, and hospice care and provide references. Although the responses were generally accurate, some errors and omissions were still present, and unreliable references were provided.³⁶ The risk of inaccuracy, when combined with limited knowledge of palliative care and the lack of specialists to validate the chatbots’ responses, can amplify the misconceptions that already exist. This is also worrisome for those in hospice care as terminally ill patients may be less able to advocate for themselves due to their serious condition.³⁷

The more urgent concern is the possibility of users’ overreliance on these chatbots without exercising their autonomy in decision-making. Questions asked in palliative care often go beyond symptom management; they also address patient care issues intertwined with moral conflicts. Although the decision for comfort care is already made in Case 3, caregivers may second-guess this or have feelings of guilt.³⁸ Unlike conventional search engines that present a wide range of search results for users to choose from, AI chatbots provide direct and succinct answers to the user’s questions. While they may not offer explicit treatment recommendations for ethical issues, there is no guarantee that the scope of information in their succinct answer, or the way they frame it, is entirely value-neutral. Furthermore, evidence that users perceive AI as a superior form of knowledge is emerging³⁹—users may accept AI-generated responses unquestioningly, especially if they lack confidence in their own judgment or are unaware of AI’s limitations. Given the lack of social support for hospice caregivers,⁴⁰ the chatbots’ perceived non-judgmental and attentive tone could also result in a positive bond,⁴¹ which may add another reason for overreliance.

AI chatbots in palliative care may also worsen the already existing divide caused by unequal access to digital infrastructure. Even among persons with internet access, a new form of disparity may arise based on AI chatbot subscriptions, as users encounter varying information quality depending on whether they use the free or paid version.⁴² Moreover, based on the language, the chatbots can provide different quality outputs. These attributes can lead to unequal distribution of information, adding an additional layer of inequality to the conventional digital divide.

Explicability is an especially important principle to consider in Case 3. Currently, state-of-the-art AI chatbots are corporately owned and their training data are not publicly available.⁴³ It is unclear whether or how companies have implemented strategies to mitigate biases within the large amount of data and ensure fairness, particularly for marginalized populations. In fact, Zack et al.⁴⁴ demonstrated the potential for ChatGPT perpetuating societal biases by overrepresenting racial stereotypes in diseases and offering different directions when modifying the demographics in the prompts. However, despite these issues of opacity, AI chatbots cannot hold any accountability for their responses and associated impact.

In summary, AI chatbots may benefit users by improving access to health information tailored to their needs. Yet, ethical concerns persist, including the potential for overriding users’ autonomy in decision-making and the risk of harm caused by inaccurate yet persuasive responses. These issues require extra attention in palliative care, where dilemmas often involve moral conflicts and there is a general lack of understanding about its services and concepts. Moreover, new disparities may emerge based on user subscriptions or language barriers. Finally, the fact that end-users are unaware of the strategies used to address potential bias in the data raises serious concerns, as does the unclear entity responsible for the chatbots’ impact.

Discussion

In this critical paper, we have explored whether AI utilization within palliative care aligns with the principles for developing ethical AI and can facilitate the overarching goal of supporting seriously ill individuals and their families. Indeed, AI offers many opportunities within this domain, such as prompting goals-of-care discussions and ACP and promoting timely management for symptom burden. AI can also generate health information for patient-caregiver dyads personalized to their inquiries. Nonetheless, we highlight several gaps that need further attention to ensure AI better supports palliative care.

The main concern pertains to whether AI can shape or augment the decisions of the patient-caregiver dyads and palliative care providers and override their agency to retain autonomy. We pointed out cognitive biases that impact users’ decision-making across various AI applications. Tschandl et al. also demonstrated the possibility of AI misleading clinicians, even when it contradicts their initial impressions.⁴⁵ This issue becomes more concerning given the ongoing uncertainty surrounding the extent to which end-users are accountable for the harm caused by decisions involving AI.⁴⁶ The vulnerability of patients with life-limiting illnesses further amplifies the ethical concerns about autonomy and accountability in palliative care. Therefore, we strongly suggest that providers view AI tools as a supplement to their practice, rather than a substitute of human effort, at the individual, institutional, and policy level. Also, before AI predictions are deployed within standard care, specific guidelines describing the extent and context in which providers can disregard these predictions must be established to protect providers’ own decision-making autonomy without fear of retaliation.

Another ethical concern is the possibility of AI perpetuating harm through biased data. Obermeyer et al.⁴⁷ presented evidence for racial bias in a widely distributed algorithm, and Diaz et al.⁴⁸ found both explicit and implicit age-associated bias within various natural language processing tools. To mitigate these potential harms, ethical concerns could be approached as computational issues that require further attention. For instance, bias could be reframed as an issue requiring focus on label noise (i.e., meaningless or corrupted data caused by mislabeling that does not adequately reflect the true attributes of the target variable) as well as feature noise (i.e., erroneous or missing data).⁴⁹ Techniques such as finetuning (i.e., applying pretrained weights on a smaller and application-specific dataset⁵⁰) are also introduced to make the algorithm output more representative. Other computational techniques that can potentially address ethical concerns regarding the data should be further identified. Moreover, beyond generating accurate outputs, emphasis should also be placed on AI’s ability on providing explanations for the derived outputs. This is particularly crucial for AI chatbots prone to generating erroneous outputs as if they were true. Providing explanations as to why such responses were generated can enhance trust and empower users to make a decision based on the rationale given.⁵¹ These efforts will not only safeguard end-users from potential harm but will also improve transparency, ultimately enhancing users’ informed AI usage and autonomy in decision-making.

We have also highlighted the potential distress that patients and caregivers may experience when utilizing AI in palliative care. Distressing emotions could result from depersonalization, data privacy concerns, potential interference with one’s autonomy, and the lack of access to digital infrastructure that supports AI usage. A recent Pew Research Center report also found that 52% of US adults expressed concern about the increased use of AI in their daily lives and 60% would be uncomfortable if their provider relied on AI for their care.⁵² Given this public uncertainty, if AI-based predictive analytics are to be deployed in standard palliative care practice, the extent to which AI has been utilized needs to be explicitly shared with the patient. Because some patients may not want to know their computed mortality risk, obtaining informed consent before disclosing such information could also be an option to ensure patient autonomy and respect for their preferences. There is much that remains to be learned with the use of AI within the clinical arena and especially as it relates to palliative care services. There is always an opportunity to improve communication between patients and their providers and additional training in informed consent practices and disclosure of AI outputs would be beneficial. Referrals for psychological support may also be needed. If the output was later shown to be an error, providers should also be aware of how to inform patients without causing further mental harm. Moreover, the possibility of secondary use of EHR data should be more explicitly informed to the patients and families at the system level to better safeguard data privacy.

Finally, equitable allocation of resources is crucial for palliative care, as the field emphasizes support for all serious ill individuals, regardless of age, care setting, diagnosis, and prognosis.⁶ On one hand, AI could improve access for high-risk individuals who lack awareness of palliative care, and researchers could identify health disparities within the field using these algorithms. However, depending on the main outcome of interest and the quality of the data, AI may still inadvertently prioritize a certain group, which diverges from the original intent of palliative care. Further discussion is needed to agree upon which outcome should guide these algorithms to ensure equitable distribution of palliative care resources. Moreover, the costs and training required for advanced technologies introduce new disparities in access. AI developers targeting patient-caregiver dyads should consider how to make the technology accessible and user-friendly for marginalized population. For example, utilizing user-centered or participatory design approach during the development and testing of AI tools can help designers better understand the specific situation and needs of end-users.⁵³

Conclusion

Palliative care is specialized care for seriously ill individuals and families, with the goal of improving their quality of life and alleviating suffering. Guided by principles for ethical AI, we examined the ethical dimensions of three cases using AI in the palliative care context. Despite the opportunities it can offer for the field, it remains unclear whether AI can ethically support the person-centered and holistic intent of palliative care. Further discussion is needed to identify outcomes future AI designers should focus on to best reflect the aim and scope of palliative care. The needs and situations of the palliative care population should be comprehensively reflected in these tools, with a specific focus on inclusivity for marginalized stakeholders. Before its integration into practice, additional training should be established for providers in effectively communicating with patients and caregivers regarding AI usage, as well as concrete guidelines to mitigate patient distress caused by its disclosure. More attention is required in addressing biases within the data and promoting explainability. These efforts will help strengthen the ethical and responsible use of AI in palliative care, ensuring that its use facilitates the goal of supporting seriously ill individuals and their families.

Footnotes

Acknowledgments

We would like to thank Dr. Sara F. Jacoby for her invaluable feedback and support.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Oonjee Oh

Connie M Ulrich

References

Secinaro

Calandra

Secinaro

, et al. The role of artificial intelligence in healthcare: a structured literature review. BMC Med Inform Decis Mak 2021; 21: 125.

Shen

Zhang

CJP

Jiang

, et al. Artificial intelligence versus clinicians in disease diagnosis: systematic review. JMIR Med Inform 2019; 7: e10010.

Strong

DiGiammarino

Weng

, et al. Chatbot vs medical student performance on free-response clinical reasoning examinations. JAMA Intern Med 2023; 183: 1028–1030.

Steinmann

Schröder

, et al. Applications of machine learning in palliative care: a systematic review. Cancers 2023; 15: 1596.

Windisch

Hertler

Blum

, et al. Leveraging advances in artificial intelligence to improve the quality and timing of palliative care. Cancers 2020; 12: 1149.

National Consensus Project for Quality Palliative Care . Clinical practice guidelines for quality palliative care guidelines. 4th ed. Richmond, VA: National Coalition for Hospice and Palliative Care, 2018. https://www.nationalcoalitionhpc.org/ncp.

Rome

Luminais

Bourgeois

, et al. The role of palliative care at the end of life. Ochsner J 2011; 11: 348–352.

Chi

Kim

Reuter

, et al. Advanced care planning for hospitalized patients following clinician notification of patient mortality by a machine learning algorithm. JAMA Netw Open 2023; 6: e238795.

DiMartino

Miano

Wessell

, et al. Identification of uncontrolled symptoms in cancer patients using natural language processing. J Pain Symptom Manage 2022; 63: 610–617.

10.

Masukawa

Aoyama

Yokota

, et al. Machine learning models to detect social distress, spiritual pain, and severe physical psychological symptoms in terminally ill patients with cancer from unstructured text data in electronic medical records. Palliat Med 2022; 36: 1207–1216.

11.

Weger

Yeazitzis

. Conceptualizing a socio-technical model for evaluating AI-driven technology. Proc Hum Fact Ergon Soc Annu Meet 2023; 67: 1639–1644.

12.

Sarmet

Kabani

Coelho

, et al. The use of natural language processing in palliative care research: a scoping review. Palliat Med 2023; 37: 275–290.

13.

Storick

O’Herlihy

Abdelhafeez

, et al. Improving palliative and end-of-life care with machine learning and routine data: a rapid review. HRB Open Res 2019; 2: 13.

14.

Floridi

Cowls

. A unified framework of five principles for AI in society. Harvard Data Science Review 2019; 1: 1–15.

15.

Kok

Boers

Kosters

, et al. Artificial intelligence: definition, trends, techniques, and cases. Artif Intell 2009; 1: 270–299.

16.

Turing

. Computing machinery and intelligence. Mind 1950; 59: 433–460.

17.

Goodman

Kaminsky

, et al. What is machine learning? A primer for the epidemiologist. Am J Epidemiol 2019; 188: 2222–2239.

18.

Kimia

Savova

Landschaft

, et al. An introduction to natural language processing: how you can get more from those electronic notes you are generating. Pediatr Emerg Care 2015; 31: 536–541.

19.

Khurana

Koli

Khatter

, et al. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl 2023; 82: 3713–3744.

20.

Beauchamp

Childress

. Principles of biomedical ethics. 5th ed. New York, NY: Oxford University Press, 2001.

21.

Brinkman-Stoppelenburg

Rietjens

JAC

van der Heide

. The effects of advance care planning on end-of-life care: a systematic review. Palliat Med 2014; 28: 1000–1025.

22.

Bernacki

Block

. Communication about serious illness care goals: a review and synthesis of best practices. JAMA Intern Med 2014; 174: 1994–2003.

23.

Poveda-Moral

Falcó-Pegueroles

Ballesteros-Silva

, et al. Barriers to advance care planning implementation in health care: an umbrella review with implications for evidence‐based practice. Worldviews Evid Based Nurs 2021; 18: 254–263.

24.

Toguri

Grant-Nunn

Urquhart

. Views of advanced cancer patients, families, and oncologists on initiating and engaging in advance care planning: a qualitative study. BMC Palliat Care 2020; 19: 150.

25.

Parikh

Manz

Nelson

, et al. Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study. Support Care Cancer 2022; 30: 4363–4372.

26.

Gardner

Doherty

Bates

, et al. Racial and ethnic disparities in palliative care: a systematic scoping review. Fam Soc 2018; 99: 301–316.

27.

Chu

Nyrup

Leslie

, et al. Digital ageism: challenges and opportunities in Artificial intelligence for older adults. Gerontol 2022; 62: 947–955.

28.

Giddings

Joseph

Callender

, et al. Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review. Lancet Digit Health 2024; 6: e131–e144.

29.

Glare

Semple

Stabler

, et al. Palliative care in the outpatient oncology setting: evaluation of a practical set of referral criteria. J Oncol Pract 2011; 7: 366–370.

30.

Kamal

Wolf

Troy

, et al. Policy changes key to promoting sustainability and growth of the specialty palliative care workforce. Health Aff (Millwood) 2019; 38: 910–918.

31.

von Eschenbach

. Transparency and the black box problem: why we do not trust AI. Philos Technol 2021; 34: 1607–1622.

32.

Bosch-Barrera

Vidal Bota

. Vulnerability at the end of life: a medical perspective. In: Masferrer

García-Sánchez

(eds). Human Dignity of the Vulnerable in the Age of Rights. Cham: Springer, 2016, pp. 167–187.

33.

Ulrich

Grady

Demiris

, et al. The competing demands of patient privacy and clinical research. Ethics Hum Res 2021; 43: 25–31.

34.

Mirza

Tang

Connolly

, et al. Using ChatGPT to facilitate truly informed medical consent. NEJM AI; 1. Epub ahead of print January 25, 2024. DOI: 10.1056/aics2300145.

35.

Flieger

Chui

Koch-Weser

. Lack of awareness and common misconceptions about palliative care among adults: insights from a national survey. J Gen Intern Med 2020; 35: 2059–2064.

36.

Kim

Admane

Chang

, et al. Chatbot performance in defining and differentiating palliative care, supportive care, hospice care. J Pain Symptom Manage 2024; 67: e381–e391.

37.

Morberg Jämterud

. Acknowledging vulnerability in ethics of palliative care - a feminist ethics approach. Nurs Ethics 2022; 29: 952–961.

38.

Lewis

. Caregivers’ experiences seeking hospice care for loved ones with dementia. Qual Health Res 2014; 24: 1221–1231.

39.

McCradden

Kirsch

. Patient wisdom should be incorporated into health AI to avoid algorithmic paternalism. Nat Med 2023; 29: 765–766.

40.

Oliver

Demiris

Washington

, et al. Challenges and strategies for hospice caregivers: a qualitative analysis. Gerontol 2017; 57: 648–656.

41.

Dosovitsky

Bunge

. Bonding with bot: user feedback on a chatbot for social isolation. Front Digit Health 2021; 3: 735053.

42.

Rahimzadeh

Kostick-Quenet

Blumenthal Barby

, et al.

Ethics education for healthcare professionals in the era of ChatGPT and other large language models: do we still need it?

Am J Bioeth 2023; 23: 17–27.

43.

van Dis

EAM

Bollen

Zuidema

, et al. ChatGPT: five priorities for research. Nature 2023; 614: 224–226.

44.

Zack

Lehman

Suzgun

, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 2024; 6: e12–e22.

45.

Tschandl

Rinner

Apalla

, et al. Human–computer collaboration for skin cancer recognition. Nat Med 2020; 26: 1229–1234.

46.

Habli

Lawton

Porter

. Artificial intelligence in health care: accountability and safety. Bull World Health Organ 2020; 98: 251–256.

47.

Obermeyer

Powers

Vogeli

, et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366: 447–453.

48.

Diaz

Johnson

Lazar

, et al. Addressing age-related bias in sentiment analysis. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery, 2018, pp. 1–14.

49.

Shanthini

Vinodhini

Chandrasekaran

, et al. Retracted article: a taxonomy on impact of label noise and feature noise using machine learning techniques. Soft Comput 2019; 23: 8597–8607.

50.

Wang

Russakovsky

. Overwriting pretrained bias with finetuning data. ICCV 2023: 3934–3945.

51.

Rai

. Explainable AI: from black box to glass box. J Acad Mark Sci 2020; 48: 137–141.

52.

Faverio

Tyson

. What the data says about Americans’ views of artificial intelligence. Pew Research Center, https://www.pewresearch.org/short-reads/2023/11/21/what-the-data-says-about-americans-views-of-artificial-intelligence/ (2023, accessed 4 March 2024).

53.

Fischer

Peine

Östlund

. The importance of user involvement: a systematic review of involving older users in technology design. Gerontol 2020; 60: e513–e523.

The ethical dimensions of utilizing Artificial Intelligence in palliative care

Abstract

Keywords

Introduction

What is Artificial Intelligence?

Principles for ethical AI

Case examples of utilizing AI in palliative care

Case 1. Mortality-predicting machine learning algorithms used for palliative care referrals

Case 2. Natural language processing systems detecting psychological distress

Case 3. AI chatbots addressing unmet information needs in palliative care

Discussion

Conclusion

Footnotes

Acknowledgments

Declaration of conflicting interests

Funding

ORCID iDs

References