Evaluating the Clinical Reasoning of Generative AI in Palliative Care: A Comparison with Five Years of Pharmacy Learners

Abstract

Context:

Artificial intelligence (AI), particularly large language models (LLMs), offers the potential to augment clinical decision-making, including in palliative care pharmacy, where personalized treatment and assessments are important. Despite the growing interest in AI, its role in clinical reasoning within specialized fields such as palliative care remains uncertain.

Objectives:

This study examines the performance of four commercial-grade LLMs on a Script Concordance Test (SCT) designed for pharmacy students in a pain and palliative care elective, comparing AI outputs with human learners’ performance at baseline.

Methods:

Pharmacy students from 2018 to 2023 completed an SCT consisting of 16 clinical questions. Four LLMs (ChatGPT 3.5, ChatGPT 4.0, Gemini, and Gemini Advanced) were tested using the same SCT, with their responses compared to student performance.

Results:

The average score for LLMs (0.43) was slightly lower than that of students (0.47), but this difference was not statistically significant (p = 0.55). ChatGPT 4.0 achieved the highest score (0.57).

Conclusions:

While LLMs show potential for augmenting clinical decision-making, their limitations in patient-centered care highlight the necessity of human oversight and reinforce that they cannot replace human expertise in palliative care. This study was conducted in a controlled research setting, where LLMs were prompted to answer clinical reasoning questions despite default safety restrictions. However, this does not imply that such prompts should be used in practice. Future research should explore alternative methods for assessing AI decision-making without overriding safety mechanisms and focus on refining AI to better align with complex clinical reasoning. In addition, further studies are needed to confirm AI’s comparative effectiveness, given the sample size limitations.