A Single Academic Site Study of Five Years Evaluating Pharmacy Students’ Palliative Care Clinical Reasoning Using Script Concordance Testing

Abstract

Introduction: This study aimed to evaluate the impact of a Pain and Palliative Care elective didactic course on enhancing clinical reasoning skills among Doctor of Pharmacy (PharmD) students using the Script Concordance Test (SCT). As palliative care education becomes increasingly vital in pharmacy training, assessing its effectiveness is essential for curriculum development. Methods: A 5-year prospective study was conducted between 2018 and 2023 at a single academic institution. The elective course was offered six times, covering various palliative care topics such as opioid management, procedural pain, and end-of-life care. A total of 130 third year PharmD candidate students, completed pre- and post-course SCT assessments. The SCT, administered via an online platform, was used to measure clinical reasoning in ambiguous scenarios, with scores compared to a panel of expert responses. Results: The study found a statistically significant improvement in SCT scores from pre- to post-course assessments (P = 0.04), with a mean score increase of 0.09. The 5-year cohort’s average score (0.56) outperformed a previous multisite cohort (0.50), although the difference was not statistically significant. Item analysis highlighted areas where students struggled, such as bowel obstruction and CYP interactions, indicating potential areas for curriculum enhancement. Conclusion: This study demonstrates the effectiveness of a targeted palliative care elective in improving clinical reasoning skills among pharmacy students. The findings suggest the need for ongoing curriculum refinement to address specific areas of difficulty and ensure comprehensive palliative care education.

Background

Palliative care provides care for patients with serious illnesses at any point in their disease progression. ¹ A few main goals of this practice include relieving physical pain, determining treatment goals, and monitoring physical symptoms. This is often done with an interdisciplinary team to provide an array of professional opinions to design the best care plan for each unique patient. The knowledge in this field is growing in the United States as the aging population is projected to double by the end of the century. ²

The role of a pharmacist in interprofessional palliative care is essential in providing unique knowledge on symptom management and pharmacological interventions. ³ Interprofessional palliative care teams often benefit from having overlapping goals and responsibilities of extending better patient care to the most intricate of patient cases. ⁴ In recent decades, the role of the pharmacist has evolved in palliative care that complements the work of a physician and interprofessional team by using expertise for future developments in patient care. ⁵

It is vital for the curriculum for the Doctorate of Pharmacy (PharmD) to go under constant revision to keep up with the most current palliative care practices. Today, more than 50% of United States accredited schools of pharmacy incorporate generalist palliative care skills training throughout the students’ time of education both didactically and through experiential clerkships. ⁵ Research towards how to track the development of these palliative care curricula has focused on students’ ability to choose between multiple possible treatments to optimize results towards a patient’s goals for any given case. Oftentimes, there can be gray areas in the presented evidence, and it is the job of the palliative care teams to curate the best treatment plan for each individual case, ⁴ balancing the proverbial “art and science” of clinical care.

The script concordance test (SCT) is a standardized test that assesses clinical reasoning of students in cases where there is ambiguity. ⁶ The basis of SCT involves taking experts as panelists to provide the best answers to uncertain cases, then using their answers to assign points to each of the possible answers based on their frequency of being chosen. This addresses that in some clinical cases there is not a single correct answer, but a possible range of approaches in real-world practice. These will be the basis to test student scores and compare them to professionals for readiness in the field and effectiveness of a palliative care course. ⁷

This study will build on our previous, multisite study of the SCT in Pharmacy school pain and palliative care electives ⁴ by reporting on 5 years’ worth of data from a single institution. Our objectives in this study were to assess PharmD students’ clinical reasoning using the SCT among students enrolled in a palliative care elective over a 5-year period.

Methods

This prospective study was completed over 5 years between 2018 and 2023 at a single academic school of pharmacy during a 10-week elective course on Pain and Palliative Care. The study was reviewed and exempted by the Institutional Review Board (Project #161035X). Over the study timeframe, the elective course was provided six times, 3 times during the fall quarter and 3 times during the spring quarter. Three instances of the elective were provided virtually due to the COVID public health emergency and associated campus restrictions on in-person instruction. The elective is primarily open to third-year Pharmacy trainees after the completion of the opioid pharmacology and therapeutics lecture series in the core curriculum, though occasional exceptions are considered on a case-by-case basis.

The pain elective course covered a range of topics, including comprehensive patient assessment, opioid selection, and opioid equianalgesia. Additional subjects included the management of opioid side effects, opioid misuse and abuse, procedural pain management, and end-of-life care. The course also addressed the role of a palliative care pharmacist, patient-controlled analgesics, methadone, buprenorphine, sickle cell disease management, and compassionate communication skills.

The survey and SCT questions were provided to all enrolled students in the elective during the first lecture (pre-elective survey) via an online platform (Qualtrics, Provo, UT). At the conclusion of the course, the SCT questions were again presented (post-elective survey) using the same platform. (See Appendix for complete survey and SCT questions.) Completion of both surveys was required for course credit, but answers were not individually tracked. Time was allotted within the classes for students to complete the surveys.

At the conclusion of the 5-year study period, the pre- and post-elective survey data was extracted from Qualtrics into Excel (Microsoft Corporation, Redmond, WA) in order to generate qualitative results and statistical analysis. Significance was defined as a P value ≤ 0.05 using t test. As there was only one non-third-year student enrolled in the 5-year study period, their data was excluded from analysis, leaving a final n of 130.

SCT results were scored using the methods described in our previous paper on this topic. ⁴ In short, the students’ answers on the SCT questions are compared against weighted responses from a panel of experts to generate a final score, generating potential scores between zero (0) and one (1). ⁸ The methodology of SCT therefore attempts to capture some of the “art” within the field of palliative care and the management of seriously ill patients.

Results

Over the 5-year period, 130 students completed both the pre- and post-test 16-SCT case-based questions, all were in their third year of Pharmacy school. Nearly 60% of the participants were between the ages of 21-25-year-old, 26-30-year-old made up 35%, and the remaining 5% were >30-year-old. The majority of students responded with plans to practice after graduation within acute care or ambulatory care residencies and the remaining 10% for community pharmacy, fellowships, or other paths. This tendency toward specialized training and acute care may have contributed to the students’ choice to enroll in the course. General demographic information can be found in Table 1.

Table 1.

Baseline Demographics of University of California San Diego Skaggs School of Pharmacy Students Completing Both Pre- and Post-assessments Over a 5-Year Period.

	n (%)
Age
<21	0 (0%)
21-25	77 (59%)
26-30	46 (35%)
>30	7 (5%)
Year in school
P1	0 (0%)
P2	0 (0%)
P3	130 (100%)
P4	0 (0%)
Graduation plans
Acute care residency	50 (38%)
Ambulatory care residency	39 (31%)
Community	12 (9%)
Fellowship	15 (12%)
Other	14 (11%)

Script Concordance Results

The individual item results are displayed in Table 2. The mean difference between the 5-year pre-assessment and post-assessment is 0.09 indicating a 9% improvement post-course improvement. There was a statistically significant improvement of scores between pre- and post- elective with a P-value of 0.04, although the overall Cohen’s d between the two groups was 0.14 suggesting a small effect size, that is noticeable but not substantial. This 5-year research cohort (average score of 0.56) outperformed the original 2018 multisite cohort (average score of 0.50), however there was no statistically significant difference between the two results. ⁴

Table 2.

Script Concordance Individual Item Analysis Comparing Pre- and Post- Elective Performance.

Case	Item	Pre-elective Mean (SD)	Post-elective Mean (SD)	Pre- post- elective mean delta	Variance	Skewness	Kurtosis	P	Cohen’s d
A	1	0.79 (0.16)	0.77 (0.20)	−0.02	0.15	2.08	3.35	0.37	−0.02
	2	0.45 (0.09)	0.70 (0.20)	0.25	0.18	1.75	2.05	0.41	0.34
	3	0.48 (0.11)	0.72 (0.23)	0.25	0.16	2.00	2.49	0.43	0.29
	4	0.46 (0.11	0.57 (0.14)	0.11	0.09	1.11	1.32	0.49	0.18
B	1	0.29 (0.09)	0.64 (0.25)	0.35	0.20	1.25	1.38	0.35	0.40
	2	0.57 (0.12)	0.57 (0.13)	0.00	0.08	1.01	2.82	0.38	0.00
	3	0.61 (0.16)	0.85 (0.25)	0.23	0.26	2.21	2.70	0.47	0.23
	4	0.46 (0.11)	0.42 (0.09)	−0.04	0.16	0.39	−0.55	0.35	−0.08
C	1	0.08 (0.03)	0.44 (0.18)	0.36	0.09	1.78	2.28	0.36	0.61
	2	0.82 (0.22)	0.61 (0.17)	−0.21	0.11	0.34	0.95	0.36	−0.20
	3	0.52 (0.11)	0.80 (0.27)	0.28	0.24	2.10	2.48	0.43	0.29
	4	0.07 (0.02)	0.24 (0.08)	0.17	0.04	0.28	−1.24	0.25	0.64
D	1	0.37 (0.17)	0.40 (0.07)	0.03	0.13	0.84	−0.44	0.41	0.12
	2	0.60 (0.20)	0.35 (0.06)	−0.25	0.06	2.36	3.43	0.27	−0.33
	3	0.66 (0.15)	0.54 (0.06)	−0.12	0.06	−0.69	1.86	0.29	−0.20
	4	0.36 (0.09)	0.38 (0.09)	0.03	0.08	1.55	1.85	0.43	0.08
Overall		0.47 (0.12)	0.56 (0.16)	0.09	0.13	1.27	1.67	0.04	0.14

Item Analysis

The average points earned by item was 0.56 (0.24-0.85) out of a maximum potential one point per item. As evident in Table 2, certain questions on topics of bowel obstruction and CYP interaction demonstrated overall poorer performance. Variance analyses were consistent with our original publication ⁴ and found positive variance from pre- to post-test performance as well as primarily right-skewed and leptokurtic distribution consistent with overall improvement in scores. Two related questions (C1 and C4) demonstrated a Cohen’s d greater than 0.6 (see Table 2), which suggest a medium effect size related to the topic of methadone conversions that is likely to be of practical significance.

Discussion

This is the first study to evaluate SCT scores among pharmacy students enrolled in a palliative care elective over a 5-year period. The significant improvement in SCT scores from pre- to post-course assessments highlights the value of targeted education in enhancing clinical reasoning skills in ambiguous situations, a common challenge in palliative care. The improvement, although modest (0.09 mean difference), is statistically significant, indicating that the elective course effectively equipped students with the skills to navigate complex clinical scenarios. The results are particularly noteworthy given that the study spanned 5 years and included adjustments to the course delivery method due to the COVID-19 pandemic, suggesting that the course was resilient and adaptable to different teaching modalities.

Comparing the performance of this cohort with the original 2018 multisite ⁴ cohort further validates the elective’s impact. Although the average score of 0.56 in this 5-year cohort outperformed the previous average of 0.50, the lack of statistical significance between the two cohorts suggests that while improvements were made, there remains room for further enhancement of the curriculum. The item analysis revealed areas where students struggled, particularly with questions related to bowel obstruction and CYP interactions. This insight is valuable for future curriculum development, indicating specific topics that may require more focused instruction or alternative teaching strategies to ensure comprehensive understanding. Alternatively, item analysis of questions related to methadone conversions demonstrated likely practical significance and supports that element of the curriculum remaining in place.

The SCT has been utilized in other studies to evaluate second-year (P2), third-year (P3), and fourth-year (P4) advanced pharmacy practice experience (APPE) students in a one-time assessment, whereas our study extended this evaluation over 5 years.^9-13 In these studies, the number of SCT case questions ranged from 30 to 75, which is more than the 16 cases included in our study.^9-13 One study used the SCT for summative purposes, while our study evaluated it as a formative tool. ¹¹ In two studies, investigators found that the SCT not only assessed clinical reasoning but also enhanced clinical discussions.^9,11 Similar to our study, two other studies assessed the SCT in didactic elective courses focused on basic emergency and psychiatry pharmacy.^12,13 Additionally, a significant improvement in SCT scores was observed over a 5-week period in APPE clinical pharmacy rotations, similar to the improvement seen in our study over 10 weeks of pre-clinical coursework. ⁹

Limitations

As individual student performance on items was not tracked, we were unable to calculate point biserial performance for each item which limits our analysis of the tool’s testwiseness. Differences in performance from question-to-question could therefore be related to problems of test question construction rather than deficits in student performance. This is a single-site study with a selection of students who elect to take a course in pain and palliative care, which could limit generalizability of results. Students were also given time in-class to complete the assessment, which may have biased results.

Conclusion

This study highlights targeted education in palliative care within the pharmacy curriculum, as evidenced by the statistically significant improvement in SCT scores over the 5-year period. Despite the modest score increase, the results highlight the elective course’s effectiveness in enhancing clinical reasoning in complex and ambiguous scenarios. The study also identifies areas for future curriculum improvements and the role of SCT, particularly in topics where students demonstrated challenges. These findings contribute to the ongoing efforts to refine palliative care education and teaching approaches, ensuring that pharmacy students are better equipped to meet the demands of this critical field.

Supplemental Material

Supplemental Material - A Single Academic Site Study of Five Years Evaluating Pharmacy Students’ Palliative Care Clinical Reasoning Using Script Concordance Testing

Supplemental Material for A Single Academic Site Study of Five Years Evaluating Pharmacy Students’ Palliative Care Clinical Reasoning Using Script Concordance Testing by Florence Labrador, Kyle P. Edmonds, Toluwalase A. Ajayi and Rabia S. Atayee in American Journal of Hospice and Palliative Medicine®