Sage Journals: Discover world-class research

Abstract

Palliative care (PC) longitudinal curricula are increasingly being recognized as important in Undergraduate Medical Education (UME). They are however, not yet commonplace, and where they do exist may be implemented without a systematic, prospective approach to curriculum evaluation. This paper describes an implementation of a new longitudinal curriculum at the Perelman School of Medicine (PSOM) at the University of Pennsylvania. We used the Context Input Process Product (CIPP) model, a holistic evaluation model, to assess the local environment, design the curriculum, guide the improvement process, and evaluate outcomes. Comprehensive models such as CIPP provide a more robust approach to curriculum evaluation than outcomes-only models and may be of use to other programs who are implementing new curricula or improving upon existing programs.

Keywords

palliative care undergraduate medical education program evaluation context input process product

Introduction

A palliative care (PC) workforce shortage and a call for all physicians to have primary PC skills highlight the need for longitudinal PC curricula in undergraduate medical education (UME).^1-3 Such curricula align with accreditation standards and other expert recommendations and are successful in improving student knowledge, skills and attitudes.^4-8 However, U.S. medical school deans, students and researchers report that integrated longitudinal UME curricula in PC are underutilized.^9-16 Among existing curricula, few include a structured approach to evaluation of program context, design and process in addition to short-term and long-term outcomes.^17-19 Without a comprehensive approach to evaluation from the inception of a program, educators run the risk of missing important information that might explain how and why outcomes were achieved. For example, a local context may have resources in terms of protected curricular time that could help explain why students gained knowledge (vs less notable knowledge gains in a context where students had to study on their own time). This paper describes the use of the Context Input Process Product (CIPP) Model to design, implement and evaluate a new longitudinal curriculum in PC at the Perelman School of Medicine (PSOM).²⁰ The program is named CARE-7; the CARE acronym stands for “Caring, Attention, Responsiveness and Empathy” while 7 references the 7 overarching goals of the curriculum (Table 1).

Table 1.

CARE-7 Goals and Objectives.

Goals	Objectives	LCME standards
1) Caring for patients throughout their disease process through knowledge of specialist palliative and hospice care	a) Defines the philosophy and role of palliative care across the life cycle and differentiates hospice from PC	7.1
	b) Describes the rationale behind the inter-disciplinary nature of the PC team and describe the roles of various team-members	7.2
	c) Describes disease trajectories for common serious illnesses	7.9
2) Effective communication that incorporates the patient’s values, cultural context and goals	d) Explores patient and family understanding of illness, goals, values and cultural beliefs and practices that inform the plan of care	7.6
	e) Identifies and responds to emotion in patients and families facing serious illness	7.8
	f) Demonstrates patient-centered communication techniques when giving bad news and discussing medical decisions
3) Consideration of psychosocial and spiritual distress and bereavement in the care of seriously ill patients and their families	g) Identifies grief and psychosocial distress in patients and families	7.1
	h) Identifies spiritual and existential suffering in patients and families	7.6
4) Connecting with one’s own emotional response to patients with serious illness	i) Reflects on personal emotional reactions to encountering patients with serious illness	7.1
		7.6
5) Comprehensive pain management for patients with serious illness	j) Describes an approach to pain management in serious illness including knowledge about opioid initiation and titration	7.1
	k) Assesses pain systematically and distinguishes nociceptive from neuropathic pain	7.2
		7.4
6) Providing comfort through excellent symptom management	l) Describes an approach to the evaluation and management of common non-pain symptoms such as constipation and nausea	7.2
6) Providing comfort through excellent symptom management		7.4
7) Care of the patient at the end-of-life	m) Identifies common signs of the dying process and describes treatments for common symptoms at the end of life	7.2
	n) Describes tasks prior to and after a patient dies, including rationale behind medical decision making, anticipatory guidance, death pronouncement and family notification	7.7
		7.8

Program History

Prior to the implementation of CARE-7, typical PC content at PSOM was scant, often voluntary, and designed by educators who were not always content experts. The structure of oversight for the PSOM curriculum was changed in July 2019 coincident with the development of a new leadership team in Medical Education under the leadership of a new Senior Vice Dean. At the same time, a new paradigm for the development of longitudinal courses was developed beginning with a list of key curricular topics that needed enhancements. A plan for definitive direction for longitudinal curricular development across the entire curriculum with defined goals, objectives and curricular elements was put into place. In the Spring 2021, 1 of the authors (S.R.) began discussions with an interested donor to support the longitudinal development in PC. Upon development and submission of a CARE-7 curriculum proposal (N.B.), this came to successful fruition. In the Fall of 2021, a curriculum director (A.S.) and co-directors (R.M., J.O.) were named, and a curriculum evaluation committee was assembled. Curriculum directors are board-certified hospice and palliative medicine physicians who practice specialty PC. The curriculum evaluation committee is composed of the CARE-7 directors, school of medicine leadership, PC leadership as well as curriculum evaluation experts.

CIPP and the Program Evaluation Model

CIPP is a program evaluation model that evaluates 4 components: characteristics of the environment in which the curriculum is to be created (Context), design of content based on the environmental context (Input), monitoring and refining the curriculum (Process) and finally outcomes assessment (Product).²⁰ We used the CIPP model because of its broad scope, its adaptability and its attention to design and process. CIPP includes outcomes assessment but avoids a narrow outcomes-only focus. Instead CIPP emphasizes a context-driven design and process that encourages diverse stakeholder involvement. Such features make it well-suited for the evaluation of complex longitudinal curricula where the evaluation model can serve as a guide for curriculum creation and implementation as well as capture of outcomes. Table 2 summarizes the questions posed and tools used in each step of the model. The curriculum evaluation protocol was submitted and deemed exempt by the Penn IRB.

Table 2.

CIPP Model Questions and Tools.

	Context	Input	Process	Product
Questions	What are the educational needs of students in this environment and what should the goals/priorities of intervention be?	What approaches and strategies should we use in our curriculum to meet our context specific goals?	Is the curriculum being implemented as intended? If not, why not?	Is program well-received by students/faculty/others?
	What are the assets and opportunities of the environment?	Are resources adequate to deliver and evaluate the proposed curriculum?	Are system capabilities strained by the curriculum?	Are students gaining knowledge/skills/attitudes laid out in curriculum objectives?
	What are the problems and barriers that might inhibit meeting the goals?		What else is happening that might facilitate or inhibit implementation?	What else is being accomplished? Are there un-intended consequences?
			Do early indicators suggest we are on target to meet our goals?	Why was our curriculum successful or not in the above measures? Does our conceptual framework explain our results? Is there another explanation?
Tools	Faculty and student interviews	Literature review	Faculty interviews	Student yearly focus groups
	Curricular mapping	Interviews with PC curriculum directors at outside institutions	Student evaluations	Surveys to assess lifelong learning indicators
	Student focus group and survey	Session observations at outside institutions	Student pilot-specific focus groups	Student evaluations
	Hospital of the university of Pennsylvania’s patient advisory council Co-chair interview	Discussion with leadership team/content experts	Student written reflections	Meetings with PSOM educational leaders
	Residency director interview	Inventory and analyze available resources	Requests for clinical PC elective/area of concentration	Student written reflections
		Piloting of interventions	Quiz scores	Requests for clinical PC elective/area of concentration
		Student focus group		Activity of PC specialty interest group
				PC consult numbers on select surgical floors
				Student medical record notes
				Quiz scores

Context Evaluation

Context evaluation focuses on assessing the needs, problems, and opportunities of the educational environment, acknowledging that a curriculum’s design, implementation, and evaluation process will depend upon contextual factors. Our context evaluation questions centered on educational needs as well as assets/opportunities and barriers. Here, we use the terms context, environment, and environmental context interchangeably.

Context Evaluation Methods

To answer context evaluation questions, we conducted semi-structured interviews with twenty-one PSOM faculty, 6 students, the internal medicine residency director, and the co-chair of the Hospital of the University of Pennsylvania’s Patient Advisory Council, to map existing PC content in the curriculum and assess stakeholder viewpoints via 5 questions ranging from current state to opportunities for improvement. A student focus group was also conducted with seven fourth-year students for the purpose of eliciting student perceptions about the current state of palliative care education at PSOM, perceived opportunities to improve and opinions about potential new curricular content. These students were given surveys assessing their thoughts on the importance of learning various palliative care topics and their current comfort with those topics (Table 3). A document mapping existing PC content at PSOM was also created.

Table 3.

Student Survey.

How important do you think it is for the following skills to be included in a medical school curriculum? (N = 6)
	Mean	Median	Mode	Range
Pain/symptom management	4.71	5	5	4-5
Communication with patients/families with serious illness	4.71	5	5	4-5
Knowledge about specialist hospice and PC	3.43	3	3	2-5
Reflecting on personal emotions around patients with serious illness	4.43	5	5	3-5
Caring for patients undergoing the active dying process	4.29	4	5	3-5
How strong do you feel in each of these competencies? (N = 6)
	Mean	Median	Mode	Range
Pain/symptom management	3.71	4	3	3-5
Communication with patients/families with serious illness	4.00	4	4	2-5
Knowledge about specialist hospice and PC	2.71	3	3	1-4
Reflecting on personal emotions around patients with serious illness	3.86	4	3	3-5
Caring for patients undergoing the active dying process	2.71	2	2	1-4

1 = Not important, 2 = Slightly important, 3 = Important, 4 = Very important, 5 = Extremely important.

1 = Very Weak, 2 = Weak, 3 = Neutral, 4 = Strong, 5 = Very Strong.

Context Evaluation Findings

All PSOM stakeholders agreed that communication skill training should be a priority. While some serious illness communication skills training existed, there was no cohesive educational plan with longitudinal integration. Training in pain and symptom management, knowledge of PC and hospice, and mandatory clinical experiences in hospice and palliative medicine with time to debrief these experiences were also identified as curricular needs.

The student survey showed that students considered symptom management and communication with patients and families to be the most critical competencies associated with PC. Knowledge about hospice and PC was considered less important, but students felt relatively weak in their skills in this domain. Students also reported weakness in their ability to care for dying patients, and this was consistent with student reports of lack of exposure to patients near the end of life. Table 2 summarizes these survey results.

We identified many assets and opportunities in the context evaluation dataset, including donor funding which provided for dedicated time for curriculum directors. PSOM had a large network of PC clinical programs with providers who were also skilled educators, and leadership who were supportive of longitudinal PC education. We also identified contextual barriers, such as a need for administrative support and the planned closure of Penn Hospice’s inpatient unit.

While context evaluation tended to validate our own experience of the environment, it elucidated various points. For example, we realized that resources would not allow us to offer a dedicated palliative care clinical rotation; therefore, it was important to provide other opportunities for PC clinical immersion. We also felt empowered to put much of our focus in our ongoing curriculum evaluation on communication skills training, as our stakeholders felt this to be a crucial piece of the curriculum.

Input Evaluation

Input focuses on the implementation and design of interventions based on the environmental context. Our input evaluation questions asked about approaches/strategies to meet curricular goals and key resources.

Input Evaluation Methods

To answer input evaluation questions, we referred to our literature review, 7 interviews with outside curriculum directors and observations of outside programming. Summaries of best practices were collected, and initial curricular design was discussed with PC leadership, content experts and students. Finally, smaller pilots were planned for most parts of the curriculum prior to implementation for all students. Student focus groups and faculty interviews were held after each pilot session to collect feedback along with other data points. For large, single-day teaching interventions, faculty interviews happened directly or soon after the session. For small recurring sessions, faculty group interviews were scheduled once all faculty had a chance to participate. Student focus groups were scheduled in a similar fashion.

Input Evaluation Findings

We used the input evaluation data along with context data to write goals and objectives (Table 1) and an outline of key sessions (Table 4). Goals and objectives were created based on our literature review and relied largely on a 2014 article by Schafer et al who surveyed national PC experts and identified important medical student PC competencies.^21-23 They were then modified based on our context data; for example, the ethics around end-of-life decision making and management of certain symptoms were de-emphasized as they were already covered in other areas of the general curriculum. Finally, goals and objectives were cross-referenced with curricular content standards from the Liaison Committee on Medical Education (LCME) and PSOM Objectives.^8,24

Table 4.

CARE-7 Core Sessions.

Session title	Description	Placement in PSOM curriculum	Student evaluation	Session evaluation	Changes made after pilot
Introduction to palliative care part one	Didactic session introducing basic concepts in PC	Pre-clerkship: Fall of 1^st year	# Large group discussion	# Student evaluations	# Decision to split up material into 2 sessions rather than 1
				# Faculty interviews
				# Student focus group
Introduction to palliative care part two	Didactic session reviewing basic concepts in PC and introducing more complex topics such as disease trajectories and PC across settings	Pre-clerkship: Spring of 1^st year	# Large group discussion	# Student evaluations	NA
				# Faculty interviews
				# Student focus group
Hospice and palliative medicine experience	Multi-part experience including an immersion experience in a hospice or PC setting followed by a small group session during which students interview bereaved family members and then debrief their experiences	Pre-clerkship: Fall of 2^nd year, part of doctoring course	# Small group discussion	# Student evaluations	# Expansion of immersion experiences to include home visits
				# Faculty interviews	# Creation of short, recorded talks to prepare students prior to immersion and small-group sessions
				# Student focus group
Pain and symptom management	Asynchronous didactic session on pain and symptom management followed by a small group evolving video case simulation	Occurs during “PC skills practicum day,” a full day of required PC content midway through clerkship year	# Quizzes	# Student evaluations	# Additional asynchronous didactic sessions added to create more time for discussion during in-person sessions
			# Small group discussion	# Faculty interviews
			# Small group discussion	# Student focus group # quizzes
Coping with serious illness and end-of-life	Small group session where students read and discuss a poem and then discuss their own emotional responses to encountering patients with serious illness	Medicine clerkship	# Small group discussion	# Student evaluations	# Addition of a second poem to the session
				# Faculty interviews	# Increased emphasis on student discussion over facilitator tips or suggestions
				# Student focus group
Palliative care assessment	Students choose 1 patient with serious illness to do a PC holistic assessment with the goal of understanding their experience of illness and practicing communication skills	Surgical clerkship	# Reflection assignment	# Student evaluations	# Improved messaging to students and faculty that the assignment is about understanding the patient experience and is not a goals of care discussion
			# Student medical record notes	# Faculty interviews
			# Discussion with surgical team	# Medical record notes
				# Student reflections
				# Number of PC consults on select surgical services
				# Student focus group
Nuts and bolts of end-of-life care	Didactic session on caring for patients at the end of life	Post-clerkship: Takes place during residency preparation electives in 4^th year	# Small group discussion	# Student evaluations	# Increased discussion about the role of hospice care
				# Faculty interviews
				# Student focus group
Communication curriculum	“Delivering serious news”: Didactic followed by small group skills practice using drills and video case simulation	Occurs during “PC skills practicum day,” a full day of required PC content midway through clerkship year	# Formative small group assessment	# Faculty interviews	# Increased student engagement/interaction during didactic sessions
				# Student evaluations
				# Surveys
				# Focus groups
	“Goals of care conversations part one: drills”: Didactic followed by small group sessions where students practice using skills through evolving cases	Integrated into 4^th year required specialty-specific tracks	# Formative small group assessment	# Faculty interviews # Student evaluations # Surveys # Focus groups	NA
	“Goals of care conversations part two: skills practice”: Small group sessions using live simulated patients and VitalTalk^© methodology	Integrated into 4^th year required specialty-specific tracks	# Formative small group assessment	# Faculty Interviews # Student evaluations # Surveys # Focus groups	# Inclusion of time for short didactic content review

Based on what we learned in the literature review and discussion with outside institutions, we included clinical exposure to patients with PC needs, simulation, and a range of teachers across specialties and disciplines.^5,18,25 For communication skills sessions we used methodology from VitalTalk^©, a nationally recognized communication skills training program.²⁶ As other successful curricula have done,⁴ we took a step-wise approach to designing the curriculum: pre-clerkship students are introduced to core concepts and skills and exposed to clinical PC; clerkship students then build on these skills by learning more advanced skills and concepts that they are prompted to apply clinically; and post-clerkship students practice advanced skills and have opportunities to pursue additional training through PC clinical electives or a PC area of concentration. CARE-7 coaches mentor students in the PC area of concentration and help teach curricular sessions. Coaches are all faculty members who practice primary palliative care rather than primarily specialist palliative care; this choice was intentional as we were hoping that coaches could model for students the importance of primary palliative care skills regardless of specialty.

Based on our input evaluation we created the following mission statement: “The CARE-7 Curriculum will prepare all medical students, regardless of future specialty, to care for people with serious illness in a multidisciplinary team through simulation and experiential sessions that foster lifelong learning of empathic patient and family-centered communication skills, holistic symptom management and support.” We felt that it was important to include lifelong learning as a principle in our curriculum vision because while students will not be experts in PC skills or topics by the time they graduate, we hope to foster an approach to learning that will help students to continue to develop these skills over their careers.

As with our context evaluation, our input evaluation revealed some unanticipated information. There was variation in expert opinion about the extent to which medical students should be expected to master advanced communication skills such as discussions about goals of care, for example, and whether students should have a mandatory PC rotation. We opted to give students an opportunity to practice having a goals of care conversation. Because our PC and hospice services did not have the bandwidth for a mandatory PC rotation, we created a short early-immersion experience and embedded PC assessment of a patient in an existing clerkship. Furthermore, advice from outside programs that signposting the cohesiveness of a longitudinal curriculum can be difficult led to the creation of a CARE-7 logo.

Process Evaluation

Process evaluation is the observation of how the implementation process is proceeding, noting any areas that diverge from the plan or that require modification. In our process evaluation, we asked if our PC curriculum was being implemented as intended and sought to learn about early indicators of desired outcomes. Given our ambitious roll-out plan, a process evaluation was especially important.

Process Evaluation Methods

A large part of process evaluation focused on faculty interviews, student evaluations and focus groups that took place after session pilots. Detailed notes were taken during these meetings and used in leadership discussions to guide changes for future sessions. One session, called the “PC Assessment,” asks students to perform a comprehensive evaluation of 1 of their patients during Surgical Clerkship and evaluate key PC domains. Students write a note in the medical record, bring pertinent findings back to their team, and write a reflection on their experience. Two CARE-7 team members are currently working on qualitative analysis of electronic medical record (EMR) notes and reflections to identify common themes. At the time of this writing 42 student reflections and EMR notes have been collected and coding is ongoing. A new PC Specialty Interest Group was created in the Fall of 2021, and we are monitoring activity as well as clinical PC elective and area of concentration requests.

Process Evaluation Findings

At the time of this writing 8 out of 10 curricular sessions have been piloted or implemented for all students. Based on the corpus of process evaluation data, curricular sessions have gone as intended and been well-received by students as well as faculty. Reflections and medical record notes show that students are applying knowledge and shifting attitudes in the desired direction. The PC specialty interest group had 7 meetings in its first year and started a program to help hospice patients write a memoir, and multiple students signed up for the PC area of concentration directly after its approval.

All sessions have undergone changes after session pilots (Table 4); changes are tracked and discussed at curriculum director meetings. Some early challenges have included faculty availability to teach sessions and attend faculty development, and logistical hurdles of having pre-clinical medical students spend time in clinical settings.

There are many changes that we made to sessions based on student and faculty input that would have been missed if we had focused on outcomes data alone. For example, we were surprised to learn from students that they valued having evolving video cases to learn communication skills rather than using simulated patient encounters for every session. Another surprise was the interest and engagement of medical students, residents, and fellows in the process of evaluating CARE-7; learners have helped by designing the logo, running focus groups, evaluating reflections and quiz scores and volunteering to help teach sessions. While all planned sessions were given and well-attended, the observation that some students are not able to attend all sessions led to the creation of make-up assignments.

Product Evaluation

Product evaluation refers to the learning outcomes of a curriculum and determines how well it has met the needs of its stakeholders. Our product assessment questions attempt to answer whether sessions are well received, effective in imparting curricular objectives, and to attempt to uncover any un-intended consequences of the curriculum. It also seeks to answer why our curriculum was successful or not in the above measures.

Product Evaluation Methods

Student surveys are being administered at various points in their medical school training to assess student indicators of a lifelong learning approach to communication skill building and to assess the learning environment. Because a growth mindset has been associated with a lifelong learning approach in medical students, surveys include an instrument measuring students’ growth mindset toward communication skills.²⁷ Yearly student focus groups are planned to assess student’s thoughts about the value of the curriculum, gain insight about their attitudes to learning curricular content, and assess the impact of the hidden curriculum. These yearly focus groups have a broader scope than the pilot-specific focus groups used in process evaluation and will undergo thematic analysis. Bi-annual meetings of PSOM faculty involved in communication skills sessions are planned to ensure integration of CARE-7 communication skills content with other communication skills sessions and to assess the impact of these sessions. Given that 1 of the CARE-7 sessions takes place during the surgical clerkship, palliative care consults to select surgical services will be monitored to assess for any changes to clinical care that might arise. Other tools used in product evaluation are listed in Table 2. Data is collected in a database and qualitative analysis of focus groups and reflections is summarized.

Product Evaluation Findings

At the time of this writing, product evaluation is still in early stages. We are encouraged by initial student enthusiasm for sessions as assessed by session evaluations, focus groups, reflections, and interest in the specialty interest group and area of concentration. Evaluations showed that the “PC Skills Day,” a part of the clerkship year “Practicum Series” was the most highly rated practicum day of the clerkship year. Initial student surveys surrounding communication skills sessions show that students feel sessions create a safe learning environment, and that student indicators of a lifelong learning approach are improved after CARE-7 skills sessions. Quiz scores show that students are learning key concepts. Further data collection as students progress through medical school is needed to understand survey implications and general curricular impact.

Conclusion

Our use of the CIPP model has been invaluable in the process of designing and implementing CARE-7. Because every curriculum will be developed in a context with unique resources, unique barriers and unique stakeholder needs, a one-size-fits-all approach to a PC longitudinal curriculum would be impractical and limiting. No curriculum is a static creation. Ideally, curriculum directors are responsive to the need to change and constantly seek to improve. A successful curriculum will design inputs based upon the context, involve stakeholders in process evaluation and design an outcomes evaluation informed by stakeholder needs. CIPP is a holistic evaluation model that can be applied across contexts and facilitates iterative review of the the curricular processes while paying attention to curricular products.

While longitudinal palliative care curricula are increasingly recognized as critical to medical training, there is a lag in emphasis on robust evaluation to support such curricula.²⁸ When such curricula are implemented, evaluators often cite prospective program evaluation as a challenge.^6,29 To our knowledge, CARE-7 is the first U.S. palliative care UME longitudinal curriculum to describe the use of an evaluation model for prospective program development as well as outcomes assessment. The resulting curriculum is an ambitious program composed of 10 core sessions over 4 years utilizing instructional tools including clinical experiences, various types of simulation, VitalTalk^© methodology and interdisciplinary teachers who can role model both primary and specialist palliative care skills.

In our own use of the CIPP framework to evaluate CARE-7 we have identified lessons learned. First, having a curriculum evaluation committee comprised of some members with extensive experience with curriculum evaluation was hugely helpful in guiding our efforts. Next, stakeholder involvement in the process has been helpful both in terms of data collection as well as fostering ongoing stakeholder investment in the project. Finally, we have found that the focus on fostering students primed for lifelong learning has provided further direction to our curriculum evaluation and shaped how we improve upon and evaluate sessions.

In conclusion, the use of a comprehensive program evaluation model such as CIPP can help with the design and implementation of context-specific PC curricula and improve the quality of evaluation. Utilizing the structured approach of the CIPP model allowed us to uncover findings we might not have otherwise and dynamically improve the curriculum, ultimately making the educational experience more relevant and impactful for our students. We look forward to learning more from product evaluation and plan for future scholarship to describe findings in this component of CIPP.

Footnotes

Acknowledgments

The CARE-7 curriculum is generously funded by Barbara M. Jordan, a Penn Medicine Board member and spouse of the late Henry A. Jordan, MD, a Penn alumnus (M’62 from the School of Medicine).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Barbara M. Jordan.

ORCID iD

Alana Sagin

References

Kamal

Wolf

Troy

Leff

Dahlin

Rotella

, et al. Policy changes key to promoting sustainability and growth of the specialty palliative care workforce. Health Aff. 2019;38(6):910-918. doi:10.1377/hlthaff.2019.00018. PubMed PMID: 31158018

Head

Schapmire

Earnshaw

Chenault

Pfeifer

Sawning

, et al. Improving medical graduates’ training in palliative care: Advancing education and practice. Adv Med Educ Pract. 2016;7:99-113. doi: 10.2147/amep.S94550. Epub 20160224. PubMed PMID: 26955298; PMCID: PMC4772917.

Lloyd-Williams

Carter

. Can medical education extend palliative care? Palliat Med. 2003;17(7):640-642. doi:10.1191/0269216303pm831xx. PubMed PMID: 14594157.

Ellman

Fortin

Putnam

Bia

. Implementing and evaluating a four-year integrated end-of-life care curriculum for medical students. Teach Learn Med. 2016;28(2):229-239. doi:10.1080/10401334.2016.1146601. PubMed PMID: 27064725.

Bush

Gratton

Kabir

Enright

Grassau

Rice

, et al. Building a medical undergraduate palliative care curriculum: lessons learned. J Palliat Care. 2021;36(1):29-37. doi: 10.1177/0825859720916565. PubMed PMID: 32284024. Epub 20200413.

Radwany

Stovsky

Frate

Dieter

Friebert

Palmisano

, et al. A 4-year integrated curriculum in palliative care for medical undergraduates. Am J Hosp Palliat Care. 2011;28(8):528-535. doi: 10.1177/1049909111406526. PubMed PMID: 21518705. Epub 20110425.

Weissman

Quill

Block

. Missed opportunities in medical student education. J Palliat Med. 2010;13(5):489-490. PubMed PMID: 20420547. doi:10.1089/jpm.2010.9838

Liason committee on medical education. https://lcme.org/. Available from: [November 17, 2022].

Sullivan

Warren

Lakoma

Liaw

Hwang

Block

. End-of-life care in the curriculum: A national study of medical education deans. Acad Med. 2004;79(8):760-768. doi:10.1097/00001888-200408000-00011. PubMed PMID: 15277133.

10.

Sullivan

Lakoma

Block

. The status of medical education in end-of-life care: A national report. J Gen Intern Med. 2003;18(9):685-695. doi: 10.1046/j.1525-1497.2003.21215.x. PubMed PMID: 12950476; PMCID: PMC1494921.

11.

Chiu

Cheon

Lutz

Lao

Pulenzas

Chiu

, et al. Inadequacy of palliative training in the medical school curriculum. J Cancer Educ. 2015;30(4):749-753. doi:10.1007/s13187-014-0762-3. PubMed PMID: 25487030.

12.

Hammel

Sullivan

Block

Twycross

. End-of-life and palliative care education for final-year medical students: A comparison of Britain and the United States. J Palliat Med. 2007;10(6):1356-1366. doi:10.1089/jpm.2007.0059. PubMed PMID: 18095815.

13.

Smith

Schaefer

. Missed opportunities to train medical students in generalist palliative care during core clerkships. J Palliat Med. 2014;17(12):1344-1347. doi:10.1089/jpm.2014.0107. PubMed PMID: 24971609.

14.

Horowitz

Gramling

Quill

. Palliative care education in U.S. medical schools. Med Educ. 2014;48(1):59-66. doi:10.1111/medu.12292. PubMed PMID: 24330118.

15.

Billings

Engelberg

Curtis

Block

Sullivan

. Determinants of medical students' perceived preparation to perform end-of-life care, quality of end-of-life care education, and attitudes toward end-of-life care. J Palliat Med. 2010;13(3):319-326. doi: 10.1089/jpm.2009.0293. PubMed PMID: 20178433; PMCID: PMC2883506.

16.

Dickinson

. Thirty-five years of end-of-life issues in US medical schools. Am J Hosp Palliat Care. 2011;28(6):412-417. doi: 10.1177/1049909110397608. PubMed PMID: 21282128. Epub 20110130.

17.

DeCoste-Lopez

Madhok

Harman

. Curricular innovations for medical students in palliative and end-of-life care: A systematic review and assessment of study quality. J Palliat Med. 2015;18(4):338-349. doi: 10.1089/jpm.2014.0270. PubMed PMID: 25549065. Epub 20141230.

18.

Kozhevnikov

Morrison

Ellman

. Simulation training in palliative care: state of the art and future directions. Adv Med Educ Pract. 2018;9:915-924. doi: 10.2147/amep.S153630. PubMed PMID: 30574008; PMCID: PMC6292390. Epub 20181207.

19.

Fitzpatrick

Heah

Patten

Ward

. Palliative care in undergraduate medical education-how far have we come? Am J Hosp Palliat Care. 2017;34(8):762-773. doi: 10.1177/1049909116659737. PubMed PMID: 27468933. Epub 20160728.

20.

Stufflebeam CFMaTK

. Evaluation Models. Boston, MA: Kluwer Academit Publishers; 2000.

21.

Meekin

Klein

Fleischman

Fins

. Development of a palliative education assessment tool for medical student education. Acad Med. 2000;75(10):986-992. doi:10.1097/00001888-200010000-00011. PubMed PMID: 11031142.

22.

Schaefer

Chittenden

Sullivan

Periyakoil

Morrison

Carey

, et al. Raising the bar for the care of seriously ill patients: Results of a national survey to define essential palliative care competencies for medical students and residents. Acad Med. 2014;89(7):1024-1031. doi:10.1097/acm.0000000000000271. PMC4077186. PMCID. PubMed PMID: 24979171.

23.

Bush

Roze des Ordons

Chary

Boyle

. The development and validation of updated palliative and end-of-life care competencies for medical undergraduates in Canada. J Palliat Med. 2019;22(12):1498-1500. doi: 10.1089/jpm.2019.0160. PubMed PMID: 31486699. Epub 20190905.

24.

Perelman school of medicine: Medical education program objectives webpage. https://www.med.upenn.edu/student/medical-education-program-objectives.html. Available from: [cited 2023 January 21].

25.

Boland

Barclay

Gibbins

. Twelve tips for developing palliative care teaching in an undergraduate curriculum for medical students. Med Teach. 2019;41(12):1359-1365. doi: 10.1080/0142159x.2018.1533243. PubMed PMID: 30689479. Epub 20190128.

26.

VitalTalk. https://www.vitaltalk.org. Available from: [November 9, 2022].

27.

Babenko

Daniels

Ross

White

Oswald

. Medical student well-being and lifelong learning: a motivational perspective. Educ Health. 2019;32(1):25-32. doi:10.4103/efh.EfH_237_17. PubMed PMID: 31512589.

28.

Lloyd-Williams

MacLeod

RDM

. A systematic review of teaching and learning in palliative care within the medical undergraduate curriculum. Med Teach. 2004;26(8):683-690. doi:10.1080/01421590400019575. PubMed PMID: 15763870.

29.

Denney-Koelsch

Horowitz

Quill

Baldwin

. An integrated, developmental four-year medical school curriculum in palliative care: A longitudinal content evaluation based on national competency standards. J Palliat Med. 2018;21(9):1221-1233. doi: 10.1089/jpm.2017.0371. PubMed PMID: 29768072. Epub 20180516.

Evaluation of a Palliative Care Longitudinal Curriculum for Medical Students Using the Context-Input-Process-Product Model

Abstract

Keywords

Introduction

Program History

CIPP and the Program Evaluation Model

Context Evaluation

Context Evaluation Methods

Context Evaluation Findings

Input Evaluation

Input Evaluation Methods

Input Evaluation Findings

Process Evaluation

Process Evaluation Methods

Process Evaluation Findings

Product Evaluation

Product Evaluation Methods

Product Evaluation Findings

Conclusion

Footnotes

Acknowledgments

Declaration of Conflicting Interests

Funding

ORCID iD

References