Abstract
OBJECTIVES
Medical school curricula have increasingly incorporated topics and content related to health equity and affiliated social determinants of health. However, there is limited literature to guide how programs might measure the success of these initiatives. Previous studies assessed medical student attitudes and perceived knowledge, preparedness, and skills. Based on self-reported measures of these attributes, we compared within-group and between-group differences at the onset of a novel equity-focused curriculum implementation.
METHODS
A multi-component approach to “thread” lectures, panel discussions, and other content dedicated to health equity concepts was assessed using adapted versions of two validated survey instruments of the measured constructs. Baseline data were collected prior to coursework and at follow-up early in students’ second year assessed change attributable to the equity-focused curriculum thread, with additional comparison to a cohort of second-year students who had no exposure to the curriculum. Data were collected at the beginning of academic years 2021–2022 and 2022–2023.
RESULTS
The multivariate analysis of variance indicated significant change over time (p < 0.001) with the analyses of variance identifying students’ perceived current skills and topic knowledge increasing over time. No significant differences were found between two separate groups of M2 students.
CONCLUSIONS
Students’ perceived skills at working with diverse patient populations and knowledge of topics focused on health equity increased across the study, despite a much smaller response rate for the same student cohort at follow-up. Students’ perception that they are prepared to care for patients of diverse backgrounds was unaffected. Attitudinal assessment revealed a ceiling effect at baseline, which should be explored further with longitudinal assessment. For the ongoing effort to evaluate the success of equity-focused curricula and programs, this study contributes evidence of change on some but not all outcomes, and can help guide other programs in determining which outcomes best reflect areas of programmatic need and impact.
Introduction
Medical training programs and their accrediting bodies 1 have made progress toward incorporating curricula on health inequities and affiliated social determinants responsible for these inequities. This is critical to ensure that U.S. medical students are trained to recognize and address inequities in patient care due to a host of social, environmental, and economic factors that have created longstanding inequities in health, such as insufficient education, unemployment or job insecurity, housing and transportation needs, access to affordable healthcare, and the various impacts of living in impoverished environments. 2 Recognizing that an initial focus on the topics of social determinants and inequities is fundamental to teach students to interact with diverse patient populations and ultimately to the improvement of health outcomes, the Liaison Committee on Medical Education (LCME) now requires that instruction include topics of cultural competency, resource-limited populations and communities, health care disparities and inequities, and diversity and inclusion. 3 In addition, we want to be sure that our curriculum illuminates how the lack of health equity manifests itself in our state (Kentucky, a state where inequitable social and health risk factors are well documented, and where life expectancy is the second lowest among the 50 states 4 ) and other states with very poor health outcomes.
We intentionally chose to focus on foundational content for our initial investigation to uncover the perceptions of students during their first 2 years of medical school. Our hope was to be able to measure changes in these perceptions as students progress through 4 years of medical school. This work is now in progress. Nonetheless, we acknowledge that there are other important topics such as patient advocacy, cultural awareness and competence, and social accountability that should have a place in curricula for medical students and residents. Studies demonstrating the success of equity-focused education are especially necessary, given recent state-level legislation decisions that significantly restrict diversity, equity, and inclusion teaching at publicly funded K-12 schools 5 and similar restrictions beginning in higher education. 6 Should any coming limitations imposed upon the curriculum offered in medical education and clinical training, harkened by recent efforts to limit racial or ethical background considerations during the medical school admission process, 7 it will alter the support that medical programs can offer students and limit their skills in treating their future patients. It may also threaten the accreditation, and potential workforce availability of, medical and other health professional schools in affected states. 8
The components of such curricula have been the subject of systematic review,9,10 and a comprehensive programmatic tracking guide for monitoring the various educational offerings has been made available as a “Tool for Assessing Cultural Competence Training (TACCT)” by the Association of American Medical Colleges (AAMC). 11 For most programs, training aims to help students develop skills in identifying and addressing inequities within the scope of physician practice, as part of an over-arching competency-based education. 12 However there has been little guidance for programs on suitable assessment methods that might indicate areas of programmatic success for these training initiatives.
This study was not intended for use as a student-level analysis, but rather, the goal was to add to the limited literature on the measurement of attitudes, knowledge, preparedness, and skills of medical students on health equity-related concepts as a programmatic evaluation need. The study focuses on the assessment of students’ baseline ratings when entering medical school, before delivery of a suite of equity-focused curricular change interventions. In a recent review of similar interventions, 9 only three prior studies reported on evaluation during the early training years; of these, one was evaluation of a specific student-run clinic experience 13 and one provided evaluation only of attitudinal change. 14 Only one reported on more than one dimension of change, but used a survey that was created internally and contained a limited number of items (nine items across three self-evaluation constructs). 15
Developed curricular elements were initiated within our university's medical training course sequence at the beginning of the fall semester 2021 as an innovative four-year curriculum—the health equity and advocacy “thread.” The thread was designed to scaffold various concepts pertaining to inequities and social determinants of health; the accompanying competencies related to the HEAT thread are shown in Appendix 1. Figure 1 depicts the approach to curricular changes instituted during the first year of the curriculum.
Progression of equity and advocacy content throughout the M1 year.
Our primary aim was to characterize the impact of the thread on measures of equity-focused attitudes, knowledge, preparedness, and skills of medical students. Comparison of the baseline measurement from prior to the start of medical year (M1) was made to assess the impact of the thread approximately 1 year later, at the start of year 2 (M2). An additional comparison was made to the previous M2 class, who had not been presented with this curricular intervention during their M1 year. Thus, this study helps to characterize areas of change through both within- and between-group comparisons.
Current literature citing these tools has not reported a pre-training baseline, and no prior study made comparison to students not formally exposed to such content in the same setting. Additionally, as we adapted prior assessment tools for this purpose, we added to the limited literature that tests the suitability or sensitivity of existing tools for assessing change in these constructs. Such a study may then serve as a potential guide for programs seeking to assess possible outcomes of their work to incorporate content on health inequities into the medical curriculum.
Method
The survey was first introduced in July 2021 alongside the new curricular content and continues to be annually administered to each cohort within our medical school. The focus of this article is the initial iteration of the survey administered to two preclinical medical student cohorts enrolled in the University of Kentucky College of Medicine between July 2021 through August 2022.
Survey creation
We created a survey comprised of Likert-scale items from two validated surveys, the Self-Assessment of Perceived Level of Cultural Competence (SAPLCC) 16 and the Cross-Cultural Care Survey (CCCS). 17 In addition to items from the SAPLCC and CCCS described below, the survey included questions specific to our medical school and whether these were useful in preparing for interactions with diverse patient populations (qualitative and quantitative). Demographic questions included race/ethnicity, academic year, gender, and age.
The SAPLCC was initially created to assess perceived levels of cultural competency in pharmacy students 18 but has been adapted for use in other medical and allied health professions.19,20 Among the six domains of the SAPLCC, we selected—with slight wording changes—11 items for perceived subject area knowledge (eg, “How knowledgeable are you about…”: social and cultural issues in geriatrics; health risks experienced by diverse patient populations). Responses on these Topic Knowledge items were on a 4-point Likert scale (Not at all, A little, Quite a bit, Very).
The CCCS has been formally validated with residents,21,22 and questions were previously revised to suit medical students’ experiences in a version obtained by permission from the survey author (the HMS-CCC V.2: 7.29.09). 23 We made minor changes to the 12 items from the HMS-CCC “Attitudes” Domain and added one item to comprise our assessment of Opinions of Health Equity: following the question “I don’t consider myself to be part of any particular culture” we added, “I don’t consider myself to have strongly held religious beliefs.” Items on this scale were measured on a Likert scale (Strongly disagree to Strongly agree) that differed from the original version by including a “neutral” response option as a middle point (7-point Likert). We used all 10 items from the “Skillfulness” Domain (eg, skill when interacting with diverse patient populations in activities such as taking a social history or identifying patient mistrust or need for an interpreter). We also included all 12 items from the “Preparedness” Domain without alteration. Within the Preparedness question set (“how well prepared you feel to care for patients who are…”), we added one item: following the question “Who are persons with disabilities?” we added, “Who are illiterate?” Both the Current Preparation and Current Skills question sets were on a 5-point Likert (Not at all prepared/skillful to Very prepared/skillful).
Survey administration
Data reported here were from electronic surveys (Qualtrics, Provo, UT) administered at three timepoints: baseline (July 2021, just before entering medical school for the class which entered in 2021) and at follow-up early in these same students’ second year (August 2022), and in September 2021 with students from a second cohort who were then beginning their M2 year (for the class which entered in 2020). The survey was shared with an entire cohort of students at once. Selection was therefore limited to all members of the cohorts who were exposed to the curricular thread which is the focus of this survey. In total, the invited sample included all 212 members of the entering cohort of 2020 (M2 s surveyed) and all 204 members of the entering cohort of 2021 (surveyed before and after M1 year). For the incoming M1 baseline data collection, surveys were sent out immediately following an in-person introduction by the College's Associate Dean for Diversity, Equity and Inclusion and the study PI during an in-person, weeklong orientation. For the two groups of M2 s, the survey went out asynchronously without an opportunity for an in-person introduction. To enhance our response rate, we sent two reminder e-mails to students separated by 5–10 days; participants had a total of 2 weeks to respond. Completion and return of the survey constituted consent. All procedures were approved by the University of Kentucky Institutional Review Board (protocol #67746). A waiver of formal documentation of informed consent was granted such that, in lieu of written consent, all survey participants read a cover letter for the research study including details about associated risks and precautions prior to accessing the survey.
Analysis
To evaluate both the cross-sectional and repeated measures comparisons available from our survey administrations, we considered a sample comprised of both the cohort that matriculated in the academic year 2020 (2021 M2 Cohort) and the cohort that matriculated in the academic year 2021 (the 2021 M1/2022 M2 Cohort).
An initial multivariate analysis of variance (MANOVA) was conducted to assess differences across waves (data collected in 2022 compared to data collected in 2021) and between cohorts in their respective M2 years (2022 M2 students compared to 2021 M2 students) on Topic Knowledge, Current Preparation, and Current Skills as outcomes. However, as assumptions of both normality and sphericity were violated, the MANOVA was conducted using a rank-based Wilks’ lambda. No power analysis was conducted as the sample consisted of all students who participated in the survey, and the primary analyses included all students for whom we had complete data.
Analyses were performed using the R Statistical Software Package version 4.1.0 for Windows (R Core Team, Vienna, Austria).
Results
The demographic composition and descriptive statistics for the primary outcomes are described in Table 1 as the “Full Sample” (top) and “2021 M1/2022 M2” (bottom, reflecting only those 2021 M1 students who also took the survey as M2 students in 2022), respectively. Completion rate was 21.2% for the 2021 M2 cohort. For the 2021 M1/2022 M2 cohort, participation was 77.9% (2021 M1 s) and 45.1% (2022 M2 s). A small number of participants (range: 3.1–20%; see Table 1) did not complete the scales of interest and, as such, were excluded from the primary analyses.
Demographics of samples and descriptive statistics for outcomes.
Results from the Opinions of Health Equity of the survey indicated significant ceiling effects across all items—that is, scores were heavily skewed toward favorable opinions with few scores indicating unfavorable opinions—and concomitantly low reliability (Cronbach's αFull Sample = 0.66; Cronbach's α2021 M1/2022 M2 Cohort = 0.57). Average ratings were at or above 5.5 on a 7-point scale for each of the two survey administrations. Notably, scores on the baseline M1 2021 administration, a group for which no formal medical school training had yet been initiated, were the highest and showed the least variability. These data are shown in the demographics of Table 1, but no further analyses of these data are described due to the significant ceiling effect.
The initial MANOVA assessed differences between cohorts and waves on Topic Knowledge, Current Preparation, and Current Skills as primary outcomes; Figure 2 depicts these comparisons. This MANOVA simultaneously included all participants, including those students who only took the survey in either their M1 or M2 year, as well as a within-subjects comparison of only the 60 students who participated in both their M1 and M2 years. These comparisons are shown in Table 2.
Map of primary comparisons.
MANOVA and pairwise comparison results.
Note: Wave = 2021 cohort only, 2022 M2 scores compared to baseline 2021 M1 scores; Cohort = 2022 M2 scores (2021 cohort) compared to 2021 M2 (2020 cohort) scores.
p-values obtained from post hoc t-tests adjusted using the Benjamini-Hochberg correction.
Results from the ranked MANOVA indicated a significant overall effect (Ranked Wilks’ Lambda = 0.85, p < 0.001) with a parametric MANOVA remaining consistent and yielding significant effects for both Wave (p < 0.001) and Cohort (p < 0.01). Subsequent a priori specified pairwise comparisons on each outcome found a significant change in both Current Skills (M1 = 2.61, M2 = 3.18; t = −4.791 (230); p < 0.001) and Topic Knowledge (M1 = 2.11, M2 = 2.40; t = −4.124 (215); p < 0.001) over time (Wave) for the 2021 M1/2022 M2 cohort. No significant differences were found between 2021 M2 s and 2022 M2 s on any measure (ps > 0.05). Means and confidence intervals for all outcomes are shown in Figure 3.
Factor mean scores and confidence intervals by wave and cohort.
Follow-up one-way analyses of variance on only the 2021 M1/2022 M2 cohort then yielded significant changes over time for Current Skills (M1 = 2.57, M2 = 3.14; F = 19.83 (1, 51); p < 0.01) and Topic Knowledge (M1 = 2.14, M2 = 2.36; F = 4.52 (1, 49); p < 0.05), as was the case in the full sample. These results largely suggest that students’ perceptions of their current skills to work with diverse patient populations and their knowledge of topics focused on health equity were consistent factors that improved across the duration of the study as they were the factors that were significant across both samples. Further, these results indicate that the 2022 M2 s had similar perceptions of their knowledge and skills for working with diverse patient populations as compared to the 2021 M2 s.
Discussion
In a 2019 review of cultural competency interventions for medical students, fewer than 10% of interventions, largely comprised of targeted lectures, were formally evaluated. 19 This study fills the need for comparative data and multi-year assessment, including a benchmark for the baseline level of key attributes (attitudes, knowledge, preparedness, and skills related to health inequities) prior to entering medical school. This is especially needed as more academic medical centers begin to initiate and seek to evaluate similar programs.
Our findings highlight four important discoveries. First, given the novelty of these assessments, our data helps in assessing sensitivity of the items to change across the selected constructs. Importantly, we found that opinions regarding various related topics of health equity (derived from the CCCS “Attitudes” Domain; eg, “Understanding a patient's social circumstances is vital to developing a treatment plan”) reflected strong levels of agreement across all assessments (RangeMeans = 5.47–5.62 on a 7-point scale). The significant ceiling-level responses from medical students on the attitude items was not noted in the formative CCCS administrations (data on Attitudes were either omitted 23 or reported only for residents rather than medical students, 22 but other studies have shown significant attitude change using similar scales (eg, the Crandall Medical Student Attitudes Toward the Underserved).14,24–26 Potential factors driving this ceiling effect are numerous. Response biases toward social acceptability could contribute to the high opinions at baseline and lack of change with time; also this subscale had a low reliability coefficient for both the full sample and for the 2021 M1/2022 M2 group. Alternatively, responses may reflect either larger social trends or a potential University-specific recruiting bias toward students who are highly aware of, or have relevant experiences on issues of health equity.
Second, as shown in a prior study,15,27 we found the expected increase in self-reported increase in knowledge across program years from M1 to M2 in the within-group comparison. This indicates the chosen assessment is sensitive to students’ perceived advancement in core curricular competencies. There was no difference in stated knowledge between the two class years (M2 groups).
Third, accompanying the change in knowledge in the 2021 M1/2022 M2 group, we found significant change in these students’ rating regarding their skills but paradoxically no change in student perception of their preparation for diverse patient encounters. This was an unexpected result as aside from standardized patient encounters there is no formal patient exposure during the first year of medical school. Given the level of training in the M1 and M2 years, no change in skill level ratings was expected. However, medical students are still receiving training on how to interview patients with a focus on social determinants of health and how to apply this with standardized patients, even if not applied in formal patient encounters. This may explain why students felt more skillful (eg, on taking a social history), even while not reporting a change in preparedness (eg, in feeling prepared to care for patients who are transgender or from cultures that differ from their own). Students in general might not feel prepared yet to interact with patients unsupervised because they are in the preclinical years, regardless of patient complexity or background. As with all content in the preclinical years, we are evaluating the effectiveness of the thread with student evaluations after each course, and with evaluation methods such as multiple-choice questions, asynchronous group projects, and reflection papers. Over time, this will allow us to assess if our interventions are making a meaningful impact in the student education beyond the self-perception of the students.
Our final observation relates to the returning M2 (2021) class, representing students who enrolled prior to the curricular implementation, and who therefore would have had core introductory clinical medicine coursework without the curricular changes of the thread depicted in Figure 1. While none of the differences were significant, their scores were lower overall than the class entering medical school in 2021. As described above, participating M1 s were primed for their baseline assessment by an in-person introduction during a lengthy orientation session. It is unclear if this difference reflects the lower completion rate of the survey by initial or returning M2 participants, or any positive priming due to the in-person introduction session that was provided to the M1 s at the time their survey was introduced. Further longitudinal data will be needed to assess differences in cohorts.
Limitations and opportunities
Limitations of our study include its length (a full 28 questions, many of which required multiple points of response or were open-ended) and its voluntary, unpaid format, which may have limited engagement especially among students who do not already value equity-related content. Fewer students participated in the survey after the baseline administration, and those who did may have reflected more passionate perspectives on this topic. However, we noted that follow-up participants included a range of both highly positive and highly negative views of the curricula, so while responses were limited without funding, we believe the voluntary follow-up format still captures a broad range of student perspectives. Additionally, the absence of an a priori power analysis is a limitation, though the exploratory and observational nature of this study did not allow for clear determination of the effect sizes requisite to conducting an effective power analysis.
One of the surprising findings of the study was the overall positive attitude that the students displayed toward learning about patients with diverse backgrounds. However, students with a more negative attitude likely chose to not participate in the survey study. In the future, we would like to evaluate further how attitudes of students who have a negative view about health equity are shaped by the interventions of the thread. It should also be noted that attitudes and other measures could increase over time as a result of being in medical school and with time for personal growth. As we would like all students to be taught the health equity related curriculum, there is no control group of students within each cohort that is not exposed to the curriculum.
Conclusions that can result from our data are also limited by the relatively small sample size noted regarding our 2021 Cohort sample. Relatedly, testing of the survey was limited due to the limited population size. Because the study population is limited to specific current medical school cohorts, we tested the survey outside of the study population among more advanced students; students on the research team; and faculty. As similar equity-focused curricular efforts expand across other medical schools, we anticipate that larger studies with greater opportunities for methodological testing and review will become possible.
Additionally, we recognize that the sample characteristics of those enrolled in our medical program may not represent those in other parts of the country in many respects. White students are overrepresented in our sample (68.3% of the sample overall) and over 50% of the sample that completed the survey is female. In addition, our study involved a single institution, and although students from three campuses participated, only one participating cohort had experienced the curriculum changes under review. While many of the limitations associated with this study are related to the sample size, we also consider this institutional setting a unique asset in that we were able to potentially capture the beginning of a shift in how equity-related content is viewed and mastered in a population where it has not previously been systematically centered. As many institutions struggle with how, to what extent, and with what challenges they should effectively teach equity-related content in health professions education, the experience of students at this predominantly white institution may offer useful insight. In particular, the lack of a ceiling effect among participants’ skills and preparedness measures may indicate that students recognize their need for additional learning on these topics, as well as the impact of a nascent equity-related curricula. Opportunity to link student exam performance with these attributes awaits future research.
Conclusion
This study provides a benchmark for showing yearly progress as we begin to track longitudinal changes in our learners as they progress through medical school, and while we continue to revise and further incorporate content that highlights health inequities in the core competencies. For programs seeking to incorporate similar interventions and forms of assessment data, our findings call attention to the need to select measures of attitudes, knowledge, preparedness, and skills of medical students that are sensitive to change and appropriately measure the desired constructs. As ours and other programs 28 expand on the use of such tools, the assessment might also guide curricular development, for example, by impacting how programs track the implementation of health equity content. Through these efforts, we seek programmatic success in creating a physician workforce that is better informed and better prepared to meet the needs of their patients.
Supplemental Material
sj-docx-1-mde-10.1177_23821205241230755 - Supplemental material for Evaluation of a Novel Equity-Focused Curriculum for Early-Stage Medical Students
Supplemental material, sj-docx-1-mde-10.1177_23821205241230755 for Evaluation of a Novel Equity-Focused Curriculum for Early-Stage Medical Students by Karen L. Roper, Anna-Maria South, Skyler Palmer, Julia Johnson, Lillian Sims, Carol Hustedde and Anthony A. Mangino in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
We are indebted to Dr A.R. Green for providing access to the CCCS (modified) survey. We also wish to thank Dr Jacqueline Pope-Tarrence for her persistence in building this educational study team and the energy she brings to the development of the health equity curriculum at our institution.
Author contributions
Karen L. Roper conceived and designed the analytic approach, wrote the paper, and handled all correspondence with support from Carol Hustedde. Anna-Maria South and Lillian Sims oversaw the design and implementation of the curricular implementation within the University of Kentucky College of Medicine; each also assisted with data collection and provided writing and editorial assistance. Lillian Sims oversaw software development and distribution of the survey. Skyler Palmer and Julia Johnson provided critical student perspective in design of the survey and manuscript review; each also assisted Carol Hustedde with the literature review. Anna-Maria South and Carol Hustedde designed and created Figure 1. All other Tables and Figures for the final manuscript were prepared by Anthony A. Mangino who performed all statistical analyses and wrote the Results section of the manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article, including a list of Competencies related to the HEAT thread assessed during year 1 and the survey items are available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
