Abstract
OBJECTIVES
In recent years, Indigenous health curricula have been integrated into medical education in response to international calls to improve Indigenous health care. Instruments to evaluate Indigenous health education are urgently needed. We set out to validate a tool to measure self-reported medical student preparedness to provide culturally safe care to Indigenous Peoples. We then applied the tool to evaluate the effectiveness of the Northern Ontario School of Medicine University's (NOSM U) Indigenous health curriculum.
METHODS
We conducted psychometric testing of a 46-item draft NOSM Cultural Competency and Safety Tool (CAST). Testing included principal components analysis, subscale and item analysis, and the use of paired sample t-tests to examine pre- and posttest change to measure learner outcomes. The NOSM CAST was transposed to create a retrospective pre–posttest survey with single-point-in-time scoring.
RESULTS
Respondents included five cohorts of first-year undergraduate medical students, with 305 of 320 participating (response rate of 95.3%). The validated survey subscales included knowledge, confidence/preparedness, attitudes, intentions for advocacy, antidiscrimination, and self-reflective practice, measured using 36 scale items. Cronbach's alpha showed good to excellent internal consistency for the scales (α range = 0.82–0.91). Composite reliability values were acceptable. The pre–posttest analysis showed statistically significant increases on four scales: knowledge [t(254) = 15.10, P < .001], confidence/preparedness [t(254) = 15.85, P < .001], intentions for advocacy [t(251) = 3.32, P = .001], and self-reflective practice [t(254) = 8.04, P < .001]. The largest mean increases were for knowledge (d = 1.07) and confidence/preparedness (d = 1.15).
CONCLUSIONS
The NOSM CAST tracks student progress in Indigenous health curricula. NOSM U's classroom and immersion-based Indigenous health curriculum enhanced students’ self-reported preparedness for culturally safe care. NOSM CAST implemented together with an assessment of Indigenous patient experiences with the same learners constitutes a rigorous evaluation approach to Indigenous health curricula.
Keywords
Introduction
In the context of the United Nations Declaration on the Rights of Indigenous Peoples 1 (UNDRIP) and the Canadian Truth and Reconciliation (TRC) Calls to Action, 2 policy directions have prioritized reducing access barriers to culturally congruent health care as an essential strategy to move toward the restoration of Indigenous Peoples’ health and wellness. 3 Culturally competent and culturally safe care are internationally recognized as quality indicators that reduce healthcare barriers, improve provider-patient communication, patient-centered care, patient satisfaction, and, eventually, health outcomes experienced by Indigenous Peoples. 4 Conversely, the lack of culturally competent health systems and culturally safe providers can negatively affect Indigenous patients’ health through overt and unconscious biases in diagnostics and treatment. 5 Further, Indigenous Peoples may avoid accessing care altogether if interpersonal experiences with health service providers lack cultural safety. 6
In Canada, the Association of Faculties of Medicine of Canada (AFMC) and the Indigenous Physicians Association of Canada (IPAC) released a Joint Commitment to Action on Indigenous Health (JCAIH) Report in 2019. 7 Subsequently, Health Canada funded the National Consortium for Indigenous Medical Education (NCIME) whose six working groups are focused on developing Indigenous cultural safety and antiracism curriculum frameworks, as well as recommendations regarding admissions, recruitment and support, and physician well-being. 8 Furthermore, the Committee on Accreditation of Canadian Medical Schools (CACMS) standard 7.6 addresses cultural competence and healthcare disparities. 9 In order to improve the cultural competence of healthcare providers so that they can provide culturally safe care, these two concepts require succinct definitions within health education. It is important to note that these concepts are still actively debated in the literature with varied interpretations between countries.10–14 We have found the following definitions helpful in guiding the development of Indigenous health curricula:
Cultural competence is the mastery of measurable skills, knowledge, attitudes, and behaviors in which practitioners become self-aware of their own culture in providing quality care to diverse populations. 15 Potential limitations of cultural competence in medical education is the danger of essentializing Indigenous people or reducing cultural concerns in the clinic to the mastery of a few technical skills. To counteract this constraint, the AFMC has advocated for an emphasis on skill and attitudinal competency along with provider self-reflection to prepare students to engage in culturally safe care.7,12 Cultural safety extends beyond cultural competence in that self-awareness requires active engagement and reflection of the practitioner in cross-cultural settings that interrogate personal biases, assumptions, and inherent power imbalances between the healthcare provider and the recipient of care13,16 and “underpin health inequalities/disparities.” 17 A second key concept in cultural safety is that its quality must be evaluated from the perspective of the patient. 18
In order to create culturally safe healthcare systems required to reduce barriers to optimal health for Indigenous Peoples, healthcare providers require thoughtful Indigenous health curricula to support the development of the foundational skills and knowledge to provide care that supports health equity. 19 This preparatory training should help providers strengthen their therapeutic relationships with Indigenous patients, increase Indigenous patient satisfaction, and ultimately improve clinical outcomes. The need for cultural competency training is recognized in nursing and medical associations in many countries with settler colonial histories, including Australia, Canada, New Zealand, and the United States. 18 As a result, nursing, medical, and allied healthcare professional schools in these countries have begun to redress health inequities by implementing Indigenous health in their curricula.12,20–23 Undergraduate education presents an ideal opportunity to teach skills underlying culturally safe care10,24 but there is also a need to offer this through continuing professional education.25,26
NOSM University
The Northern Ontario School of Medicine (NOSM U) was established in 2002 with an explicit social accountability mandate focused on improving the health of the people and communities in Northern Ontario, including Indigenous Peoples and communities.27,28 Initially, it served as the faculty of medicine for Lakehead and Laurentian Universities and then in 2021 became a standalone University as a result of the NOSM University Act. In this context, as well as concurrent broader global developments in medical education, NOSM U developed an Indigenous health curriculum in consultation with Indigenous communities that has been described in detail elsewhere.27–36 Briefly, it consists of lectures on a spectrum of topics related to Indigenous Peoples’ health including diversity of culture, history, colonialism, legislation, health policies, Indigenous health services, traditional medicine, and mental health and child welfare. In addition, relevant medical and clinical science curriculum is taught. Some of these topics are explored in small and large in-class groups during the first year of medical school. The first year culminates with a major curriculum commitment to Indigenous health, the world's only mandatory 4-week placement in Northern and rural Indigenous communities. It includes community-guided cultural and clinical experiences, and reflective tutorials with experienced faculty to help students make sense of the cultural, experiential, and health systems learning and the relevance of this curriculum to their future medical practice, and in particular, relevance to providing culturally safe care to Indigenous Peoples. The need to enhance cultural competency objectives with a focus on cultural safety and the inclusion of community led curriculum was realized early on; our 2005 pilot showed that without this intentionality student learning could be limited.33,35
Evaluating Indigenous health curricula
While the UNDRIP
1
and TRC policy statements (eg TRC Call to Action 24 concerning medical education)
2
have been adopted by educational institutions and Indigenous health curricula implemented, the research evaluating these curricula is mainly limited to learner satisfaction data. Rigorous research assessing the outcomes of cultural competence training and its translation into culturally safe care is still largely lacking.26,37–40 In a systematic review, Clifford et al.
41
concluded that There is a lack of evidence from rigorous evaluations on the effectiveness of interventions for improving cultural competency in health care for Indigenous Peoples. Future evaluations should employ more rigorous study designs and extend their measurement of outcomes beyond those relating to health professionals, to those relating to the health of Indigenous Peoples.
Similarly, based on a systematic review of the literature, Hardy et al.
39
also called for more research because currently, there is limited evidence regarding the effectiveness of specific content and approaches to cultural safety training on improving non-Indigenous health professionals’ knowledge of and skills to deliver quality, non-discriminatory care to Indigenous patients.
Clearly, research that measures the impact of cultural competency curriculum on learners’ progression in the relevant skills for culturally safe care is needed. Furthermore, evaluating skills for culturally safe care should include an evaluation of learners by Indigenous patients. However, methodological challenges make it difficult to measure patient perspectives, especially during undergraduate training. We propose that the first step toward rigorous evaluations of the Indigenous health curriculum requires foundational research to establish validated outcome measures and to implement benchmarking to track progress against long-term learning goals. Although some of this work has been started in midwifery education, 42 our review of the literature indicated a lack of appropriate, validated scales to evaluate medical education of Indigenous health curricula.
Exacerbating this knowledge gap is the current lack of a sophisticated understanding of the incremental learning levels that should be attained to meet the learning outcome of provider preparedness for culturally safe care. In addition, there is little research evidence of teaching strategies that support students in acquiring these skills. Medical educators need a better understanding of the specific aspects of Indigenous health curricula that enhance the knowledge, skills, and professional behavior of students, healthcare providers, and physicians as they prepare to become culturally safe providers.19,43 We argue that discourse about the effectiveness of Indigenous health curricula is currently almost exclusively based on anecdotes and personal perceptions. Validated instruments are therefore urgently needed to evaluate teaching strategies and learning outcomes related to Indigenous health curricula. 44
We seek to address these knowledge gaps by developing, validating, and applying such an instrument through multiyear research with medical students at NOSM U in Canada. NOSM U is an ideal site for this research because the School designed its undergraduate medical curriculum to focus explicitly on Indigenous health as part of its social accountability mandate. 29 NOSM U's Indigenous health curriculum was designed to prepare medical students to address health disparities through longitudinal, knowledge-based, reflective, experiential, and immersive learning about Indigenous cultures, communities, health and health services, and culturally safe care.30–36
To measure the effectiveness of the curriculum, our team of researchers first created the Northern Ontario School of Medicine Cultural Competency and Safety Tool (NOSM CAST), designed to measure self-reported progression in students’ readiness to provide culturally safe care to Indigenous patients over eight months of participating in the NOSM U integrated Indigenous health curriculum. In this article, we (a) describe the development process of learning constructs that prepare students for culturally safe care, (b) outline the validation process of an instrument designed to measure these constructs, (c) present the validated instrument, and (d) report Indigenous health learning outcomes for five cohorts of NOSM U students.
Methods
We set out to develop a validated learner self-report survey instrument to measure the effectiveness of the Indigenous health curriculum in preparing students for culturally safe care. Our reporting follows the Standards for Quality Improvement Reporting Excellence for Education (SQUIRE-EDU) which was developed to enhance the quality of health professional education research (Appendix 1 in the online supplemental materials). 45
Situating our team
Our team is composed of Indigenous and non-Indigenous faculty and staff who have led and supported the development and implementation of the Indigenous health curriculum at NOSM and then NOSM U over the past 20 years. Past and present roles include: Pilot (GH) and Module Coordinator of the Indigenous cultural immersion placement module (GH, MM, LM, DM); Chair of the Northern and Rural Health Course (KJ, MM); Chair of the Personal and Professional Aspects of Medical Practice Course (GH); Chair of the Social and Population Health Course (GH, MM, DU); Module Content Coordinators for one or more of these courses for the Indigenous Placement Module (GH, MM, LM, DM); Director of the Centre for Rural and Northern Health Research (WW, DU); AMS-NOSM U Hannah Chair in the History of Indigenous Health and Indigenous Traditional Medicine (DM); Indigenous Community Coordinator (SS); Director of Indigenous Affairs (DM, SS); and Founding Dean of NOSM U (RS). All team members have engaged and collaborated with Indigenous community members and Indigenous healthcare providers in formal and informal ways to shape the Indigenous health curriculum at NOSM U. Three coauthors are also members of Indigenous communities who host students in NOSM U's month-long Indigenous cultural immersion curriculum (DM, LM, SS).
Overview of the research approach
During Phase 1 of the research, we drafted an instrument to assess cultural safety preparedness learning constructs, populated in part with items adapted from the published literature. Item generation and content validity of a draft instrument to measure learner preparedness to provide culturally safe care to Indigenous Peoples were also established. In Phase 2, we conducted principal component analysis (PCA) of the survey items using pretest data from five cohorts of medical learners to assess the internal structure of the instrument. PCA is an established exploratory technique used in test development for examining relationships among variables; the purpose is to identify dimensions or components from a larger pool of items. 46 We used the method to determine viable subscales for inclusion in our tool.
We also generated additional indices of reliability and validity, including pre- and posttest survey results to measure learner outcomes of the Indigenous health curriculum at NOSM U. In Phase 3, we transposed the pre- and posttest instrument to create a retrospective pre–posttest tool to simplify future data collection to a single-point-in-time scoring, to support realistic evaluation processes in everyday teaching environments. 47
Phase 1: survey development
Current theoretical competency frameworks and policy statements on cultural competence and cultural safety were reviewed12,21,48 alongside NOSM U's Indigenous Community Gathering reports.49,50 Based on this review and the research team's collective experience in Indigenous health education, we developed a conceptual framework of enablers of provider preparedness for culturally safe care. Preparedness for culturally safe care was conceptualized based on earlier work by Ramsden, 11 Kurtz, 18 the AFMC, 7 and NCIME 8 which combined the attainment of relevant competencies with the process of reflection to understand biases and power imbalances between the patient and the provider. The following conceptual domains comprised the framework of enablers of preparedness to provide culturally safe care: (1) knowledge/awareness of Indigenous Peoples’ health and its determinants; (2) confidence/perceived preparedness to provide care for Indigenous Peoples; (3) attitude toward providing care for Indigenous Peoples; (4) intentions for advocacy for Indigenous health issues; (5) anti-discrimination attitudes with respect to Indigenous Peoples; and (6) engagement in self-reflective practice related to culturally safe care (see Figure 1).
Conceptual framework of enablers of readiness for culturally safe care.
Scale items from published scales on cultural awareness, sensitivity, competency, and safety were reviewed and appropriate items selected for adaptation to the Indigenous context. The search identified nine published scales that could provide adapted or verbatim scale items for the NOSM CAST (Appendix 2 in the online supplemental materials). Only one scale of cultural safety was identified, however it was aimed at the recipients of care rather than providers of care. 10 Therefore, questions geared to identify provider preparedness for culturally safe care for Indigenous Peoples were created by cross-referencing learning objectives for Indigenous health curricula by Indigenous physician and nursing associations.12,21 Items to assess attitudes of antidiscrimination were adapted from Gawronski's discrimination scale. 51 Finally, items assessing our constructs of provider self-reflection and reflective practice and intentions for advocacy were developed de novo.
A total of 146 potential survey items were identified for inclusion into the NOSM CAST, mapped to our conceptual domains, and revised or removed if they were redundant or questionably relevant. Each item was assigned a 5-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree, and the research team then again reviewed the tool for content validity. Eleven items were reverse coded to minimize response bias. Using a Delphi approach modified for Indigenous research, 52 the team members revised the items iteratively until a consensus on the content was reached, and the measure was reviewed again in its entirety by the research team for relevance and clarity. The content review and revision process resulted in the inclusion of 46 items in the survey. The resulting scale items and their conceptual origins in the literature are presented in Appendix 2 in the online supplemental materials.
Using the 46-item tool, NOSM CAST data were collected for five cohorts of undergraduate medical students to measure learner outcomes of the Indigenous health curriculum delivered in their first year at NOSM U. Pretest data were collected at the beginning of the first academic year and the posttest data were collected approximately eight months later at the end of the same year. During the eight months, the students experienced a variety of Indigenous health learning opportunities, including lectures and small group learning, led by NOSM U faculty. Opportunities included learning from Indigenous faculty, patients, and knowledge keepers (described elsewhere), and culminated with a 4-week Indigenous community health immersion placement within a 7-week module focusing on Indigenous health.29–36
Phase 2: survey validation
The draft NOSM CAST survey was validated in the second phase of the study. Inclusion criteria was enrollment as a student in the first year of the NOSM U UME program at the time of participation in the survey. The survey was administered to the 2013–2014 cohort of 64 NOSM U first-year undergraduate medical education (UME) students, and again with the next four consecutive cohorts of incoming students: the 2014–2015, 2015–2016, 2016–2017, and 2017–2018 cohorts of 64 students, respectively. The purpose of Phase 2 was to validate the instrument against the conceptual domains identified in Phase 1. We conducted statistical analyses to test (a) the internal structure of the survey using principal components analysis (PCA); (b) the internal consistency (Cronbach's α) and composite reliability (CR) of each scale; (c) item-total correlations; and (d) pre–posttest change using paired samples t-test with Eta square for effect size. In addition, descriptive statistics for the items and scales (means, standard deviations, skewness, and kurtosis) in the preliminary and final surveys were examined.
Guidelines suggest various minimum sample size estimates for the development of psychometric tools using PCA. 53 For example, Comrey and Lee 54 have recommended sample sizes of 50 as very poor, 100 as poor, 200 as fair, 300 as good, 500 as very good, and 1000 as excellent. In general, minimum required sample size has focused on the number of variables, the number of factors, the number of variables per factor, and the magnitude of communalities (with higher communalities, >0.40; factor loadings, >0.35; and more variables/per factor suggesting increased adequacy and interpretability). However, simulation analyses have shown that as variable to factor ratios and levels of communality increase, required sample sizes decrease. 55 We sought to adopt a balanced approach in the development of our instrument, and based our validation process (PCA, reliability and pre–post testing) on a minimum sample size of 300. The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was also examined. 53
Phase 3: NOSM CAST transposed as a retrospective pre–posttest tool
After validating the NOSM CAST, we transposed the content into a retrospective pre–posttest tool. 47 The retrospective pre–posttest tool was not used for data collection in this study.
Results
In this section we describe the five cohorts of medical student participants, followed by the survey validation analyses, including the PCA analyses of the NOSM CAST survey items, scale and item analyses, and a pre–posttest examination of NOSM CAST scales and items pre- and postimmersion in the first-year Indigenous health curriculum in the medical education program at NOSM U. We also provide the NOSM CAST retrospective pre–posttest instrument.
Participants
Participants included 305 of 320 (response rate of 95.3%) first-year undergraduate medical education students. Approximately two-thirds of the sample were female (63%), and one-third (37%) were male. The sociodemographic profile of the five student cohorts from the 2013–2014 to 2017–2018 academic years is presented in Table 1. The average age of participants was 25 years. Most students resided in northern Ontario for more than seven years (89–95%), and, on average, approximately 8% of students resided in rural or remote areas. Indigenous students comprised between 5% and 9% % of each cohort, whereas about 17% self-identified as Francophone.
Sociodemographic characteristics for the undergraduate student cohorts (academic years 2013–2014 to 2017–2018).
NOSM CAST survey validation
Principal component analysis (PCA)
PCA of the 46 items was performed on the data from the 305 undergraduate students using IBM SPSS 29.0. 56 Initial PCA with an oblique rotated solution (Promax with Kaiser Normalization) was specified. Components extracted were assessed at Eigenvalues > 1.0, and factor loadings with a cutoff of 0.35. The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy (0.906) indicated that the data were suitable for PCA. Likewise, Bartlett's Test of Sphericity (<0.001) suggested sufficient correlation between the items to proceed. The initial analysis resulted in nine components with eigenvalues greater than 1.0. Examination showed that two components failed to retain sufficient items (factor loadings < 0.35). However, further specification of seven components still showed an inadequate number of retained items for two components. When the number of components was fixed at five for extraction, an acceptable number of items per component were achieved. The five components accounted for 50.3% of the total variance.
Thirty-seven items were retained; the factor loadings and communalities are illustrated for these items in Table 2. Items 1 and 34 had low factor loadings (<0.35) and poor communalities (<0.40). Items 3, 8, 9, 20, 22, 23, and 28 also had communality extraction scores below 0.40; this suggests relatively little variance explained by the component structure for these items. Overall, the obtained component structure corresponded to the proposed conceptual domains, with one main exception. The items associated with the confidence/perceived preparedness domain loaded on the same component (Component 1) as the knowledge/awareness items. In addition, one item from the attitude domain (item 26), and two items from the self-reflection domain (items 31 and 46), loaded on this component. Item 26 was related to one's comfort within Indigenous communities, and item 46 concerned one's understanding of how to integrate traditional approaches into the care of Indigenous patients. These items reflected confidence in, and perceived preparedness regarding the provision of care to Indigenous Peoples. Item 31 included learning to enhance one's skills to provide effective care, which aligned with the knowledge/awareness domain. Component 1 thus comprised 13 items that reflected two subscales: (1) knowledge/awareness of Indigenous Peoples (6 items), and (2) confidence/perceived preparedness in providing care to Indigenous Peoples (7 items).
Rotated component matrix of NOSM CAST items (academic years 2013–2014 to 2017–2018).
Items removed from the scale are gray shaded (9 items). Factor loadings below 0.35 are suppressed.
Items reverse coded so that strongly agree = 1, strongly disagree = 5.
Component 2 was defined by 7 items assessing the concept of advocacy for Indigenous health issues, including raising awareness, and supporting public initiatives and outreach activities around Indigenous health. Item 27 (attitude domain) also loaded on this component and reflected continued involvement in Indigenous communities upon completion of one's medical training. Component 3 loaded with 6 items associated with antidiscrimination with respect to Indigenous Peoples, including item 6 (attitude domain), which assesses one's perspective on providing care to Indigenous Peoples without learning about a patient's culture. Component 4 was loaded with 6 items on the concept of attitudes, including the importance of trust and understanding cultural, spiritual, and religious beliefs in one's decision-making for Indigenous patients. Last, the 5 items loading on Component 5 were associated with one's self-reflection and awareness of their cultural identity and biases as a care provider. The component matrix of NOSM CAST items with factor loadings is shown in Table 2.
Scale and item analyses
Results for the NOSM CAST scale and item analyses are presented in Appendix 3 in the online supplemental materials, for the 37 items remaining after the exploratory PCR analysis. Cronbach's alpha showed good to excellent internal consistency for each of the scales (ranging from α = 0.82–0.91), and the CR values were acceptable (knowledge scale = 0.78; preparedness scale = 0.85; attitude scale = 0.76; advocacy scale = 0.90; antidiscrimination scale = 0.84; self-reflective practice scale = 0.84). The Cronbach alpha for the advocacy scale was >0.90, and CR, 0.90, which indicates some redundancy for the items on this scale. 57 The item-total correlations for the items on this scale were somewhat high (Appendix 3 in the online supplemental materials). However, the decision was made to retain all advocacy items as each captured distinctive educational objectives, including advocacy for public awareness of Indigenous health and social issues. The item-total correlations for the remaining items across each of the scales were within the acceptable range (ie 0.30–0.70), with the exception of the correlation for item 31 (Pearson r = 0.15). This item was subsequently removed from the knowledge scale, due to the low item-total correlation for this item.
Pre–posttest change—immersion in the NOSM U Indigenous health curriculum
The combined average pre- and posttest scores for the NOSM CAST subscales are presented in Table 3 for the 36-item measure. There were statistically significant increases pre- to postcurricular immersion in learner knowledge of Indigenous People's health [t(254) = 15.10, P < .001], preparedness to provide care for Indigenous Peoples [t(254) = 15.85, P < .001], intentions for advocacy for Indigenous health issues [t(251) = 3.32, P = .001], and engagement in self-reflective practice related to culturally safe care [t(254) = 8.04, P < .001]. The largest mean increases were for knowledge [M change = 0.73, 95% CI (0.64, 0.83), d = 1.07] and preparedness to provide clinical care (M change = 0.76, 95% CI (0.66, 0.85), d = 1.15] to Indigenous Peoples. All item responses significantly increased on these scales (see Appendix 4 in the online supplemental materials for detailed item statistics). Results showed the largest effect sizes for items covering knowledge of local Indigenous community contacts and community support structures (d = 1.21); understanding of the Indigenous community healthcare team and how to involve them in the care of patients (d = 1.11); knowledge of the strengths of regional Indigenous communities (d = 1.0); and understanding how to integrate traditional approaches to healing in the care of Indigenous patients (d = 0.97).
NOSM CAST average subscale scores and paired sample statistics.
N = 255.
*Significant at P < .001, two-tailed.
The pre–posttest change for the self-reflective practice scale was also relatively large [M change = 0.31, 95% CI (0.24, 0.39), d = 0.54], with the largest average increases observed for item 45 (understanding one's own biases and preconceived notions about Indigenous Peoples) [M change = 0.34, 95% CI (0.22, 0.45), d = 0.44)] and item 7 (reflective practice related to power imbalances) [M change = 0.33, 95% CI (0.21, 0.45), d = 0.38]. However, change for the advocacy scale was more modest [M change = 0.14, 95% CI (0.06, 0.23), d = 0.20], with item 37 having the largest effect size among individual items, and which assesses advocacy for Indigenous health or social issues with family and friends (d = 0.23). Average pre–posttest scores for the attitude and antidiscrimination scales showed only small, nonstatistically significant changes (Table 3; Appendix 4 in the online supplemental materials).
Retrospective pre–posttest instrument
Implementing a pre- and posttest tool requires research time and resources. It may be unrealistic to implement these in everyday teaching environments. 47 Therefore, we transposed the validated scales into a retrospective posttest tool that requires only a single testing time for use in future studies and ongoing program evaluation of Indigenous health curricula in a variety of training sessions. The instrument is provided in Appendix 5 in the online supplemental materials.
Discussion
An important emergent area of scholarship in health education is research on the effectiveness of social accountability-related education such as Indigenous health curricula in order to enable health educators to engage in continuous quality improvement to ensure that curricula are meeting the intended outcomes. 58 While there has been a flurry of development of Indigenous health curricula as a strategy to improve Indigenous patient care, little has been done to establish outcome measures for these curricula and to evaluate their effectiveness. While ultimately these measures should include Indigenous patient data such as perceived cultural safety and Indigenous patient health outcomes, our research has created a rigorous first step in this direction by creating a validated instrument to measure outcomes of Indigenous health curriculum in health education that tracks changes in students’ preparedness for culturally safe care.
Survey development and validation
We found that psychometric testing generally supported the domain constructs that the content experts developed; however, several items had to be removed or moved to a different domain based on the results of the PCA and item analyses.
Based on the PCA, nine items were removed due to poor factor loadings and communalities. Two items related to learners’ understanding of the barriers that lead to health inequities (I-1 “I have an understanding of the barriers faced by Indigenous Peoples that contribute to health inequities”) and the multigenerational effects of colonial violence (I-23 “Health effects of the residential school system are propagated through several generations”). It is possible that the understanding of these items require knowledge of chronic illness and comorbidity care which may be too advanced for students at this early stage in medicinal education. Item I-20 (“I feel confident in my ability to be nonjudgmental”) was also removed from the instrument. Anticipated was that this item would have a stronger relationship with the knowledge/confidence domains/component, but it did not contribute to the factor structure. This may indicate that the idea of nonjudgment has a weak or complex relationship with the constructs set forth in the NOSM CAST. It may also suggest that judgment does not align with confidence in one's ability around engaging in traditional healing practices and providing culturally safe care.
Likewise, the item designed to measure learners’ ability to disagree with a preceptor (I-28 “I have enough knowledge that I could disagree with a clinical preceptor who holds a discriminatory view toward Indigenous patients”) was removed. In retrospect, this item requires the confidence to disagree with a preceptor, which is often impeded by feelings of vulnerability as learners (such as the first-year students in this case) and the fear of risking poor evaluations instead of having an expected relationship with knowledge or preparedness to deliver culturally safe care. A recent Canadian study on sexual harassment of medical learners underscored their inherent fear of challenging preceptors in the medical education system due to the power imbalance in this relationship. 35 Further, the item I-22 (“When I am surrounded by culturally diverse individuals, I feel that my own beliefs and values are being threatened”) was removed; the item appears to require a high level of awareness of personal triggers when confronted with unfamiliar cultures and belief systems.
It is interesting to note that the one item assessing advocacy removed from the scale is the only item that is a general statement about the responsibility of the medical profession concerning advocacy (I-8 “I believe that physicians should use their position and status to redress existing health inequalities in Indigenous populations”). All other statements are focused on learners’ willingness to contribute to advocacy personally. It appears that the general responsibility may be a separate construct which contrasts with the students’ interest in becoming an advocate. While the internal consistency reliability are high for this scale, and therefore highly related, we believe the items are not redundant as they cover diverse areas of advocacy in healthcare, including at the interpersonal, organizational, and health systems level. 59
The antidiscrimination domain was perhaps the most challenging to measure, and three items were removed due to low communality extraction scores. These items included: (I-9) “There may be a few exceptions, but in general, Indigenous Peoples are pretty much alike,” (I-3) “Indigenous Peoples would be best served by policies that integrate them fully into Canadian culture and society,” and (I-34) “Indigenous Peoples have the same desire to raise a healthy family as most other Canadians.” This is an emerging field of inquiry of a complex construct and more work needs to be done to better understand the underlying concepts of discrimination and racism, both from the perspective of experiencing racism as well as implicit attitudes. That the items were poorly performing suggests lack of clarity in the wording presented to respondents, and/or variations in the interpretation of these items. To reduce acquiescence bias, these items were reverse coded, but reverse coding may affect psychometric estimates.60,61 However, we sought to minimize the influence of coding method by reverse scoring all of the items in this construct instead of using mixed response option coding. 62
Results of the item analyses yielded a poor relationship between one knowledge item [(I-31) “I realize I have more to learn in order to enhance my ability to be a good physician for Indigenous Peoples”] and its respective knowledge scale. The item-total correlation for this item was r = .15; thus, the decision by the research team to remove this question. Although this item was included in the NOSM CAST tool as an indicator of knowledge of Indigenous People's health, the results suggest that it measures other underlying concepts, or nuances of knowledge acquisition that are not measured by the other items in the scale. It is also possible that social desirability affected the results; this item warrants further examination, eg using a retrospective pre–posttest design or the pre–posttransposed version of the NOSM CAST.
Effectiveness of the NOSM U Indigenous health curriculum in fostering readiness for culturally safe care among medical students
The NOSM U Indigenous health curriculum has been described in detail elsewhere.27–36 Applying the NOSM CAST tool to five cohorts of first-year medical students, there is clear evidence of statistically significant growth in knowledge and awareness, confidence to provide care to Indigenous patients, intentions to advocate and reflective practice, all of which are key prerequisites for culturally safe care.
Knowledge/awareness of Indigenous Peoples’ health and its determinants
An increase in knowledge and awareness of Indigenous Peoples is likely connected to curriculum content focused on foundational information concerning the impact of historical, colonial, political, economic, and socio-cultural dynamics on the health of Indigenous Peoples, as well as basic knowledge of the diverse cultures and communities. This content is taught didactically as well as through Indigenous patient stories and the cultural immersion experience.
Confidence/perceived preparedness to provide care for Indigenous Peoples
Confidence and preparedness increases are particularly important as we, as educators, often observe a level of hesitancy among many non-Indigenous learners to engage with Indigenous patients, often out of fear of embarrassment, and fears of inadvertently offending Indigenous Peoples in conversation and, for some students, shame of the settler colonial history.31,32,34 Our research with Indigenous patient actors has also shown that some Indigenous students may lack confidence with Indigenous patients because they perceive themselves to lack cultural knowledge and may worry about judgment by peers and patients. 32
Intentions for advocacy for Indigenous health issues
Advocacy for Indigenous health issues in curriculum requires a coordinated approach between institutional practices and the educational curricula. 19 Increases in advocacy for social justice of marginalized populations, including raising awareness, and supporting public initiatives and outreach activities are known to result from cultural immersion experiences. 19 Involving health humanities aspects of education such as the incorporation of meaningful interaction with elders, significant consideration of the history of health and medicine, and exposure to alternative representations of knowledge and self-reflection contributes to emotional growth, especially in empathy and compassion, 63 in part arising in an iterative process from relevant cues in an immersion self-study project.29–36 The NOSM U Indigenous health curriculum addresses the TRC Call to Action 24 including the requirement that all students study “…the history and legacy of residential schools, the United Nations Declaration on the Rights of Indigenous Peoples, Treaties and Aboriginal rights, and Indigenous teachings and practices.” 2 In addition, the curriculum provides students with “skills-based training in intercultural competency, conflict resolution, human rights and anti-racism” as described in the TRC Call to Action 24. 2 Perhaps most importantly, students are able to observe health inequities and reflect on the need for advocacy during their month-long cultural immersion experience.
Antidiscrimination
The mean pretest score for this subscale was very high even at the pretest stage (4.5 out of 5) and showed no statistically significant change. This finding indicates that incoming NOSM U students, tested within weeks of beginning medical studies, hold perceptions that are aligned with positive attitudes toward Indigenous patients, and nondiscrimination attitudes, and it would be difficult to increase these scores even further. The high average pretest score may be a result of NOSM U attracting and preselecting students who identify with the values of social accountability through the admissions process, although social desirability bias may also be an important factor in this high score especially since students had recently studied and passed a socially accountable admissions process.
Pre- and posttest show only statistically insignificant variations and we can conclude that these personal convictions and commitment to antiracism and antidiscrimination appear to remain stable, while knowledge, confidence, and interest in advocacy increase. It is possible however that this construct may not only be affected by a social desirability bias, but also a response-shift bias, meaning students may be idealistically overestimating their antidiscriminatory perspectives at the pretest. Some researchers have argued that a retrospective pre–posttest may be a better measure when the internal frame of reference of a construct has the potential to change pre- and posttest. In these cases, a retrospective pre–posttest can reduce the effect of the response-shift bias. 47 We are very interested in testing this subscale further with the retrospective pre–posttest version of NOSM CAST and analyzing differences in this measure when the two different approaches are applied.
Engagement in self-reflective practice related to culturally safe care
Increase in one's self-reflection and awareness of one's cultural identity and biases as a healthcare provider is a key prerequisite for culturally safe care. The curriculum creates space during weekly small group reflective sessions in preparation for, during, and after the Indigenous community placement. The placement culminates in a research and reflection project that students present in their placement community as well as to faculty and peers once they return to campus. At NOSM U, these intentional curricular spaces for reflection are a cornerstone for integration of knowledge, gaining of confidence, and interest in advocacy and this is evident in the increased use of reflection as indicated in this subscale.
Implications
Cultural competence embodies abstract values and social constructs, embedded within the affective domain of learning that challenge many medical learners. Kumagai and Lypson 64 argue that when humanistic values are incorporated into medical education, “[the orientation] is directed towards fostering critical self-awareness, acquiring an understanding of social issues (both interpersonal and societal) in medicine, and developing an individual approach to reflective, ethical practice.” The enactment of culturally competent and culturally safe care requires complex growth in emotions and attitudes that are difficult to teach directly compared with clinical and medical sciences where the objective is to teach a foundation of knowledge to be applied in practice. 64
We argue that an added challenge of teaching preparedness to provide culturally safe care is that learning outcomes are difficult to pinpoint and evaluate. To advance medical education scholarship in this area, we developed a conceptual framework of learning outcomes that assess changes in cultural competence and readiness to provide culturally competent care to Indigenous patients among medical learners who undertake specialized cultural safety training. Our study demonstrates that the NOSM CAST is a pre–posttest tool that measures the changes in the levels of cultural competence and readiness to provide culturally safe care among prelicensure medical learners in response to exposure to the Indigenous health curriculum. To our knowledge, this is the first validated tool of its kind.
NOSM U is known for its social accountability mandate which may be reflected in admissions process preselecting for students who are, on average, highly committed to health equity as evidenced in their high scores in antidiscrimination and positive attitudes toward Indigenous Peoples in the pretest. The high average antidiscrimination scores did not increase any further, but were maintained, which is an important finding for the NOSM U Indigenous health curriculum as some cultural curricula have been shown to be counterproductive13,65–69 which was not the case here.
The results also show that students can attain measurable learning goals such as skills, behaviors, and attitudes that prepare them for culturally safe care when exposed to the comprehensive, integrated Indigenous health curriculum at NOSM U. It is important to emphasize that this measure is not a substitute for the patients’ perspective on how safe they feel with a medical learner. At NOSM U, Indigenous community staff at each site gather information about how safe and respected community members felt when interacting with the student. Staff then complete a score sheet to evaluate perceived cultural safety in their interactions from the perspective of the Indigenous community members. This is an important aspect of the evaluation of the Indigenous health curriculum and complements the NOSM CAST which uses a self-reported measure of progression in students’ readiness for culturally safe care.
Creation of the NOSM CAST retrospective pre–posttest
The methodology of a pre- and posttest is difficult and time-consuming to implement in educational institutions. We believe that a retrospective pre–posttest tool can be very useful for evaluating Indigenous health curricula as it is more efficient to implement. Therefore, we have transposed the tool as a retrospective pre–posttool that can be administered at the end of any specific educational component for use in regular evaluation cycles (Appendix 5 in the online supplemental materials). In this application, learners score changes in their perceived knowledge, skills, and practices retrospectively after the teaching has taken place. This tool should help to reduce students’ bias of overestimating their competence before the educational intervention and account for shifts in personal frames of reference as students progress through a demanding and life-altering medical curriculum. 47 This version of the tool should undergo further validation research in the future.
Either version of the tool provides the first real opportunity to assess the effectiveness of different approaches to Indigenous health education. The tool can potentially contribute to health education knowledge globally in the area of evidence-based understanding of the impact of cultural safety teaching in health professional education specific to Indigenous health. As medical schools and other health professions invest in Indigenous health curricula to support the health-related goals of UNDRIP and the TRC, it is essential to monitor the effectiveness of Indigenous health curricula.
Limitations
The assessment of the provider's attainment of skills and understanding deemed to be underlying cultural safety is only an intermediate evaluation of the ability to provide culturally safe care. In this study, our scope was limited to assessing the learner's readiness to provide culturally safe care. While the content validity of the NOSM CAST is strong from the perspective of preparing students for culturally safe care, further research should be conducted to test the tools aligned with Indigenous patient perspectives about the safety of the care provided by the students at different stages of their learning as students and clinical providers.
Another limitation is that the tool was developed for and only tested in one Canadian medical school with an overtly strong commitment to preparing students to provide culturally safe care. Therefore, adjustment of terminology may be required for use of NOSM CAST in other countries. However, adding to the strength of our methodology is that the tool was tested on learners who participate in school-wide mandatory cultural training (ie students did not self-select into cultural streams) with a 95.3% response rate.
Conclusion
Teaching cultural competence and safety in medicine is one pathway that is required to realize Indigenous health equity and is encouraged by medical school accreditation. As we teach about culture and colonialism as determinants of health in medicine in response to recommendations by UNDRIP and TRC, it is extremely important to monitor if these cultural competency training initiatives help to achieve the desired outcomes. The PCA of the NOSM CAST tool demonstrated that a five-component model performed well in measuring “preparedness for culturally safe care” outcomes among first-year undergraduate medical students. The validation process should be extended to the NOSM CAST retrospective pre–posttest tool in future studies because its simplified data collection process makes it a practical tool to use at the end of any level of an Indigenous health curriculum.
Under its social accountability mandate and through community engagement, NOSM U includes the world's only mandatory integrated Indigenous health curriculum that includes a cultural immersion experience for each first-year learner. The month-long immersion experience is the main component of the first-year Indigenous health curriculum. Applying the NOSM CAST to evaluate the Indigenous health curriculum shows that NOSM learners progress significantly in acquiring skills for culturally safe care for Indigenous Peoples during their first year of medical school. The NOSM CAST results underscore the importance of the Indigenous immersion curriculum at NOSM U in supporting Indigenous health learning goals. Based on our results, we encourage consideration of mandatory extended stay Indigenous community placements at all medical schools with settler colonial histories. This enables all students and not just those who select an elective opportunity to augment classroom learning about Indigenous health in a meaningful and effective manner in the community. As part of rigorous evaluation of Indigenous health curricula, we advocate for the use of NOSM CAST in conjunction with an assessment of Indigenous community and patient experiences of safety with the same learners.
Supplemental Material
sj-docx-1-mde-10.1177_23821205241286292 - Supplemental material for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument
Supplemental material, sj-docx-1-mde-10.1177_23821205241286292 for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument by Marion Maar, Diana Urajnik, Geoffrey L. Hudson, Darrel Manitowabi, Lorrilee McGregor, Sam Senecal, Roger Strasser, Wayne Warry and Kristen Jacklin in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-2-mde-10.1177_23821205241286292 - Supplemental material for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument
Supplemental material, sj-docx-2-mde-10.1177_23821205241286292 for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument by Marion Maar, Diana Urajnik, Geoffrey L. Hudson, Darrel Manitowabi, Lorrilee McGregor, Sam Senecal, Roger Strasser, Wayne Warry and Kristen Jacklin in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-3-mde-10.1177_23821205241286292 - Supplemental material for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument
Supplemental material, sj-docx-3-mde-10.1177_23821205241286292 for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument by Marion Maar, Diana Urajnik, Geoffrey L. Hudson, Darrel Manitowabi, Lorrilee McGregor, Sam Senecal, Roger Strasser, Wayne Warry and Kristen Jacklin in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-4-mde-10.1177_23821205241286292 - Supplemental material for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument
Supplemental material, sj-docx-4-mde-10.1177_23821205241286292 for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument by Marion Maar, Diana Urajnik, Geoffrey L. Hudson, Darrel Manitowabi, Lorrilee McGregor, Sam Senecal, Roger Strasser, Wayne Warry and Kristen Jacklin in Journal of Medical Education and Curricular Development
Supplemental Material
sj-docx-5-mde-10.1177_23821205241286292 - Supplemental material for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument
Supplemental material, sj-docx-5-mde-10.1177_23821205241286292 for Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument by Marion Maar, Diana Urajnik, Geoffrey L. Hudson, Darrel Manitowabi, Lorrilee McGregor, Sam Senecal, Roger Strasser, Wayne Warry and Kristen Jacklin in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
We would like to thank the Indigenous community partners of NOSM who provided cultural immersion placements for NOSM medical students. Indigenous community partners since 2005 have included Fort William First Nation; Kitchenuhmaykoosib Inninuwug (Big Trout Lake); Deer Lake; Fort Frances Tribal Health Authority; Northwest Bay First Nation; Fort Frances Tribal Health Authority; Lac La Croix First Nation; Couchiching First Nation; Naotkamegwanning First Nation (Whitefish Bay); Sandy Lake; Lac Seul First Nation; Constance Lake; Eagle Lake; Wasaaygiizhik Nanandawe’iyewigamig (Kenora Area Health Access Centre); Fort Frances Gizhewaadiziwin Health Center (Couchiching First Nation); Thunder Bay (Metis Nation of Ontario); Kingfisher Lake; Muskrat Dam; Nibinamik (Summer Beaver); Eabamet Lake (Fort Hope); Fort Severn; Atikameksheng Anishnawbek (Whitefish Lake); Attawapiskat First Nation; Brunswick House First Nation; Fort Albany First Nation; Garden River First Nation; Mattagami First Nation; M’Chigeeng First Nation; Metis Nation of Ontario—Timmins; Mississauga First Nation; Mnaamodzawin Health Services; Moose Cree First Nation; Nipissing First Nation; Ojibways of Batchewana First Nation; Ojibways of Pic River First Nation; Sagamok Anishnawbek; Serpent River First Nation; Temagami First Nation; Whitefish River First Nation; and Wikwemikong Unceded First Nation. We also thank Drs. Doris Mitchell and Robert Pastre for providing advice on the study. We would like to thank Lisa Boesch for her support in preparing this manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article. This study received funding from the NOSM Medical Education Research Grant fund (KJ and MM).
Author's contributions
RS made substantial contributions to the early conception and design of the work, and to revisions of the text for important intellectual content. GLH made substantial contributions to the early conception and design of the work, and to revisions of the text for important intellectual content, especially related to the development and implementation of the curriculum. WW provided substantive and critical advice on the design of the research, the conceptual development of the knowledge domains, questions de novo, and the early sorting of principal components analysis outcomes. KJ and MM served as the co-PIs on the research grant, co-designed the research project, including conceptual design and survey development, and made substantial contributions to early drafts, writing, and revisions of the text for important intellectual content. DU led the data cleaning, statistical analysis, and writing of the statistical sections of this paper. DU, SS, LM, and DM all made substantial contributions to the paper.
Ethics
Research ethics approval was provided by Laurentian (REB #2013-06-14) and Lakehead (REB #045 13-14) Universities.
Consent
Written informed consent was not obtained from the students in an effort to keep participation anonymous. Instead, consent was implied by completing and returning the survey.
Supplemental material
Supplemental material for this paper is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
