Effects of an Experiential Learning Curriculum on Systems-Thinking

Abstract

Introduction

Though health systems science (HSS) is referred to as the third pillar of medical education along with the pillars of basic science and clinical care, the effects of learning theories used to teach components of HSS including quality improvement/ patient safety (QI/PS), are poorly understood. Experiential learning theory is often referenced in QI/ PS education but its effects on QI/PS education are not well-described.

Objective

To examine the effects of teaching QI using experiential learning theory on resident systems-thinking.

Methods

Data was gathered from 30 resident participants in a 3-h QI workshop designed using experiential learning theory. Using a mixed-methods design, aspects of learner systems-thinking were analyzed both before and after the workshop. Learners were asked about their confidence in systems-thinking behaviors, and their QI plans were evaluated qualitatively for themes as well as quantitatively via the QIKAT-R.

Results

There was a significant increase in self-reported confidence in systems-thinking behaviors post-workshop. Odds ratio of the QI aim statement focusing on the systems-level of the problem after the workshop was 41.4 with a 95% CI of 8.9142 to 192.2721, p-value .0001. Thematic analysis of QI plans revealed a change in residents’ thinking about healthcare problems. They shifted from attributing problems to individual actors to thinking about the problem as lying in the interaction between systems.

Conclusion

Experiential learning theory can be an effective framework for QI to transform learners into systems-thinkers.

Keywords

Quality improvement medical education systems thinking experiential learning

Introduction

Though health systems science (HSS) is referred to as the third pillar of medical education along with the pillars of basic science and clinical care, the role of learning theories in HSS instruction is poorly understood.¹ Multiple articles have called for explicit descriptions of how theory is used in quality improvement and patient safety (QI/PS) education, which are components of HSS.^2-6 Though there are many theories that could act as a lens through which to view and create QI/PS curriculum, experiential learning theory is often referenced in QI/PS education, but its application to QI/PS is not always explicitly described.² To address this need for empirical investigation, we designed a QI curriculum for residents using experiential learning. Our objective was to examine the effects of this QI curriculum and its underpinning experiential learning theory framework on residents’ systems-thinking. We will first describe experiential learning theory and then outline the major tenets of systems-thinking in the context of HSS. Other theories will also be referenced in this article insofar as they relate to experiential learning theory in the context of our study.

Because the literature offers varied interpretations of experiential learning theory, an explanation of the theory in the context of our QI/PS curriculum may be helpful. The AMEE Guide provides a template for experiential learning theory that we found helpful in thinking about our curriculum.⁷ The Guide describes both individual and group aspects of experiential learning theory.

The individual aspect of experiential learning theory

In Kolb's experiential learning cycle, illustrated in Figure 1 below, individuals reflect upon a concrete experience and then an abstract conceptualization occurs that can then be applied to new situations.

Figure 1.

Kolb's experiential learning cycle.⁸

Kolb's experiential learning cycle⁸ has some overlap with constructivism, another separate but related theory. Constructivism is a theory that views new knowledge creation as something that learners construct by linking new information to what was previously learned.⁹ In this way, knowledge is not regarded as absolute but tied to an individual's own perspective.^10,11

The group aspect of experiential learning theory

The AMEE Guide also addresses the social or group aspects of experiential learning theory by focusing on the part of Kolb's learning cycle where learners move from reflection to abstract conceptualization. Mezirow's work on transformative learning theory explains how, during this part of the learning cycle where learners move from reflection to abstract conceptualization, discourse with a group of people can transform our concrete experiences into new knowledge as we reflect with a group by taking our own perspectives or “taken-for-granted frames of reference” and make them more “inclusive and discriminating” of other people's perspectives to form new beliefs and opinions “that are more true” and help guide future actions.⁷

Using the group aspect of experiential learning theory to teach systems-thinking

Per Colbert et al, this ability to think about a problem from multiple perspectives is necessary for systems thinking.¹² According to Colbert et al, systems-thinking is the cognitive prerequisite for systems-based practice and can be defined as “the ability to analyze systems as a whole, including the recognition of essential interrelationships within the system and between subsystems, and any changes and patterns that arise out of the networks of relationships and interactions”.¹²

Systems-thinking is a necessary component of health systems science education in that it allows medical students to move beyond the diagnosis and treatment of disease to an understanding of how the whole patient fits within the multiple systems that impact the care of the patient.^1,13-17 For example, if a medical student understands what medication to give to a patient with pneumonia (clinical care of the patient), but the patient is unable to obtain the medication due to problems with transportation or insurance (health systems science care of the patient), the patient's care is compromised. Systems-thinking is seen as a professional responsibility for medical students as well as an integral foundation for quality improvement and patient safety.^1,13-17

Using systems-thinking to guide learners through the experiential learning cycle

One of the problems with experiential learning theory in general and specifically relevant to QI/PS education is that many curricula may not provide sufficient mentorship or guidance during the transformative learning process, which results in learners having difficulty creating new meaning out of old.^7,10 Expert consensus suggests teaching systems-thinking via a curricular progression that starts with the application of quality improvement concepts to a familiar personal experience and then to a health systems problem.¹⁸ This is also in line with constructivist learning theory, which holds that learners build new knowledge on pre-existing knowledge and experiences. To address this common oversight, we designed a QI workshop using experiential learning theory as a framework and included structured small-group learning activities (described below) to support learners’ progression toward systems-thinking.

Methods

The reporting of this study conforms to the SRQR guidelines¹⁹. [Supplementary File]

Context

The context for our study was a 3-h QI workshop conducted in Fall of 2021 for 30 residents at a single institution in the Southern United States. Participants were divided into non-homogeneous small groups of 4–5 residents each. Small groups were asked to complete two learning activities based on a shared experience across learner disciplines, that of preparing for rounds as an intern. Before and after the workshop, participants were asked to individually make a QI plan to increase immunizations in a fictional clinic and to complete a questionnaire (see supplemental file for a copy of the questionnaire). The learning activities are described below.

Learning activity #1: key driver diagram

In their small groups, learners developed a key driver diagram for why the intern was not prepared for rounds. The small group role-played the different stakeholders involved in the intern's preparation for rounds: the significant other, the residency program director, a fellow intern, a nurse, a patient, and the senior resident. Learners then brainstormed key drivers of why the intern was unprepared for rounds from each stakeholder's perspective.

This exercise tapped into the relativist nature of constructivist theory, meaning that there are competing versions of reality depending on which stakeholder's perspective you take. Perhaps the senior resident sees that it takes the intern twice as long to chart check as it does other interns, but the significant other sees that the intern takes ten minutes to make coffee in the morning and is late to their shift. Each of these key drivers could lead to an intern being unprepared for rounds. Depending on which version of reality (or which stakeholder's perspective) is taken, an intervention is then chosen to address the key driver. The intervention will be different depending on which key driver is chosen. For example, if making coffee takes too long in the morning, the intervention may be setting an automatic coffee marker. However, if chart checking is taking too long, the intervention might be printing out the labs rather than handwriting them. Per Colbert et al, systems-thinking is predicated on this ability to think about a problem from multiple perspectives.¹² This may be because each stakeholder's perspective captures a different problem in a different system.

Learning activity #2: systems-based list

Elaborating on the first activity, small groups were asked to use a systems-based list to reflect upon their shared prior experience of preparing for rounds as interns. The list included systems such as the intern's residency program, the hospital system, the intern's prior education, the intern's social system, and the intern's transportation system. Learners were asked to think about problems in each system that may be contributing to an intern's ability to be prepared for rounds, as well as how those systems may interact with each other to create obstacles for the intern's preparation. Figure 2 (below) illustrates how the learning activities were applied to Kolb's experiential learning cycle:

Figure 2.

Application of experiential learning theory to learning activities.

Data collection

All 30 residents enrolled in the workshop participated, and any residents not enrolled were excluded from the study. The residents had all completed their intern year and were from multiple subspecialties.

Data collection was done by gathering written participant responses via Qualtrics. Each resident individually completed this form electronically just before and just after the QI workshop. The form consisted of 12 questions rated by the residents on a 5-point Likert scale and one question asking them to write out a QI plan²⁰. The data collection instrument is included as a supplementary file. Participants did not enter any identifiable information into Qualtrics. Data were stored in a password-protected, encrypted online file system.

Study design, outcome measures, & data analysis

We situate this research in the constructivist research paradigm in alignment with the theories of learning driving the curriculum's development as described above. We used a concurrent triangulation mixed-methods design to examine systems-thinking, meaning that we collected and analyzed qualitative and quantitative data simultaneously to explore with greater depth the elements of systems-thinking²¹. Outcome measures included the following: (1) QIKAT-R scores, (2) Whether the participant identified the system's level of the problem, (3) Participant-reported confidence in being able to work with residents from other subspecialties and confidence in being able to gather stakeholder input, and (4) themes from qualitative content analysis of the QI plans (see Figure 3). Formal sample size calculations were not performed for this study as the sample included all residents available for analysis (ie, the full population). Triangulation refers to methods used to enhance the trustworthiness of a study by cross-checking multiple data sources²². Concurrent triangulation designs allow for the use of quantitative and qualitative data sources and analytic methods to see if they converge to similar conclusions. Since both types of data are collected and analyzed concurrently, they have equal priority and can serve to offset either method's potential weaknesses²³.

Figure 3.

Measuring the elements of system thinking.

Ethical considerations and IRB

The University of Texas at Austin Institutional Review Board (IRB) determined that the study is not research involving human subjects as defined by DHHS and FDA regulations. As the data collected was a part of required course activities, no written or verbal consent was obtained at the time of data collection. The activity was determined to be “Not Human Subjects Research” (NHSR): Program evaluation for educational programs using non-identifiable secondary use data.

Quantitative analysis

QIKAT-R scores

We wanted to understand how participants transferred the abstract conceptualization of systems-thinking to solving a health systems problem of increasing vaccines in a fictional clinic (see Figure 3 below). As referenced above, expert consensus has suggested that QI may be effectively taught by starting with teaching QI using a prior experience, like intern preparation for rounds, and then progressing to asking learners to apply QI to a health systems problem (like increasing vaccine rates in a fictional clinic).¹⁸ Participants were asked to individually make a QI plan to increase immunizations in a fictional clinic before and after the workshop. Individual QI plans were graded using the QIKAT-R, a validated rubric for assessing QI knowledge.¹⁹ QI plans were graded by KK, SG, and JR; differences in scoring were resolved through discussion. The Wilcoxon rank sum test was used to determine whether differences in pre-workshop and post-workshop quality improvement knowledge were significant.

Identification of the systems-level of the problem

One evaluation criterion within the QIKAT-R asks whether the AIM statement is “focused on the system-level of the problem presented”²⁴. For example, a system may include an outpatient clinic, a hospital, or the vaccine manufacturing system. An odds ratio was used to determine the likelihood of whether QI plans “focused on the systems-level of the problem presented” after the workshop as compared to before the workshop²⁴.

Participant confidence

Learner confidence in working with residents from other subspecialties and learner confidence in incorporating stakeholder input were assessed before and after the workshop by questionnaire using a 5-point Likert scale (See Table 1)²⁰. Due to the non-normal distribution of the data, differences in confidence were assessed via the Wilcoxon rank sum test.

Table 1.

Mean difference in scores Among all participants (N = 30).

Topic (Confidence in …)	Mean Baseline Score^a	Mean Post simulation Score^a	P-value
Working with residents from other subspecialties	3.6	4.37	< 0.001
Incorporating stakeholder input	2.03	4.1	< 0.001

^aRated on a 5-point Likert Scale (1 = Strongly Disagree, 5 = Strongly Agree)²⁰

Qualitative analysis

Researcher positionality and reflexivity

QI plans were also analyzed qualitatively via content analysis by KK and GG to discover how learner QI plans changed after the workshop. GG is a PhD medical education researcher with training in qualitative analysis. His research focuses on hierarchy and power in medical education systems. KK is a medical doctor (MD) with subspecialty board certification in pediatric hospital medicine, a subspecialty that focuses on quality improvement within hospital systems and is also a faculty instructor in interprofessional education. JR and SG are internal medicine hospitalists with additional training in quality improvement. KK, JR, and SG created the workshop curriculum and taught it to the 30 residents in this study.

Content analysis

Using an iterative, inductive approach, each QI response was independently coded first as to the type of intervention suggested. These codes were then grouped by what factors and systems were involved. For example, a process within the clinic, such as intake procedures, would be coded as part of the clinic system, whereas interventions involving patient education would be coded as a “patient factor” at the system level. Finally, the entries were analyzed for descriptions of interactions between systems, if any. GG and KK coded several entries independently, then discussed codes to form the codebook. After completing independent coding, all coded entries were discussed to reach consensus. See Table 3 for examples of codes.

Results

QIKAT-R scores

QIKAT-R scores demonstrated a significant improvement in the residents’ quality improvement knowledge. QIKAT-R scores significantly improved from a mean of 2.17 to 6.57 (p-value < .001.)

Identification of the systems-level of the problem

Odds ratio of the QI aim statement focusing on the systems-level of the problem after the workshop compared to before the workshop was 41.4 with a 95% CI of 8.9142 to 192.2721, p-value .0001.

Participant confidence

There was also a statistically significant improvement in working with residents from other subspecialties and incorporating stakeholder input. (See Table 1).

Qualitative analysis

A greater number of systems, and a greater number of interactions between systems were identified after as compared to before the workshop. (See Table 2)

Table 2.

Pre-Post number of systems identified per response and interaction between systems.

Number of Systems Identified	Pre-Workshop (Number of Responses)	Post-Workshop (Number of Responses)
No Systems Identified	No systems identified (19)	No systems identified (9)
Patient Factors Only	Patient factors only (5)	Patient factors only (3)
	Specific Interventions:	Specific Interventions:
	4 – Patient Education	1 – phone call reminders
	1 – Text Reminder	1 – text reminder
	1 – Text Reminder	1 – motivational interviewing
One System	Clinic factors (2)	Provider Factors (2)
One System	Clinic factors (2)	Clinic Factors (6)
Multiple Systems	Patient/ Clinic factors (2)	Patient/ Clinic Factors (3)
	Clinic/ Manufacturing System (1)	Clinic/ Manufacturing System (1)
	Patient/ Clinic/ Provider (1)	Clinic/ Community (1)
		Patient/ interprofessional team (1)
		Provider/ Patient (1)
		Clinic/ Provider (2)
		Patient/ Clinic/ Community (1)
		Patient/ Clinic/ Provider (1)

Table 3.

Codebook.

Theme (System addressed)	Code (Intervention)
Clinic factors (Clinic processes)	Ask patients if they are vaccinated
	Provide handout on where to obtain the vaccine
	Offer vaccine at all visits
	Offer specific clinics for vaccinations
	Make immunization status part of intake procedure
	Nurse alerts to provider
	Offer vaccine to high risk groups
	Nurse educates patient due to provider time constraints
	Same-day appointments
	Vaccine room
	Miscounting the number of vaccines
Patient factors	Discuss vaccination with the patient
	Verbal education by provider
	Motivational interviewing
	Phone call reminders
	Community education campaign
	Educational handout about vaccination
	Text reminders
Community factors	Work with local groups to provide vaccines
	Same day vaccination at a community site
	Work with local organizations to offer incentives
Provider factors	Provider awareness
Provider factors	EMR alerts to provider
Interprofessional team factors	Educate office staff about the vaccine
Manufacturing system	Vaccine supply

Systems-Thinking versus individual factors - representative quotes

Pre-workshop, residents proposed interventions that addressed a fault in an individual person by providing education to the individual:

“…Poll unvaccinated patients on the reason they do not receive the flu vaccine. Try to objectively quantify this data and start by addressing the top reason. For example, if it was misinformation, then provide a handout with FAQs…”

“Improve patient awareness with flyers, handouts…”

Post-workshop, residents displayed systems-thinking by proposing interventions that addressed interactions between systems, such as (1) the interaction between clinic processes (providing separate flu clinics and dedicated space for vaccine administration) and patient access to the vaccine and (2) the interaction between electronic health record processes like EMR alerts and provider factors like forgetting to ask if the patient has been vaccinated:

“Interventions: recurring flu vaccine clinics, offer flu vaccines in a room adjacent to a waiting room to increase convenience, have a check box on EMR to say that flu vaccinations were discussed at visit.”

Discussion

This study examined the effects of teaching QI using experiential learning theory on systems-thinking. Learning activities guided learners through Kolb's experiential learning cycle, first starting with a concrete prior experience (intern preparation for rounds) and then reflecting on that experience with other residents in their small group, using a key driver diagram and systems-based list, to then propel them to the next step in Kolb's learning cycle – formation of the abstract concept of “systems-thinking.” Finally, learners were pushed to the final step of Kolb's learning cycle – active experimentation by writing a QI plan. (See Figure 4).

Figure 4.

Using the experimental learning cycle to create system's thinking.⁸

The familiar, concrete experience chosen for our workshop was intern preparation for rounds as our learners had just finished their intern year. They were asked to individually create a QI plan to solve a health systems problem (that of increasing immunizations in a clinic) before and after the workshop. Our study offered a unique way to examine this progression as the same health systems problem was presented both before and after the workshop without any other intervention. Residents did not go back to the wards or clinic or even their own lives prior to solving the same health systems problem again.

Prior to the workshop, learner QI plans reflected an abstract conceptualization that saw individual actors (patient education being the most common problem identified) as the problem that needed intervention. After the workshop, learner QI plans displayed systems-thinking in which the systems-level of the problem was identified, and interventions focused on addressing problems within multiple systems. We relied on experiential learning theory in asking our learners to think about a past concrete experience of preparing for rounds as interns and reflect upon it to come up with an abstract conceptualization that could then be applied to new situations (like improving vaccinations in a clinic.) Because one of the problems with experiential learning theory is that sometimes insufficient guidance is provided to help learners create abstract conceptualizations that can then be applied to new situations, learners were asked to create a key-driver diagram and systems-based problem list of the systems that are involved in intern preparation for rounds as a framework for the abstract conceptualization of “systems-thinking.”

We relied on theories that emphasized the importance of discourse with others and taking the perspective of others in transforming learners into systems-thinkers. Though there was an improvement in learner confidence in working in groups after the workshop and there was also an improvement in systems-thinking after the workshop, it is unclear to what extent the improvement in confidence in working in groups led to solving QI problems with systems-thinking within the context of our study. A strength of teaching QI/PS in groups of learners who each have different perspectives is that it more closely mimics the real-life experience of quality improvement in which the perspectives of stakeholders are engaged to improve our understanding of the systems in which each stakeholder works and thereby implement more effective solutions. Another strength of our study is that we triangulated the qualitative results of systems-thinking in our learner's QI plans with a validated rubric (the QIKAT-R) as well as subjective improvements in learner confidence in QI to demonstrate improvements in aspects of systems-thinking in all areas.

Intern preparation for rounds may have been a particularly potent concrete experience as learners (especially learners who have been successful at navigating multiple systems to get to residency) may be particularly likely to think about this problem as a problem with the individual intern as opposed to the systems involved in intern preparation for rounds. However, there may be multiple systems interacting to hinder intern preparation for rounds. For example, the hospital system may be set up in such a way as to impede intern preparation for rounds in the case where a hospital system opens a new residency program, and there are changes that need to be made within the hospital system processes to accommodate residents and to provide appropriate levels of autonomy and supervision. There may also be problems within the residency program where insufficient preparation is given to interns prior to starting on the inpatient wards. There may be problems within the intern's prior educational system – perhaps the intern has a learning disability that would need to be accommodated by the residency program (problem in the interaction between systems) or had a medical school experience in which shadowing other physicians rather than active participation in patient care was the primary mode of teaching. This “systems-thinking” abstract conceptualization of problem-solving where learners are contemplating interactions between systems may differ from a “taken-for-granted frame of reference” in which a problem is seen as the fault of individual actors rather than the fault of the complex interactions between systems.¹²

Systems-thinking is an important construct in teaching systems-based practice, particularly in a healthcare system that has grown increasingly interconnected and that relies on the healthy interaction between multiple systems to improve patient care. As healthcare is moving to better address social determinants of health, systems-thinking may also be a valuable lens through which to view health disparities and the systems that either perpetuate or disrupt health inequities. Rather than seeing a patient with poor control of their diabetes as the fault of the individual patient alone, systems of care can be rethought to improve the likelihood that a person with diabetes can have better diabetes control. In this way, physicians may come to see problems not as the fault of individual patients, providers, or staff but as the complex interplay of systems that is more or less likely to perpetuate good or bad outcomes.

The study adds to the body of work that suggests that systems-thinking is a foundational mindset for teaching QI/PS (as opposed to teaching QI/PS as a set of skills to be learned) and suggests that using Kolb's experiential learning cycle could be a way to teach the systems-thinking mindset^1,12-17

Limitations include challenges to generalizability given the study was conducted at a single institution and lacked an a priori power analysis. Additionally, the questionnaire was not validated or pilot-tested. This is mitigated by the triangulation inherent in the mixed-methods design. The additional quantitative measures from the previously validated QIKAT-R and the qualitative measures provide confirmability for the findings and add context for the reader to make judgments about applicability. However, multi-institutional studies and comparisons between QI courses taught without the framing of experiential learning theory would provide additional insights into future systems-thinking curricula.

Conclusion

Using experiential learning theory can be an effective way of transforming learners into systems-thinkers. Further study should be given to the effects of teaching QI/PS in groups of learners as opposed to teaching QI individually, particularly with the proliferation of online QI modules which are intended for the individual rather than the group and have the potential drawback of not being able to simulate the perspective-taking needed for systems-thinking.

Supplemental Material

sj-docx-1-mde-10.1177_23821205241262210 - Supplemental material for Effects of an Experiential Learning Curriculum on Systems-Thinking

Supplemental material, sj-docx-1-mde-10.1177_23821205241262210 for Effects of an Experiential Learning Curriculum on Systems-Thinking by Kelly Kovaric and Gareth Gingell in Journal of Medical Education and Curricular Development

Supplemental Material

sj-docx-2-mde-10.1177_23821205241262210 - Supplemental material for Effects of an Experiential Learning Curriculum on Systems-Thinking

Supplemental material, sj-docx-2-mde-10.1177_23821205241262210 for Effects of an Experiential Learning Curriculum on Systems-Thinking by Kelly Kovaric and Gareth Gingell in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgements

We would like to acknowledge Saurin Gandhi, DO, and Jananie Ramesh, MD, for their contributions to creating the curriculum.

Contributions

Dr Kovaric conceptualized the curriculum and quantitative methodology. Dr Gingell conceptualized the qualitative methodology. Dr Kovaric drafted the initial manuscript. Dr Gingell drafted the qualitative methodology, edited, and revised the final manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Ethics approval

The University of Texas at Austin IRB determined that the proposed activity is not research involving human subjects as defined by DHHS and FDA regulations. The activity was determined to be NHSR: Program evaluation for educational program using non-identifiable secondary use data.

Statements or declarations

On behalf of all authors, the corresponding author states that there are no conflicts of interest. The authors received no financial support.

ORCID iD

Gareth Gingell

Supplemental material

Supplemental material for this article is available online.

References

Gonzalo

Chang

Dekhtyar

Starr

Holmboe

Wolpaw

. Health systems science in medical education: unifying the components to catalyze transformation. Acad Med. 2020;95(9):1362-1372.

Goldman

Smeraglio

Kuper

Wong

. Theory in quality improvement and patient safety education: a scoping review. Perspect Med Educ. 2021;10(6):319-326.

Goldman

Kuper

Wong

. How theory can inform our understanding of experiential learning in quality improvement education. Acad Med. 2018;93(12):1784-1790.

Davidoff

Dixon-Woods

Leviton

Michie

. Demystifying theory and its use in improvement. BMJ Qual Saf. 2015;24(3):228-238.

Jagpal

Fant

Bianchi

Kalnow

Teaching quality improvement: the use of education theories across the medical education spectrum. Cureus. 2022;14(7):e26625.

Varpio

Paradis

Uijtdehaage

Young

. The distinctions between theory, theoretical framework, and conceptual framework. Acad Med. 2020;95(7):989-994.

Yardley

Teunissen

Dornan

. Experiential learning: AMEE guide No. 63. Med Teach. 2012;34(2):e102-e115.

David

. Experiential learning - experience as the source of learning and development. FT Press; 2014.

Dong

Lio

Sherer

Jiang

. Some learning theories for medical educators. Med Sci Educ. 2021;31(3):1157-1172.

10.

Dennick

. Constructivism: reflections on twenty five years teaching the constructivist approach in medical education. Int J Med Educ. 2016;7:200-205.

11.

Loyens

SMM

Gijbels

. Understanding the effects of constructivist learning environments: introducing a multi-directional approach. Instr Sci. 2008;36(5–6):351-357.

12.

Colbert

Ogden

Ownby

Bowe

. Systems-based practice in graduate medical education: systems thinking as the missing foundational construct. Teach Learn Med. 2011;23(2):179-185.

13.

Johnson

Miller

Horowitz

SD.

Systems-Based practice: improving the safety and quality of patient care by recognizing and improving the systems in which we work In: Henriksen

Battles

Keyes

Grady

, eds. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Advances in Patient Safety. US: Agency for Healthcare Research and Quality; 2008. Accessed May 20, 2024. http://www.ncbi.nlm.nih.gov/books/NBK43731/

14.

Plack

Goldman

Scott

, et al. Systems thinking and systems-based practice across the health professions: an inquiry into definitions, teaching practices, and asessment. Teach Learn Med. 2018;30(3):242-254.

15.

O’Brien

Bachhuber

Teherani

Iker

Batt

O’Sullivan

. Systems-oriented workplace learning experiences for early learners: three models. Acad Med. 2017;92(5):684-693.

16.

Gonzalo

Ogrinc

. Health systems science: the “broccoli” of undergraduate medical education. Acad Med. 2019;94(10):1425-1432.

17.

Frenk

Chen

Bhutta

, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376(9756):1923-1958.

18.

Ogrinc

Headrick

Mutha

, et al. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med J Assoc Am Med Coll. 2003;78(7):748-756.

19.

Singh

Ogrinc

Cox

, et al. The quality improvement knowledge application tool revised (QIKAT-R). Acad Med. 2014;89(10):1386-1391.

20.

O’Brien

Harris

Beckman

Reed

Cook

DA.

Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014; 89(9): 1245-1251,

21.

Likert

. A technique for the measurement of attitudes. Arch Psychol. 1932;22(140):1-55.

22.

Cuschieri

Sant

Vella

Demanuele

Schembri

Cuschieri

. Allied health professional students’ perceptions on research: a multi-disciplinary mixed-methods study. Med Sci Educ. 2022;33(1):129-138.

23.

Yin

RK.

Case study research design and methods. Vol 5. Sage publications; 2014.

24.

Creswell

Clark

VLP

. Designing and conducting mixed methods research. Sage publications; 2017.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB