Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation

Abstract

OBJECTIVES

Medical schools considering longitudinal integrated clerkships (LICs) have access to literature that provides recommendations for planning, implementation, and sustainability. However, LIC development and implementation remain notoriously challenging. University of Utah’s LIC development process was informed by the documented experiences of long-established LIC programs. A literature gap was identified pertaining to more recently implemented LICs. The aim of this study was to explore the experiences of faculty in the early stages of LIC development.

METHODS

Thirteen representatives from eight LICs implemented after 2015 participated in 2 Zoom focus groups (5 participated in the first and the other 8 participated in the second). Participants were asked questions to assess key supports, barriers, and recommendations. Following the focus groups, participants were asked to rank the responses based on their level of importance.

RESULTS

Highest ranked supports included stakeholder and partner involvement; a dedicated coordinator or team; and strong, committed leadership. Highest ranked barriers included difficulty recruiting preceptors and clinical sites; underestimation of the amount of work required to coordinate the LIC; and challenges in providing the needed faculty development. Top recommendations for new LICs included investing in the needs of clinical partners; staffing or assigning a dedicated coordinator early in the development and implementation process; and frequent communication with all stakeholders.

CONCLUSION

Despite variation among the types of new LICs represented, there was consensus among participants on the importance of key supports, barriers, and recommendations. Knowledge of these factors can help new schools plan and allocate resources during their LIC development process. Participants found the focus group process and follow-up discussions useful and have formed an ongoing workgroup which meets quarterly.

Keywords

longitudinal integrated clerkships LIC clinical clerkships medical education

Introduction

While undergraduate medical education varies across schools, third-year medical students typically move out of the classroom and begin an intensive clinical education. These students are often assigned to traditional block rotations (TBRs) in which they engage in a series of brief (eg, 4-8 weeks) clinical placements within different disciplines (eg, pediatrics, family medicine, obstetrics/gynecology, surgery, internal medicine, psychiatry). Students are immersed sequentially in each discipline and work with clinical faculty within each specialty before moving on to the next rotation. An alternative to the TBR model is the longitudinal integrated clerkship (LIC) model. In an LIC, students engage with multiple specialties simultaneously, for less time per specialty in any given week, but over a longer period of time (the full length of the LIC). LICs allow for an educational experience that is grounded in longitudinal relationships with preceptors, patients, peers, places, and pedagogy.

In the half-century since LICs began emerging as alternatives to TBRs in clinical medical education, a great deal of literature has been published on this model. The model's types and program logistics have been described,^1–10 along with considerable benefits. Benefits for students include closer relationships with preceptors^11–17 and higher quality of feedback^{3,11,14,15,18–20}; professional identity development^{11,12,21–23}; and academic outcomes, clinical experiences, skill, and confidence.^{2,13,15–17,19,24–33} LIC students are more likely to enter primary care^24,34–37 or rural practice^{15,23,37–39}; and demonstrate greater patient-centeredness.^{15,19–21,40,41} Benefits have also been shown for patient satisfaction and perceived patient health benefits^42–46; as well as preceptor satisfaction.^{8,14,27,47,48} Additionally, recent review articles have compiled recommendations for planning, implementing, and sustaining LICs.^7,10,49–52

LICs represent a significant departure from the TBR model in which most faculty and clinicians completed their education. Additionally, the training and logistics within most existing clerkships have been designed around the TBR model. Accordingly, the initiation of a new LIC remains a complex and difficult process that bears continuing investigation. Moreover, much of the literature focuses on important factors for ongoing program and student success rather than those factors critical in early-stage LIC implementation. In their 2021 literature review, Hense et al noted that although they originally wanted to focus on barriers and facilitators of LIC development in the early implementation phase, most studies they reviewed did not indicate where they were in the development process.⁵⁰ Bartlett et al's 2020 review intentionally gave more weight to longer-running programs, as these have demonstrated sustainability.⁷ Additionally, many of the studies in recent reviews were published more than 10 years ago.^1,7,10,49,50 Brown et al (2019) note a “relative dearth of recent literature” on factors enabling LIC success.⁴⁹ Given the rapidly evolving landscape of undergraduate medical education, including the accessibility of guidance from literature and faculty from well-established LIC programs, we undertook a qualitative study of the factors affecting the early phases of LIC development and implementation.

Previous Findings

The literature on LIC implementation includes factors relevant to the initial development of LICs as well as ongoing LIC success. During the early development stages, the literature indicates that medical schools need to determine their goals and objectives in choosing the LIC model, and based on these, make informed decisions regarding the type of LIC to be adopted (eg, amalgamative, blended, or comprehensive¹), setting and duration of the LIC, and percentage of students who will participate in the LIC.^7,10,49 Other factors that should be considered at the outset include how the LIC will integrate with the broader curriculum, including online resources^7,10,51,53; how the LIC will recruit and support students, mentors, and clinical sites^4,7,10,50,54; and how assessment, evaluation, and quality assurance will be managed.^7,10,49–51 Significant time and resources must be allocated to the development process.^{7,49,50,53,54}

The same considerations that are important for the ongoing success and sustainability of LICs seem likely to be important for successful development and initiation. These include recruiting and preparing engaged students^7,26,49,55 and enthusiastic preceptors^{7,49,52,54,55}; eliciting and responding to evaluation data and stakeholder feedback^7,49,51,53; securing strong support from school of medicine and clinic leadership^{2,3,7,10,49,53,55}; encouraging student sense of belonging at clinical sites^7,26,49,55; facilitating use of the Electronic Health Record system and notifications to enhance continuity with patients^19,49,51; engaging in cross-disciplinary collaboration^3,49; providing professional development and other support to preceptors^{2,4,7,10,17,49,51,53,54}; and ensuring the program provides mutually beneficial outcomes to both academic and clinical partners.^2,7,50 Barriers to ongoing success and sustainability include failure of faculty to adapt to a different style of teaching^7,49,50,54; student isolation in remote placements^7,17,49,50; and conflict or poor match between student and preceptor.^10,49 Recruitment of new faculty and clinics not accustomed to teaching can present challenges^3,4,8 and recruitment of faculty and clinics that already work with traditional block students can present a different set of challenges.^4,10,49 Costs associated with LICs may be daunting, both in terms of added expenses for students placed in remote locations and costs of faculty time for developing and running the LIC. Research suggests that these expenses may be recovered through eventual benefits to the teaching clinics and their larger communities.^{10,49–51,56–59}

While the aforementioned works shed light on previously reported factors related to LIC development, faculty at the Spencer Fox Eccles School of Medicine at the University of Utah were interested in exploring the experiences of programs that had initiated LICs in the last 5 years in hopes of identifying the most important facilitators, most challenging barriers, and top recommendations from a group of early-stage LICs. An ongoing understanding of these factors is needed to inform evidence-based LIC development, particularly as this model continues to be an increasingly popular approach to undergraduate medical education worldwide.

Methods

Participant recruitment

Faculty and staff of LICs were eligible to participate if they were implementing an LIC in the United States that was initiated less than 5 years ago or was still in the development stage. There were no other inclusion or exclusion criteria. To identify LICs established within the last 5 years, we first downloaded the listing of all LIC programs provided on the website of the Consortium of LICs (CLICs).⁶⁰ We identified 45 LIC programs located in the United States. If the LIC initiation year was not included in the program listing, we reviewed program websites to determine their start date or anticipated start date. (If the program website did not provide the initiation year, the program was removed from consideration.)

Ten LICs established in 2016 or later were identified, and leadership from nine of those programs were able to be reached via email or phone and invited to participate in a 1.5-h focus group. Five of these nine LICs participated. Additionally, the University of Utah participated in an ad hoc group of 10 medical schools awarded Value-Based Medical Student Education Training Program grants from the Health Resources and Services Administration (HRSA) in 2020. Seven of these grantees (including the University of Utah) were developing LICs and were also invited to participate. Three of these seven LICs participated. Thus, staff from 16 LICs were invited and 8 LICs participated (13 individual participants). Reasons for nonparticipation included lack of time and being too early in the development process to have information to contribute. Figure 1 illustrates our recruitment process. Table 1 shows that participating schools varied in terms of their LIC length, focus, and percentage of students participating.

Figure 1.

Consolidated criteria for reporting qualitative research (COREQ) diagram.

Table 1.

Participating new LICs.

Participating LICS	LIC initiation year	Length in weeks	AY21-22 cohort size	LIC focus	Percentage of students in LIC
MedStar Franklin Square Medical Center/Georgetown University	2016	22	12	Urban underserved	6%
Elson S. Floyd COM, WA State University	2018	46	80	General	100%
Sidney Kimmel Medical College at Thomas Jefferson University	2018	48	7	General	2%
TX Christian University and University of North TX Health Science Center SOM	2019	40	60	Urban, suburban	100%
University of MO SOM	2021	50	4	Rural	3%
Spencer Fox Eccles SOM at the University of UT	2021	26	3	Urban underserved in pilot; expanding to multiple types	2% in pilot Transitioning to 100%
University of Houston COM	2022	44	30	Community-based urban, suburban	100%
University of OK School of Community medicine	2025	48	N/A	Tribal, rural, and medically underserved	100%

LIC, longitudinal integrated clerkships.

Data Collection

Two 1.5-h focus groups were held over Zoom in June 2021. Five participants attended the first session, and the remaining eight participants attended the second. Both sessions were recorded. The three questions posed to the members of each focus group were:

What have been key supports or key actions which have supported or led to your successful LIC launch?

What significant barriers have been encountered in your LIC implementation process?

What recommendations do you have for those seeking to start an LIC (which would be helpful in successful implementation)?

We used a modified nominal group process (see Figure 2).^61,62 Participants were emailed the questions beforehand, and asked to consider their responses prior to the focus group. During the focus group, answers to each question were solicited from participants in a round-robin fashion. A moderator summarized each answer on a shared screen and asked for clarification as needed. During the focus groups, SG led the discussion and SKE summarized participant answers. LLB was a participant in the first focus group as the LIC Director of the University of Utah Spencer Fox Eccles School of Medicine. Once participants had no more answers to share for a given question, they were given time to review the written summaries and select the 2 answers that they felt were most important. Once everyone made their selections, they shared their choices with the rest of the group. (In the first focus group, participants shared their selections privately with the moderator using the chat feature, and the moderator summarized the results back to the group. During the second focus group, the chat feature was inadvertently disabled, so participants shared their selections out loud to the full group.)

Figure 2.

Modified nominal group process.

Due to time constraints, after each initial vote, we did not have the customary discussion and final vote. Instead, following the second focus group, the authors compiled all of the responses that received at least one vote across the 2 groups. Participants were asked via email to rank order what they felt were the top 5 most important answers to each question. All 13 focus group participants provided their final rankings.

Data Analysis

Because the researchers summarized the focus group responses during the focus group and the participants provided clarification as needed, we were confident that the written summaries accurately captured the participants’ sentiments. However, because participants had prepared their answers in advance, in many cases there were similar answers provided by multiple respondents, both within and across the 2 focus groups. Therefore, prior to sending the responses back to the participants for their final ranking, SKE and SG combined similar answers from both focus groups into concise statements to prevent redundancies from diluting the ranking process. Table 2 provides an example of multiple statements combined into a single item for ranking. Once all 13 respondents had submitted their rankings, the percentages of participants who ranked each answer in their top 5 were calculated and graphed.

Table 2.

Example of combined statements.

Statements recorded during the focus group	Combined statement for ranking
Steering committee that included all different levels of stakeholders; met regularly, everyone is kept abreast as changes take place in the development process	Stakeholder/key partner involvement—includes buy-in, communication, collaboration, and agreement with the direction of the LIC; includes a multidisciplinary approach that involves all levels (from students to leadership) including both the university and the health system.
Buy in of key partners—ability to have all the key partners bought in and on board
Multidisciplinary representation in the LIC meetings; buy-in
Support from the university and from the medical facility. Support from both arenas is critical
Time for collaboration, facetime for problem-solving and building collaboratively; clerkship directors and regional deans; need for a shared mental model for the LIC.
Willing partners: from the preceptor level all the way to systems level and SOM leadership. Key players must buy in.

LIC, longitudinal integrated clerkships.

Despite a small sample size, our methodology, and the content of the participants’ answers gave us confidence that we were able to reach saturation of the most important supports, barriers, and recommendations for new LIC implementation as experienced by these participants. The nominal group method encourages participants to spend time prior to the focus group thinking about their answers. Additionally, the round-robin format ensures that all participants have an opportunity to provide their answers. We did not end the discussion of any question until all participants indicated they had provided all their answers. Most importantly, there were redundancies among the answers both within and between the groups that indicated we had reached data saturation.⁶³

Ethical Considerations

This study was determined to be exempt by the Institutional Review Board of the University of Utah (IRB_ 00141272) in 2021. Participants were informed of the study purpose, questions, and procedure prior to the focus groups. Verbal consent was obtained at the beginning of both focus groups with the approval of our IRB because the study presented no more than minimal risk of harm to subjects and involved no procedures for which written consent is normally required outside of the research context.

Results

Key Supports for Successful Launch of an LIC

The first question discussed in the focus group addressed key supports or actions that supported the successful launch of the LIC. The first group (5 participants) provided 15 key supports, and the second group (8 participants) provided 23 key supports. The vote tallies taken during the Zoom group for this question in both groups revealed diversity in opinions about the most important key supports, with the majority of key supports receiving only one vote. The final key supports are provided in Figure 3, along with the frequency with which respondents ranked each factor in their top 5. Table 3 provides additional information about each factor as well as additional supports that were not ranked as top factors.

Figure 3.

Percentage of respondents who ranked each key support in their top 5.

Table 3.

Details about key support.

Key supports	Details
Stakeholder and partner involvement	Willing partners are needed from all subspecialties, university faculty and administration, community clinical staff, and the student body.
	Frequent communication is needed to ensure all stakeholders understand and support the structure, purpose, and benefits of the LIC model.
	Dedicated time needs to be built into the process to develop relationships and resolve problems collaboratively.
	Roles and responsibilities of partners need to be clearly defined.
	“I don't think this would be possible if we hadn't had partnership from all of those aspects, all of those different key players.” (FG1, P1)
Dedicated coordinator or team	Developing the LIC takes more time than expected. The coordinator or coordination team needs “a significant amount of dedicated time for developing curriculum and relationships with [partners.]” (FG1, P4)
Dedicated coordinator or team	Ideally, the coordinator or coordination team should be in place early in the planning process.
Leadership	LIC Directors need experience, knowledge, passion, commitment, and dedication. The Director needs to have or be able to develop relationships that allow them to facilitate connections with the community and clinicians.
	School of Medicine Leadership needs to provide support and champion the LIC, and effectively address challenges that arise.
	“It's having buy-in from your top leadership, and it really is that it's unwavering like the thing that our school is committed to is this LIC above a lot of other priorities and that has certainly helped our success.” (FG2, P4)
Clerkship directors identified prior to launch	Clerkship directors for each of the included specialties should be identified as early in the process as possible. Roles and responsibilities should be defined before the LIC launch.
Clerkship directors identified prior to launch	“Really having clerkship director roles identified […] as well as you know, clearly defining what their responsibilities are and how they each work with the staff has been really helpful. To just have that already set before the launch of LIC.” (FG2, P3)
Consultation with external experts	Consultation with experts who have experience directing or developing a similar type of LIC can provide valuable insight into strategies and challenges.
Consultation with external experts	“One of the things that really helped us a lot, at least getting started, was consultation with outside experts […] a number of times to give us guidance and give us the overall contours and help us be aware of what the challenges are going to be and how we do this or that and so that's been invaluable for us in the in the planning process, and I think in getting successfully approved to now implement the LIC.” (FG2, P2)
Assessment and evaluation systems developed early	Assessment and evaluation systems should be developed with careful forethought ahead of the LIC launch so that problems can be identified early and adjustments can be made before problems get too big.
Assessment and evaluation systems developed early	“Go ahead and build out your evaluative feedback process, again, for multiple stakeholders. You need feedback from students, from faculty, from your clerkship directors to allow for that flexibility and rapid cycle continuous quality improvement. You're going to need to be able to make changes, but they need to be driven by data, so you need that system in place really before you get started.” (FG2, P4)
Relationships between primary care and subspecialty providers	Relationships between primary care providers (PCPs) and specialty physicians can help students follow patients longitudinally and ease scheduling challenges.
	“Relationships already established between our primary care physicians and some of our specialty physicians eased the transition [to LIC] for them.” (FG2, P5)
Student champions	Students committed to the LIC model can help their classmates see the value and embrace the new model.
	Student champions can also serve as near-peer mentors to new LIC students. LIC students in inaugural classes may benefit from connections with LIC alumni at other medical schools.
	“When we finally had students who themselves had decided they were all-in and love the model, we had student champions, which just helped the rest of their classmates really embrace and see the LIC experience a little differently.” (FG1, P2)
Additional supports not rated in the top 5	Clerkship directors experienced in TBRs can help ensure the LICs meet the same competencies.
	Clinical coaches can support students and alert the team to student needs.
	A creative educational technology team can help keep curriculum delivery integrated.
	Grant funding for the coordinator, the development process, and evaluation can be instrumental in getting the LIC off the ground.
	Dedicated recruiters can help with the process of hiring preceptors and getting buy-in from clinical stakeholders.
	Directors with a history at the institution can leverage personal relationships and institutional knowledge to explain the new model and gain needed approvals.
	Careful selection of students can help ensure a successful LIC launch.
	Software and strong tech support can help with scheduling and evaluation.
	The flexibility and independence of the LIC can be leveraged to meet clerkship requirements.
	Electives such as emergency medicine can be offered if students will be getting the experiences in the course of the LIC.
	The relationship with on-site medical coordinators is important and can be facilitated by careful selection of clinical sites and frequent meetings. Likewise, working with a single medical director can help avoid the need to please too many people.
	The extension school may be able to facilitate relationships with rural communities.

LIC, longitudinal integrated clerkships.

Significant Barriers Encountered During LIC Implementation

The second question discussed in the focus group addressed significant barriers to the successful launch of the LIC. The first group provided 17 barriers, and the second group provided 22 barriers. The vote tallies taken during the first group showed diversity in opinions on the most important barriers. The vote tallies in the second group showed more convergence in opinions, with all respondents voting for preceptor and clinical site recruitment and 5 of the 7 participants voting for underestimating coordination work. This convergence was also seen in the online rankings of the responses. Figure 4 provides the responses that received at least one vote, and the percentage of respondents who ranked each factor in their top 5 after the focus groups. Table 4 provides additional information about each factor as well as additional barriers that were not top factors.

Figure 4.

Percentage of respondents who ranked each barrier in their top 5.

Table 4.

Details about barriers.

Barriers	Details
Preceptor and clinical site recruitment	Clinical sites may be inundated with requests for preceptors and may already have a high concentration of learners.
	Funding for preceptors is an ongoing concern. Clinicians, especially those operating on the relative value unit (RVU) model, may fear financial costs of precepting. Medical schools that do not offer preceptor compensation may be competing with schools that do.
	Preceptors who trained in the TBR model may not understand the LIC model and may find it daunting.
	It can be challenging to find preceptors for all subspecialties of the LIC within a community site.
	The COVID-19 pandemic was a substantial barrier as some health systems withdrew precepting support due to patient overload or social distancing needs. Recruitment of new preceptors was also stymied by system overload and the inability to meet face-to-face with administration at potential sites.
	“I think one of our biggest challenges is in identifying our preceptor network with our community partners - you know, having preceptors motivated to supervise students, because of the complexity experienced with the pressures for clinical revenue within their healthcare systems.” (FG2, P2)
	“For us, recruitment has been challenging as well, and what keeps coming up is funding for these very busy preceptors who are RVU-based [relative value unit]. […] They're not professors or assistant professors here at the School of Medicine, so they don't have to teach and they're doing it out of the goodness of their heart for the most part.” (FG1, P1)
Underestimating coordination work	New LICs underestimated the need for coordinator time, commitment, and allocation of resources within each of the clerkships.
	Clinical sites underestimated the need for a dedicated coordinator.
	New LICs specifically noted that adding LIC responsibilities to existing positions was not effective.
	Underestimation of work and time led some sites to take on too much responsibility.
	“The healthcare system that we're working with in our rural community greatly underestimated the need for a dedicated person to coordinate on their behalf. What they did was take a person who already had a full-time job and just shoved all the responsibilities onto them, and that has been a nightmare for all of us.” (FG2, P6)
Faculty need professional development	New community faculty may not have experience precepting.
	Experienced preceptors may need support to shift their teaching strategies from the TBR model to the LIC model.
	Logistical and communication issues can make professional development and support for community-based faculty difficult to provide effectively.
	“I think one of the hardest things for us … as you know, the distributed model can be great, but it also has been challenging with a lot of our partners really never having had a student before or not being familiar with medical education. That's definitely been a challenge that's needed additional faculty development.” (FG2, P4)
Logistics and structural misalignment	The medical education system and the health systems may have different priorities. Institutional policies reflecting these priorities can create a misalignment in the structure of the two systems and cause challenges in LIC implementation.
	The medical school must coordinate with all participating health systems, each of which may have different policies and requirements. Likewise, each health system may have to learn the policies and procedures of multiple medical schools.
	Legal requirements (affiliation agreements) require a lot of work in a short amount of time. University organizational processes that conflict with external partner processes can cause challenges to finalizing formal agreements, and result in implementation delays.
	“It is difficult to develop academic programs in non-academic spaces.” (FG1, P4)
Resistance to change	LICs can be confusing to stakeholders who are unfamiliar with the model. New programs reported needing to explain the model and its benefits repeatedly.
	Preceptors may not initially understand the difference between learning in the classroom (TBR model) and learning in the workplace (LIC model).
	“Something that we have struggled with is letting the preceptors know that change isn't bad. We know this is a brand-new model and it's scary - it's scary for everybody. And it comes with a new system, a new software system, it comes all these different things, but it's not necessarily a bad thing […] So facing that but encouraging change.” (FG2, P7)
Funding	Funding was a barrier for the new LICs, particularly finding funding for enough LIC staff, including clerkship and site directors to properly run the program, as well as funding for clinical sites.
Funding	“Obtaining funding. For people that are valuable to the implementation of the LIC such as clerkship directors or site directors. That would be very helpful in maintaining relationships and faculty development.” (FG1, P4)
Additional barriers not rated in the top 5	Core principles (eg, “continuity”) were perceived differently by different stakeholders.
	Initial scheduling looked very different across sites.•
	Developing an assessment model that tracked longitudinally was difficult.
	Preceptor drop-out (due to administration or RVU pressures) can create chaos.
	Students may be resistant to the inconvenience of rural placements. The right balance between commute times and educational outcomes must be considered.
	Integrating didactics for TBR and LIC students created challenges.
	Fees for student licenses for electronic health records may cause clinics to limit student participation.
	It may be difficult to obtain effective technology to manage the complex LIC model.
	The health system administration did not understand what it meant to teach in an LIC and sometimes blocked payment of preceptor stipends.
	It was challenging to systematize all of the hospital-based logistics, such as credentialing, onboarding, and training.

LIC, longitudinal integrated clerkships.

Recommendations for Starting an LIC

The final focus group question addressed recommendations for the successful launch of the LIC. The first group provided 14 recommendations, and the second group provided 21 recommendations. As with the first question, the vote tallies taken by both groups showed diversity in opinions about the most important recommendations. Figure 5 provides the combined recommendations that received at least one vote, and the percentage of respondents who ranked each factor in their top five after the focus groups. Table 5 provides additional information about each factor as well as recommendations that were not top factors.

Figure 5.

Percentage of respondents who ranked each recommendation in their top 5.

Table 5.

Details about recommendations.

Recommendations	Details
Invest in needs of clinical partners	Establish clinical partners and get commitments early in the development process.
	Invest extensively in addressing needs of clinical partners, taking care to understand their contexts and involving their leadership early.
	Plan enough time to provide professional development for faculty; communicate regularly with preceptors before they take their first student. “Start faculty development early, even before you get the LIC off the ground.” (FG2, P1)
	Make the time and effort to really understand the clinical sites you will be making requests from. For example, determine whether it is appropriate to initiate the conversation with faculty or leadership first. “Success may depend on going about the process properly, which depends on what their definition of properly is.” (FG1, P1)
Hire a qualified coordinator early	Have a coordinator who can do everything that needs to be done (planning, implementation, and networking) and have that person involved from the very beginning (in the planning phase).
	Allocate dedicated time for LIC responsibilities.
	“Be cautious about thinking that because the LIC is only a few students, it can be added on to an existing job, or that it's just another rotation. It is a lot of work and people with dedicated responsibility for the LIC are needed.” (FG2, P1)
Frequent communication with all stakeholders	Communication must be clear, frequent, open, and consistent to avoid miscommunication and misunderstanding.
	Make sure all stakeholder voices are heard.
	Meet frequently with the initial student cohort so they can accurately champion the program.
	Have frequent meetings with leadership to ensure their ongoing support and commitment.
	“Over-communicate to all levels of everyone involved in the planning process, both internal and external stakeholders; make sure everyone is on the same page at all times.” (FG2, P3)
	“Exercise the art of persuasion with people who may not understand LICs and may be afraid of change.” (FG2, P2)
Consult other LICs	Research other LICs, and consult with the ones that match your desired outcomes and structure.
Consult other LICs	“It was such a departure from the traditional way the curriculum has been delivered…so I even consulted with an outside institution's admissions and records officer to find out how they did all this…and that was all very helpful.” (FG2, P2)
Focus on the student experience	Keep a focus on the students; their experience will ultimately determine the success of the LIC.
Focus on the student experience	“Don’t forget to put yourself in the shoes of the students. Think about the user, your students, and what their experience will be like; always be thinking of students.” (FG2, P4)
Start with desired outcomes	Determine what you want to accomplish with the LIC at the very beginning of the development process.
Start with desired outcomes	“Before you start, define what the non-negotiable outcomes are for your LIC, and then figure out how that matches your community and resources. There are lots of ways to [run an LIC]; determine what will work for your community.” (FG2, P4)
Make the LIC curriculum equitable (not equivalent) to TBR⁷	Develop an equitable (not equivalent) curriculum that mirrors the TBR. Accept that the LIC and TBR experiences will not be equivalent.
Make the LIC curriculum equitable (not equivalent) to TBR⁷	“We had to make sure that the equity was there in the amount of education the student was getting with the LIC versus traditional blocks.” (FG2, P5)
Monitor and address issues immediately	If the LIC has enough students, have dashboards in place to help monitor the metrics of the learning environment so that problems can be identified and addressed early (before becoming larger problems.)
	Develop your evaluative feedback system before starting the LIC to allow for flexibility and data-driven continuous quality improvement.
	“Have a program of evaluation in place before starting; this saves a lot of trouble later on.” (FG1, P2)
Choose the right LIC for your stakeholders	Choose the right model for your specific context and goals.
	Identify a clear mission and rationale for your LIC, especially one that can stand out as unique.
	“Do what is right for the context of your school and elevates the mission of both the school and the community health care organization.” (FG2, P1)
Additional recommendations not rated in the top 5	Build in enough time to identify your community, faculty, and student champions.
	Build relationships with clinic managers or the process will fail even if preceptors are engaged.
	Discuss and prioritize the specific dimensions of continuity that the LIC should support (eg, students and preceptors, patients, peers, and place).
	Maintain flexibility during the planning phase. “There's nothing like flying a plane, while building it.” (FG1, P3)
	Account for driving distance so that it does not detract from clinical education for students in community placements.
	Create a detailed timeline for both planning and implementation.
	Conduct multiple trial runs with all technology. Use an accessible electronic resource to track process and progress.
	Make physician identity development a focus of the LIC.
	Start well in advance and include 2-3 months beyond the time you think you will need.

LIC, longitudinal integrated clerkships.

Discussion

Overall, our qualitative research indicates that the challenges described in earlier studies continue to apply, and lead to a conclusion that was woven throughout the study regarding facilitators, barriers, and recommendations: The early stage of LIC development takes a significant amount of time, generally more than anticipated, and needs to be resourced accordingly. Moreover, having those resources and staff in place early in the process was strongly recommended.

While there was general consensus around the key supports, barriers, and recommendations, there was divergence in participants’ votes regarding the most important factors. The greatest convergence was found among barriers. All of the participants selected preceptor and clinical site recruitment and all but one selected underestimating the amount of time and effort needed to develop and launch an LIC as a top barrier. These two barriers are not unrelated, considering the time and effort participants described as necessary for the preceptor recruitment process. The minor divergences in importance ratings represent the unique context of each school of medicine. Despite dissimilarities in the top two rankings of supports and recommendations collected during the focus group, participants agreed that the factors discussed were relevant to their programs. During the focus group, it was clear that participants could relate to all of the key supports, barriers, and recommendations.

Across the three categories, the majority of the factors listed as most important were related to human capital. As the LIC is fundamentally a model of learning through relationships,^26,53 it makes sense that relationships with stakeholders form the backbone of the early development process. Within this overarching topic of relationships were contributing factors of good relationships: frequent communication, investment of time, and reciprocity in terms of needs and priorities. Participants also noted the importance of repetition of a consistent message and updates as changes occur to avoid misunderstandings. Despite the increasing popularity of the LIC model, a lack of understanding of the model persists, even among medical school faculty, but especially among clinical partners.

Limitations

This study limited participation to new LICs located in the United States, in order to increase the likelihood of finding consensus. While there are likely common issues for new LICs regardless of location, medical schools considering LICs in other geographical areas may have different barriers, facilitators, and recommendations based on their specific local and cultural contexts. Additionally, while the current study included LICs with a range of characteristics in terms of length, size, focus, and percentage of students in the LIC, we did not have enough participants to be able to draw conclusions about different subtypes of LICs.

Next Steps

Preliminary results were presented virtually and discussed at the CLICs 2021 Conference in Stellenbosch, South Africa.⁶⁴ Subsequent efforts led to a collaboration of several participating medical schools on a Personally Arranged Learning Session (PeArL) on recruiting, retaining, and supporting preceptors in the early stages of LICs at the CLIC 2022 Conference in Limerick, Ireland,⁶⁵ as well as a roundtable discussion on community preceptor recruitment for LICs at the Society of Teachers of Family Medicine (STFM) 2023 Conference on Medical Student Education in New Orleans.⁶⁶

After the initial focus groups were held, we realized there was an opportunity for a continuing workgroup to problem solve and support one another in their early stage LIC implementation. We invited participants to continue to meet over Zoom on a quarterly basis. LIC staff from all eight schools have continued to participate, as well as a faculty member from another school who joined after hearing about our project at the 2021 CLIC conference. Discussion topics have included barriers and strategies for preceptor recruitment; how the LIC model compares to the TBR model in terms of the numbers of preceptors needed and the amount of time precepting requires of preceptors; administrative staff-to-student ratios needed for LICs; and how to engage preceptors in professional development and how much professional development to provide. Additional research is needed on the critical factors in early stage implementation of LICs, including expanding the geographical scope and comparing different subtypes of LICs.

Conclusions

The early stages of LICs continue to be challenging. While there was divergence in the participants’ preliminary votes regarding the most important supports, barriers, and recommendations, there was consensus regarding the final importance rankings of each of these factors. These findings demonstrate both the commonality of the issues in the LIC development process and the uniqueness of the specific needs and challenges in early-stage implementation within each LIC. Knowledge of these factors can help new schools allocate resources during their LIC development process. Participants found the focus group process and follow-up discussions useful and have formed an ongoing workgroup to meet quarterly.

Authors’ information

SG (MS, PhD, MPH) is an Adjunct Professor in Family and Preventive Medicine at the University of Utah Spencer Eccles Fox School of Medicine. He is also a Professor Emeritus at East Stroudsburg University of Pennsylvania. At the time of this study, he served as the lead program evaluator for the Health Resources and Services Administration (HRSA) grant-funded Value-Based Medical Student Education and Training Program. He has extensive experience with mixed methods program evaluation, including teaching research methods and statistics to Public Health graduate students.

SKE (PhD) is a Research Associate at the University of Utah Spencer Eccles Fox School of Medicine and the Evaluation Manager for the HRSA grant. She has extensive experience with mixed methods program evaluation.

LLB (MD) is a practicing pediatrician. At the time of the study, she was the LIC Director and clinical faculty at the University of Utah Spencer Eccles Fox School of Medicine.

Supplemental Material

sj-pdf-1-mde-10.1177_23821205231217894 - Supplemental material for Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation

Supplemental material, sj-pdf-1-mde-10.1177_23821205231217894 for Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation by Stacy K. Eddings, Laura L. Brown and Steven Godin in Journal of Medical Education and Curricular Development

Supplemental Material

sj-docx-2-mde-10.1177_23821205231217894 - Supplemental material for Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation

Supplemental material, sj-docx-2-mde-10.1177_23821205231217894 for Qualitative Findings of a Nominal Group Process to Identify Critical Factors in New LIC Implementation by Stacy K. Eddings, Laura L. Brown and Steven Godin in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgments

The researchers wish to thank all of the focus group participants for their time and their insightful answers to our questions. We would also like to thank the three anonymous reviewers who provided valuable feedback on our initial draft.

DECLARATION OF CONFLICTING INTERESTS

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Disclaimers

The views and content are those of the authors and not an official position of their institutions or funder.

FUNDING

This project was supported by the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) as part of the Value-Based Medical Student Education and Training Program (#T99HP39203) award totaling $17,644 661 with 10% financed with nongovernmental sources. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement, by HRSA, HHS or the U.S. Government.

ORCID iD

Stacy K. Eddings

Supplemental Material

Supplemental material for this article is available online.

References

Worley

Couper

Strasser

, et al. A typology of longitudinal integrated clerkships. Med Educ. 2016;50(9):922-932. doi:10.1111/medu.13084

Poncelet

Mazotti

Blumberg

Wamsley

Grennan

Shore

. Creating a longitudinal integrated clerkship with mutual benefits for an academic medical center and a community health system. Permanente Journal. 2014;18(2):50-56. doi:10.7812/TPP/13-137

Poncelet

Bokser

Calton

, et al. Development of a longitudinal integrated clerkship at an academic medical center. Med Educ Online. 2011;16(1):5939. doi:10.3402/meo.v16i0.5939

Epstein-Lubow

Cineas

Yess

Anthony

Fagan

George

. Development of a longitudinal integrated clerkship at The Warren Alpert Medical School of Brown University. R I Med J. 2015;98(9):27-31.

George

Tunkel

Dollase

, et al. The primary care-population medicine program at The Warren Alpert Medical School of Brown University. R I Med J. 2015;98(9):16-21.

Gheihman

Jun

Young

, et al. A review of longitudinal clinical programs in US medical schools. Med Educ Online. 2018;23(1):1-1. doi:10.1080/10872981.2018.1444900

Bartlett

Couper

Poncelet

Worley

. The do’s, don’ts and don’t knows of establishing a sustainable longitudinal integrated clerkship. Perspectives on Medical Education. 2020;9(1):5-19. doi:10.1007/s40037-019-00558-z

Bartlett

Dowell

Graham

, et al. Dundee’s longitudinal integrated clerkship: drivers, implementation and early evaluation. Educ Prim Care. 2019;30(2):72-79. doi:10.1080/14739879.2018.1564889

Hirsh

Ogur

Thibault

Cox

. “Continuity” as an organizing principle for clinical education reform. N Engl J Med. 2007;356(8):858-866. doi:10.1056/NEJMsb061660

10.

Ellaway

Graves

Berry

Myhre

Cummings

Konkin

. Twelve tips for designing and running longitudinal integrated clerkships. Med Teach. 2013;35(12):989-995. doi:10.3109/0142159X.2013.818110

11.

Bates

Konkin

Suddards

Dobson

Pratt

. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47(4):362-374. doi:10.1111/medu.12087

12.

Hauer

Hirsh

, et al. The role of role: learning in longitudinal integrated and traditional block clerkships. Med Educ. 2012;46(7):698-710. doi:10.1111/j.1365-2923.2012.04285.x

13.

Shahi

Walters

Ward

Woodman

Prideaux

. Clinical participation of medical students in three contemporary training models. Med Educ. 2015;49(12):1219-1228. doi:10.1111/medu.12815

14.

Snow

Gong

Adams

. Faculty experience and engagement in a longitudinal integrated clerkship. Med Teach. 2017;39(5):527-534. doi:10.1080/0142159X.2017.1297528

15.

Walters

Greenhill

Richards

, et al. Outcomes of longitudinal integrated clinical placements for students, clinicians and society. Med Educ. 2012;46(11):1028-1041. doi:10.1111/j.1365-2923.2012.04331.x

16.

Wamsley

Dubowitz

Kohli

Cooke

O’Brien

. Continuity in a longitudinal out-patient attachment for year 3 medical students. Med Educ. 2009;43(9):895-906. doi:10.1111/j.1365-2923.2009.03424.x

17.

Couper

Worley

Strasser

. Rural longitudinal integrated clerkships: lessons from two programs on different continents. Rural and Remote Health. 2011;11(1):226–236. doi:10.22605/RRH1665

18.

Mazotti

Adams

Peyser

Chretien

Duffy

Hirsh

. Diffusion of innovation and longitudinal integrated clerkships: results of the clerkship directors in internal medicine annual survey. Med Teach. 2019;41(3):347-353. doi:10.1080/0142159X.2018.1472369

19.

Ogur

Hirsh

Krupat

Bor

. The Harvard Medical School-Cambridge Integrated Clerkship: an innovative model of clinical education. Acad Med. 2007;82(4):397-404. doi:10.1097/ACM.0b013e31803338f0

20.

Teherani

Irby

Loeser

. Outcomes of different clerkship models: longitudinal integrated, hybrid, and block. Acad Med. 2013;88(1):35-43. doi:10.1097/ACM.0b013e318276ca9b

21.

Konkin

Suddards

. Creating stories to live by: caring and professional identity formation in a longitudinal integrated clerkship. Advances in Health Sciences Education. 2012;17(4):585-596. doi:10.1007/s10459-011-9335-y

22.

O’Doherty

Culhane

O’Doherty

, et al. Medical students and clinical placements - a qualitative study of the continuum of professional identity formation. Educ Prim Care. 2021; 32(4):1–9. doi:10.1080/14739879.2021.1879684

23.

Roberts

Daly

Kumar

Perkins

Richards

Garne

. A longitudinal integrated placement and medical students’ intentions to practise rurally. Med Educ. 2012;46(2):179-191. doi:10.1111/j.1365-2923.2011.04102.x

24.

Latessa

Beaty

Royal

Colvin

Pathman

Heck

. Academic outcomes of a community-based longitudinal integrated clerkships program. Med Teach. 2015;37(9):862-867. doi:10.3109/0142159X.2015.1009020

25.

O’Brien

Poncelet

Hansen

, et al. Students’ workplace learning in two clerkship models: a multi-site observational study. Med Educ. 2012;46(6):613-624. doi:10.1111/j.1365-2923.2012.04271.x

26.

Roberts

Daly

Held

Lyle

. Social learning in a longitudinal integrated clinical placement. Adv in Health Sci Educ. 2017;22(4):1011–1029.

27.

Teherani

O’Brien

Masters

Poncelet

Robertson

Hauer

. Burden, responsibility, and reward: preceptor experiences with the continuity of teaching in a longitudinal integrated clerkship. Acad Med. 2009;84(10):S50–S53. doi:10.1097/ACM.0b013e3181b38b01

28.

Worley

Silagy

Prideaux

Newble

Jones

. The parallel rural community curriculum: an integrated clinical curriculum based in rural general practice. Med Educ. 2000;34(7):558-565. doi:10.1046/j.1365-2923.2000.00668.x

29.

Hirsh

Walters

Poncelet

. Better learning, better doctors, better delivery system: possibilities from a case study of longitudinal integrated clerkships. Med Teach. 2012;34(7):548-554. doi:10.3109/0142159X.2012.696745

30.

Myhre

Woloschuk

Pedersen

. Exposure and attitudes toward interprofessional teams: a three-year prospective study of longitudinal integrated clerkship versus rotation-based clerkship students. J Interprof Care. 2014;28(3):270-272. doi:10.3109/13561820.2013.829425

31.

Woloschuk

Myhre

Jackson

McLaughlin

Wright

. Comparing the performance in family medicine residencies of graduates from longitudinal integrated clerkships and rotation-based clerkships. Acad Med. 2014;89(2):296-300. doi:10.1097/ACM.0000000000000113

32.

Ogur

Hirsh

. Learning through longitudinal patient care—narratives from the Harvard Medical School–Cambridge Integrated Clerkship. Acad Med. 2009;84(7):844-850. doi:10.1097/ACM.0b013e3181a85793

33.

Zink

Halaas

Finstad

Brooks

. The rural physician associate program: the value of immersion learning for third-year medical students. J Rural Health. 2008;24(4):353-359. doi:10.1111/j.1748-0361.2008.00181.x

34.

Ford

Patel

Sierpina

Wolffarth

Rowen

. Longitudinal continuity learning experiences and primary care career interest: outcomes from an innovative medical school curriculum. J Gen Intern Med. 2018;33(10):1817-1821. doi:10.1007/s11606-018-4600-x

35.

O’Donoghue

McGrath

Cullen

. How do longitudinal clerkships in general practice/primary care impact on student experience and career intention? A cross-sectional study of student experience. Educ Prim Care. 2015;26(3):166-175. doi:10.1080/14739879.2015.11494336

36.

Pfarrwaller

Sommer

Chung

, et al. Impact of interventions to increase the proportion of medical students choosing a primary care career: a systematic review. J Gen Intern Med. 2015;30(9):1349-1358. doi:10.1007/s11606-015-3372-9

37.

Zink

Center

Finstad

, et al. Efforts to graduate more primary care physicians and physicians who will practice in rural areas: examining outcomes from the University of Minnesota–Duluth and the rural physician associate program. Acad Med. 2010;85(4):599-604. doi:10.1097/ACM.0b013e3181d2b537

38.

Myhre

Bajaj

Woloschuk

. Practice locations of longitudinal integrated clerkship graduates: a matched-cohort study. Canadian Journal of Rural Medicine (Joule Inc). 2016;21(1):13-16.

39.

Somporn

Ash

Walters

. Stakeholder views of rural community-based medical education: a narrative review of the international literature. Med Educ. 2018;52(8):791-802. doi:10.1111/medu.13580

40.

Gaufberg

Hirsh

Krupat

, et al. Into the future: patient-centredness endures in longitudinal integrated clerkship graduates. Med Educ. 2014;48(6):572-582. doi:10.1111/medu.12413

41.

Krupat

Pelletier

Alexander

Hirsh

Ogur

Schwartzstein

. Can changes in the principal clinical year prevent the erosion of students' patient-centered beliefs? Acad Med. 2009;84(5):582-586. doi:10.1097/ACM.0b013e31819fa92d

42.

Beard

Candy

Anderson

, et al. Patient satisfaction with medical student participation in a longitudinal integrated clerkship: a controlled trial. Acad Med. 2020;95(3):417-424. doi:10.1097/ACM.0000000000003021

43.

Hudson

Knight

Weston

. Patient perceptions of innovative longitudinal integrated clerkships based in regional, rural and remote primary care: a qualitative study. BMC Fam Pract. 2012;13(1):72-79. doi:10.1186/1471-2296-13-72

44.

Flick

Felder-Heim

Gong

, et al. Alliance, trust, and loss: experiences of patients cared for by students in a longitudinal integrated clerkship. Acad Med. 2019;94(11):1806-1813. doi:10.1097/ACM.0000000000002812

45.

Poncelet

Wamsley

Hauer

Lai

Becker

O’Brien

. Patient views of continuity relationships with medical students. Med Teach. 2013;35(6):465-471. doi:10.3109/0142159X.2013.774335

46.

Weston

Hudson

. Clinical scholarship among preceptors supervising longitudinal integrated medical clerkships in regional and rural communities of practice. Aust J Rural Health. 2014;22(2):80-85. doi:10.1111/ajr.12090

47.

Walters

Prideaux

Worley

Greenhill

. Demonstrating the value of longitudinal integrated placements to general practice preceptors. Med Educ. 2011;45(5):455-463. doi:10.1111/j.1365-2923.2010.03901.x

48.

Krehnbrink

Patel

Byerley

, et al. Physician preceptor satisfaction and productivity across curricula: a comparison between longitudinal integrated clerkships and traditional block rotations. Teach Learn Med. 2020;32(2):176-183. doi:10.1080/10401334.2019.1687304

49.

Brown

Anderson

Finn

. A narrative literature review considering the development and implementation of longitudinal integrated clerkships, including a practical guide for application. J Med Educ Curric Dev. 2019;27(6):1–12. doi:10.1177/2382120519849409

50.

Hense

Harst

Küster

Walther

Schmitt

. Implementing longitudinal integrated curricula: systematic review of barriers and facilitators. Med Educ. 2021;55(5):558-573. doi:10.1111/medu.14401

51.

Hudson

Poncelet

Weston

Bushnell

Farmer E

. Longitudinal integrated clerkships. Med Teach. 2017;39(1):7-13. doi:10.1080/0142159X.2017.1245855

52.

Bernstein

Wood

Latessa

Hirsh

. Teaching in longitudinal integrated clerkships: the seven ‘C’s. Clinical Teacher. 2019;16(2):101-107. doi:10.1111/tct.12947

53.

Hudson

Farmer

Weston

Bushnell

. Using a framework to implement large-scale innovation in medical education with the intent of achieving sustainability. BMC Med Educ. 2015;15(1):1-9. doi:10.1186/s12909-014-0282-1

54.

Heddle

Roberton

Mahoney

Walters

Strasser

Worley

. Challenges in transformation of the “traditional block rotation” medical student clinical education into a longitudinal integrated clerkship model. Educ Health: Change Learn. 2014;27(2):138-142. doi:10.4103/1357-6283.143744

55.

Bonney

Albert

Hudson

Knight-Billington

. Factors affecting medical students’ sense of belonging in a longitudinal integrated clerkship. Aust Fam Physician. 2014;43(1):53-57.

56.

Worley

Kitto

. Hypothetical model of the financial impact of student attachments on rural general practices. Rural and Remote Health. 2001;1(1):83. doi:10.22605/RRH83

57.

Woolley

Sen Gupta

Stewart

Hollins

. A return-on-investment analysis of impacts on James Cook University medical students and rural workforce resulting from participation in extended rural placements. Rural Remote Health. 2021;21(4):6597. doi:10.22605/RRH6597

58.

Hudson

Weston

Farmer

. Medical students on long-term regional and rural placements: what is the financial cost to supervisors? Rural Remote Health. 2012;12(2):1951. doi:10.22605/RRH1951

59.

Walters

Worley

Prideaux

Lange

. Do consultations in rural general practice take more time when practitioners are precepting medical students? Med Educ. 2008;42(1):69-73. doi:10.1111/j.1365-2923.2007.02949.x

60.

CLIC - The Consortium of Longitudinal Integrated Clerkships. CLIC. Accessed November 4, 2022. https://clicmeded.com/

61.

Delbecq

de Ven A

Glenview

. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Scott, Foresman and Company; 1975.

62.

McMillan

King

Tully

. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38(3):655-662. doi:10.1007/s11096-016-0257-x

63.

Saunders

Sim

Kingstone

, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52(4):1893-1907. doi:10.1007/s11135-017-0574-8

64.

Eddings

Brown

Godin

. Qualitative findings of a nominal group process to identify critical factors in LIC implementation. Paper Presented at: CLIC - Consortium of Longitudinal Integrated Clerkships Conference; October 10-13, 2021; Stellenbosch, South Africa.

65.

Brown

Cerqueira

Eddings

, et al. Recruiting, retaining, and supporting preceptors in the early stages of longitudinal integrated clerkship (LIC) implementation. Paper Presented at: CLIC - Consortium of Longitudinal Integrated Clerkships Conference; June 17-20, 2022; Limerick, Ireland.

66.

Morris

Touchet

Porter

Meyers

LaCour-Chestnut

. Recruiting and retaining community preceptors in a longitudinal clerkship. Paper presented at: Society of Teachers of Family Medicine (STFM) Conference on Medical Student Education; January 26-29, 2023; New Orleans, LA.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.47 MB