Sage Journals: Discover world-class research

Abstract

Introduction:

Primary care clinicians are presented with hundreds of new clinical recommendations and guidelines. To consider practice change clinicians must identify relevant information and develop a contextual framework. Too much attention to information irrelevant to one’s practice results in wasted resources. Too little results in care gaps. A small group of primary care clinicians in a large health system sought to address the problem of vetting new information and providing peer reviewed context. This was done by engaging colleagues across the system though a primary care learning collaborative.

Methods:

The collaborative was a grass roots initiative between community and academic-based clinicians. They invited all the system’s primary care clinicians to participate. They selected new recommendations or guidelines and used surveys as the principal communication instrument. Surveys shared practice experience and also invited members to give narrative feedback regarding their acceptance of variation in care relate to the topic. A description of the collaborative along with its development, processes, and evolution are discussed. Process changes to address needs during the COVID-19 pandemic including expanded information sharing was necessary.

Results:

Collaborative membership reached across 5 states and included family medicine, internal medicine, and pediatrics. Members found involvement with the collaborative useful. Less variation in care was thought important for public health crises: the COVID pandemic and opioid epidemic. Greater practice variation was thought acceptable for adherence to multispecialty guidelines, such as diabetes, lipid management, and adult ADHD care. Process changes during the pandemic resulted in more communications between members to avoid practice gaps.

Conclusion:

An internet-based learning collaborative in a health system had good engagement from its members. Using novel methods, it was able to provide members with feedback related to the importance of new practice recommendations as perceived by their peers. Greater standardization was thought necessary when adopting measures to address public health crisis, and less necessary when addressing multispecialty guidelines. By employing a learning collaborative, this group was able to keep members interested and engaged. During the first year of the COVID pandemic the collaborative also served as a vehicle to share timely information.

Keywords

practice variation COVID learning collaborative learning contextual learning primary care learning collaborative

Introduction

Primary care clinicians are presented with hundreds of clinical recommendations and guidelines. The United States Preventive Services Task Force (USPSTF) alone has nearly 100 recommendations, 23 of them new in the past 2 years.¹ Specialty societies and disease or organ specific organizations have over 1000 guidelines related to primary care.² Translating research into practice is difficult, often taking many years.³ Incorporating new recommendations requires practice change, which has costs; costs which may take away from other elements of practice.⁴ Before a decision to invest in practice change clinicians must identify relevant information⁵ and then develop a contextual framework relating it to one’s practice.^6-9

With so much competing and at times conflicting information both intra practice and inter practice variation will result. Inter practice variation may be intentional due to factors such as clinician, administrative, or community preference.^10,11 Or it may be unintentional due to knowledge or resource deficits. Likewise intra practice variation may be intentional due to customization or unintentional due to inconsistent processes.¹² To compound the problem, there are often conflicting recommendations such as those with colon cancer¹³ or prostate cancer screening.^14,15 Whereas customization may enhance the quality and satisfaction of care,¹⁶ unwanted variation may result in added costs, inefficiency, and potential gaps in care.¹⁷

Although individualized care (personalization) is the goal of all primary care clinicians, within specialties, regions, health care systems, or practices, patients should expect consistent care (standardization) surrounding certain elements of practice.^2,15 In our large health system 5 primary care clinicians (3 community-based, 2 academic-based) became aware and intrigued by wide variation within and between practices. Observed variation centered around the application of new recommendations such as those by the USPSTF. The group met with leaders from the Department of Family Medicine to discover ways to better understand, and when appropriate address practice variation and potential gaps. A learning collaborative model was adopted.^18,19 The collaborative would provide an interactive, inclusive experience that would not only disseminate information, but allow peer-to-peer learning and leverage the collective experience of its members. The goal of the collaborative was to ensure all members were aware of selected recommendations and through shared experiences place them into context for their own practices. Over the first 3 years the years the collaborative continually evolved. For example, changes were made in response to member feedback to include more topical information within the surveys and expand the discussion sent with the results. The SARS-CoV-2 pandemic prompted greater information sharing outside the survey process to help members stay abreast of information not available on the system’s intranet. This is a description of a primary care learning collaborative in a large health system.

Methods

The Primary Care Learning Collaborative

The collaborative resides within a large health system that covers five states including Arizona, Florida, Iowa, Minnesota, and Wisconsin. All share the same electronic health record and intranet. The system houses a medical school and postgraduate programs including family medicine, pediatrics, and internal medicine. The Department of Family Medicine (DFM) agreed to provide infrastructure support and facilitate the development process. A roadmap outlining the collaborative development and process is shown in Table 1. The initial steps were to identify a steering committee (SC) and a medical director. The SC was recruited through personal invitation of known colleagues and through the initial survey with the intention of creating broad representation and diversity. The SC was charged with developing the process, with the medical director taking on responsibility as the principal investigator. It became the responsibility of the SC to identify survey topics, develop and send the surveys, analyze and deliver results, and provide a summary with discussion to members. The role of the medical director was to oversee these activities, serve as the point of contact for collaborative members and interested stakeholders outside the network, ensure compliance with IRB regulations, and maintain alignment with the health system’s values. The initial SC was 12 members and represented all states and included family medicine, primary care internal medicine, primary care pediatrics, nurse practitioners, physicians, and physician assistants. To increase representation after 2 years the SC expanded to 20 members. Participation in the SC was voluntary. In June 2017 the initial email invitation to become a member of the health system’s primary care learning collaborative was sent to 506 primary care clinicians identified from department (family medicine, general internal medicine, general pediatrics) lists, trying to include as many of the system’s clinicians practicing in any outpatient primary care setting as possible. Membership enabled participants to propose survey topics, provided avenues for feedback to practice leadership, created opportunities for benchmarking peers, and to belong to the only systemwide group of practicing primary care clinicians. In July 2019, using an expanded distribution list, 851 primary care clinicians providing any outpatient care were again sent an invitation to promote expanded participation. Both invitations requested completion of a personal and practice demographic survey from participants. Topical surveys were only mailed to members who agreed to participate through the demographics survey. All survey results were deidentified.

Table 1.

Learning Collaborative Road Map.

Steering Committee Development

Initial Steering Committee (SC) 12 members
• Five initial interested physicians
• Department of Family Medicine members
○ Important as research and content experts
• Personal invitation to colleagues
○ Broaden representation and inclusiveness
SC growth 20 members
• Invitation to join SC sent to all members sent at year 2
○ Four new clinicians accepted SC role
• Personal invitation to colleagues
Maintain and broaden representation and inclusiveness

Initial Charge and Network Development

Develop interactive process
• Survey Description
○ Take <3 min to complete
○ Based on practice behavior, not knowledge
○ Include topical recommendations/guidelines in survey
○ Questions to be non-judgmental and inviting
• Return results promptly
○ Include supplementary discussion and references
○ Share all member narrative responses unless requested otherwise
• Answer questions through appropriate venue
○ Personal email if sent directly to MD or SC member
○ Survey responses were deidentified so responses would be to general membership when necessary
Submit proposal for IRB approval
• Medical director (MD) to serve as principal investigator
Invite all clinicians providing primary care in the system to become members

Steering Committee Roles

Identify survey topics
Develop surveys
Sent out surveys
Analyze results
Provide results summary and discussion to members
Maintain alignment with health system mission and values

Medical Director Roles

Oversee SC
• Ensure the survey process proceeds
Ensure alignment with health system
• Ensure compliance with system and IRB regulations
Serve as point of contact for network members and other stakeholders

Member Privileges

May propose topics for surveys
May serve as authors on results discussions
Access to full data sets
• For personal review
• For publication or presentation
Become member of the only enterprise-wide primary care initiative

Example of Survey Timeline

Survey development
• Survey topics are received throughout the year
• Source: members, SC, or leadership
• Survey topic is selected 4 to 6 weeks before expected mailing
• Principle author (PA) and MD develop survey with input from SC
Survey is submitted to IRB 2 to 3 weeks before anticipated mailing

Survey mailing
• Initial mailing mid-week
• Second mailing to non-responders 2 weeks after initial mailing
Results and discussion
• 1 to 2 weeks after initial mailing PA, MD, and SC start reviewing results
• 3 to 4 weeks after initial mailing PA, MD, and SC start writing discussion
• 4 to 6 weeks after initial mailing MD sends out results and discussion

Major Changes During First 3 Years

Developed process to compare perceived member relevance of topic
• The “ranking question”
Enhanced discussion and results review
Increased informational component within surveys
Shared supplementary educational materials during first year of COVID
• Shared materials sent by members to all membership
• Based on survey results, developed materials for caring for post-acute COVID

Surveys

A survey-based collaborative was chosen as the interactive tool to allow participation by busy primary care clinicians; something they could fill out quickly and easily. Based on the steering committee’s previous experience, surveys would be brief, interactive, relevant, informative, inclusive, and allow 360-degree communication-survey with supplementary information sent, receive responses, and promptly return results. Any direct emails to the medical director were answered promptly. Questions generally focused on practice behaviors, not subject knowledge. Examples are shown in Table 2. To ensure appropriate oversight and to utilize the system’s internal survey system, a version of RedCap, all surveys were approved by the Mayo Clinic Institutional Review Board (IRB). IRB review is required for surveys using RedCap.

Table 2.

Selected Survey Questions.

Selected General Survey Questions

1. What best describes your current practice of screening for hepatitis C virus (HCV)?
2. Which best describes your use of Urine Drug Testing related to opioid management?
3. How do you currently address questions or concerns your patients may have about breast density?
4. What best describes your patients’ access to Suboxone?
5. Do your patients have barriers to the availability to tobacco cessation counseling?
6. Do you share information about oral-pharyngeal cancer with your patients or families when they ask why young men should be immunized against HPV?
7. What are your barriers to participating in team huddles?
8. Which instrument(s) do you use most often when managing children 6 to 12 years old with ADHD?
9. When providing a Goals of Care conversation, do you use the Advance Care Planning code in addition to the E/M code for billing?
10. Which of the following factors do you find as useful in identifying patients who may benefit from palliative services or a Goals of Care conversation?
11. What best describes your treatment role of adults ages 18 to 55 with attention deficit disorder (ADHD)?
12. When do you perform pelvic examinations on non-pregnant, asymptomatic women (also exclude those for whom you have a high index of concern for abnormality)?
13. What best describes your response to the new USPSTF guideline for cervical cancer screening for low-risk women age 30 to 65? (May do rHPV testing only.)
14. What reflects your practice regarding PrEP for those at high risk for HIV?
15. What best describes your approach to lipid screening on children 9 to 11?
16. What is your experience with the newer lipid lowering agents, PCSK9 inhibitors?
17. As the COVID-19 pandemic unfolded and we prepared for a possible patient surge: What were some of the clinical challenges you perceived as necessary to address?
18. What have been some of your personal concerns during the first 6 weeks of the pandemic? (sent April 2020)
19. Which of the following patients with COVID infections have you helped manage? (month 8 of pandemic)
20. In your practice how often do you see patients that you suspect have at least one social factor that negatively influences their health?

Selected Survey Ranking Questions

1. How important do you think it is that all primary care teams be allowed and encouraged to have team huddles?
2. How critical do you think it is that practice gaps be addressed related to attention deficit disorder in pediatric patients?
3. How critical do you think it is that practice gaps be addressed related to chronic opioid management-universal use of medication agreement, checking the PDMP, and urine drug testing?
4. How important is it that we standardize how we manage the refilling of chronic medications?
5. How critical do you think it is that practice gaps be addressed related to advanced care planning and goals of care conversations?
6. How critical do you think it is that practice gaps be addressed related to attention deficit disorder in adult patients?
7. How critical do you think it is that practice gaps be addressed related to screening and referral for intimate partner violence?
8. How important is it to minimize variation from guidelines when managing patients with diabetes?
9. How important is it to minimize variation across our practices when providing pre-exposure prophylaxis for HIV?
10. How critical is it to minimize variation between practices when discussing lipid management and risk modification in adults?
11. How important is to minimize variation between practices in our approach to patients with suspected of confirmed COVID-19?
12. How important is it for providers to continue sending a consistent message to patients on COVID-19 related issues, including safety and prevention, based on guidelines from the CDC?

Survey topics were generated from collaborative members, SC members, and system leadership. There was an attempt to select topics that were between early adaptation and widespread implementation or when wide practice variation was expected.²⁰ In the first 3.5 years there were 18 surveys on clinical or practice management. Surveys were sent via email 6 to 8 times a year, took less than 3 min to complete, and results were communicated to participants in less than 6 weeks from survey closure. The results report included a brief summary, expanded discussion, access to the raw data and often a link to supporting materials. Communication from members occurred within the survey (anonymously) or through the medical director if the member wanted a direct response. All surveys provided respondents the opportunity to add comments which were shared with the collaborative membership unless the writer indicated a preference otherwise. To create better context around the topic, beginning with the fifth survey, we introduced a ranking question addressing the acceptability of practice variation ranked from 1 (variation acceptable) to 10 (consistency important). Question examples are shown in Table 2.

During the second year a survey was sent to better understand the value of the survey components. Respondents were asked to rate each component as useful, somewhat useful, slightly useful, or not useful.

Expanded Communications

Throughout the COVID-19 pandemic in response to changing clinical and personal demands the SC developed additional communications with members. Personal concerns were shared anonymously with appropriate leadership. In the early months of the pandemic the health system posted regularly updated material on the systems website. Even so, rapidly changing information on a variety of subjects left knowledge gaps members would obtain through various methods they would then share with steering committee members. In the spirit of information sharing, this would then be shared with all members when appropriate in separate communications outside the usual survey process. Later in the pandemic, with a paucity of information but an ever-expanding need for addressing post-COVID care, the network again initiated vigorous information sharing on the evaluation and management of those recovering from acute COVID-19.

Statistical Analysis

For the ranking questions median rankings were used rather than means because the data was non-normal and skewed.²¹ The 95% intervals were estimated for each median using resampling methods.^22,23 Non-parametric resampling methods were used because the rankings were integers not continuous numbers and the data lacked normality. For the resampling, 2000 iterations were done. During each iteration, the ratings provided for each item were resampled to obtain the median and confidence interval. We used R, version 4.0.2 (R Core Team 2020) for all analysis.²⁴

Results

Results of the 2 demographic surveys are shown in Table 3. In the 2017, 24.7% of invited clinicians elected to participate in the learning collaborative, while in 2019 23.7% participated. The second demographic survey identified more primary care clinicians and more responded from internal medicine (5.6%-18.3%), nurse practitioners (13.6%-24.8%), and women (49.6%-64.7%). Starting in 2019 members of the collaborative were more likely female (65%-57%) and physicians rather than NP/PS’s (64%-57%) when compared to the system’s family medicine clinicians.²⁵ Respondents of the second survey were more likely to practice in a rural community (12%-21.8%), provide outpatient only care (43%-68%), and provide obstetrical care (21.4%-34.2%).

Table 3.

Primary Care Learning Collaborative Member Demographics July 2017 and July 2019.

Survey year	2017	%	2019	%
Response rate	125/506	24.7	202/851	23.7
Average age	48.4		46.3
Gender
Male	63	50.4	71	35.3
Female	62	49.6	130	64.7
Ethnicity
African American	1	1.6	3	1.5
Asian	4	3.2	14	6.9
Caucasian non-Hispanic	109	87.5	172	85.1
Hispanic	4	3.2	8	4
Other/decline	6	4.8	5	2.5
Specialty
Family medicine	103	82.4	140	69.3
Family medicine physicians			84	41.5
Internal medicine	7	5.6	37	18.3
Pediatrics	13	10.4	15	7.4
Other	2	1.6	10	5
Primary credentials
MD/DO	95	76	130	64.4
APNP	17	13.6	50	24.8
PA	13	10.5	18	8.9
Other			4	1.9
Other credentials
Certificates of Added Qualifications (1 or more)	25	20	38	18.8
Other degree (MS/MPH/PhD/Other)	23	18.4	42	21.8
Community size
>100 000	39	31.2	82	40.6
10 000-100 000	71	56.8	68	33.4
<10 000	15	12	42	21.8
Practice elements
Outpatient only	52	43	139	68
Inpatient non-critical access hospital	15	12.4	39	19.3
Inpatient critical access hospital	16	12.8	23	11.4
Ob with delivery	17	14	28	13.9
Ob with delivery family physician			18	21.4
Ob pre/postnatal only	9	7.4	41	20.3
Ob pre/postnatal only family physicians			13	15.5
Residency faculty	19	15.7	34	16.8
Other teaching			90	44.6
Research			42	20.8
Nursing home	27	22.3	43	21.3
Administration			81	40.1
Emergency department
Work in department or cover admissions			20	9.9
Call which requires one to go in			44	21.8

Between July 2017 and November 2020 18 topical surveys were sent. Of the 18 surveys, 7 related to USPSTF recommendations, 6 on clinical management, 3 on national guidelines, and 2 on practice management. Overall participation by the learning collaborative in the topical surveys was 1221/2470 (49.4%) with individual survey participation varying from 38.8% to 70.6%. Thirteen percent of individuals completed greater than 85% (11 completed all) and 12% completed less than 15% (4 completed 0) of the surveys.

Inter practice variation was noted throughout most surveys. Some of this is reflected in the written responses with examples shown in Table 4. With the additional ranking question (examples of ranking questions in Table 2) members expressed an expectation for the most consistency and the least variation in practice related to COVID-19 public health management and opioid prescribing. For diabetic guidelines, lipid screening guidelines, fluoride treatment, and adult ADHD they accepted more variation (Figure 1) compared to the other topics.

Table 4.

Selected Survey Results. A. Selected Narrative responses. B. Assessment of Learning Collaborative Elements. C. New or Increased Clinical Problems During COVID Pandemic Surge.

A. Selected Comments from Survey Respondents
1. Based on the questions, I think I am missing opportunities to screen2. Thought provoking3. I learned from this survey4. While it is important to utilize guidelines, we need to individualize our management of our patients using careful discussion with the patient. We need to avoid “cookbook” medicine or the impression that we just follow algorithms. Medicine is an art.5. I’ve found these surveys very useful, especially since you began adding the supplementary information in the survey itself. Thanks for all of your work on this.6. Standardization of the general process can still be done while individualizing the clinical care that we provide.7. There is enough “gray” in this topic to allow for some variation8. I did not realize people did that!9. I’m not sure the statement in the first question is necessarily true, and certainly a source of recent debate10. I like the idea of breaking results into “pearls” that we can incorporate into practice (or changes we can make to our practices to improve care).11. I am well aware of potential benefits of PCSK9 inhibitors and choose to not use them. Costs do not outweigh benefits at current price point. These medications have a significant financial toxicity on individual patients and the US healthcare system for minimal improvement in patient-oriented outcomes.12. I like your format, with additional information available within the survey, but it is not intuitive. I just clicked to see what happened.
B. Results of a Survey to Assess the Perceived Usefulness of Elements of CollaborativeThose ranking the component useful/somewhat useful
	N = 51	%
Overall response	51/116	44
Any component useful/somewhat useful	46	90
Educational material sent with survey	43	84
Brief summary	42	82
Expanded summary and discussion	38	74
Participation in the surveys	35	69
Access to the raw data	24	47
C. New or Increased Clinical Problems During COVID Pandemic Surge: November 2020
Which of the following patients with COVID related problems have you helped manage?
Post illness follow up, ongoing problems	77	82.8
Post illness follow up, asymptomatic	73	78.5
Acute illness, outpatient	64	68.8
Notification by remote home monitoring my patient is being monitored	52	55.9
Which of the following have you seen in your patients who do NOT have COVID as an indirect effect of the pandemic?
Increased anxiety	89	95.7
Increased depression	86	92.5
New anxiety	82	88.2
Loneliness	81	87.1
Delayed or deferred care for chronic illness (heart disease, diabetes, etc.)	74	79.6
New depression	73	78.5
Delayed or deferred cancer screening	71	76.3
Other mental illness related to societal/pandemic affects	47	50.5

Figure 1.

Relative rankings of acceptable variation versus consistency by topic. Respondents’ perception of criticality of variation in care related to survey topic.

A survey to assess the perceived usefulness of the process/survey components was sent after the second year. Results are shown in Table 4. Of 51 respondents, 46 (90%) found at least one of the components useful. Eighty-four percent found topical information sent with the first survey the most useful followed by the brief summary with discussion, 82%. This survey had the lowest response rate of all surveys (44%).

The collaborative continuously evolved. The ranking question was added starting with the fifth survey. The recommendations and information related to the topics under review were initially sent with the results, but progressively more information or links were sent within the survey. The discussion sent with the results was expanded and better referenced.

During the pandemic, there were 2 surveys over the first 9 months. A survey in May 2020 addressed both personal and clinical concerns. Personal concerns were shared anonymously with system leadership in real time. The survey also identified common practice concerns where information was limited. Members shared experience and identified resources which were passed along to all members. A survey in November 2020 identified all primary care clinicians were caring for post-acute COVID patients and a growing number of patients with new or worsened mental health problems. (Table 4). At the time information to address these issues was scare so the SC developed and shared related educational materials with members.

During the first 3.5 years of the collaborative over 50 clinicians and scientists participated in survey development, results discussion authorship, or presentations at academic venues including international meetings. Hundreds of emails have been exchanged across the country not only in the development and dissemination of the process, but by members reaching out with question, comments, and providing input.

Discussion

This is a description of the development and evolution of a learning collaborative of primary care clinicians in a single health system across disparate geographic regions. All participants were in the same health system, but practiced in a wide variety of communities, in a diversity of practices, with varying practice demands and resources. It was initially developed over concerns of practice variation in the adoption of USPSTF recommendations. It expanded to include a range of topics including disease specific guidelines and practice management. The original motivation for developing the learning collaborative was to identify broadly applicable topics thought important and allow members to collectively place them in a contextual framework relevant to their practices. It sought to function as a primary care learning collaborative and in doing so, develop a sense of community among primary care clinicians within the health system.²⁰

Selection of appropriate topics was successful overall based on member feedback and participation. Topic or survey ideas were submitted from a variety of sources. The SC provided most of the ideas initially, but as members understood the process, they contributed most of the topics for later surveys. When the number of requests exceeded the number of surveys, by merging topics or including 1 to 2 supplementary questions in a survey, most requests could be accommodated.

The collaborative was able to help members gage the relevance of the topic to their practice. This was done in 3 ways. First, the questions were behavior-based, that is, “What are you doing . . .?” providing members insight into their peers’ practices. Second, narrative responses provided rich, honest feedback, and deeper insight. Despite the design as a quick “point and click” survey process, this narrative response was robust and grew. Absolute participation varied across surveys, but frequently members spent the time and effort to write 60 to 100-word responses. Finally, the ranking questions generated member interest in the results while providing stratification of their peer’s perception.

Several trends were identified over the first 3.5 years of the collaborative as shown in Figure 1. When applying practice elements to public health crises such as the COVID pandemic or opioid epidemic, little variation was deemed appropriate. Applying USPSTF or system practice management principles, slightly more variation was viewed as acceptable. When implementing clinical guidelines greater variation and perhaps more customization was tolerated or even expected. The ranking question, though not a validated instrument, was a simple method to provide the SC and members with feedback about the relative tolerance for variation in care when adopting or implementing new or changing recommendations.

As a grassroots initiative, it was a “bottom-up” collaborative, run by members with infrastructure support from the DFM. Most of the SC and over 80% of the members were community-based, not academic-based clinicians. This ensured the collaborative would remain sensitive to members priorities and stay open to input from members.

The COVID pandemic disrupted all aspect of life, including medical education.^26-28 During the first several months of the COVID pandemic using the quick turnaround process certain patterns emerged. Most notably, the need to address those with post-acute COVID illness, a problem seen by most primary care clinicians in this group (Table 4). At the time there was a paucity of information for caring for those post-acute COVID, for example the NIH guidelines on the care of those with SARS-CoV-2 contained a single paragraph in a 250-page report. The network identified and shared information among themselves.²⁹ Likewise, the mental health crisis identified throughout 2021 by national organizations was apparent within collaborative practices in 2020.³⁰ Although the collaborative lacked the resources to address the clinical needs brought on by the crisis, could provide an awareness and validation of members’ experiences.

There were limitations to the generalizability of this project. It was a review of a collaborative within a single healthcare system. Although involvement of the SC and members was voluntary additional resource availability included access to an IRB, intranet survey system, library, and infrastructure support from an academic department. Membership of the collaborative was not representative of primary care clinicians nationally; members were more likely to be female, younger, and Caucasian-Non-Hispanic since most members were clinicians practicing in small cities or rural upper Midwest.³¹ The largest demographic, family physicians, (81, 41.5%), were more likely to practice obstetrics than their peers nationally (27% vs 18%).³¹ The instruments used in surveys themselves, the ranking questions, and the survey assessment were all unvalidated. Even so, the process was expected to be subjective and generate dialog. The collaborative was designed to share knowledge, experience, and collegiality, not to be objective research. Responses were based on self-reported practice behavior, a source of bias.^32,33

This project demonstrated primary care clinicians in a large health system will participate and find value in a primary care learning collaborative. Although neither the project nor this study had the resources to support practice change initiatives or review whether these activities created practice change, it provided early building blocks to increase clinician awareness.^34,35 Future growth and success will require the collaborative to facilitate not only information exchange, but practice change and the ability to measure that change without disrupting the current collegial process. While part of a single health system, the ability of the collaborative to reach across a diversity of practices suggests the principals applied here could be adopted among other primary care groups.^8,18,36 The collaborative achieved early success for many reasons, among them inclusivity, strong identification with community-based clinicians, and an intentionally diverse SC. While many academic departments struggle to engage physicians in their communities with scholarly activity, those associated with this collaborative were able to do so. This was through listening and supporting the members. It succeeded through the goodwill, time, and efforts of the SC members. But most importantly it was the members that made the collaborative successful. Collectively they generously gave hours of their time on each topic for the betterment of their colleagues. This project demonstrates that a group of primary care clinicians dedicated to their practices, colleagues, and especially their patients will work together for the benefit of those they serve.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rodney Erickson

Augustine Chavez

Ramona DeJesus

Richard Presutti

Steven Rosas

Erin Westfall

Terrance Witt

Thomas Thacher

References

US Preventive Services Task Force. Published recommendations. Published 2021. Accessed December 16, 2021. https://www.uspreventiveservicestaskforce.org/uspstf/topic_search_results?topic_status=P

Tello

Barbazza

Waddell

Review of 128 quality of care mechanisms: a framework and mapping for health system stewards. Health Policy. 2020;124(1):12-24.

Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century . National Academy Press; 2001.

Schifalacqua

Shepard

Kelley

Evidence-based practice: cost-benefit of large system implementation. Qual Manag Health Care. 2012;21(2):74-80.

Medleg

Choi

, et al. Integrating shared decision-making into primary care: lessons learned from a multi-centre feasibility randomized controlled trial. BMC Med Inform Decis Mak. 2021;21(1):323.

Dadashi

Golightly

Sharples

Seeing the woods for the trees: the problem of information inefficiency and information overload on operator performance. Cogn Technol Work. 2017;19(4):561-570.

Selman

Brighton

Robinson

, et al. Primary care physicians’ educational needs and learning preferences in end of life care: a focus group study in the UK. BMC Palliat Care. 2017;16(1):17.

Hale

Freed

Alston

Ricotta

DN.

What are we really talking about? An organizing framework for types of consultation and their implications for physician communication. Acad Med. 2019;94(6):809-812.

Bowman

Seehusen

Victoria Neale

Interventions must be realistic to be useful and completed in family medicine. J Am Board Fam Med. 2018;31(1):1-4.

10.

Sinsky

Bavafa

Roberts

Beasley

JW.

Standardization vs customization: finding the right balance. Ann Fam Med. 2021;19(2):171-177.

11.

Hoff

The challenges of consumerism for primary care physicians. Am J Manag Care. 2020;26(1):e1-e3.

12.

Sutherland

Levesque

JF.

Unwarranted clinical variation in health care: definitions and proposal of an analytic framework. J Eval Clin Pract. 2020;26(3):687-696.

13.

Davidson

Barry

Mangione

, et al. Screening for colorectal cancer: US Preventive Services Task Force Recommendation Statement. JAMA. 2021;325(19):1965-1977.

14.

Holt

Gerayli

Prostate cancer screening. Prim Care. 2019;46(2):257-263.

15.

Walters

Stern

Robertson-Malt

The measurement of collaboration within healthcare settings: a systematic review of measurement properties of instruments. JBI Database System Rev Implement Rep. 2016;14(4):138-197.

16.

Miller

McDaniel

Jr. Crabtree

Stange

KC.

Practice jazz: understanding variation in family practices using complexity science. J Fam Pract. 2001;50(10):872-878.

17.

Nouhi

Hadian

Jahangiri

Hakimzadeh

Gray

Olyaeemanesh

The economic consequences of practice style variation in providing medical interventions: a systematic review of the literature. J Educ Health Promot. 2019;8:119.

18.

Leroy

Rittner

Johnson

Gerteis

Miller

Facilitative components of collaborative learning: a review of nine health research networks. Health Policy. 2017;12(3):19-33.

19.

Haraldseid-Driftland

Aase

Wiig

Billett

Developing a collaborative learning framework for resilience in healthcare: a study protocol. BMJ Open. 2021;11(8):e045183.

20.

Miake-Lye

Mak

Lam

, et al. Scaling beyond early adopters: a content analysis of literature and key informant perspectives. J Gen Intern Med. 2021;36(2):383-395.

21.

Sun

VanderWeele

Tchetgen Tchetgen

EJ.

A multinomial regression approach to model outcome heterogeneity. Am J Epidemiol. 2017;186(9):1097-1103.

22.

Good

PI.

Resampling Methods. Springer; 2006.

23.

Banjanovic

Osborne

JW.

Confidence intervals for effect sizes: applying bootstrap resampling. Pract Assess Res Eval. 2016;21(1):5.

24.

Suebnukarn

Haddawy

A Bayesian approach to generating tutorial hints in a collaborative medical problem-based learning system. Artif Intell Med. 2006;38(1):5-24.

25.

Bernard

Laabs

Nagaraju

, et al. Clinician care team composition and health care utilization. Mayo Clin Proc Innov Qual Outcomes. 2021;5(2):338-346.

26.

Chen

Nagendran

Postgraduate medical education after COVID-19: reflections on decision-making and the state of UK medical Grand Rounds. Postgrad Med J. 2020;98(e2):e94-e95.

27.

Kitto

Meeting the challenges of COVID-19 at the close of 2020. J Contin Educ Health Prof. 2020;40(4 Suppl 4):215-216.

28.

Krist

DeVoe

Cheng

Ehrlich

Jones

SM.

Redesigning primary care to address the COVID-19 pandemic in the midst of the pandemic. Ann Fam Med. 2020;18(4):349-354.

29.

NIH. Coronavirus disease 2019 (COVID-19) treatment guidelines. Published 2021. Accessed December 17, 2021.

30.

General US. Protecting youth mental health. Published 2021. Accessed December 16, 2021.

31.

Petterson

McNellis

Klink

Meyers

Bazemore

The State of Primary Care in the United States: A Chartbook of Facts and Statistics. Robert Graham Center; 2018.

32.

Gagliardi

Brouwers

Finelli

Campbell

Marlow

Silver

IL.

Physician self-audit: a scoping review. J Contin Educ Health Prof. 2011;31(4):258-264.

33.

Peersman

Pasteels

Cambier

De Maeseneer

Willems

Validity of self-reported utilization of physician services: a population study. Eur J Public Health. 2014;24(1):91-97.

34.

Bowe

Armstrong

Assessment for systems learning: a holistic assessment framework to support decision making across the medical education continuum. Acad Med. 2017;92(5):585-592.

35.

Armson

Perrier

Roder

, et al. Assessing unperceived learning needs in continuing medical education for primary care physicians: a scoping review. J Contin Educ Health Prof. 2020;40(4):257-267.

36.

Arora

Mate