CAR Peer Learning Guide

Abstract

Peer learning is a quality initiative used to identify potential areas of practice improvement, both on a patient level and on a systemic level. Opportunities for peer learning include review of prior imaging studies, evaluation of cases from multidisciplinary case conferences, and review of radiology trainees’ call cases. Peer learning is non-punitive and focuses on promoting life-long learning. It seeks to identify and disseminate learning opportunities and areas for systems improvement compared to traditional peer review. Learning opportunities arise from peer learning through both individual communication of cases reviewed for routine work, as well as through anonymous presentation of aggregate cases in an educational format. In conjunction with other tools such as root cause analysis, peer learning can be used to guide future practice improvement opportunities. This guide provides definitions of terms and a synthetic evidence review regarding peer review and peer learning, as well as medicolegal and jurisdictional considerations. Important aspects of what makes an effective peer learning program and best practices for implementing such a program are presented. The guide is intended to be a living document that will be updated regularly as new data emerges and peer learning continues to evolve in radiology practices.

Introduction

The Canadian Association of Radiologists Working Group on Peer Learning was assembled to review the literature on peer review and peer learning and provide an up-to-date report for the radiology community. This guide provides definitions of terms and a synthetic evidence review regarding peer review and peer learning. It presents important aspects of what makes an effective peer learning program, as well as best practices for implementing such a program. This guide is intended to be a living document that will be updated regularly as new data emerges and peer learning continues to evolve in radiology practices.

Professional peer review is the evaluation of a colleague’s work to ensure that current standards of care are met. ¹ The 2011 CAR Guide to Peer Review Systems ² defined peer review as “a generic term for a process of self-regulation by a profession or a process of evaluation involving qualified individuals within the relevant field. Peer review methods are employed to maintain standards, improve performance and provide credibility.” Peer review was introduced to radiology 20 years ago, and the dominant model used for peer review was score-based, such as the American College of Radiology (ACR) RADPEER system.^3,4 However, the traditional model of peer review has been criticized for several reasons, as outlined in the following section.

Drawbacks of Traditional Peer Review

In response to the 1999 Institutes of Medicine report “To err is human,” the American College of Radiology developed RADPEER, a score-based peer review system aimed at reducing diagnostic error. This system is based on a “fair” evaluation of a radiologist’s performance by a peer in order to identify opportunities for self-improvement, additional education, and error reduction. ⁵ However, multiple studies have identified problems with traditional score-based peer review, including: its perceived punitive and adversarial nature; tendency to subjectivity and bias; perceived waste of time; negative impact on organizational culture; poor compliance among radiologists; and lack of efficacy to improve radiologist performance.^3,4,6,7 A summary of these issues is provided in a report from the American College of Radiology 2020 Peer Learning Summit. ⁴

One survey-based study found that only 246/1240 (19.8%) of radiologists felt their practice patterns changed with peer review, and only 186/1405 (13.2%) performed peer review for reasons other than as a hospital or accreditation requirement. ⁸ A survey of one academic institution’s radiologists found that only 30% felt peer review improved performance of themselves or others, 44% felt that it was a waste of time, and 58% felt it was only done to meet hospital/regulatory requirements. ⁹ Randomly selected cases are still prone to bias, for example, such an error rate may not account for differences in disease prevalence, case and modality complexity, history provided, systematic and organizational factors, and subjectivity, inherent biases and interobserver variability amongst reviewers.^7,10,11 One study found that radiologists were more likely to submit a discrepant peer review report within a semi-monthly block of time after receiving one, suggesting another source of bias. ¹²

An important drawback to peer review of randomly selected cases is that they rarely reveal clinically significant discrepancies, and therefore have low yield for learning. In a study that reviewed over 42 000 reports over 42 months, the potentially meaningful discrepancy rate was only 0.5%, and usage of the peer review system declined by 56% between the first and last quarters of the study period. ⁴ In another study, of 1,690 randomly selected cases, 1,646 (97.4%) were scored as 1 (no discrepancy) and the remaining 44 (2.6%) were scored as 2 (minor discrepancy); no randomly selected case was scored as 3 (significant discrepancy) or 4 (major discrepancy), suggesting a significant error rate of 0%. ¹³ In contrast, 190 non-random cases were scored from 1 to 4, respectively, as follows: 0, 60 (31.6%), 94 (49.5%) and 36 (18.9%). A number of teaching points and practice quality improvement measures were developed based on the non-random case review, and the authors concluded that non-random peer review has a higher yield for learning. ¹³

It is important to contextualize reports on discrepancy rates or interpretive errors within radiology.^14,15 This is subject to the definition of what a true discrepancy is in the first place as differences in imaging interpretation do not necessarily indicate a mistake; objective reference standards such as pathological correlation are often unavailable in many instances. ¹⁶ Furthermore, errors or discrepancies in interpretation do not equate to negligence as there are several steps where errors can occur involved with imaging examination request, examination protocol and triage, image acquisition and final interpretation. ¹⁴

Transitioning From Peer Review to Peer Learning

Following its introduction in 2002, the ACR implemented several modifications in the RADPEER program to transition to a model where collaborative peer learning would be achievable. ¹⁷ Peer learning is a newer paradigm focused not on evaluating competence, but rather on providing peer feedback and learning opportunities with a goal of future improvement. ³ This model is based on a 2015 report by the Institute of Medicine, ¹⁸ which outlines specific goals for improving diagnosis and reducing diagnostic error, as well as cultural values for continuous learning in healthcare systems. The primary purpose of peer learning is improvement; this applies not only to individual physicians, but also to physician groups, systems and processes. ³ Under the peer learning paradigm, scoring peer review cases is no longer recommended. ³ The peer learning process has also been coined peer collaborative improvement (PCI) by Donnelly et al. ¹⁹ Studies that have evaluated the effect of transitioning from traditional peer review to peer learning have shown positive results, including increased radiologist engagement, more opportunities for learning, and positive feedback.^19–21

Collaborative learning is a term used in the realm of peer learning. It is a general term used to describe educational approaches involving joint efforts by groups of two or more individuals. This type of learning has been shown in quantitative educational studies to improve individual student achievement, as well as increase positive interpersonal relationships, and overall well-being. ²² Peer coaching is a newer term coined by Parker et al. ²³ This process is grounded in attributes of mutual growth, and development of both parties in a way that is devoid of power dynamics.

Ultimately, the goal of a peer learning program is to improve patient care and outcomes as well as to mitigate future errors and discrepancies. Over time, efforts to support peer learning are likely to provoke sustainable practice change with positive downstream effects for radiologists, imaging departments, and patient care.

Characteristics of Successful Peer Learning Programs

Overarching Goals

Facilitate a Just Culture

A necessary component for peer learning is a just culture, which is a philosophical framework for responding to errors in healthcare. Just culture entails a learning-focused culture that constantly seeks to improve and is oriented towards patient safety as well as the psychological safety of staff. ²⁴ A just culture holds individuals accountable to organizational standards, but accepts that even the most competent professionals make errors, and that imperfect organizational systems create error-prone conditions. ²⁵ A just culture requires understanding and support from institutional leaders, coinciding with alignment of institutional goals and policies with the improved culture. Consistent application of the just culture to quality improvement processes necessitates ongoing leadership and community engagement in the process.

Peer learning seeks to avoid the deleterious and adversarial effects of traditional peer review on organizational culture by removing the punitive aspects of peer review and focusing solely on identifying and learning from error. ²⁶ Peer learning incorporates just culture by acknowledging the inherent fallibility of human nature but seeking to mitigate this through a focus on education, system improvement, and commitment to shared learning. A successful program shifts the focus from the insular “me” to a collaborative “we,” resulting in increased satisfaction with program goals and educational value for participating radiologists. ²¹

Key elements include identification of behavioral components of failure which include human error, at-risk behavior, or reckless behavior, and the more common “systemic” elements of failure. While the focus is on risk reduction and learning, accountability of decisions is achieved through the evaluation of decisions for their consistency and compliance with patient safety and organizational values.

Identify Opportunities for Continued Learning

To ensure sustainability of the peer learning program, it is necessary to build on initial efforts at encouraging stakeholder buy-in, and to continually seek opportunities for learning. Multidisciplinary case conferences (MDC) create opportunities for radiologists to learn what is important to other healthcare practitioners when they request imaging, and to demonstrate strengths and weaknesses of various imaging modalities to address the clinical question at hand, thereby creating learning opportunities for requesting physicians. These case conferences have shown discrepancy rates to be in the range of one third of cases. ²⁷ This is thought to be due to the complexity of the patient issues as well as the direct input from clinicians, surgeons, and pathologists about important details of the patient’s clinical status and management. The radiologists who attend MDC should be encouraged to provide direct feedback in the peer learning system to the reporting radiologists, and to document the cases for group learning as well. The benefit of including these cases into peer learning is that the imaging is usually contemporaneous, facilitating a closed loop focused on patient outcomes and improvement.

Peer learning can also make use of on-call cases from residents and fellows, since learning can occur in near-real time for the trainees providing the preliminary report as well as for a larger audience (in an anonymized fashion). This can also be performed in such a way that it creates another closed loop system allowing communication with the learning from the radiologist as well as communicating important findings and potential discrepancies with requesting physicians.

An easily accessible opportunity for peer learning is during review of prior imaging examinations used for comparison. Although the ability to contemporaneously alter patient care is diminished by the time interval between studies, and there is a risk of hindsight bias on retrospective review, ²⁸ many studies have found that non-random case reviews yield far more learning than randomly selected ones. This type of peer learning can be implemented in any setting and could also facilitate the creation of a system whereby radiologists who prospectively make a diagnosis would have the option of receiving follow-up on cases where surgical or endoscopic intervention is likely. A similar system can be created for technologists with feedback from radiologists as well as other technologists performing follow up imaging examinations.

Other peer learning opportunities include direct feedback about reports from clinicians and surgeons, direct feedback from patients (including through a patient relations department or other organizational feedback mechanisms), as well as regular review of pathologic-surgical discrepancy reports.

Foster Group Learning and Education

The designated peer learning education coordinator or team should choose cases that have the most potential to impact shared learning, improve practice, and patient care. When presenting cases, the format can vary but general principles apply. Cases need to be anonymized, adequate clinical information should be provided, and relevant images included. Some departments may submit cases for subspecialty peer learning rounds^24,29 while others present cases in a more general format^20,25 either out of necessity (structure and type of practice) or priority (review of learning points from a variety of subspecialty cases). The timing and frequency of peer learning rounds will depend on each department’s clinical demands and structure but should be regularly scheduled. Rounds may be during the day, after hours, or recorded for convenience.

Create Lasting Improvements to Patient Care

Linking peer learning with quality improvement strategies is necessary to improve long term patient care. Root cause analysis of specific peer learning instances as well as aggregation of data regarding commonly encountered peer learning lessons can lead to long term change at many levels in the imaging department. This can include optimizing patient scheduling and patient flow, updating imaging protocols, department-wide use of general standard template reporting or specific synoptic reporting templates (for a particular disease entity), as well as encouraging standard terminology use among radiologists, and avoiding acronyms.

Concrete Steps to Create a Peer Learning Program

Conduct a Readiness Assessment

Before implementing a peer learning program, a department should conduct a readiness assessment. This may be as simple as outlining the structure of peer learning case submission and presentation ²⁶ to an in-depth assessment of departmental acceptance and readiness for a peer learning program. Further considerations include whether the program will replace a peer review program, or if the two programs will operate in parallel within the department, in addition to administrative support and regulatory requirements in a specific jurisdiction. ²⁴ The “readiness” of any department can be enhanced by a peer learning champion who can help navigate any potential setbacks or obstacles. ³⁰ To navigate these challenges, a program champion must be empowered to make changes. For a series of readiness assessment questions for imaging departments or clinics looking to implement peer learning programs, please see Supplementary Appendix A: Readiness Assessment and Implementation Considerations.

Identify and Equip Leaders

Empowering a local champion to lead the transition to peer learning can have an enormously positive impact. This individual must have the public support of departmental leadership and appropriate administrative and infrastructure resources to navigate the uncertainty and resistance that comes with change. To minimize the impact of this change, programs should engage with radiologists and administrators in their department prior to implementing a peer learning program. Priorities should include highlighting the value of peer learning, educating radiologists about high-value case submission, ensuring regulatory compliance and legal protection, and striving to provide Continuing Medical Education (CME) credits for participation in submissions and meetings.^24,29 High-value case submission can be achieved by educating radiologists about different types of errors which, when corrected, are most likely to result in quality improvement in the department. These include such considerations as errors of perception, errors of interpretation, or errors of communication as well as near misses and good calls. ²⁹ Using high-quality cases will ensure a high level of satisfaction amongst radiologists and ultimately an improvement in departmental accuracy, while making the program itself more valuable.

Consider Software and Workflow

Another key component of successful peer learning programs is an appropriate departmental workflow. Case submissions should be as seamless as possible. Some programs have achieved this through integration of case submission for peer learning into existing peer review software^6,21 while others have developed proprietary software exclusively for peer learning cases. ²³ Smaller groups have left case collection up to individual radiologists. ²⁶ Each method has pros and cons, but the underlying principle is constant: invoke principles of crowdsourcing to encourage relevant case submission and foster buy-in from radiologists in the department. By engaging radiologists to submit cases of interest, the peer learning coordinator can encourage participation and enhance the relevance of the peer learning process to local practice conditions. Software packages should also ensure adequate anonymization, avoiding major concerns raised in traditional peer review,^1,5,17,19 including discomfort associated with disagreeing with a colleague, especially a superior or more experienced radiologist. In addition, cloud-based solutions may allow smaller community-based practices and independent health facilities with a limited number of radiologists in establishing a peer learning program as well as in further developing the concept of peer coaching.

Implement Program Assessment Tools

Most departments that have implemented peer learning have established some sort of feedback mechanism to assess radiologist satisfaction with the program.^21,24,26,31 While some participants initially doubted the degree of difference between peer learning and peer review, follow up questionnaires have consistently revealed perceptions of positive collaboration and improvement with peer learning rather than the punitive or retributive perceptions with peer review. This highlights the value of peer learning in and of itself. In addition, continuous feedback is required to address issues and ensure that the peer learning experience is positive and constructive.

The process of setting up a peer learning program is as unique and specific as each radiology department. Focusing on the underlying principles of peer learning as well as established principles of change management and just culture can ensure a successful transition from peer review to a more collaborative, educational, and collegial process of quality assurance and improvement.

When establishing a peer learning program, a frequent consideration is what the minimum number of learning opportunities or peer learning group sessions a department should conduct. While there is no established answer for those questions, the most important step is to initiate a peer learning process. Recently, the American College of Radiology established minimum requirements for peer learning through the Physician Quality Assurance Pathway for Accreditation, ³² in which annual documentation of peer learning accomplishments should be documented. ³³ Each institution is required to define its own expectations for minimum radiologist participation in peer learning submissions and in learning activity participation. Similarly, the Royal College of Physicians and Surgeons in Canada has initiated changes to make its Maintenance of Certification program more “improvement-focused to enable physicians to achieve care outcomes and/or continued personal-professional growth important to them and their patients.” ³⁴

Pearls to Engage Stakeholders in Peer Learning

To promote positive and sustainable change, multiple factors need to be considered for a peer learning program. Buy-in and commitment is needed at all levels: from administrators, legislators (where relevant), radiologists, radiology trainees, technologists, and ideally from practitioners receiving radiology reports, given that future changes to report structure and communication of learning may result from the peer learning program. Involving end-users in quality improvement initiatives via awareness of the peer learning program can enhance the value of radiology. ³⁵

1). For departmental and hospital leadership: outline the principles of the program, establish robust communication workflows, and focus the program on learning opportunities rather than punitive measures. Engage with hospital leadership to make sure that they understand the unique working environment of radiologists, complexity of modern healthcare, and the inherent nature of error when working in complex systems. Medicolegal concerns should be addressed early by engaging relevant local legal representatives.

2). For peer learning leadership: designate a peer learning leader with secured administrative time, to ensure consistent management of the case submission process, feedback to individual radiologists, preparation of peer learning conferences and process improvement. Support from a peer learning committee may be important depending on the size of the department.

3). For radiologists/learners: strive for user friendliness by creating a system with the fewest possible “clicks” required to input or retrieve the information. An ideal system seamlessly integrates into existing workflows and includes intuitive forms which are aesthetically pleasing.

4). For radiologists and technologists: offer opportunities for CME credit for participation. This can be applied for through various accreditation bodies and can be used to garner initial participation as well as an incentive for continued participation. The option of having radiologists prospectively flag cases for follow up in situations where the reporting radiologist is unsure of the diagnosis and the patient is likely to have surgical/pathology follow up would be ideal for a peer learning system.

5). For administrators: aim for cost-effectiveness and demonstration of how closed loop systems can improve patient outcomes. For example, systems which automatically alert the original interpreting radiologist of a potential learning opportunity based on additional clinical information, including laboratory values or pathology reports, as observed by Trinh et al. ⁴ in their discussion of rooting their peer learning tool (PLT) into their existing critical alert notification system. This allowed a closed loop two-way communication between the radiologist interpreting the current study and the radiologist who interpreted the prior study, as well as facilitating time-sensitive communication with the most responsible physician. There are other potential ways of creating closed loops which may be advantageous in various work settings, and these should be tailored to the environment where peer learning is being implemented.

6). For IT support staff: integrate the program into the existing workflow, without additional webpages, logins, or hardware. Some institutions have developed a PLT embedded into their radiology information system to aid in the ease of use and decreased time. ³⁶ Integrated solutions are necessary to send automated reminders for case contribution to radiologists, the ability to organize and collate data and to create anonymized educational material for both group and personal learning.

Provincial and Jurisdictional Experiences in Canada

The development, implementation, and deployment of radiology peer learning programs can vary widely. There is a paucity of published literature and official documentation about the existence of peer review and peer learning programs in Canada. In preparing this guide, the working group conducted informal surveys with radiologists practicing in various jurisdictions and practice environments across Canada.

British Columbia has a provincial peer review system which is implemented within all health authorities except for Fraser Health Authority and Vancouver Coastal Health Authority; additionally, peer learning rounds occur in many radiology groups within the province. In Alberta, the Diagnostic Imaging Quality Assurance Program supports a peer learning initiative. In Ontario, peer review is mandated by the College of Physicians and Surgeons of Ontario for Independent Health Facilities (IHF). The individual IHF is allowed to implement their own program and ensure that all radiologists participate. In practice, the program is more akin to a peer learning approach: the results of the program are not shared externally, and the goal is to implement positive change through learning. In other provinces, radiology departments and hospital systems have implemented peer learning in different ways, or not at all. Educational rounds in the spirit of peer learning happen in many radiology groups, even in the absence of provincial or regional mandates.

The experiences of radiologists attempting to institute peer review and peer learning programs demonstrate that the programs are more effective and lasting when radiology champions lead the efforts and participate in governance. The more successful examples of peer learning programs emphasize collaborative learning, rather than the implementation of audit-based programs that may be perceived as punitive instruments at the intra- or interdepartmental level. Avoidance of such behavior supports the credibility of the program and the governing structure. Peer learning is a fluid process which should engage debate as knowledge expands. At the regional or local level, strong internal peer learning programs can supplement or replace quality assurance initiatives.

Historically, there has been a tension between provincial health quality assurance achieved via routine audits and physician-oriented efforts to implement true peer learning initiatives. Budgetary questions about who will bear the cost of implementing and running formal programs are common to many efforts to institute peer review or peer learning at the radiology group, hospital, health authority, or provincial level. Anecdotal evidence suggests that a lack of buy-in and engagement from local radiologists can be mitigated by engaging colleagues in the process of developing and implementing the peer learning program and process from the very beginning. In any publicly operated facilities, radiology leaders can be expected to engage with local, regional and/or provincial quality improvement authorities to enact and operationalize the peer learning program.

Medicolegal and Professional Considerations

Numerous studies conducted in the United States that assessed radiologists’ perspectives on peer review have found that radiologists are concerned about a perceived lack of anonymity and vulnerability for medicolegal action. In one survey, the most common concern raised by 833/1273 respondents (65.4%) was the discoverability of the data in legal proceedings. ⁸ In another survey, 26/36 (72.2%) of radiologists were concerned that peer review data is not sheltered from malpractice, and 33/36 (91.6%) responded that peer review should be anonymous. ³⁶

Although Canadian healthcare is generally less litigious than its American counterpart, Canadian radiologists have raised similar concerns of medicolegal risk. It is important to state explicitly that this document is related to the issues regarding how to establish and sustain a peer learning environment and program. This document does not deal with setting appropriate standards of reporting. To provide more information regarding medicolegal risk under the newer peer learning paradigm, members of the working group sought advice from the Canadian Medical Protection Agency (CMPA). CMPA does not review or endorse societal guidelines and declined to specifically comment on this guideline or provide medicolegal advice regarding peer learning. CMPA directed interested readers to their 2009 ³⁷ peer review document available online. Canadian radiologists are advised to ensure that any peer learning program complies with regional administrative and medicolegal requirements.

Future Work

Transitioning from peer review to peer learning advances our field and highlights radiology as a specialty that is willing to embrace change and to seek continuous improvement. ³⁶ This report aims to guide this transition. As a living document, future updates will incorporate new knowledge and will help to expand peer-learning in radiology practices across Canada.

An important concern raised by CAR members in response to this document is how to incorporate peer learning in smaller centers, which may lack infrastructure and sufficient members to implement a successful program. One solution to help both small and larger practices would be to create networks of multiple hospitals, or even a national, subspeciality-based peer coaching platform. This network could help foster collaborative relationships and establish strong peer learning opportunities. A promising avenue to support such a network is through CAR affiliate subspecialty radiology groups (thoracic, pediatric, emergency medicine, abdominal imaging), which could potentially provide subspecialty peer coaching as well as sharing of peer learning lessons in an anonymized fashion.

For more information about the CAR’s ongoing work on peer learning, please visit https://car.ca/education/peer-learning/

Supplemental Material

sj-pdf-1-caj-10.1177_08465371211065454 – Supplemental Material for CAR Peer Learning Guide

Supplemental Material, sj-pdf-1-caj-10.1177_08465371211065454 for CAR Peer Learning Guide by Felipe Soares Torres, Andreu F Costa, Yoan Kagoma, Martin Arrigan, Malcolm Scott, Brian Yemen, Casey Hurrell and Ania Kielar in Canadian Association of Radiologists Journal