Abstract
Purpose:
All postgraduate residency programs in Canada are transitioning to a competency-based medical education (CBME) model divided into 4 stages of training. Queen’s University has been the first Canadian institution to mandate transitioning to CBME across all residency programs, including Diagnostic Radiology. This study describes the implementation of CBME with a focus on the third developmental stage, Core of Discipline, in the Diagnostic Radiology residency program at Queen’s University. We describe strategies applied and challenges encountered during the adoption and implementation process in order to inform the development of other CBME residency programs in Diagnostic Radiology.
Methods:
At Queen’s University, the Core of Discipline stage was developed using the Royal College of Physicians and Surgeons of Canada’s (RCPSC) competence continuum guidelines and the CanMEDS framework to create radiology-specific entrustable professional activities (EPAs) and milestones for assessment. New committees, administrative positions, and assessment strategies were created to develop these assessment guidelines. Currently, 2 cohorts of residents (n = 6) are enrolled in the Core of Discipline stage.
Results:
EPAs, milestones, and methods of evaluation for the Core of Discipline stage are described. Opportunities during implementation included tracking progress toward educational objectives and increased mentorship. Challenges included difficulty meeting procedural volume requirements, inconsistent procedural tracking, improving feedback mechanisms, and administrative burden.
Conclusion:
The transition to a competency-based curriculum in an academic Diagnostic Radiology residency program is significantly resource and time intensive. This report describes challenges faced in developing the Core of Discipline stage and potential solutions to facilitate this process.
Introduction
The traditional model for educating and evaluating resident physicians is being replaced by competency-based medical education (CBME) across Canada and globally. 1 This innovative approach involves an outcome-based assessment system, in which educational targets are explicitly defined and upon which promotion is based, shifting from the prior time-based system which presumed that competence was gained in a fixed period. 2,3 With CBME, assessment is tailored toward observable abilities. Education is based on defined competencies relevant to the resident’s particular stage of training. 4,5 In this model, teaching is customized to the stage of training and gaps in training are more readily detected. 6 Additionally, this approach provides increased transparency for faculty and residents. 7 The aim of this model is to improve the development of skills and knowledge required by physicians to meet increasing patient and societal healthcare demands. 8
In 2014, The Royal College of Physicians and Surgeons of Canada (RCPSC) initiated the transition to CBME by developing a postgraduate medical education (PGME) framework known as Competence by Design (CBD) for implementation across Canada. 9 The RCPSC also granted Queen’s University permission to begin transitioning its residency programs to CBME as part of the Fundamental Innovations in Residency Education (FIRE) proposal in 2015. As of July 2017, new trainees in all 29 specialty programs at Queen’s University entered into the CBME model.
Queen’s University’s CBME curriculum model contains 4 stages of training 1 : Transition to Discipline, 2 Foundations of Discipline, 3 Core of Discipline, and 4 Transition to Practice. Within each stage, discipline-specific entrustable professional activities (EPAs) and milestones provide the foundation for resident assessment. 10 -12
EPAs are major tasks performed by a physician in their discipline which can be entrusted to residents upon sufficient standardized demonstration of competency. 4 EPAs consist of milestones which are observable objectives for the resident to achieve successfully upon assessment. EPAs and milestones are associated with corresponding CanMEDS roles and provide the basis for resident assessment. They were developed within the Queen’s University Diagnostic Radiology residency program in consultation with faculty.
The purpose of this paper is to describe the Queen’s University Diagnostic Radiology Residency Program’s development and implementation of the EPAs and milestones associated with the Core of Discipline stage, and associated challenges. The aim is to inform the development of other CBME Diagnostic Radiology residency programs at the national level. This paper is Part 2 in the series on Diagnostic Radiology CBME implementation at Queen’s University. 13
Methods
The method for implementing CBME in the Diagnostic Radiology Residency program at Queen’s University includes a restructuring approach that focuses on 4 specific areas:
1
program lead and administrative support,
2
the creation of a Competence Committee,
3
an Academic Advisor system, and
4
the faculty role in resident assessments. 1) Program Lead and Administrative support
A CBME Program Lead position with protected administrative time (20% of total full time hours) was created and designated by the Department Head, and is separate from the role of Program Director. The Program Lead role is involved in developing and implementing the CBME curriculum including EPAs and milestones, acting as chair of the Competence Committee, and creating assessments. An educational consultant with expertise in curriculum development was recruited to provide administrative support, facilitate staff and resident training, assist in curricular development, analyze and compile evaluations, and coordinate meetings. These 2 positions were funded by the department. 2) Competence Committee
To monitor resident progress in satisfying educational requirements and EPA completion, a Competence Committee was established as a sub-committee of the Residency Program Committee (RPC). Membership is determined by the RPC, with guidance from the Program Director. Members serve 2 year appointments, with the possibility of renewable terms. There are no specific criteria for selection, but senior faculty with an interest in resident education are ideal candidates. Membership is composed of 3 faculty members, the Program Director, and the CBME Program Lead. This committee meets once at the end of each stage to discuss resident progress and make decisions about resident promotion and remediation as necessary. Special meetings can be issued if there is a particular issue to discuss such as remediation. 3) Academic Advisors
Each resident is assigned an Academic Advisor (AA) for the duration of residency. Faculty members serve in this role and do not receive protected administrative time. There are no specific selection criteria. Currently, AAs have only one assigned resident. AAs meet with residents quarterly, follow a template for reviewing their advisee’s progress, and for identifying strengths and weaknesses. AAs identify concerns, discuss these with the resident, and summarize the resident’s progress on EPAs prior to a Competence Committee meeting. If the Competence Committee identifies concerns regarding a resident’s progress, these are communicated to the AA who meets with the resident to create a plan to address any gaps. AAs also serve an important mentorship role, available for residents to approach with any concerns or advice pertaining to residency, future career options, or other issues as they arise. 4) Faculty Role in Resident Assessments
In transitioning to CBME, regular workplace-based assessments on the residents’ abilities by faculty assessors correlating to EPAs were implemented. Upon initiation by the resident, the assessor is emailed a unique link to the assessment, the results of which can be seen by the resident upon completion. Assessors escalate concerns to the CBME Program Lead and Program Director but have no further role in relation to the residents’ overall learning plan.
Results
In this report, we summarize the third stage, Core of Discipline, the associated EPAs, milestones, and assessment strategies. This stage spans 38 blocks and consists of 4 EPAs that a resident is required to attain (Table 1). EPAs for this stage build upon EPAs from the 2 prior stages and are “nested” by design—they progress from stage to stage. Each EPA is comprised of several underlying milestones which provide detailed objectives to be achieved within the larger EPA. EPAs and milestones were developed and implemented by the CBME Program Lead in combination with faculty input and RCPSC guidelines.
Summary of Core of Discipline EPAs and Assessment Strategies.
Abbreviation: EPAs, entrustable professional activities.
Volumetric parameters for case reporting, procedures, and assessment frequency were determined by consultation with diagnostic radiology faculty members from each subspecialty within the department, as there is a lack of volumetric parameters in the literature. Currently, assessment of case volume is manually curated. Assessments associated with EPAs and milestones are completed through an online tool created by the local Elentra™ group at Queen’s University. Elentra™ is an integrated teaching and learning platform that allows residents, faculty, and administrators to access scheduling, learning materials, and assessments.
Stage 3: Core of Discipline EPAs
The third stage, Core of Discipline, consists of 4 EPAs: C1: Code requisitions for various modalities and body systems and procedures C2: Generate complete radiology reports C3: Perform basic and advanced adult and pediatric procedures C4: Disseminate QA/research project results
C1: Code Requisitions for Various Modalities and Body Systems and Procedures
Description
This EPA focuses on protocoling requisitions, including for CT (Computed Tomography), MRI (Magnetic Resonance Imaging) and GI/GU (Gastrointestinal/Genitourinary) fluoroscopic procedures. The resident will demonstrate the ability to appropriately triage patients, protocol the use of various contrast agents and their delivery, and in cases of decreased renal function or allergic reaction, the resident should show knowledge of appropriate hydration and premedication protocols.
Entrustment
The resident is entrusted to protocol requisitions for various modalities and body systems and procedures, with the appropriate priority code and contrast agent usage (if indicated).
Assessment
Review of protocoled requisitions by staff
Minimum of one Supervisor form (online assessment completed by faculty supervisor) per month
Field notes (narrative comments linked to EPA) as appropriate
C2: Generate Complete Radiology Reports
Description
This EPA focuses on reporting radiology studies of various modalities, both during the day and on-call. The resident is expected to recognize complex anatomic and physiologic findings on fluoroscopy, radiology, ultrasound, CT, MRI and NM studies as well as on more advanced studies (e.g., obstetrical ultrasound, pediatric radiology).
Entrustment
The resident is entrusted to detect, describe, and interpret all major, minor, and uncommon imaging findings and document this in an accurate radiology report which requires only minor editing by staff.
Case Mix
Approximate case volumes are provided to guide the resident during each block (Table 2).
Sample Volumetric Guidelines for Case Reporting During Final Block (C2: Generate Complete Radiology Reports).
Abbreviations: CT, computed tomography; MRI, magnetic resonance imaging; GI, gastrointestinal; GU, genitourinary system; MSK, musculoskeletal; NM, nuclear medicine.
Assessment
Weekly staff evaluation for on-service rotations
Evaluation of weekend call by a staff radiologist a minimum of once every 2 months
Field notes as appropriate
C3: Perform Basic and Advanced Adult and Pediatric Procedures
Description
This EPA focuses on the performance of various invasive and non-invasive radiologic procedures in both the adult and pediatric populations. The resident will obtain informed consent, explaining to the patient the risks, rationale, alternatives, and verifying patient understanding. The resident will optimize the environment to ensure patient comfort, safety, and privacy. The resident will demonstrate understanding of the relevant anatomy, pathologic findings, sterile technique, use of local anesthesia when appropriate, and use of various imaging modalities. If a complication arises, the resident will identify this and respond appropriately (e.g., Advanced Cardiovascular Life Support/Basic Cardiac Life Support), seeking help from staff as needed. Post-procedure, the resident will complete orders for patient care, communicate findings to the patient, and document in a report.
Entrustment
The resident is entrusted to perform adult and pediatric invasive and non-invasive procedures involving a variety of organ systems which may require ultrasound, radiographic, and fluoroscopic guidance.
Case Mix
Suggested cases and frequency of assessments are provided to guide the resident (Table 3).
Sample Invasive and Non-Invasive Radiological Procedures (C3: Perform Basic and Advanced Adult and Pediatric Procedures).
Abbreviations: PICC, peripherally inserted central catheter; CT, computed tomography; GI, gastrointestinal; GU, genitourinary system.
Assessment
Direct and indirect staff supervision during procedures
Weekly staff evaluation
Field notes as appropriate
Multisource feedback form as needed
C4: Disseminate Quality Assurance (QA)/research Project Results
Description
This EPA focuses on the resident’s ability to conduct a research or QA project pertaining to diagnostic/interventional radiology. The resident will complete data collection, data analysis, and write-up of the results.
Entrustment
The resident is entrusted to complete their QA/research project which will be either presented at research day or submitted for publication in a peer-reviewed journal.
Assessment
Documented assessment at least once per stage
Project supervision by staff radiologist
Field notes as appropriate and narrative account of resident progress
Milestones
For each EPA, milestones were developed to provide detailed objectives for the resident to accomplish (Tables 4 –7).
EPA and Corresponding Milestones (C1: Code Requisitions for Various Modalities and Body Systems and Procedures).
Abbreviation: EPAs, entrustable professional activities.
EPA and Corresponding Milestones (C2: Generate Complete Radiology Reports).
Abbreviations: EPAs, entrustable professional activities; CT, computed tomography; MRI, magnetic resonance imaging; NM, nuclear medicine; OB, obstetrics.
EPA and Corresponding Milestones (C3: Perform Basic and Advanced Adult and Pediatric Procedures).
Abbreviations: EPAs, entrustable professional activities; ACLS, Advanced Cardiovascular Life Support; BCLS, Basic Cardiovascular Life Support.
EPA and Corresponding Milestones (C4: Disseminate QA/Research Project Results).
Abbreviation: EPAs, entrustable professional activities.
Discussion
Currently, 2 cohorts of residents (n = 6) have progressed to the Core of Discipline stage of the Queen’s University CBME stream within the Diagnostic Radiology Residency Program. This is the longest stage of our curriculum (38 blocks, where each block is 4 weeks in duration) where all major competencies are met. Thus, it is crucial to describe issues as they arise within this stage and propose potential solutions. Notably, most significant issues that arose were either global or specific to the C2 and C3 EPAs.
Procedural Volume Requirements
Issues arose pertaining to residents meeting procedural volume requirements. Specifically, fluoroscopic guided lumbar punctures are scheduled for outpatients every Wednesday. This did not provide adequate opportunities for the resident on the neuroradiology service to meet volumetric requirements. In addition, the resident was often away due to vacation or post-call time off.
To overcome this barrier, a department-wide policy was implemented to allow residents to participate in off-rotation procedures as needed. Specifically, residents not on the neuroradiology rotation were allowed to perform lumbar punctures if the resident on the neuroradiology rotation was away. This afforded additional opportunities to residents who could not meet volume requirements on this rotation in cases where the on-service resident had time off. Notably, there has been no issues encountered with respect to residents needing to compete for these opportunities, possibly due to the small size of our residency program. If such issues arise, a more standardized system to allocate these opportunities may be beneficial.
Procedural Tracking
There were also issues that surfaced pertaining to procedural tracking. For example, there were instances in which staff preference in evaluating residents on each individual procedure increased the difficulty of tracking procedural volumes. Specifically, interventional radiology staff preferred to evaluate residents for each individual biopsy instead of per batch of biopsies. Completing an EPA for each individual biopsy is not feasible due to the length of the assessment, making it difficult to accurately track procedural volumes.
To solve this, a procedural log was implemented to indicate the number of procedures that a resident performed, while assessments continued to be completed by staff based on cumulative performance over a batch of procedures. This procedural log is based on an honor system and is tracked, completed, and electronically submitted by the resident for review.
Administrative Burden
Increased administrative burden is a known concern in the transition to CBME. 14,15 Under the CBME model, there is a far greater amount of data from assessments which must be compiled and organized. 16,17 This is a laborious task and not feasible for the staff radiologist to perform themselves. Aggregating data that can be filtered to provide more context for the resident’s performance for immediate and longitudinal time periods is achieved with our e-portfolio software, Elentra™. Features in Elentra™ are continuously rolled out based on priority requests to automate analysis of the data.
The administrative burden has been addressed by our residency program via the hiring of an educational consultant whose role includes compiling this aggregate data for the residents and relaying it their AAs for interpretation. This is an efficient and cost-effective solution to ensure timely analysis of resident performance.
Assessments of resident performance occur much more often under CBME compared to the traditional model. 15,18 In other radiology competency-based curricula, a variety of assessment tools including online radiology modules, quizzes, and self-assessment modules have been employed. 19,20
Faculty Feedback
Regular feedback is an important aspect of CBME to address identified gaps in knowledge and skills. 21,22 The primary tool used to deliver this feedback in the Core of Discipline stage is an online assessment form that includes rating scales and written comments. The advantage is the opportunity to track performance and skill mastery longitudinally. 23,24
However, a challenge associated with rating scales is that they can be interpreted differently by different evaluators. For example, the scale for assessments of milestones in most of the radiology assessments, is “Not observed,” “Needs attention,” “Developing,” and “Achieved.” The “Needs attention” item can be interpreted subjectively by the evaluator. However, if the milestone is clearly defined, it narrows the scope for interpretation. We have noted these inconsistencies and adjusted the wording of the milestones based on feedback. The scales are also constructed such that if the milestone is flagged as “Needs attention,” a comment is required before the assessor can proceed. Faculty were given targeted training on providing optimal feedback in live sessions with the educational consultant prior to the transition to CBME.
Resident Engagement With Assessment Completion
There can be significant differences in the quantity of assessments gathered on a particular rotation between residents. Residents are responsible for triggering assessments to ensure compliance with the assessment strategy associated with the EPA. This raises the possibility of residents selectively choosing their most successful direct observation performances for assessment.
This has partially been addressed with the scheduling of regular meetings with the AAs where documentation from Elentra™ may reveal discrepancies, although there is currently no robust mechanism in place to detect selective triggering of evaluations.
Change Management
Differences exist in perceived barriers to CBME implementation between faculty and residents. A prior study examined the most concerning barriers for faculty and residents within the Neurology Residency Program at Queen’s University. 24 For faculty, the failure of residents to take on leadership roles within CBME implementation was the most concerning barrier. This contrasted with residents’ identification of responsiveness of faculty to providing feedback during their assessments as the most concerning. 24
CBME outcomes can be optimized by involving residents, as issues apparent from the trainee’s perspective may be identified and addressed more readily. 25,26 New cohorts entering training do not have a pre-existing mental model of assessment for PGME, allowing for more seamless engagement in the CBME process. However, senior faculty have spent many years assessing without the process of documentation that CBME entails. This involves a paradigm shift that is difficult for busy practicing radiologists. One-on-one professional development has been provided for all faculty in the department. This includes live CBME sessions with the educational consultant prior to the start of CBME, ongoing grand rounds on CBME topics, and 1-3 workshops annually sponsored by the PGME office on CBME topics. Topics include providing optimal feedback to trainees, recognizing and escalating concerns with competency, and assessments. Additional “refresher” training could be added but logistical issues in coordinating this for available faculty have been a consistent barrier.
CBME implementation at Queen’s University continues to evolve as more data become available. Discussion of the fourth stage, Transition to Practice, remains forthcoming as presently no current residents have yet completed the Core of Discipline stage.
Conclusion
CBME represents a shift to an outcome-based learning and assessment model. A systematic program design developed on a competency-based framework will readily reveal gaps in resident knowledge and skills during each stage of training.
The transition to CBME poses unique challenges. We have described the development of a CBME curriculum in the Diagnostic Radiology Residency Program at Queen’s University, with a focus on the Core of Discipline stage. Encountered challenges included difficulty meeting procedural volume requirements, inconsistent procedural tracking, increased administrative burden, timely and standardized faculty feedback, and resident engagement with assessment completion. Implementing solutions to these challenges is ongoing, requires customization to the local environment, and is guided by input from all stakeholders.
As the transition to CBME progresses toward full implementation and is further optimized, it remains important to continue lobbying for the required financial resources to successfully implement CBME. The goal is to standardize resident training and address deficiencies to produce physicians better able to provide quality care in their communities. This paper may inform CBME implementation in other training programs within and outside of Queen’s University.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
