Abstract
Background
Competence by Design (CBD) is a model of medical education developed by the Royal College of Physicians and Surgeons of Canada. CBD emphasizes an outcomes-based approach to residency training, replacing traditional models of time-based training. CBD focuses on the achievement of Entrustable Professional Activities (EPAs), which are key tasks of a discipline. Misalignment of educational priorities between teachers and learners can negatively impact learning and quality of care. This study aims to compare the learning priorities of plastic surgery residents and attendings, as well as explore perceptions of CBD.
Methods
An online survey was administered to plastic surgery residents and attendings identified via the Canadian Society of Plastic Surgeons mailing list. Participants ranked milestones by learning priority using a 5-point Likert scale. Participants were also asked their opinions on CBD, and responses were analyzed thematically.
Results
A total of 76 participants (25 residents and 51 attendings) responded to the survey (15% response rate). Residents ranked gathering a relevant clinical history as less important than attendings (odds ratio = 0.38, 95% confidence interval 0.14-0.98, P = .049). Residents and attendings had comparable rankings for the other 18 EPA milestones. Thematic analysis revealed that while CBD is perceived to have benefits for trainees, concerns pertain to excessive administrative burden and subjective evaluation. There was also inconsistency in the way EPAs are utilized by attendings with regard to when they were initiated/completed and how milestones were scored.
Conclusions
Overall, this exploratory study showed good concordance between resident and attending plastic surgeon prioritization of EPA milestones. Perceived benefits and limitations of CBD in plastic surgery are similar to those reported in other specialties. Many attendings have knowledge gaps in the implementation of EPA evaluations.
Introduction
The traditional, time-based system of postgraduate training has been criticized for not explicitly defining the outcomes and abilities expected of graduating residents. 1 Over the last decade, the Royal College of Physicians and Surgeons of Canada (RCPSC) has implemented a new model of postgraduate medical education called Competence by Design (CBD). CBD is based on competency-based education principles, which de-emphasize time-based training and promises greater accountability, flexibility, and learner-centredness. 2
Competency is assessed through Entrustable Professional Activities (EPAs), which are key discipline-specific tasks that trainees are expected to be able to perform at the completion of their training. EPAs are comprised of individual milestones that correspond to specific, observable behaviors, and are categorized based on various stages of postgraduate training. 3 EPAs are designed to be scaffolded, building depth and complexity as residents progress in their training and require less support. Each EPA corresponds to various stages of training as defined by the RCPSC: Transition to Discipline, Foundations of Discipline, Core of Discipline, and Translation to Practice. 4 There are 53 EPAs defined in the current iteration of the CBD curriculum for plastic surgery.
Proper student-teacher alignment between learning objectives, teaching activities, and evaluations is required to achieve coherency in a curriculum. Curricular alignment can be approached at four levels: content and structure of the course, program-level structure of curriculum, students’ perspective of the experience of learning, and lastly, the collaborative dynamic between student and teacher. 5 In medical education, curriculum alignment has mostly focused on the first three levels.6–8 There remains a paucity of research focused on alignment of learning priorities between student and teacher. Misalignment of the perceived educational priorities can negatively impact learning, educational satisfaction, and quality of care. 9
Currently, there are no studies comparing the perception of EPAs between plastic surgery residents and attendings. We hypothesize that residents and attending surgeons may prioritize different EPA milestones based on their knowledge, experience, and cognitive load. This exploratory study aimed to (1) compare learning priorities of Canadian plastic surgery residents and attending surgeons using the framework of EPA milestones and (2) explore resident and attending perceptions of CBD.
Methods
This study was designed as a behavioral survey-based study. Ethics approval was obtained from the University of British Columbia Children's and Women's Research Ethics Board (H24-01273).
Study Participants and Recruitment
An anonymous online survey was disseminated to plastic surgeons as well as residents at English-speaking institutions, identified via the Canadian Society of Plastic Surgeons email list. Subspecialty trainees (fellows) and residents at programs where English was not the primary language of instruction were excluded from this study. The total number of eligible participants was estimated at 523. The survey remained open for 13 weeks from June 26, 2024 to September 24, 2024. Follow-up reminder emails were sent at 2 and 4 weeks after the initial distribution. Survey data was collected using Qualtrics survey software hosted by the University of British Columbia. 10
Survey Design
Two versions of the survey were created, one for residents and a second for attending surgeons (Supplemental File 1). The survey collected demographic characteristics including gender, province/territory, practice type, and primary clinical focus. Participants were asked to rank EPA milestones by their learning priority using a 5-point Likert scale (response options ranged from “least important” to “most important”). Residents were instructed to rank each milestone's importance relative to their stage of training. Information regarding practice type, number of EPA assessments completed per month, and the frequency/timing of EPA assessments was collected. Open-ended questions were designed to capture feedback regarding EPAs and the CBD framework, and attendings were also asked about factors that are considered when completing EPA evaluations for trainees.
Data Analysis
Residents’ and attending surgeons’ responses were compared using Mann-Whitney U tests. Associations between participant characteristics and rating of EPA milestones were analyzed using ordinal logistic regression models. Quantitative analysis was conducted in RStudio with statistical significance set at 0.05. 11 Open-ended text responses were approached inductively using content-analysis to develop themes. All responses were coded by 2 authors (CL, SC) and differences were discussed and resolved with the senior author (TKW). The codes were then organized into themes, and each theme was described with illustrative quotes.
Results
Study Participants
A total of 76 participants responded to the survey (response rate 76 of 523, 15%), comprising 25 residents and 51 attendings (Table 1). The majority of respondents identified as male (49%), while 26% of respondents preferred not to disclose their gender. The majority of respondents resided in Ontario, British Columbia, or Alberta. Respondents who identified as attending surgeons mostly practiced at academic institutions located in urban settings (Table 2). The most frequently reported areas of primary clinical focus were hand/peripheral nerve, general reconstruction, and pediatrics. Table 3 describes the level of training of resident respondents, and the timing of EPA completion relative to the learning activity. The majority of attendings reported that residents requested for an EPA to be completed after the learning activity had already occurred. Residents reported that they only sometimes initiated the discussion of EPA assessments with attendings before the activity.
Demographic Characteristics.
*As of September 2024, there are no plastic surgery residency programs in Saskatchewan.
Attending Plastic Surgeon Characteristics.
a8 participants did not answer this question. Abbreviation: EPA, Entrustable Professional Activities.
Resident Plastic Surgeon Level of Training.
Abbreviation: EPA, Entrustable Professional Activities.
Quantitative Analysis
Table 4 shows results from responses to Foundations EPA #3: Performing an initial assessment of patients with craniofacial trauma. Residents ranked gathering a relevant clinical history as lower importance than attendings (odds ratio [OR] = 0.38, 95% confidence interval [CI] 0.14-0.98, P = .049) (Figure 1). Generally, each milestone was ranked highly and there was concordance between resident and attending rankings. The milestone “Recognize urgent problems and seek assistance” was ranked as the most important according to both residents and attendings, with a median ranking of 5 of 5 (interquartile range [IQR] = 4-5). The milestones “Determine the need and timing of referral to another service” and “Share information and explanations with the patient/family” were tied as the lowest importance for attendings and residents, with median scores of 4 of 5 (IQR = 3-4). There were no statistically significant associations between prioritization of milestones and subspecialties, practice location, or years in practice among attending physicians for this EPA (Supplemental Tables S1 and S2).
Foundations EPA #3: Performing an initial assessment of patients with craniofacial trauma.
Foundations Entrustable Professional Activity #3 Performing an Initial Assessment of Patients With Craniofacial Trauma.
*P < .05.
Abbreviation: IQR, interquartile range.
Table 5 shows results from responses to Core EPA #13: Providing assessment and definitive management for patients with simple craniofacial trauma. Residents ranked the milestones “Apply knowledge of anatomy, key landmarks, and the surgical procedure” and “Perform safe dissection of relevant structures and tissue layers with gentle tissue handling” as the most important, with median scores of 5 (IQR = 4-5) (Figure 2). Although attendings ranked these 2 milestones lower, with a median score of 4 (IQR = 4-5), this difference was not statistically significant. Residents and attendings were concordant in their ranking of remaining milestones. The least important milestone was “Participate in the surgical safety checklist” with a median score of 3 (IQR = 3-4) and 3 (IQR = 2-4) for residents and attendings, respectively. There were no statistically significant associations between subspecialties, practice location, or years in practice and prioritization of milestones for this EPA.
Core EPA #13: Providing assessment and definitive management for patients with simple craniofacial trauma.
Core Entrustable Professional Activity #13 Providing Assessment and Definitive Management for Patients With Simple Craniofacial Trauma.
Abbreviation: IQR, interquartile range.
Qualitative Analysis
A total of 24 participants responded to the optional open ended text questions. Five themes were identified: (1) challenges in implementation/excessive administrative burden of EPAs; (2) subjective and inconsistent EPA evaluation; (3) EPAs are not a valuable tool for learning and feedback; (4) neutral/mixed opinion on the utility of EPAs for learning and feedback; and (5) EPAs are a valuable tool for learning and feedback (Table 6).
Thematic Analysis of the Responses to the Question: “What are Your Thoughts About EPAs and/or the CBD Framework?”
Abbreviation: EPA, Entrustable Professional Activities.
Challenges in Implementation/Excessive Administrative Burden of EPAs
This theme arose with the greatest frequency in responses to the open-ended questions. Four residents and 6 attending plastic surgeons provided comments relating to this theme. Respondents reported that EPAs added additional administrative work to complete beyond regular teaching and clinical duties. Additionally, there were concerns that the implementation of CBD is challenging given the heterogeneity of training programs. Respondents also identified the challenge of attendings’ attitudes toward CBD and negative perspectives on its effectiveness in medical education: “Ultimately, the training programs are all different, so it may be difficult to achieve certain EPAs at certain sites. There has to be good staff buy in, but from talking to people within our program and across the country, this isn't universal, even within a particular training program.”—Resident
Subjective and Inconsistent EPA Evaluation
Concerns regarding subjectivity and difficulty with interpreting the results of EPAs were identified by 2 surgeons and 1 resident. Respondents noted that scoring on an EPA can be dependent on the evaluator; a trainee's scoring can be influenced by external considerations such as the nature of the relationship between the resident and evaluator. Concerns were also expressed that the number of EPAs required to be completed during residency may be insufficient to represent mastery of the procedure or skills being evaluated. “EPAs are incredibly subjective. One staff's evaluation can be completely different than another's… It also seems dependent on relationships with staff… This ends up leading to residents only asking staff that they know will get them a useful EPA (ie a one and done completed task).”—Resident
EPAs Are Not a Valuable Tool for Learning and Feedback
Three residents and 4 attending surgeons felt EPAs were not a valuable tool for learning and feedback. Many comments falling under this theme were also coded under the theme that EPAs create excessive administrative burden. Several residents expressed skepticism about the accuracy and utility of EPAs, highlighting concerns that the high volume of required assessments may compromise their quality. One resident noted that EPAs are “often rushed through or given a pass out of convenience rather than competency.” This comment reflects a broader sentiment that the process can prioritize form completion over meaningful assessment.
Neutral/Mixed Opinion on the Utility of EPAs for Learning and Feedback
Two residents and 4 attendings responded with neutral/mixed opinions surrounding the utility of EPAs for learning and feedback. EPAs were viewed as useful for identifying residents that would benefit from additional educational support and increased opportunities for formative assessment and feedback. One resident noted that “As a whole, it seems like a good way to identify residents that are behind or may need some type of intervention.” However, a perceived limitation of CBD was the persistence of time-based training—respondents found that there were difficulties in extending training for struggling residents, or conversely, for excelling residents to graduate early.
EPAs Are a Valuable Tool for Learning and Feedback
Two residents and 4 attendings expressed favorable perspectives about the utility of EPAs for learning and feedback. Residents indicated that EPAs provided opportunities for receiving constructive feedback and identifying areas for further learning. Attendings felt that EPAs were a valuable instrument for providing feedback, but some respondents noted challenges including increased administrative load, increased resources required to support CBD, and a lack of buy-in from other attendings. “Initial adoption of the system has growing pains (as with all systems) but once everyone was familiar with the system it [became] very useful.”—Resident “CBD provides a method to document proficiency and directs the learner and the instructor to the various aspects of the successful conduct of patient care.”—Attending
Discussion
Our study suggests attending surgeons and residents have good alignment in their prioritization of EPA milestones, although attendings prioritized gathering a relevant clinical history more highly than residents. Analysis of the qualitative data revealed five themes highlighting perceptions and attitudes of surgeons and trainees toward the CBD framework. These themes included the perceived utility of CBD, subjectivity and inconsistency in evaluations, and challenges in implementation.
It is difficult to directly compare our results to other reports as there is a paucity of studies comparing attending and resident perceptions of EPAs in medical education. Prior reports have focused on comparisons between self-evaluations versus formal evaluations at various residency programs. Sterkenberg et al. 12 surveyed anesthesiology attendings and residents at a single institution on their expectations of residents, perceived abilities of residents (from attending and resident viewpoints), and actual practice. They found that residents tended to rate their abilities more highly than attendings, suggesting discordance in perceived versus real-world clinical performance. However, a similar study conducted by Chow et al. in a plastic surgery training program demonstrated good concordance between resident self-evaluations and milestone evaluations completed by staff surgeons on a set of six core competencies. 13
Transition from traditional time-based, apprenticeship models of residency training to competency-based models have been driven by factors such as work-hour restrictions, patient safety, and recognition of individual learning needs. CBD adopts an outcomes-based approach, which emphasizes demonstration of key skills and competencies rather than the completion of fixed time periods of training. This framework promotes deliberate practice, timely feedback, and a more accurate assessment of each trainee's abilities. Prior research has demonstrated reduction in trainee errors and improvement in overall surgical outcomes through active engagement in a competency-based training model. 14 Providing practical assessment and timely feedback of surgical skills can be difficult, but certain tools such as the Ottawa Surgical Competency Room Evaluation (O-SCORE) have good inter-rater reliability and may be useful for providing structured and timely feedback for trainees. 15 Moreover, CBD allows for flexible training progression tailored to the individual needs of each learner. This allows trainees to receive the support that they need on areas of weakness and facilitates a more efficient learning process.
Although CBD has many benefits over time-based training models, our study found that implementation of CBD may be limited due to incomplete understanding of CBD principles. For instance, EPA evaluations are intended to be done proactively; residents should inform their attendings that they wish to be evaluated on an EPA before the activity occurs. However, this rarely occurred in practice. Only 1 resident (4%, 1 of 25) and 6 attendings (11.8%, 6 of 51) were regularly discussing EPA evaluations before learning activity occurred (Table 3), suggesting either a lack of familiarity in the EPA process or that proper evaluation protocols are disregarded for the sake of convenience.
Another area of divergence between CBD best practices and real-world application was regarding the benchmark that residents were to be compared against. According to RCPSC guidelines, when filling out an EPA evaluation, an assessor should consider a resident's ability to perform a task independently regardless of their year of training. For example, a score of 1 of 5 would indicate that the assessor had to give hands on guidance, whereas a score of 5 of 5 would indicate that the resident would be able to perform the task completely independently. 16 However, only 23.5% (12 of 51) of attendings correctly chose level of competence as the primary factor considered during evaluation. And 52.9% (27 of 51) reported that they considered both year of training and level of competency when evaluating EPAs. Six percent (3 of 51) responded that they considered year of training only. These findings suggest that some faculty view the EPA evaluation scale as based on stage-specific expectations, rather than on a continuum to independent practice. This misunderstanding can result in under- or overestimation of a resident's abilities.
Overall, the themes identified in this study align with broader research on Competency-Based Medical Education (CBME) perceptions across other specialties, including urology and anesthesiology programs.17,18 Advantages of CBME included identification of residents who require extra assistance and the ability for residents to receive more personalized, regular feedback. However, these studies also identified challenges such as increased administrative burden, low faculty buy-in around CBME implementation, and challenges faced by residents in completing EPAs for low-prevalence procedures or conditions. These shared findings suggest that barriers surrounding CBME implementation extend beyond individual specialties.
Challenges surrounding CBME are being actively addressed by the RCPSC. Ongoing feedback has driven the development of the CBD Adaptations Plan by the RCPSC, which aims to address key challenges of CBD implementation. 19 Goals include addressing the burden of assessments, lost opportunities for feedback and coaching, and misconceptions surrounding CBD. For instance, in order to reduce the burden of assessments, the RCPSC proposes to reduce the total number of EPAs and the number of EPA milestones where appropriate. The proposal also aims to increase opportunities for in-the-moment and longitudinal feedback. While it is encouraging to see active engagement and refinement of the CBD curriculum by the Royal College, this study highlights further areas of potential investigation and improvement.
Limitations
There are several limitations to our study. Firstly, anchoring bias may have influenced participants’ responses on the rank order of EPA milestones. The majority of the rankings for EPA milestones were clustered toward the higher end (median scores of 4-5). This trend could reflect that physicians tend to view every milestone as uniformly important; however, it could also indicate anchoring bias, where early items or prior beliefs may influence the ranking of milestones. Anchoring bias is known to affect the reliability of survey-based studies and could limit the variability of answer choices. 20 Future studies could consider strategies to address this, such as randomized item order or forced-choice ranking systems to mitigate the anchoring effect. 21 The current study is also limited by the relatively small sample size and low response rate. Nevertheless, our response rate was similar to other similar studies evaluating perceptions on CBME—Boet et al. 22 had an 18% response rate for anesthesiology program directors and a 6% response rate for anesthesiology residents, while Wang et al. 23 had an overall total response rate of 20.5% in their study. Finally, our study was limited by the exclusion of primarily French-speaking residency programs.
Conclusions
To our knowledge, this is the first study to examine the prioritization of EPA milestones in plastic surgery within the context of CBD. Overall, CBD has been identified as a valuable tool for assessing resident progression. Residents and attendings generally have good alignment in the prioritization of EPA milestones. However, challenges such as administrative burden and subjectivity of assessments still remain. Importantly, this study identifies inconsistencies in the understanding and utilization of EPA assessments, highlighting the need for further education and guidance in CBD implementation. To better understand the challenges and potential improvements in CBD, future studies should use semistructured interviews to elicit detailed feedback from participants and collaboratively develop targeted solutions. Understanding the challenges faced by current iterations of CBD may help aid understanding of how postgraduate medical education can be optimized to ensure that residents become capable physicians who are well prepared to meet the needs of their patients.
Supplemental Material
sj-docx-1-psg-10.1177_22925503261448829 - Supplemental material for Alignment of Learning Objectives in Canadian Plastic Surgery Residency Programs
Supplemental material, sj-docx-1-psg-10.1177_22925503261448829 for Alignment of Learning Objectives in Canadian Plastic Surgery Residency Programs by Cameron J. Leong, Stephanie Cooper, Rebecca Courtemanche, David Tang and Terence Kwan-Wong in Plastic Surgery
Supplemental Material
sj-docx-2-psg-10.1177_22925503261448829 - Supplemental material for Alignment of Learning Objectives in Canadian Plastic Surgery Residency Programs
Supplemental material, sj-docx-2-psg-10.1177_22925503261448829 for Alignment of Learning Objectives in Canadian Plastic Surgery Residency Programs by Cameron J. Leong, Stephanie Cooper, Rebecca Courtemanche, David Tang and Terence Kwan-Wong in Plastic Surgery
Supplemental Material
Footnotes
Acknowledgments
We would like to thank Ye Shen MPH, biostatistician with the Clinical Research Support Unit at the BC Children's Hospital Research Institute for her advice and expertise, as well as the plastic surgeons and plastic surgery clinic staff at BC Children's Hospital for their collaboration.
Ethical Considerations
This project was approved by the University of British Columbia Children's and Women's Research Ethics Board (H24-01273). All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008.
Consent to Participate
All participants provided informed consent.
Author Contributions
CL, RC, DT, and TKW contributed to the study conceptualization and design. CL, SC, RC, DT, and TKW contributed to the data interpretation, manuscript writing and editing, and approved the manuscript version to be published. CL, RC, and TKW contributed to the data collection. CL and SC contributed to the data analysis.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by a grant from the Office of Pediatric Surgical Evaluation and Innovation.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Study-Related Abstracts
Leong C, Cooper S, Courtemanche R, Tang D, Kwan-Wong T. Alignment of Learning Objectives in Canadian Plastic Surgery Residency Programs. Journal of Investigative Medicine. 2024;73(1):NP1-NP778. Abstract 542. doi:10.1177/10815589241297200
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
