Abstract
BACKGROUND
The evaluation of Competence by Design (CBD) residency programs is crucial for enhancing program effectiveness. However, literature on evaluating CBD programs is limited. We conducted a 2-phase mixed-methods study to (a) assess the extent of program evaluation activities in CBD residency programs in Canada, (b) explore reasons for engaging or not engaging in these activities, (c) examine how CBD programs are conducting program evaluations, and (d) identify ways to build capacity for program evaluation.
METHODS
Phase 1 involved surveying 149 program directors from specialty/subspecialty programs that transitioned to CBD between 2017 and 2020. We calculated descriptive statistics for 22 closed-ended survey items. Phase 2 comprised interviews with a subset of program directors from Phase 1. Data analysis followed a 3-step iterative process: data condensation, data display, and drawing and verifying conclusions.
RESULTS
In Phase 1, we received 149 responses, with a 33.5% response rate. Of these, 127 (85.2%) indicated their programs engage in evaluation, while 22 (14.8%) do not. Among the 127 programs that engage in evaluation, 29 (22.8%) frequently or always develop evaluation questions, and 23 (18.1%) design evaluation proposals/plans. Reasons for engaging in evaluation included decision-making and stimulating changes in educational practices. Conversely, reasons for not engaging included lack of knowledge, personnel, and funding. In Phase 2, 15 program directors were interviewed. They reported that CBD programs face challenges such as limited resources and buy-in, rely on ad hoc evaluation methods, and use a team-based evaluation format. To enhance evaluation capacities, interviewees suggested (a) developing expertise in program evaluation, (b) acquiring evaluation resources, and (c) advocating for clear evaluation expectations.
CONCLUSIONS
Most CBD residency programs are engaged in program evaluations, but the quality is often questionable. To fully realize the potential of program evaluation, CBD programs need additional resources and support to improve evaluation practices and outcomes.
Background
In 2017, the Royal College of Physicians and Surgeons of Canada (RCPSC) started implementing Competence by Design (CBD) for postgraduate medical residency programs. It is a hybrid competency-based medical education (CBME) model that combines a time-based and an outcomes-based approach to learning. 1 To understand how CBD is implemented within individual residency programs, whether it is implemented as intended, and how it contributes to patient, learner, and faculty outcomes, these programs need to be evaluated. 2 Evaluations of CBD programs can involve the collection of data on program activities, characteristics, and outcomes. Such information can improve program effectiveness and inform programming decisions. 3
The RCPSC hosts an annual international Program Evaluation Summit where those engaging in the evaluation of CBD programs can gather to learn from each other's evaluation initiatives and explore the successes and challenges of CBD. 4 However, there are few published evaluations of CBD programs within the Canadian context as well as minimal empirical research on the evaluation of CBD programs. To date, the research on evaluation literature has focused on theoretical examples of or reflections on approaches that evaluators and stakeholders can use to evaluate CBD programs. For example, it provides tips on how to use a utilization-focused approach, 5 as well as a participatory evaluation approach. 6 It also notes how contribution analysis, a theory-based approach to evaluation, can be useful for examining the connections between CBD, physicians’ readiness for practice, and patient outcomes.2,7 Moreover, it explores the use of rapid evaluation in one CBD program. 8 Aside from these recommended approaches as well as a general recognition of the importance of program evaluation in this context, the state of evaluation across CBD programs in Canada remains unclear.
To understand and improve the evaluation of CBD programs, we need to conduct research on the evaluation of them by investigating the extent to which program evaluation activities are occurring and the reasons why programs are engaging or not engaging in them. Additionally, we must explore the actual and potential positive and negative consequences of engaging in program evaluation and the ways that these programs can build their capacities to do program evaluation. Those who are responsible for delivering CBD programs as well as ensuring their relevance and effectiveness have unique insights into program evaluation activities. Thus, by surveying and interviewing program directors in Canada whose specialty/subspecialty programs had transitioned to CBD, this mixed-methods study sought answers to the following research questions:
Phase 1 (survey of program directors of CBD):
To what extent are CBD programs engaging in program evaluation? Why are CBD programs engaging or not engaging in program evaluation? Phase 2 (interviews with program directors of CBD):
How are CBD programs engaging in program evaluation? How can CBD programs build their capacities to do program evaluation?
Methods
We followed a 2-phase, sequential mixed-methods research design. Phase 1 focused on the collection and analyses of quantitative survey data from program directors whose RCPSC specialty/subspecialty programs had transitioned to CBD between 2017 and 2020. We used the findings from Phase 1 to inform the participant-level questions for Phase 2. Phase 2 involved the collection and analyses of qualitative interview data from selected Phase 1 respondents.
Phase 1
Sample
In Fall 2021, we invited 445 program directors whose RCPSC specialty/subspecialty programs had transitioned to CBD between 2017 and 2020 to participate in the survey.
Survey development
We used the published literature on research on program evaluation and program evaluation in medical education to inform the development of our survey.3,9–11 We also adapted survey dimensions and items, originally designed to explore program evaluation activities in pediatric rehabilitation centres, 12 to the CBD context. The present study is based on 22 items that explored (a) the extent to which CBD programs are engaging in program evaluation activities and (b) the reasons why these programs are engaging or not engaging in program evaluation activities. We piloted the survey with 2 medical education researchers and 2 program evaluation experts who were ineligible for the study. We finalized the survey based on the pilot feedback.
Data collection procedures
We built the survey in SurveyMonkey for electronic distribution. We obtained program directors’ email addresses from the RCPSC's website and sent a personalized email containing the survey link and information letter to each director. To maximize participation, we emailed the survey using a modified version of Dillman et al's Tailored Design Method by sending 2 study reminders (ie, 2 weeks and 4 weeks after the initial email distribution). 13 Informed consent was obtained at the beginning of the survey, and respondents were permitted to stop the survey at any point.
Data analysis
We calculated descriptive statistics (ie, frequencies and percentages) using IBM SPSS (version 28). The Research Ethics Board at the University of Ottawa approved the study.
Phase 2
Sample
In Spring 2022, we applied convenient sampling to identify and recruit RCPSC specialty/subspecialty program directors who had consented and completed the survey in Phase 1. Of the 149 respondents, 36 (24.2%) shared their email addresses to be contacted about participating in a follow-up interview. Of the 36 program directors who agreed to be contacted, 15 program directors consented to an interview.
Interview guide development
We used the findings from Phase 1 to design the semi-structured interview guides for Phase 2. We created separate guides for respondents who indicated that they engage or do not engage in program evaluation of their CBD programs. We began both guides with an introductory script and included open-ended questions with additional probes to allow for expansion and idea prompting (see Table 1). We piloted the guides with a program evaluation researcher to ensure they were clearly worded and included relevant questions or probes. 14
Interview questions.
Data collection procedures
We emailed the program directors a letter of information and a consent form inviting them to participate in an interview. Each interview was scheduled at a time that was convenient for the interviewee, took place using Zoom, and lasted approximately 1 h. Prior to the interview, all participants signed and returned the consent form. We recorded and transcribed each interview verbatim. Each transcription was sent to the corresponding interviewee for approval and feedback. We gave each participant a $10 Amazon gift card to thank them for participating in the interview.
Data analysis
We (JM and KM) analyzed the data in NVivo using Miles’ et al's 3-step iterative process (ie, data condensation, data displays, drawing, and verifying conclusions). 15 Following each interview, we independently listened to the audio-recordings to start identifying how CBD programs are engaging in program evaluation as well as how CBD programs are building their capacities to do program evaluation. Following transcription, we then independently reviewed the transcripts and summaries to create our own coding systems, read each transcript, annotated phrases, and coded the data. Throughout the process, we embraced codes not identified a priori based on the findings from Phase 1 as well as our initial listening of the audio-recordings. We then met to discuss our independent analyses and resolved differences through discussion. Together, we merged our analyses, created data display tables, and generated conclusions, including exemplar quotations.
Results
Phase 1
Of the 445 program directors, we obtained 149 responses, constituting a response rate of 33.5%. The 149 respondents represented 16 out of 17 medical schools in Canada and 26 out of 32 specialty/subspecialty programs that had transitioned to CBD between 2017 and 2020.
Extent to which CBD programs are engaging in program evaluation activities
Of the 149 respondents, 127 (85.2%) indicated that their programs do engage in program evaluation and 22 (14.8%) indicated that their programs do not engage in program evaluation. Of the 127 respondents who indicated that their programs do engage in program evaluation, 58 (45.7%) said their programs either never or rarely work with a program evaluator, 41 (32.3%) did not know if their programs work with a program evaluator, and 28 (22.0%) noted that they sometimes, frequently, or always work with a program evaluator. Seven (5.5%) respondents reported that their programs have an employee whose primary responsibility is program evaluation and five (3.9%) said that their programs receive funding for program evaluation. Furthermore, 29 (22.8%) reported that their programs frequently or always develop program evaluation questions and 23 (18.1%) noted that their programs design program evaluation proposals/plans. Table 2 provides information on the extent to which the respondents or others in their programs engage in various program evaluation activities.
Extent to which CBD programs are engaging in program evaluation activities (N = 127).
Why CBD programs are engaging or are not engaging in program evaluation activities
Of the 127 respondents who indicated that their programs do engage in program evaluation, 65 (51.2%) indicated that their programs frequently or always do so to make decisions about the program. Fifty-one (40.2%) respondents also noted that they engage in program evaluation frequently or always to stimulate changes in educational practices. Table 3 provides a summary of the reasons why they engage in program evaluation activities.
Program directors’ perceptions about why they engage in program evaluation activities (N = 127).
The 22 respondents who indicated that their programs do not engage in program evaluation specified various reasons as to why. Nine (40.9%) indicated that they do not because the program does not know how to do program evaluation, 7 (31.8%) indicated that their program has no personnel to do program evaluation, and 7 (31.8%) noted that their programs have no funding to do it.
Phase 2
Of the 15 interviewees, 14 (93.3%) indicated that they engage in program evaluation of their CBD program, while the remaining one (6.7%) indicated that they do not.
How CBD programs are engaging in program evaluation
When asked to elaborate on how CBD programs are engaging in program evaluation, the interviewees explained that programs are (a) struggling due to limited resources and buy-in, (b) using ad hoc evaluation methods, and (c) utilizing a team-based evaluation format.
Struggling due to limited resources and buy-in
Participants noted that their CBD programs face challenges in how they do evaluation due to restricted resources and minimal stakeholder buy-in. Program directors struggle with evaluation within their specialties due to insufficient financial support, human resources, and time. As one participant put it, “the simple answer [is] it's time and money” (P1). Another added, “CBD came in, but no extra money came in. Programs have found that CBD costs a lot more than the Royal College suggested” (P3). These costs force faculty to implement CBD without additional resources for evaluation, leading to statements like, “people are needing to figure out ways to do more with the same amount of resources, but without help” (P3). With everyone stretched thin, one participant noted, “everybody else in our training committee who are doing evaluation, it's completely volunteer basically and … everybody's just got too many things to do” (P12).
Participants also pointed out that without extra evaluation resources, adding to the workload is impractical: “it's just not realistic for people who are unpaid volunteers, working full time as physicians to do this kind of work” (P12). The lack of buy-in is exacerbated by the challenge of asking staff to invest more time: “asking our staff to take more time out to do the evaluations. It's been really challenging” (P11). Additionally, some believe informal evaluation suffices: “we talk about what's going on, what's working well, what's not working well. Why do we have to do any more work for it?” (P12). Factors like burnout from surveys and the complexity of proving CBD's impact further complicate the situation: “residents [are] just too burned out from surveys because of other issues” (P6). Participants concluded that inadequate resources and time make program evaluation a “patchwork” (P14).
Using ad hoc evaluation methods
Participants described their CBD programs’ reliance on ad hoc evaluation methods, lacking formal tools or benchmarks. One program director remarked, “I don’t feel like there's a validated tool or benchmark of what we should be looking at” (P15). This gap leads to improvisation with methods such as surveys, informal feedback, and exit interviews. As a participant said, “we’re not super confident. We’re improvising” (P14). They noted that current practices involve linking evaluations to surveys and informal feedback but could benefit from more robust mechanisms: “I think there can definitely be [a] more robust and fulsome mechanism for providing that” (P10). Improvisation with “homemade surveys” (P12) and limited insight from self-designed surveys contribute to a lack of confidence: “there's not a lot of insight that can be gained from probably some of the surveys that I can design” (P10). The absence of formal evaluation limits the ability to target specific interventions: “[programs] can’t really target or pinpoint what specific interventions do” (P13).
Using a team-based evaluation format
Participants reported that their CBD programs use a team-based format for evaluation, involving various degrees of input from various stakeholders, particularly those in leadership. For instance, evaluations include input from “postgraduate, university-level leadership” (P15) and committees with representatives from program leadership and residents. Some programs benefit from “associate program director[s]” or multiple faculty members involved in evaluation (P6). Additionally, programs collaborate with administrative members for data analysis and share findings across universities: “we shared a lot of data learned from our process and what we’ve kind of been working through” (P13). Inter-university meetings facilitate cooperative learning, though participants still “don’t always know what other programs are doing” (P10).
How CBD programs can build their capacities to do program evaluation
When prompted about how CBD programs can build their capacities to do program evaluation, interviewees noted that they can (a) develop expertise in program evaluation, (b) acquire evaluation resources, and (c) advocate for clear program evaluation expectations.
Develop expertise in program evaluation
Participants suggested that CBD programs can build evaluation capacities by improving their expertise. Program directors noted their unfamiliarity with evaluation processes but found the concept promising: “I’m so unfamiliar with the process to evaluate the program… but the idea sounds very interesting” (P7). They recognized that professional development sessions and retreats offered valuable learning opportunities: “We also had some training… and decided maybe this is actually a useful thing to do” (P12). Faculty development increased understanding and encouraged systematic evaluation: “More training in program evaluation… for learners and supervisors is beneficial for proper evaluation” (P6).
Acquire program evaluation resources for guidance
Participants highlighted the need for accessible evaluation resources. Some felt unprepared due to a lack of guidance: “We haven’t done [program evaluation] really at this point… don’t know where to get started” (P10). They called for streamlined, practical resources: “A how-to resource should be streamlined, easy, reproducible” (P8). Participants also emphasized the need for funding to support human resources and time: “More time, more money would be great” (P11). Access to evaluation experts could provide valuable support: “We need a dedicated individual… to evaluate and improve our program” (P14). Additionally, “technical resources and IT support are crucial for data management and analysis” (P1).
Advocate for clear program evaluation expectations
Participants argued that clear expectations from RCPSC, postgraduate medical education (PGME), and specialty programs are necessary. They noted a lack of explicit requirements: “If the Royal College wants program[s] to have an ability to do program evaluation… they need to be explicit” (P6). They suggested that higher-level policies should provide specific guidance: “RCPSC should be saying, ‘Here are the common questions…’” (P4). Clear expectations from PGME leadership are also needed: “We need to have discussions with PGME to determine what they want to see from our conversion to CBD” (P6). Formalizing evaluation processes and providing a roadmap will ensure evaluations are valuable and consistent across programs: “A standardized roadmap could be very similarly used across the country” (P8). In summary, advocating for clear expectations will help CBD programs build their evaluation capacities.
Discussion
The purpose of this study was to investigate the extent of program evaluation activities within CBD programs, understand why some programs engage in these activities while others do not, explore how CBD programs are currently conducting evaluations, and identify ways to build their evaluation capacities. Evaluating CBD programs is crucial for identifying areas of improvement and ensuring that physicians receive optimal training to positively impact patient outcomes. Over 80% of survey respondents reported that their programs engage in evaluation activities. However, further examination revealed that these evaluations often lack proper planning, structure, methodology, support, and resources. For instance, fewer than 25% of respondents indicated that they or others develop evaluation questions or plans. While some collect quantitative or qualitative data, few design data collection tools. This pattern, also observed in the interviews, suggests that many programs use unsystematic evaluations or isolated activities rather than comprehensive evaluations.
A significant issue identified in the survey and interviews is the challenge CBD programs face in conducting evaluations with limited resources. Participants consistently noted that the financial and time commitments associated with CBD implementation exceed initial projections, leading to substantial challenges in program evaluation. The lack of additional resources and financial support has resulted in a “patchwork” approach to evaluation, where faculty members, often volunteers, are tasked with evaluation duties despite their already heavy workloads.
Evaluation theorists, including Brazil, 16 Cousins et al, 17 and Love,18,19 have long argued that resources and support are essential for effective program evaluation. Without these, evaluations are at high risk of being disorganized or nonexistent. These views are supported by researchers in medical education contexts, who emphasize that leaders in academic healthcare settings need resources, support, and intrinsic motivation to engage in evaluation, learn from its processes, and utilize its findings.9,20
The findings suggest that CBD programs would benefit from access to examples of high-quality program evaluations or adaptable evaluation templates (eg, for proposals, logic models, and evaluation matrices). 21 Alternatively, programs with overlapping interests could collaborate to share the costs of hiring trained program evaluators. Such evaluators, who would bring knowledge of Program Evaluation Standards could enhance evaluation quality in CBD programs. 22 They might also help programs implement approaches like the shoestring evaluation, which supports methodologically sound evaluations within budgetary and time constraints. 23
Interestingly, interviewees highlighted positive aspects of current evaluation practices, particularly the use of team-based formats. Many programs are leveraging collaborative efforts involving various stakeholders, including program leadership, residents, and administrative staff. They reported that this team-based format enhances the depth and quality of evaluations. Interuniversity collaborations and consultations with evaluation experts also contribute to a more comprehensive understanding of program performance. However, the variability in team involvement and collaboration suggests room for greater consistency and integration in these efforts. Program directors would benefit from additional training on, for example, participatory evaluation approaches. Successful participatory evaluations require collaboration between skilled evaluators and program stakeholders. 6 Stakeholders, who have in-depth knowledge of the program, work with evaluators to design the evaluation, collect data, analyze results, interpret findings, and disseminate information. 6 This approach enhances the evaluation process and fosters a greater sense of ownership and integration of evaluation practices into the organizational culture. 3 Patton highlights that involving stakeholders improves the relevance and clarity of findings, increasing their practical utility. 3 Strengthening program evaluation through best practices and sharing experiences across institutions could further improve evaluation processes.
Advocacy for clear expectations from regulatory bodies such as the RCPSC and PGME offices is also essential. Establishing explicit guidelines and standardized evaluation processes can provide programs with a clear roadmap, ensuring consistency and enhancing the overall quality of evaluations. As noted in the literature, a lack of policies and understanding of evaluation leads to varying levels of program evaluation. 24 Addressing these areas will help CBD programs build stronger evaluations that support continuous improvement and alignment with educational goals.
Limitations
Notwithstanding, this study has limitations. In Phase 1, the response rate for the study is low. We acknowledge that some of the program directors’ email systems may have classified the survey invitation as spam and as such, they did not participate. We also recognize that the topic may have been a low priority to some and, thus, they may have declined to participate. Moreover, since the survey was anonymous, we were unable to explore the potential of nonresponse bias. It is possible that those who did not respond were opposed to or uninterested in program evaluation. The survey relied on self-reports of evaluation activities that the respondents or other individuals in their programs engage in. A great number of respondents, even though they indicated that their program does engage in program evaluation, preferred not to answer certain items, raising validity concerns for some of the survey data. The survey also dealt with sensitive items regarding program directors’ perceptions of program evaluation. As such, respondents may have provided socially desirable responses and answered the questions more positively than is true or they may have indicated that they preferred not to answer.
In Phase 2, the limitations stem from the reliance on program directors’ self-reported perceptions during interviews. Participants might have again provided socially desirable responses due to our presence, aiming to present themselves and their interests more favorably. Additionally, the program directors who agreed to be interviewed might represent a specific subset and not the broader population of program directors, leading to potential selection bias. Consequently, there may be differences between the program directors who participated in Phase 2 and those who did not. It is likely that those who took part were more outspoken, actively engaged in their roles, and interested in program evaluation in medical education.
Conclusions
Through a survey and interviews of specialty/subspecialty program directors, this 2-phase study documented the extent to which CBD programs are engaging in program evaluation activities, the reasons why these programs are engaging in evaluation or not, how CBD programs are engaging in program evaluation, and how CBD programs can build their capacities to do program evaluation. While it confirmed that most CBD programs are doing program evaluation, the quality of those evaluations is questionable. There is a demand for program evaluator expertise to support medical education program evaluation and help CBD programs build their capacities to do program evaluation. If the value and potential of program evaluation are to be harnessed, it is evident that CBD programs need additional resources and support. To gain increased insights into the quality of these program evaluations, those engaged in the evaluation of CBD programs need to publish their evaluations and reflect on the approaches and methods used. Finally, there is a need to explore creative approaches to the evaluation of CBD programs. Such creativity in program evaluation may help CBD programs embrace an asset-based (rather than deficit-based) mindset to optimize the extent to which they are engaging in program evaluation activities.
Footnotes
List of Abbreviations
Author contributions
JM conceived the idea for this study, oversaw the day-to-day activities of the study, and drafted the manuscript. KE contributed to the design of the study and was a major contributor in writing the manuscript. KM contributed to the design of the study, facilitated data collection and was a major contributor in writing the manuscript. All authors read and approved the final version of the manuscript.
DECLARATION OF CONFLICTING INTERESTS
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethics
Before data collection, the study was approved by the University of Ottawa Research Ethics Board (S-01-21-6537).
Informed consent
Informed consent was obtained at the beginning of the survey and interviews and respondents were permitted to stop the survey or interview at any point.
