Abstract
Objectives
Engagement in research activities is a critical component of clinical residency training. It is vital to build research capacity of residents to help interpret evidence-based medicine and design quality improvement projects. A mixed methods study was conducted to assess the impact of a 1-day research training workshop conducted at Eastern Idaho Regional Medical Centre, Idaho in May 2022. The workshop was targeted to improve the research knowledge of current clinical residents of Internal Medicine and Family Medicine.
Methods
Workshop comprised of expert presentations, with assessment of difference in knowledge with a pretest and post-test. The sessions were organized around the core competencies of Institute of Medicine. Suggestions were also gathered from the audience. A pretest and post-test based on 13 questions was administered to the participants to assess change in research-related knowledge. Comments and suggestions of the participants were also recorded. Wilcoxon rank test was applied to determine statistical difference across each question and cumulative knowledge score. Conventional content analysis was applied to explore the comments and feedback.
Results
The mean score of participants improved across all 12 questions. Statistically significant results were observed for the questions about types of studies qualifying as qualitative research. The cumulative score of participants increased in the post-test from 8.57 to 9.35. The participants gained new knowledge (94.3%), and felt more comfortable in application of research methods (74.3%). Encouraging feedback was obtained from the audience. They stated that they had benefited from the workshop and felt more prepared and motivated to indulge in scholarly activities.
Conclusion
The study shows improvement in research-related knowledge of clinical residents attending a 1-day training workshop. We recommend inclusion of such workshops in the curriculum of residents for skill building and enhanced indulgence in research activities in order to prepare them as future leaders in quality improvement, health policy, and hospital administration.
Keywords
Introduction
Accreditation Council for Graduate Medical Education (ACGME) recommends participation of postgraduate trainees of all specialties, including residents and fellows in scholarly activity.1,4 It is essential to ensure not only evidence-based practice, 5 but also for capacity building of residents to be equipped with the skills required to be critical thinkers and future leaders in quality improvement, health policy, and hospital administration.6,7 During the recent COVID-19 pandemic it was widely recognized how research orientation and the ability to plan as well as critically appraise scientific literature empowers evidence-based decision making, advocacy, and guideline development. Furthermore, it is a critical action point in the career development of younger physicians, opening doors for further training and leadership in physician-scientist tracks, especially for those aiming to work at leading academic centers.8,11 It also improves resident satisfaction and dictates better patient care. 5
Individual residency programs have implemented structured curricula to increase participation of residents and fellows in scholarly activity leading to variable satisfaction and acceptance by end-users.12,17 Residency training programs usually incorporate scholarly activity in the form of research projects and publications, journal clubs, grand rounds, webinars, and service on professional committees. However, there are significant barriers to successful scholarly activity implementation recognized in existing literature, including inadequate dedicated time, supervision and mentorship, and resources. 18 Previous studies also found lack of resident interest as a major impediment, and varied attitudes lead to uncertain implementation of curricula. 19 Research capability is a significant area of need for younger physicians. It hinders application of standards of clinical practice and takes away opportunities to provide satisfactory patient care and implement quality improvement initiatives.
As a recently accredited Graduate Medical Education (GME) program at Eastern Idaho Regional Medical Center (EIRMC) for Internal Medicine (IM) and Family Medicine (FM) postgraduate training core faculty, hospital, and resident leadership have been working hard to establish a framework for action to enhance scholarly output. A resident-led research committee was created with representatives from all 3 years of residency and both programs of family and internal medicine as well as GME leadership and EIRMC quality improvement team. Residents are given priority in making decisions, in consultation with other senior members to ensure professionalism and appropriateness of initiatives. With strong, resident-driven momentum to establish a formal structure for research curriculum, a 3-pronged goal was identified: (i) to diversify educational opportunities available to residents through didactic sessions, workshops, or short-format conferences; (ii) to increase career potential by providing knowledge and media to design projects, write manuscripts, and publish in scientific journals; (iii) to increase resident participation in application of research toward quality improvement (QI) at hospital and system levels via QI committee participation, critical appraisal of QI interventions, and design of longitudinal programs at the institutional level.
With these goals in mind, the Research Committee at EIRMC organized a 1-day training workshop with instruction in the following areas: (i) developing a research proposal; (ii) critically appraising scientific literature; (iii) conducting literature reviews; (iv) organizing reference libraries and bibliography materials; and (v) utilizing qualitative study designs in clinical research and QI. The workshop consisted of a hybrid format of in-person and virtual presentations from nationally renowned experts. This study was designed to assess the workshop in (i) its effectiveness in improving participant knowledge of basic research concepts and existing resources and (ii) feasibility for scale-up as an annual event. These goals were created and implemented by consensus agreement of all members of the research committee, which include resident representatives, core faculty members, hospital quality improvement team, GME regional research director, and GME research coordinator. A framework targeting physician-scientist practice (described below) 20 was identified and core competencies salient to research curriculum studied. Subject experts were contacted for participation as expert presenters.
This workshop will be used as a pilot program for future scale-up and a baseline model for resident inclusion in curricular development. Overall goal was to inspire residents and provide opportunity to discuss impediments in the scholarly activities as well as ways to overcome them, and provide protected space and time to share ideas for new project startups as well as completion of ongoing projects. Using feedback from this workshop, the existing research curriculum will be modified to address resident needs, and introduce a regional annual event to allow residents to showcase their projects.
Methods
A 1-day training workshop was organized and conducted on May 27, 2022, at EIRMC. It was offered free of cost to the residents of FM (n = 11) and IM (n = 29). This study was reviewed by the internal HCA Healthcare Graduate Medical Education IRB and determined exempt. Informed consent was not deemed necessary as risks to participants were minimal.
Development of Session Guide
The overarching goal of the workshop was to prepare participants to carry out research activities independently. The sessions were designed to address major areas of relevance to medical residents. The framework we used is outlined by Institute of Medicine (IOM) as follows
20
:
Core competency 1: provide patient-centered care Core competency 2: teamwork and collaboration Core competency 3: employ evidence-based practice Core competency 4: apply quality improvement Core competency 5: utilize informatics
This is a wholistic framework targeting all aspects of physician-scientist practice, however for the purpose of this training workshop we addressed core competencies 2 to 5 to develop sessions that will help participants improve upon on skills necessary to conducting scholarly activities during residency training (Table 1). Field experts were contacted for each session and requested to contribute as facilitators for the workshop. The workshop had instructors from Brown University, Idaho State University, UTHealth San Antonio, Baylor College of Medicine, UTHealth Houston, Houston Methodist Hospital, and HCA Healthcare. The workshop was a blend of in-person and virtual sessions, augmented with hands-on exercises for enhanced learning of the participants.
Session outline of the workshop.
Measurement Tool
The effectiveness of the workshop was assessed via a mixed methods approach. For the quantitative analysis component, a self-administered online questionnaire (available as Supplemental material) was given to the participants (n = 34) at the beginning and end of the workshop. A small sampling frame was available depending on the currently enrolled residents in IM and FM programs. The sampling strategy therefore was nonprobability purposive sampling to include all available residents. The pretest was composed of 12 questions to assess residents’ knowledge related to research and scholarly activities. The questions were formulated by facilitators of the workshop. Each question was given a score of 1 for a correct response and 0 for an incorrect response. The same 12 questions were asked of the residents at the end of the workshop along with an opportunity to provide feedback and comments about the workshop. The mean cumulative score was calculated and compared for the pretest and post-tests. For the qualitative component structured feedback forms were provided to the participants. These contained categorical response questions and well as an open-ended section for comments. A total of 78 comments were received from the participants.
Statistical Analysis
Mean and standard deviations were determined for each question along with the cumulative score of the pretest and post-tests. Wilcoxon rank test was applied to determine statistical significance between the mean score of pretest and post-tests to assess knowledge change of the participants. A P value of <.05 was considered significant. Frequencies and proportions were calculated for the feedback given by the audience. Comments of the participants were analyzed using conventional content analysis and organized into themes, subthemes, and codes. Word clouds were generated. The quantitative and qualitative data were analyzed using Stata.v.13 and MAXQDA, respectively.
Results
Quantitative Analysis (n = 34)
Table 2 depicts the results of pretest and post-tests. The mean score of participants improved across all 12 questions however, statistically significant results were only observed for the question identifying types of studies that qualify as qualitative research (pretest: mean = 0.23, post-test: mean = 0.47, P value < .05). The cumulative mean score of participants also increased from the pretest to the post-test from 8.57 to 9.35 (P = .01). Figure 1 shows a visual representation of the difference in cumulative pretest and post-test scores as well as median scores. Although there exists a visually appreciable improvement in mean as well as median scores from pretest to post-test evaluation, the 95% confidence intervals (CIs) overlap.
Visual comparison of change in cumulative mean knowledge score (95% confidence interval [CI]).
Pretest and post-test results of the conference (complete questionnaire with response options available in Supplemental materials) (n = 34).
*Wilcoxon rank test was applied to test for significant difference between the pretest and post-test scores, as data was not normally distributed, and nonparametric tests are warranted.
**P value < .05 considered statistically significant.
Table 3 shows participant feedback around the conference. Participants reported feeling as though they gained new knowledge related to scholarly activities from the event (94.3%), felt more comfortable about doing research (74.3%) and applying newly learned knowledge (74.3%), and would recommend such workshops to others (85.3%).
Feedback of the participants for the conference (n = 34).
Qualitative Analysis/Workshop Evaluation by the Participants
The participants (n = 34) were asked to evaluate and comment about the workshop. Thematic content analysis of the comments (n = 78) revealed that participants appreciated the workshop and had benefitted from it. The major themes emerging from the comments were (a) the benefits of the workshop (n = 17 comments), (b) positive feedback about the speakers (n = 3 comments), and (c) recommendations for improving the workshop in future (n = 25 comments, Figure 2).
Themes and codes’ matrix from qualitative analysis (comments = 78, themes = 3, codes = 56).
Participants narrated that the workshop was very useful in learning about the available research resources, databases, reference managers, data analysis software, and the means to access them. They added that the sessions helped them learn the steps of systematically conducting a literature review and appraising scientific literature. Participants reported developing a better understanding of the approach to research in clinical practice as demonstrated by the following participant statements: “It got me thinking about how to get research started” (Participant 23). “I got some possible ideas for how to incorporate qualitative research into QI projects” (Participant 9).
The participants also felt more motivated to indulge in research scholarly activities, for example: “I am more interested in doing research now” (Participant 10). We received positive feedback about the speakers; “Great Presenters!” (Participant 28). “The guest speakers were very helpful” (Participant 30). The in-person speakers were especially admired by the participants as they could better connect with them.
The participants gave pertinent suggestions to improve the workshop in the future. Specifically, participants suggested making the workshop more interactive by including hands-on activities: “I think we would benefit from having residents present and talk about their research and experiences.” (Participant 16). Another suggestion was to have the guest speakers present their research and their driving factors for research. Additional suggestions for how to improve the workshop were diversifying topics and dedicating more time to discussing research fundamentals in further detail. Participants also had suggestions for optimizing the amount of time and participant retention/engagement by increasing the number of breaks in between the sessions thereby allowing time for the participants to absorb new knowledge and refresh themselves for the next topics (n = 5 comments). Another recommendation was to stretch the workshop over 2 days for better understanding of the topics: “Good information, but it was too much all at once.” (Participant 22) and “It was a long day in one chair, made it hard at times.” (Participant 11). Lastly, it was proposed to have all speakers in person—for better interaction and participation of the audience. The themes, codes matrix, and word cloud of the comments are presented in Figures 2 and 3, respectively.
Word cloud generated from participants’ comments (n = 78).
Discussion
Evaluation of this workshop has demonstrated positive results in improving research-related knowledge of the participants. Encouraging feedback was given by the participants to expand the workshop in future. This study is an indication that dedicated time for research workshops and similar activities (eg, research symposia, retreats, conferences, etc) may have a significant impact on perceived interest in scholarly activity in residency programs to meet the core competency of IOM. This training workshop was unique in that it was planned, prepared, and executed through resident leadership entirely. Administrative and advisory support was provided by program and GME leadership. This enabled resident peers to have a greater buy-in to the proposed format and to the potential recommendations of making it a recurring event. Future development and studies may include expanding the scope to include more residency programs for a more collaborative educational experience.
Previous literature shows the interventions that have had a significant impact on scholarly output by residents. A national survey of all FM programs listed in ACGME showed that “Resident recognition for scholarship” (odds ration [OR] = 2.2, 95% CI = 1.1-4.1), “Dedicated resident time for research” (OR = 2.3, 95% CI = 1.2-4.4), “Local Research Day” (OR = 2.5, 95% CI = 1.3-5.1), “Academic advancement linked to scholarship” (OR = 1.9, 95% CI = 0.9-3.9), and “Residency director performs research” (OR = 2.7, 95% CI = 1.4-5.1) were associated with participation by ≥50% of a program's residents in a research project. 21 Another retrospective cohort study points out the importance of having an annual research day (Evening of Scholarship), in which trainees presented their research work to a wider audience; participants of the research day were more likely to publish in the future (OR: 3.7, 95% CI = 2.5-5.6), and these residents had significantly more publications as compared to nonparticipants (OR: 4.7, 95% CI = 2.7-8.3). 12 In our study, participants shared this opinion of allowing current residents to showcase their research work to enhance scholarly output, which intuitively will encourage active residents to continue their engagement with ongoing projects and motivate others to initiate their own research and QI projects. This study also helped increase knowledge of residents with regard to the process of starting a new project and also the resources that are available from the program.
Another study, demonstrated that residents in programs having a structured research curriculum and research requirement had a significantly higher mean publication rate during and after residency (2.7 vs 0.4; P = .006). 14 Earlier reports also show other interventions that have proven efficacious to increase scholarly output in training institutions. Seehusen et al recommended induction of a point system which provides a benchmark for scholarly activity that fulfills criteria for graduation and awards/honors, as well as incentivizes residents to create a positive competitive atmosphere. 16 This coincides with the current momentum at EIRMC IM and FM residency programs with improvement in quality of scholarly output being targeted with multiple interventions. New hiring of a GME-Research Coordinator, creation of a Resident-led GME Research Committee, and ongoing efforts to develop a longitudinal research curriculum echo the residents’ enthusiasm to partake in scholarly activities. A new intervention at GME level will be an annual research conference as a scale-up from the current training workshop with a 3-pronged approach: (i) in-person presentations from subject experts related to specific research areas; (ii) poster and oral presentations by residents at EIRMC; and (iii) training workshop to jump start research projects.
To the best of our knowledge, this is the first attempt to scientifically assess the impact of a 1-day research training workshop for clinical residents in Idaho Falls, Idaho. The mixed methods study design adds robustness to results of the evaluation. The qualitative component assessed comments and suggestions of the participants and provided a roadmap for improvement of the workshop in the future. However, the study does not come without limitations. The sample size was small and as a result, the generalizability and ability to detect an effect are both limited. The sampling for this study was based on nonprobability purposive sampling, to gather feedback specifically from all available residents, however as the sample size was limited by the number of residents currently enrolled in internal and family medicine programs, the external validity of the results is limited. The questionnaire for pretest and post-test assessment was not pilot-tested, therefore internal validity may be limited. This questionnaire was validated on face value by the members of the research committee as a screening tool to help dictate a more robust structure of the follow-up workshops. Improvement in knowledge was demonstrated by the study, however the application of knowledge was not tested. The positive impact on residents’ knowledge in this study is the difference subjectively felt by the participants, which will need objective knowledge assessment for triangulation of data. A subsequent study following impact over time would also allow measurement of impact on resident attitude and practice leading to overall scholarly output. The qualitative analysis of participant responses showed that data saturation was not reached, however further data collection was limited by the number of respondents available. Nonetheless, the results are encouraging and the authors plan to improve the workshop based on suggestions of the participants. A resident-led research training workshop will be an annual event of the FM and IM residency training programs at EIRMC. For the second resident-led workshop, it will take place over 2 days and fundamental concepts of research will be taught and discussed more elaborately. The previously discussed results indicate that such a workshop will not only improve research knowledge and skills of clinical residents, but possibly also motivate residents to engage in scholarly activities.
Conclusion
This study demonstrates that resident-led workshops are feasible and can show improvement in research-related knowledge of clinical residents attending a 1-day research training workshop. It also signifies the importance of protected and dedicated research activities in increasing resident engagement and interest in scholarly activities. We recommend inclusion of such workshops in the curriculum of residents for skill building and enhanced indulgence in research activities in order to prepare them as future leaders in QI, health policy, and hospital administration. Such resident-driven initiatives may be rolled out in a formal fashion and the outlined curriculum can be added as a salient feature of all residency programs. Monitoring and evaluating the curriculum and assessing the impact will open up new avenues for QI and research with the goal to inculcate an atmosphere of scholarly activity and evidence-based practice.
Supplemental Material
sj-pdf-1-mde-10.1177_23821205231182043 - Supplemental material for Impact Evaluation of a Resident-Driven Research Training Workshop in Idaho: A Feasibility Study
Supplemental material, sj-pdf-1-mde-10.1177_23821205231182043 for Impact Evaluation of a Resident-Driven Research Training Workshop in Idaho: A Feasibility Study by Muhammad Arslan Khan, Meesha Iqbal, Trevi Ramirez Mancilla, John Grider, Justin Solomon and Katie Suleta in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgements
Our thanks to Min Ji Kwak and Angela Catic for participating in the conference and providing feedback on the initial draft of the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported (in whole or in part) by HCA Healthcare and/or an HCA Healthcare affiliated entity. The views expressed in this publication represent those of the author(s) and do not necessarily represent the official views of HCA Healthcare or any of its affiliated entities.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
