Abstract
Reliance on the apprenticeship model of education in the clerkship years of medical education persists despite concerns with variability in educational delivery and outcomes. Although many institutions are addressing this variability, there needs to be a clear and objective method to assess what is working. Evaluating these educational experiences is an essential component to ensure that students graduate prepared to enter residency. In 2014, A.T. Still University’s School of Osteopathic Medicine in Arizona (ATSU-SOMA) introduced a curricular change to address clerkship variability by implementing an online curricular component for the core clerkship courses in the third and fourth years of medical student education. Subsequently, a new structured and objective process to evaluate these courses was designed to improve student learning outcomes in the clerkship years. A Curriculum Year Three and Four Work Group was created to develop the new process for curricular evaluation of the clerkship courses. In the pilot phase of its implementation, described herein, the process fostered stakeholder participation and buy-in, enhanced communication of expectations, increased accountability in clerkship course design, and effectively employed objective evaluation tools in determining what curricular changes were needed. The Curriculum Year Three and Four Work Group continues to revise the tools and methods to enhance the efficiency of the evaluation process and to analyze whether recommended course revisions have improved student outcomes.
Background
Reliance on the apprenticeship model of education in the clerkship years of medical education persists despite concerns with variability in educational delivery and outcomes. Although calls for change are occurring, they are slow. 1 The primary goal of the learning environment in these 2 years is working with students to practice situated cognition, in which students learn to apply what they previously learned in the classroom to actual patient care under the guidance of experienced physicians. 2 A downside of this apprenticeship model is that student learning during this time tends to be inconsistent. Some factors may be the variety of physicians with whom students spend time and the variability in training, practice, and teaching patterns.2–4 Another factor is that students’ didactic exposure in the third and fourth years is often dependent on the educational sessions and patient cases available at their rotation site. Furthermore, accurate assessment of student clinical competence requires a multifaceted approach that extends beyond the use of activity logs 5 and preceptor evaluations, 6 especially because these types of evaluations fail to differentiate between learner abilities. 7 The Flexner Report in 1910 and the subsequent Carnegie Foundation report in 2010 both emphasize the importance of standardization and integration of formal and clinical learning. 8
Context
To address these concerns and to generally improve student outcomes on national board examinations, in 2014, A.T. Still University’s School of Osteopathic Medicine in Arizona (ATSU-SOMA) decided to enhance its educational offerings by adding weekly, required coursework via an online platform in the third and fourth years. The 11 core clerkships for which online content was designed included Family Medicine, Internal Medicine, Pediatrics, Obstetrics and Gynecology, General Surgery, Psychiatry, Neurology, Cardiology, Critical Care, Emergency Medicine, and Osteopathic Principles and Practice. The Clerkship Course Directors (CCDs) for these clerkship courses created the learning content and delivered it through the school’s learning management system (LMS), Blackboard, so all students could easily access it from locations across the country and internationally. This was the primary responsibility of the CCDs. Unlike other institutions, 9 at ATSU-SOMA, preceptor oversight and development does not fall under the purview of the CCDs and instead is the responsibility of the Regional Directors of Medical Education, who work directly with preceptors in furthering their development as clinical educators. To help close the loop of communication, the CCDs created short written guides for preceptors to better support the online course curricula and increase consistency across sites.
Problem
After the first year CCDs taught these new clerkship courses, ATSU-SOMA realized that there was a marked variation between courses and that the Curriculum Committee needed a systematic method to evaluate and document course effectiveness. The Curriculum Committee typically interviewed the CCD and reviewed data points of interest for the course, such as examination scores. However, there was no way for the Curriculum Committee to systematically and objectively rate the data and discussion points to define minimum expectations and determine what improvements the CCDs might need to make. Thus, we saw the need to create a more robust and documented evaluation system to decrease ambiguity for both CCDs and the Curriculum Committee.
This article describes the innovative, structured, and objective process that ATSU-SOMA created in 2016 to evaluate the core clerkship courses and details how this work led to enhanced communication, encouraged stakeholder engagement, increased accountability of course directors, and effectively employed objective evaluation tools to determine what curricular changes were needed.
Creating the Evaluation Process
Forming Curriculum Committee work groups
Curriculum committees at most educational institutions generally maintain oversight of the curriculum and usually include administrators and faculty who have critical roles in the planning, creation, and delivery of curricular content. 10 After noting how other institutions effectively reorganized their curriculum committees, 11 ATSU-SOMA formed working groups of the Curriculum Committee, based on the experiences of these other institutions, to subdivide the work of curriculum review to allow for a deeper evaluation of different academic years and assessment of specific tasks by those with pertinent experience. The Curriculum Committee created a Curriculum Year One and Two Work Group and a Curriculum Year Three and Four Work Group to evaluate the curriculum in those respective years.
Evaluation of the clerkship courses was a collaborative process between the CCD and the Curriculum Year Three and Four Work Group (hereafter referred to as the Work Group), and the Curriculum Committee reviewed and approved any final recommended changes. The Curriculum Committee purposefully selected Work Group members who had varied roles, perspectives, and levels of experience to ensure that the clerkship evaluation process addressed the diverse needs and concerns of CCDs, administrators, and students. The formation of these work groups allowed faculty and other key stakeholders who did not hold formal seats on the Curriculum Committee to have a say in the process of curriculum evaluation.
The chair of the Work Group was a clinical faculty representative of the Curriculum Committee who also happened to be a CCD. Members of the Work Group included 3 clinical faculty who were also CCDs of clerkship courses, 1 basic science faculty who had experience as a course director for first-year and second-year courses, 1 osteopathic medicine faculty who could evaluate the integration of osteopathic principles in each of the courses, the director of the Office of Evaluation and Effectiveness (assessment) who had analytic experience in developing methods that ensured usable outcomes, the staff member who facilitated delivery of course materials, and the assistant and associate deans of Clinical Education who had oversight of the third- and fourth-year curriculum. The Work Group also included the associate dean of Curricular Integration as an ex officio member who needed to stay informed of the Work Group’s progress. Like other institutions that have created teams or focus groups of students to obtain input on the curriculum, 12 ATSU-SOMA recognizes the value of student input in curricular reform and has used student focus groups for feedback of courses from the first and second years. The chair of the Work Group considered adding a student to the team but found that students on the Curriculum Committee were rarely available to attend or contribute because of academic obligations, and focus groups are more challenging to form during the third and fourth years when education is so widely distributed at ATSU-SOMA sites across the country. Thus, the Work Group used components of the course feedback survey from students as the primary and most efficient source of student input (Table 1). At this early stage of development, the ultimate stakeholders, the patients, were not involved in the initial course evaluation process, but we hope to include patient feedback in the future.
Student evaluation of the rotation (course evaluation).
LMS, learning management system.
The overall results of student feedback determined the rating assigned for the first item in the Clerkship Course Evaluation rubric.
Creating a timeline and communication plan
At the beginning, it took 6 months to develop the course evaluation process. Creating an annual project timeline for reviewing and revising the clerkship courses was also essential and ensured that the Work Group would meet the project’s goals and deadlines each year (Figure 1).
Annual timeline for clerkship course evaluations.
The group used backward planning to ensure it would meet target dates. During the period for course reviews from July to December, the group met for 2 hours every other week to evaluate 11 clerkship courses. The role of the CCD for each course typically accounted for 0.1 to 0.2 full-time equivalent depending on the duration and complexity of the course they were developing. Because this was not their sole teaching responsibility, CCDs typically needed several weeks to months to make course changes in this early phase of curricular development.
To minimize surprises and disruptions to usual workload and schedules and to increase the willingness to make changes, the Work Group determined that all persons involved or impacted by the evaluation process needed to know what to expect. Therefore, to ensure clear and accurate communication of the course evaluation process, the chair created a document that described the committee’s process for curricular evaluation of the clerkship courses and shared it with all CCDs, curriculum committee members, and administrators. This document provided clarity by describing the step-by-step process to evaluate clerkship courses and indicated who the responsible parties were for each step in the process.
Throughout the clerkship course review process, the Work Group documented and shared the notes from each weekly meeting with group members. A rapid editorial and dissemination procedure allowed for accurate recall of the discussions, and any questions or concerns could be addressed before the process was complete. With such a wide representation of stakeholders on the committee, web conferencing was essential for equal participation of those located on or off campus and broadened our pool of potential input.
Identifying necessary data
The Work Group and CCDs worked collaboratively to define what data were needed for the evaluation process. At ATSU-SOMA, because it takes a full academic year for an entire class of students to complete a given clerkship course, a full set of performance data and student feedback results for an individual course was not available until July of each academic year. To compensate for this delay of information, the Work Group provided mid-year course data packets to CCDs in January to allow them to assess student progress mid-year (Figure 1). The course data packets included the following reports for each clerkship course:
Clerkship course final grade distribution histogram;
Histogram of clerkship national subject examination performance;
Histogram for weekly coursework scores delineated by week;
Summative student feedback with comments from the Student Evaluation of the Rotation form;
Histogram of Level 2 national board examination scores with discipline subscores;
Anonymous student feedback comments.
Depending on the goals and interests of any given institution, different data sets may need to be included in the review packet to capture each school’s unique programs.
Developing the course evaluation process and evaluation tools
Establishing and codifying the evaluation processes ensured clear communication of expectations for all involved. The clerkship course evaluation process included reviewing essential information about each course and CCD commentary about his or her course. We decided that a course rating evaluation system based on predetermined objective criteria was necessary.
When creating the evaluation tools, we determined that they needed to be collaborative to ensure that expectations were clear between the CCDs being evaluated and the evaluators. The purpose of the evaluation tools was to gain insight into the course, answer key questions about course design and outcomes, and provide objective means of evaluation. Using Google Forms, the Work Group created and distributed a questionnaire for CCDs to complete (Table 2).
Questionnaire for Clerkship Course Directors.
CRE, clinical rotation evaluation; GI, gastrointestinal; RDME, Regional Director of Medical Education; SER, student evaluation of the rotation; SOMA, School of Osteopathic Medicine in Arizona.
The goal of the questionnaire was to gain insight into the course development process of the CCDs. The questionnaire included several items about course design, such as CCD goals for the course, methods of communication and feedback to students, methods of student assessment, interpretation of the course outcomes, and planned changes for the future. While creating the questionnaire, group members were especially interested in the CCDs’ impressions of the questions and worked to revise those that created ambiguity, such as the differentiation of course goals from programmatic competencies.
The questionnaire represents a portion of what ATSU-SOMA wanted to evaluate in this given year; however, the questionnaire could easily be modified to meet the unique goals and interests of a given institution.
Next, the Work Group created a course evaluation rubric to evaluate courses consistently and objectively (Table 3). Like the questionnaire, the rubric includes a portion of what ATSU-SOMA wanted to evaluate for this particular year and could be modified as needed.
Clerkship Course Evaluation Rubric.
CCDs, Clerkship Course Directors; LMS, learning management system; SER, student evaluation of the rotation; SOMA, School of Osteopathic Medicine in Arizona.
Identifying benchmarks in rubric format had not been previously performed in curricular reviews, resulting in inconsistent evaluations between courses. During the first year of the evaluation process, therefore, the intention of the rubric was to ensure a basic level of consistency between courses rather than to evaluate the academic content of the courses. The rating level of “exceeds expectations” was included in the rubric to encourage clerkship directors to set goals above the minimum expectations and stimulate innovation. Although evaluation categories might vary from year to year depending on the specific goals and needs of the course reviews, we felt it was important to gather baseline information about course goals and objectives, structural design, and student feedback during this pilot phase of using this new evaluation process.
Complete evaluation of courses required the Work Group to review questionnaire responses of the CCDs, course performance data, and course documents provided to the students through the LMS. After reviewing this information, the Work Group conducted live meetings with each CCD to discuss the information and ask questions about the clerkship course curriculum. Scheduled meetings adhered to the following process:
CCD gave a general overview of the intention for the curriculum and the process used in planning the course;
CCD provided suggestions for recommended changes for the next year;
Work Group members asked questions of the CCD as needed;
Work Group members shared and discussed their rubric ratings;
Work Group members made suggestions for changes to the clerkship course curriculum, which may have included or excluded recommended changes from the CCD.
Course evaluations took place from July to December (Figure 1). During the first year of evaluations using this new process, meetings lasted 1 hour per course. In the next year, the time was reduced as the Work Group and CCDs were familiar with the process and asked fewer questions. In some cases, no meeting was needed with specific CCDs because there were no apparent deficiencies and the CCD provided sufficient information in the questionnaire, course data packet, and other course materials.
At the end of each course review meeting, the Work Group and CCD discussed the proposed recommended course changes. After all courses were reviewed, Work Group members voted on all the proposed course changes. Waiting to decide on final recommended changes to each course allowed for analysis of patterns between courses and for recommendations to focus on the best practices. These recommendations were subsequently shared with the Curriculum Committee, which reviewed and modified them as necessary. The Curriculum Committee’s final recommendations were then shared with each of the CCDs in January to allow time to make course changes before the start of the next academic year in July (Figure 1).
Documenting the process and findings
Google Forms and Google Sheets were essential for keeping track of the Work Group responses and for thoroughly documenting procedures and recommendations for the Curriculum Committee, especially when questions arose. Google Forms were used to collect the ratings of each of the rubric items for individual Work Group members. This method ensured that decision-making contributions were made by all members of the group rather than using group discussion of each item in the rubric where input from those who were less vocal or less opinionated may not have been accounted for. During the first evaluation year, all Work Group members rated all of the rubric categories, and an average score was calculated for each category. In the second year, to reduce the time commitment, each group member worked on an average of 2 items in the rubric that were evaluated across courses. The chair selected qualified evaluators for each item and allowed Work Group members to provide additional commentary on other rubric items.
Google Sheets were used to track progress and improvement from one year to the next and to share information with multiple individuals using a live document. This method of information sharing was especially beneficial in the case of group member turnover and for keeping the Curriculum Committee and administrators informed of progress. Each course had its own tab containing the evaluation rubric in the Google Sheet and included a column for CCD comments and a column for the Work Group members’ comments. The Work Group created a separate Google Sheet to keep a summary of discussions from each meeting and record attendance. The group’s decision-making process was democratic and decisions made were kept in a separate tab in the Google Sheet with dates and times.
Impact of the Clerkship Course Evaluation Process and Limitations
During the pilot phase of this new process, the holistic view the Work Group gained from this innovative clerkship course evaluation process (Figure 2) indicated that much work remains to be done to improve the clerkship courses. In particular, the responses to the questionnaires for CCDs revealed that several CCDs needed additional guidance in course design and pedagogy. Thus, the Work Group created a document that detailed clerkship course design and content expectations for CCDs as a basic guide to structuring clerkship courses. A key feature of this document was to guide CCDs in creating assignments with SMART (specific, measurable, achievable, relevant, and time bound) design structure. Feedback during group CCD meetings indicated that CCDs appreciated the clarity of expectations and guidelines to follow for their courses. As a result of this process, the Work Group noted that improvements were made in course design and student assessment methods, and CCDs reported receiving fewer student complaints the next year.
Steps in creating the evaluation process.
The evaluation process also revealed that additional work on curriculum mapping for the clerkship courses needs to occur at ATSU-SOMA. Currently, our mapping involves using the 7 osteopathic core competency domains and includes 33 competencies we have defined as SOMA curricular goals. When we looked at which SOMA curricular goals were addressed by all parts of the clerkship curricula, we identified 1 goal of the 33 goals that was not addressed; the missing goal was in the area of systems-based practice. Furthermore, future steps will likely involve investigating how our curricula address the Entrustable Professional Activities of undergraduate medical education. In addition, we are assessing how clerkship courses address ATSU’s Core Professional Attributes, which are the unifying competencies expected of all graduates of our university regardless of degree program. 13
Another positive outcome of using this evaluation process was that CCDs considered more deeply what they were trying to accomplish in their clerkship curriculum. The process made them more accountable for their courses and the potential impact on student educational outcomes. This accountability also enhanced faculty collaboration because CCDs became more engaged in discussions with their peers about course design and online student assessment methods.
One of the most important benefits of using a working group for our clerkship course evaluation process included the buy-in from an array of stakeholders, which ensured the success of the process. Furthermore, the collaboration between the Work Group and CCDs led to other improvements in the overall structure of the clerkship curriculum, including necessary adjustments to the global clerkship grading scales, revision of preceptor evaluation forms, and enhancement of the student rotation feedback surveys.
We did encounter challenges during the process. For example, a few Curriculum Committee members, who were not in the Work Group, may not have read the Google Sheets and meeting notes and had questions about the clerkships when we presented our findings to the Curriculum Committee. These discussions resulted in their offering solutions without having the Work Group’s earlier input. In addition, the Work Group was not as empowered in evaluation and decision-making as it had initially presumed, and the Curriculum Committee’s additional recommendations for change came as a surprise to the CCDs, resulting in CCD dissatisfaction. In the future, the Work Group will need the explicit support of the Curriculum Committee to effectively conduct the tasks asked of it.
To ensure that repeated cycles of curricular evaluation adequately determine the effectiveness of the process, commitment from the group members and the CCDs is essential. At ATSU-SOMA, we were pleased to have 100% participation of the CCDs; each CCD had a vested interest in seeing the curriculum and, ultimately, student outcomes improve. However, the nature of academia, with its frequently shifting roles, responsibilities, and changes in leadership, places newly implemented changes at risk of dissolution.
Overall, ATSU-SOMA’s new curriculum evaluation process enhanced communication, stakeholder engagement, and accountability of course directors; and it effectively employed objective evaluation tools to determine what curricular changes were needed. Ideally, the Curriculum Year One and Two Work Group will successfully adopt and implement a similar evaluation process for their classroom-based curriculum in the next academic year. What is clear is that for sustained, robust curricular change to occur, the CCDs, department heads, and deans need to support and direct a rigorous course review process, such as that described in this article.
Subsequent evaluation and research at ATSU-SOMA is ongoing and will involve assessing the impact this process has had on student learning outcomes by monitoring score improvements on national board examinations, subject examinations, and preceptor evaluations of student performance. Other enhancements being considered are the use of student focus groups to gain deeper insight from students about how the clerkship course curricula can be improved. Ultimately, solicitation of patient feedback and input would also be included in this process. Furthermore, the Work Group will revise the course evaluation rubric each year to address evolving areas of needed evaluation. We hope that having described our process will enable other institutions to develop and revise methods of evaluating their course curriculum.
Footnotes
Acknowledgements
The author wishes to thank the members of the Curriculum Year Three and Four Work Group for creating the process described in this article: Lisa Watts, DO; Valerie Sheridan, DO; Marjorie Buick Kinney, MEd; Milton Pong, PhD; Jay Crutchfield, MD; Aaron Allgood, DO; Greg Hollick, DO; and Nichole Goldman. The author additionally acknowledges Clerkship Course Directors: Susan Steffans, DO; Grace Stewart, MD; Harvey Simon, MD; Robert Coppola, MD; and Eve Ashby, MD, for participating in the course evaluation process. Finally, the author acknowledges the ATSU-SOMA Curriculum Committee for their forward thinking in the creation of curriculum work groups.
Funding:
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
KG is responsible for writing and revising the entirety of the manuscript. KG conceived the concepts for the article, wrote the first draft and made critical revision to the article. The author reviewed and approved of the final manuscript.
Ethical Approval
This study was approved by Mesa Institutional Review Board of A.T. Still University on April 4, 2018 (IRB protocol no. 2018-083).
