Abstract
Background:
Health care research is a common undergraduate health sciences requirement. There is limited literature regarding course structure, content, or learning outcomes; most courses have traditionally been taught through didactic lecture. This is misaligned with Generation Y learner values, as they desire guided learning, real-world examples, active engagement, learning through doing, and psychological safety.
Methods:
A “journal club” approach to teaching health care research was implemented at Northeastern University in Fall 2018. Each session involved (1) a moment of reflection; (2) an introduction to the topic; (3) 1 student methods report presentation; (4) 2 student “journal club” self-directed structured article summary presentations; (5) large-group discussion; (6) plus/delta feedback to instructor. Each student completed 2 “journal club” and 1 methods presentations, 6 peer reviews, CITI research training, a quality improvement survey, and a final course reflection. We utilized a convergent mixed-methods educational evaluation, integrating data from 3 distinct sources—a quality improvement survey, final student course reflections, and Plus/Delta feedback—which were analyzed via thematic analysis. The Northeastern University Institutional Review Board exempted the study.
Results:
Students appreciated the course structure and reported confidence in their critical appraisal abilities. Four qualitative themes emerged: (1) enabled a high degree of growth as students and scholars; (2) designed in thoughtful and unique format; (3) initially intimidated students and was academically challenging; and (4) prioritized and enabled psychological safety.
Conclusions:
Although initially intimidating and admittedly challenging, undergraduate health sciences students applauded the course’s curricular design and enabling of psychological safety, which aligned with Generation Y learner values, ultimately leading to growth in perceived and realized confidence and ability to critically review research articles.
Introduction
Health care research is a common graduation requirement for undergraduate health sciences or nursing students who will enter graduate allied health programs. 1 Nearly 70% of senior medical students have received research training, which have traditionally been taught through didactic lecture, more recently with PowerPoint slides.2-4 The didactic teaching style focuses on transferring content from teacher to learner and supports only surface knowledge, minimal retention of learning, and in some cases, rote memorization. 1
Prior health care research course designs have utilized workbooks or templates to scaffold learning and promote critical reasoning skills.5-7 Small group work is commonly used which allows for peer feedback and active participation in discussion.2,8 Some have implemented approaches to group work such as the Jigsaw Technique, a collaborative approach in which each student’s grade depends on the other’s performance, or peer mentoring and tutoring to facilitate active participation and discussion.8,9 More innovative forms of teaching have included experiential learning which encourages the students to draw from and reflect on their own experiences. 10 These prior studies have used quantitative quiz scores, in-class feedback, and end-of-course learner satisfaction surveys or reflections to gather feedback from students about course structure, format, and learning outcomes.5-7,9-11
Importantly, journals clubs are an established method for increasing exposure to research methods and supporting critical appraisal skills in medical school, residency, and fellowship.12-16 However, undergraduate students are rarely introduced to this format prior to advanced medical training, suggesting they are unprepared for the level of baseline research knowledge and critical appraisal skills necessary to participate in increasingly complex research conversations. 3 Moreover, the journal club approach has not been adequately applied or studied as part of undergraduate research training, and it is unclear whether the approach effectively prepares students to understand research fundamentals with the intention for them to competently enter graduate allied health programs.7,17
Moreover, generation Y students—born after 1980—prefer more active learning, often including instructor guidance, active engagement with others, learning through doing, and course material related to real-world experiences.2,18,19 They also desire a strong classroom structure with consistent psychological safety.2,18,20
We developed a convergent mixed-methods educational evaluation for a journal club approach to teaching undergraduate health care research to Generation Y students, complemented by multiple key educational design strategies to support deep learning. We add to the literature by illuminating students’ perspectives on this approach, detailing their perceived educational impact on their understanding of and confidence with research methods via the use of a quality improvement survey, a final course reflection, and Plus/Delta feedback.
Methods
Setting and participants
All participating students were upper-level students in the Health Science major (550 total students) at the Bouvé College of Health Sciences (2000 total students) at Northeastern University in Boston, MA. In Fall of 2018, the senior author—a medical and health professions educator—taught 2 sections of Health Care Research at Northeastern University. The morning course had 16 students while the afternoon course had 24 students (total n = 39). All classrooms contained a computer with a ceiling mounted projector and screen that enabled PowerPoint presentations, as well as a chalkboard.
Curriculum design
Kern’s 6-step curriculum development process guided curriculum development. 21 Initially, the first author reviewed prior literature regarding how health care research is taught and spoke with other Northeastern University faculty members about their perceptions of what the students needed to be successful.
The instructor embedded multiple core cognitive science and adult learning principles in the course design, as outlined below.21-23 First, students were consistently informed and reminded that research methods would be essential for their future careers as health professionals, and each course session included a “who is this?” component to introduce students to practicing health care researchers. Second, although journal club assignments were scaffolded by the instructor who provided a clear template for students, significant self-directed learning and autonomy among students was required. 15 Third, the instructor encouraged students to draw on their prior learning experiences in research and statistics. Fourth, readiness to learn was supported by the use of articles of increasing complexity throughout the course so that students had to continually revisit complex course concepts. Fifth, students’ motivation to learn was encouraged through the development and maintenance of a “safe learning space,” in which students consistently asked questions as well as provided and received feedback from the instructor and engaged in peer review. The intention was to cultivate a desire among all students to complete assignments at the highest level of their ability. Sixth, psychological safety was additionally encouraged via the use of a minute at the beginning of class to hit pause and “situate ourselves in our learning space” to remind students to actively choose to be engaged in their own learning for the session. 23 Finally, the journal club design enabled active learning, largely through large group discussion with student-led questions and supported spaced repetition through the interleaving of main concepts.
Course structure
The courses spanned 14 weeks, with 2 sessions per week at 140 minutes, amounting to 23 course sessions. Students were assigned a research article, and utilized the “self-directed structured summary” template to develop a 1-page summary. 15 This summary included the introduction, methods, results, and the conclusion, highlighting challenges or concerns students noted (Appendix 1). The students presented study findings to the instructor and their peers, using PowerPoint presentations. Students were encouraged to make presentations fun, interesting, and engaging. They were also responsible for leading discussion about the journal article after their presentation, with input, guidance, and questions from the instructor. Articles varied in scope and methodology, and topics were sequentially organized by complexity to align with the articles for the day’s session. The Journal Club Implementation Flowchart details the process by which the instructor and students navigate each course session (Figure 1).
Journal Club Implementation Flowchart.
The sessions were organized as follows: situate self in learning space, the day’s outline, plus/delta review from the previous course session, student methods report presentation, introduction to the day’s topic, 3-minute break, student journal club structured summary and presentation #1, “Who is this?,” student journal club structured summary and presentation #2, and completion of the plus/delta for the course session. On 4 sessions, guest speakers with expertise provided the introduction to the day’s topic, as well as served as moderators and discussants for the student journal club structured summary and presentations.
All students completed 6 methods reports or journal club structured summary and presentation peer reviews throughout the semester. They offered feedback in 3 areas: “What the presenter did well,” “What the presenter could have done better,” and “What additional comments do you have for the presenter?” To protect student confidentiality, there were 2 peer reviews for every presentation. The peer reviews were reviewed by the teaching assistant for appropriateness and then aggregated, blinded, and sent back to students as part of formative feedback. Students were instructed in subsequent presentations to indicate to the class 1 way they incorporated this feedback into their presentation.
Most assignments were completed independently, although a small number of students in the afternoon class completed 1 methods report presentation or journal club structured summary and presentation in teams of 2 due to a slight excess of students compared with course topics.
Course grading
Course grading was on a 1- to 100-point scale, with 1 methods report presentation (20 points), 2 journal club structured summaries and presentations (40 points), 6 peer reviews (15 points), course engagement (15 points), CITI training (5 points), and a final course reflection (5 points). After each journal club structured summary and methods report presentation, students were emailed a rubric with their grade and written formative feedback from both the instructor and their peer reviewers. The rubrics detailed whether learners followed citation and formatting instructions, clearly explained topics, and related topics to prior course content or readings. Journal club rubrics specifically indicated whether students were accurate in their interpretation of the research question, design, methods, data analysis, results, and conclusions. At the course midpoint, students were given a grade update, with their raw score as well as a scaled percentage.
Data collection
As part of routine educational evaluation and quality improvement, multiple sources of course evaluation data were collected by the instructor for educational program evaluation. These data sources included the following:
Anonymous, voluntary, final quantitative course quality improvement survey developed by the instructor and collected as part of routine end of the semester educational evaluation.
Deidentified, final 1-page course reflection assignments in which students reflected on their learning throughout the semester by drafting a 1- to 2-page reflection paper which connects course content and learning throughout the course.
Anonymous Plus/Delta data completed by all students after every class session to inform educational quality improvement throughout the semester. Students were asked to take 1 minute at the end of each session to answer 2 simple questions: (1) what went well, and (2) what could have gone better. The instructor reviewed and summarized this feedback and presented it to all students at the beginning of the next session. When possible, the instructor made small iterative changes to improve student experience. When changes could not be made, the instructor was transparent as to why.
This project was reviewed by Northeastern University Human Subjects Research Protection and exempted from further review.
Data analysis
This mixed-methods convergent educational evaluation utilizes the quantitative quality improvement survey, qualitative final course reflections, and Plus/Delta feedback as data sources. Our purpose was to understand how the course design was experienced by students, as well understand the perceived educational impact on their understanding of and confidence with research methods.
Descriptive statistics were obtained for the quality improvement survey. Plus/Delta feedback were collated and reviewed to determine key recommendations from learners. Thematic analysis following the 5 stages to qualitative research framework was utilized for the qualitative final course reflections. 24 Dedoose (QSR International Inc, Burlington, MA) was used to facilitate data management while coding. First, 2 primary coders (K.D. and M.F.) familiarized themselves with the data, each independently reviewing the first 5 reflections to create a preliminary list of codes. They compared codes and refined definitions as a team with discrepancies reconciled in person by both coders until the codebook was finalized and entered into Dedoose. Interrater reliability between the coders was established using 5 transcripts at random with a 0.75 kappa indicating good to excellent interrater reliability. Subsequently, codes were systematically applied to all reflections. After independently coding all reflections, the 2 coders discussed recurring patterns in the data and codes were combined into categories and then themes—a cluster of codes that, when combined, provided a meaningful statement about students experience in the course. The coders independently reread all coded data within each theme to ensure consistency and identify illustrative quotations. The authors then considered how the 3 data sources aligned and integrated the finding in the results to meaningfully report the impact of the course on students.
To confirm findings, a student who took the class served as a member checker by critically reviewing the manuscript prior to journal submission and providing feedback which was incorporated into the final version. The Northeastern University Institutional Review Board exempted the study.
Results
Thirty-nine students completed the quality improvement survey (97.5%) and 40 completed the final course reflection (100%). All were expected to complete the plus/delta data anonymously after each session. Table 1 presents the quality improvement survey results. These results, as well as the Plus/Delta feedback, were integrated within the 4 themes constructed from the qualitative analysis of the final course reflection (Table 2) for results reporting.
Quality improvement survey results.
Note: Thirty-nine students completed the quality improvement survey. Statements were rated on a 5-point Likert-type scale ranging from strongly disagree (1) to strongly agree (5).
Selected Illustrative Quotes by Theme.
Enabled a high degree of growth as students and scholars
Students noted a high degree of perceived growth as students and scholars. They reported gaining an appreciation for research through the course and indicated that they now knew how important it was to be a critical reader of the literature. They indicated feeling that they grew as scholars and gained academic confidence, in part through improvements in both oral and written communication. The journal club self-directed structured summary provided an instructor-scaffolded approach that allowed the student to “chunk” the article in smaller sections, thus enabling them to understand the article as a whole. This activity required significant self-directed learning and was positively received by students. Overall, students felt enabled to read research articles and understand research methods, which they attributed in part to the methods report presentations by students during each session. This was supported by responses to multiple survey questions, which indicated that self-directed structured summaries, methods reports, and presentations were an efficacious way to teach critical appraisal. Importantly, students reported utilizing skills learned in the course in another course that term, or expectations that what they learned would be helpful in future classes or training.
Designed in thoughtful and unique format
Throughout their reflections, students offered praise for the thoughtful and unique course design, which they experienced as novel and greatly appreciated. Students found the mindfulness exercise at the beginning of each class refreshing and appreciated the opportunity reset from a busy day and refocus energy on learning. Students were highly supportive of the multiple opportunities for class discussion and felt comfortable engaging with their instructor and peers. Guest speakers were welcomed and students found them knowledgeable, inspiring, and helpful to expand their breadth of learning. The plus/delta activity at the end of each course to obtain student feedback was perceived as respectful, especially when changes were implemented during the next session. The consistent session structure was comfortable to students who also noted finding the varied course activities and variety of topics stimulating. Finally—as was also indicated in the survey—the utilization of peer teaching and learning including peer review was valued. In many cases, students incorporated this feedback into future assignments.
Initially intimidated students and was academically challenging
Students admitted to entering the course with a high degree of apprehension and anxiety about the topic and did not feel prepared by prior courses. They expected to be bored and uninterested. They noted feeling overwhelmed by the syllabus and apprehensive about the multiple required presentations. Once the course was underway, they felt academically challenged by the course and assignments, and felt the level of work required for the course was appropriate. Some disliked the methods report presentations and felt it was of lower educational value than other course activities.
Prioritized and enabled psychological safety
Students reported not being “stressed out” as a function of the course through the safe, welcoming, and comfortable learning environment created and sustained throughout the semester. They felt they could learn and grow and that the instructor and, subsequently, their peers were invested in their academic growth. They were comfortable engaging in discussion and noted doing so more than in other courses. Plus/Delta feedback was consistently received and was transparent—for example, offering instances when articles were too complex to facilitate learning, or times when the instructor may have been more effective in explaining key concepts. When possible, this immediate feedback informed iterative course changes, such as implementation of a 3 minute course break for restroom use and reduction in number of course readings. Overall, feedback was overwhelmingly positive, with students indicating that the instructor was high-energy and invested in their growth, and that their time and input as students was respected. These findings were supported by survey results, in which nearly all students reported the course being a positive learning experience worth their time and effort. Importantly—and further evidence of the psychological safety enabled through the course—students noted that future improvements could include a wider variety of in-class activities, inclusion of a group project, more integration between the methods report and journal club structured summaries, and additional engagement through discussion and the limiting of student laptop use.
When considered in the aggregate, the 3 data sources converge and highlight the perceived positive impact of the course on learners, who found it challenging yet noted that the class was worth their time and effort. Moreover, they ultimately developed great respect for the instructor and the course design, and would recommend it to a friend in the same major. Perhaps most importantly, they both reported and were proud of their own subsequent resultant growth as scholars and researchers.
Discussion
We describe the development, implementation, and evaluation of a undergraduate health care research course utilizing a journal club format which Generation Y students indicated allowed them to better understand research articles and improve as scholars. Psychological safety was reinforced consistently and students expressed comfort in and out of the classroom that led to their overall success in the course. Students also felt appreciated throughout the course, which in turn encouraged them to be more involved and therefore they were able to reach a higher level of learning. Students also greatly appreciated that the instructor maintained a consistent structure of the course while still offering varied activities and covering diverse topics in each session. Importantly, students noted that improvement was possible and gave multiple recommendations for how that could be achieved.
Importantly, in the United States, the Accreditation Council for Graduate Medical Education mandates that programs support advancement in resident knowledge of basic scientific inquiry principles, including how research is designed, conducted, assessed, explained to patients, and applied to patient care. 25 Those trainees who have a strong undergraduate research experience may be more likely to readily engage in research and quality improvement projects as trainees, and may be more equipped to encourage critical thinking and evidence-based medicine among their peers.
Our study has multiple limitations. First, this course was implemented in just 2 sections at 1 university and thus may not translate to other settings. Moreover, data from 2 course sections were combined for more robust data analysis and differences in student composition may have influenced class culture. The survey does not have existing validity evidence and was created by the instructor. We also did not design our study as research but instead report the evaluative results of an educational intervention. Finally, students were not asked how their learning experience compared to other courses at their institution, which offered no counterfactual or benchmark from which to compare the course evaluation.
Conclusion
Teaching styles that focus heavily on didactic teaching are misaligned with the teaching and learning desires for Generation Y students. The student-led “journal club” course structure scaffolded with a self-directed structured summary template allowed students to gain mastery over the challenging task of critically appraising health care research articles in a way which was meaningful and impactful to them. This educational evaluation demonstrates evidence that the course structure was well-received by students and resulted in perceived growth as students and scholars. We attribute our success in part due to the course’s high degree of psychological safety, which was noted, appreciated, and valued by students.
Footnotes
Appendix 1
Acknowledgements
We acknowledge the students who participated in and evaluated the course.
Declaration of Conflicting Interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding:
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Authors’ Note
Institution at which the research was conducted: Northeastern University.
Author Contributions
KD developed the curriculum, designed the study, and collected the data. MF and KD contributed equally to the data analysis and results, and drafting of the final manuscript.
