Abstract
In previous years students in the advanced bachelor course Developmental Psychopathology often failed examination questions where they had to apply theoretical knowledge to realistic clinical issues. Therefore, a new teaching method was implemented to improve students’ understanding. Flipped classroom (FC) and team-based learning (TBL) techniques were implemented, using six challenging authentic assignments. This study describes how these adaptations went along with student performance for three subsequent cohorts, where each cohort received more FC and TBL elements. In cohort 2 a semi-experimental design was used to study the effects of specific techniques. Moreover, it was investigated if students considered this approach useful, and whether it changed their learning strategy. Findings suggest that both FC and TBL techniques contributed to a higher level of understanding, particularly if group work took place in supervised tutorials with teacher feedback and discussion. Further research is required to investigate which specific elements were most effective.
Introduction
The aim of the advanced bachelor course in Developmental Psychopathology is to teach students to understand problematic behavior in children and youth from a developmental perspective. As such, we teach how problematic behaviors can be explained by developmental and transactional theories. Although in previous years we discussed these issues in the lectures, students often failed open questions in the examination if they had to apply these theories to unknown case studies. They were able to explain the theory in general terms but applying the theory to “real life” cases proved difficult. Moreover, students complained that the course contained too much repetition of topics discussed in earlier introductory clinical courses, indicating that they did not properly understand what the “advanced” aim of the course was, despite our explanation and examples in the lectures. We concluded that lectures alone, may not be enough for the required higher level of understanding in terms of Bloom’s taxonomy of educational objectives that runs from memory and comprehension of facts to their application, analysis, synthesis, and evaluation (Anderson & Krathwohl, 2001; Bloom et al., 1956). Therefore, a new teaching method was implemented to stimulate active learning techniques that foster higher levels of understanding. Six challenging authentic assignments were provided using flipped classroom (FC) (DeLozier & Rhodes, 2017; O’Flaherty & Phillips, 2015) and team-based Learning (TBL) (Michaelsen & Sweet, 2008, 2011) procedures to train the desired higher level of understanding.
The FC approach was implemented by means of the assignments. Students had to individually study lecture sheets from previous courses on the same topic, read a book chapter, and submit an assignment prior to a lecture. In the assignment they had to apply theory to clinical questions and case studies. Because students prepared the topic, teaching in the lecture could be focused on higher-level issues, including those required for the assignment. We expected that students would now better understand what the “advanced” level meant and would experience what they already knew and what they now learned. Although robust research on students’ academic performance is scarce, there is evidence that FC as compared to traditional teaching engages students more in deep learning (see O’Flaherty & Phillips, 2015 for a review).
In a TBL sequence, individual preparation and testing is followed by a group discussion and group test, that is subsequently discussed in class where the teacher clarifies what is still unclear. TBL aims to stimulate discussion, team spirit, and critical thinking (Michaelsen, & Sweet, 2008, 2011). In medical and health sciences TBL seems to show positive effects on learning (Haidet et al., 2014). Although in psychology teaching well-designed empirical studies are scarce, and the importance of each of the elements is unclear, TBL also seems to have a positive effect on student performance, including better performance on application questions (Madson et al., 2020). More generally, peer discussions in the classroom have been shown to increase students’ understanding (Smith et al., 2009), particularly when combined with teacher feedback and clarification (Smith et al., 2011). In the present course, the individual assignment was followed by a TBL-tutorial, where each student discussed two assignments in small groups (5–7 students), and prepared a presentation in front of the class with peer and teacher feedback. We asked students to present their answer to the other students because it is known that teaching enhances learning and test performance (Carberry & Ohland, 2012; Nestojko et al., 2014). Moreover, by listening to presentations of other groups, students were also expected to learn about the other four assignments that they had only prepared individually.
Study Design and Hypotheses
The present study describes how these adaptations in teaching went along with student performance for three subsequent cohorts, where each cohort received more FC and TBL elements. The first cohort acted as the baseline. This cohort did not have assignments, FC or TBL-tutorials, but the relevance of developmental theory for clinical cases was illustrated and discussed by means of examples in lectures, where interactive techniques such as clickers and think-pair-share were used.
In the second cohort six challenging assignments were provided belonging to the topics of six out of 10 lectures, using FC and TBL techniques to train the desired higher level of understanding. The specific effects of FC and TBL techniques on examination grades were investigated using a semi-experimental design. For two topics these assignments had to be prepared before the lecture (FC), whereas the other four could be done after the lectures (but prior to the TBL-tutorials). It was expected that topics supported by assignments would result in better grades for related open questions in the examination, particularly if students had to individually submit the assignment prior to the lecture (O’Flaherty & Phillips, 2015).
Students further discussed one of the FC assignments and one non-FC assignment in small groups of 5–7 students during a structured but unsupervised TBL-tutorial. They had to prepare a presentation for these two assignments for a symposium where they received peer and teacher feedback. In the examination, one question required a high level of understanding on a topic in which the student was trained in one of the FC assignments. It was expected that students who worked on this assignment in small groups (half of the class) during the TBL-tutorial would show higher grades for this question than those who did not (Carberry & Ohland, 2012; Haidet et al., 2014; Madson, et al., 2020; Smith et al., 2009).
In the third cohort, all six assignments had to be submitted prior to the lectures (FC). Now the TBL consisted of two supervised tutorials, where three small groups (5–7 students) each prepared a presentation for one of the three assignments, which were immediately presented with more extensive feedback and discussion with peers and the teacher. In cohort 3 we expected even higher grades for the open questions because of the increase in FC assignments as compared to cohort 2. Also, the supervised tutorials and extended discussions in smaller classes were expected to contribute to better results (Smith et al., 2011).
Finally, students in the second and third cohort were asked to evaluate the FC and TBL procedures in both written and oral evaluations. It was investigated if the course elements stimulated motivation of the students (Jones, 2009). We expected that the assignments, FC, and TBL elements would contribute to the experienced growth in knowledge and would be regarded as interesting and useful. In cohort 3 we added a question about their learning strategy. It was expected that FC techniques would help students to implement a schedule of practice that spreads out study activities over time instead of only learning at the end of the course. Such a distributing practice for learning is known to improve effectiveness (Dunlosky et al., 2013).
Methods
Participants
Participants were three cohorts of second- or third-year psychology students, who already followed a course in Developmental Psychology and an introduction course in Clinical Psychology—cohort 1 N = 132, cohort 2 N = 79, cohort 3, N = 105. None of the students had previous experience with FC or TBL teaching. In cohorts 2 and 3 95% of the students followed the FC and TBL procedures.
Procedure
Table 1 shows the three cohorts and how teaching procedures differed between them.
Study Design for the Three Cohorts.
Note. TBL = team-based learning; FC = flipped classroom.
FC
In cohort 2, two lectures used an FC-design, where all students had to read presentation slides on the topic from earlier courses and read a chapter from the more advanced textbook of the present course. They should apply this theoretical knowledge to a challenging authentic assignment on clinical issues or case studies, and should individually submit this assignment prior to the lecture. These authentic assignments contain characteristics of “just-in-time teaching” techniques such as “warm ups” and “goodfors” (Marrs & Novak, 2004), but were also difficult enough to require further discussion. The lectures focused mainly on advanced content and additional information not provided in the textbook, thereby discussing applications to examples on video or case studies, and topics where scientists or practitioners might not agree upon. The specific assignment was not discussed, but the content should help students to improve the assignment. Four lectures had similar supporting assignments but did not use the FC approach. Another four lectures did not have supporting assignments. In cohort 3, six out of 10 lectures required individual assignments prior to the lecture.
TBL
The TBL procedure in this study contained most of the seven guidelines of the TBL procedure (Haidet et al., 2012). Individual assignments preceded small-group work in tutorials. These individual assignments were not graded, but they were checked to see if they were serious attempts and also for plagiarism. Each student further worked on two of these assignments in small groups of 5–7 students during TBL-tutorials. These groups were created by the teacher. Students received instructions for cooperation in small groups, explaining techniques to make sure that all participants actively engage in discussing arguments and disagreements. To increase motivation for serious group work students were rewarded by means of a bonus system. For each group assignment they could earn additional points for their examination if the presentations were “sufficient” (0.1) or “good” (0.2). They could earn this bonus if they individually submitted at least five of the six assignments prior to the TBL-tutorial, including the two “flipped” assignments, thereby effectuating readiness assurance. A bonus system was preferred, because it stimulates and rewards good performance, whereas failure does not have consequences for the final grade.
To study the effect of small-group work on examination results (over and above FC effects), in cohort 2 two groups of students were made: one group worked on flipped assignment 1 (transactional model) in the TBL-tutorial; and the other on flipped assignment 2 (cognitive theories). In the examination one question asked for a transactional explanatory model as in assignment 1, but now applied to another case study and another disorder. Group 1 was expected to show better results for this question.
Presentations were given in front of the class and the teacher. In cohort 2 students presented in front of the complete group and several small groups prepared a presentation for the same assignment. In response to negative evaluations for the symposium in cohort 2 (see Results section), in cohort 3, presentations took place immediately after the small-group discussion in the tutorial. Although there was not a rival group for the same assignment, the other students did individually make the assignment and the other groups had to provide peer feedback to the presentations of each group.
Materials
The six assignments (available from the author) focused on desired learning goals, namely knowledge about/understanding of:
transactional developmental explanatory models for problem behavior and age-dependent characteristics of disorders in childhood and adolescence; (social)cognitive explanatory models and how they explain specific problem behaviors; differential analysis; explanations for age-dependent prevalence of disorders and gender differences; and the link between therapy and (age-dependent) explanatory models.
All assignments used an authentic content, that is, students had to apply the knowledge to case studies, and had to explain a theory or preferred therapy to the mother tailored to a specific child, etc.
Measures
Examination
The examination consisted of 36 multiple-choice questions (counting for 60% of the final grade) and four open questions, each with 4–5 sub-questions (counting for 40% of the final grade). Grades in the Dutch system run from 1 (chance score) to 10 (all answers correct), and with grades ≥5.5 students pass the course. In the experiment in cohort 2, one question related to the topic that was not supported by an assignment, one question related to FC assignment 1, and the other two questions related to non-FC assignments.
Evaluation
Written anonymous evaluations were done in each cohort at the end of the course, directly after the examination. Students were asked how much they agreed that elements were fun, useful or informative. Answers were given on a 5-point scale, running from “do not agree at all” to “agree very much.” The present study only reflects on the questions that specifically related to the assignments and the FC and TBL elements of the course. In cohort 3 a question was added to investigate if students thought that they started studying earlier because of the FC assignments.
Oral evaluations with a teacher took place in cohorts 2 and 3, where the tutorial teacher specifically asked if assignments, FC, and TBL procedures were considered interesting/nice or useful, and if the students learned from these assignments (if so, what). The teacher audio recorded the evaluations and made a summary of the results.
In cohort 2, evaluation also took place in a focal group (nine participants), led by a student who did not follow this course. Similar themes were discussed as with the teacher, but the format also stimulated discussion between students. The student-leader of the focal group audio recorded the discussion and selected the relevant parts that were anonymously communicated to the coordinator.
Results
Grades
Table 2 shows the means for student grades on the multiple-choice and open parts of the examination. As the examination questions in the cohorts were not the same, no statistical analyses were performed, so findings are descriptive. The means show that the grades for the open questions gradually increased over the cohorts, whereas the grades for the multiple-choice questions remained more stable. Without training, students in cohort 1 generally failed the open question part of the examination as the mean was lower than 5.5. This was no longer true in cohort 2, although scores were still in the borderline range. In cohort 3 the means were clearly above the minimum requirement. The distribution of grades further showed, that in cohorts 1 and 2 grades higher than 7 were rare, 6.8 and 8.9%, respectively, indicating that questions were regarded as being too difficult. In cohort 3 this was 24.8%. These findings suggest that the changes in teaching fostered students’ ability to answer application questions in the examination.
Mean Grades for the Multiple-Choice Examination (Mc) and the Open Questions Examination (Open) in The Three Cohorts of Students.
Table 3 shows the mean grades for the four open questions of cohort 2. Three of the questions related to a TBL-assignment, of which one had to be submitted prior to the lecture (FC). The fourth question was not related to an assignment. Repeated analyses of variance (ANOVAs) indicated a significant difference between the grades, F(74, 3) = 80.1, p < 0.001, with significant linear, F(76, 1) = 36.4, p < 0.001, quadratic, F(76, 1) = 131.5, p < 0. 001, and cubic effects, F(76, 1) = 42.6, p < 0.001. Ad hoc contrast analyses indicated that the grade for the question that was not related to an assignment was significantly lower than the other three grades (p < 0.0001), that grade for the question that related to the FC-assignment was significantly higher than the other three grades (p < 0.0001), and that grades for the two questions that both related to non-FC assignments did not significantly differ (p = 0.053). These findings suggest that assignments improve students’ performance, particularly when using an FC-procedure.
Mean Grades (Maximum 10) for Each of the Four Open Examination Questions.
Note. FC = flipped classroom.
Table 4 shows the mean grade points (maximum four) for the sub-questions that were like the content of FC-assignment 1 (transactional model) in cohort 2. ANOVA indicated a significant difference between students who worked on this assignment during the TBL-tutorials and those who did not but worked on another FC-assignment, F(74, 1) = 4.7, p < 0.034. This shows that there seems to be a small (7.5%) but significant additional gain of small-group work, over and above individually working on an FC assignment and listening to presentations of other groups and discussion with the teacher.
Mean Points Earned (Maximum Four) for Sub-Question that Specifically Related to FC Assignment 1.
Note. TBL = team-based learning; FC = flipped classroom.
Evaluations
In cohort 2 the written evaluations indicated that about 50% of the students agreed (= score 4 + 5) that the assignments made the link between theory and practice clearer. However, the majority (68%) did not regard the presentations of the assignments in the symposium as useful. Also, most students (80.9%) were not satisfied with the group work in the unsupervised TBL-tutorials. Most students (62.2%) did agree that it was useful to prepare and make the assignment prior to the lecture, and that they learned more from these lectures.
In the oral evaluations many students in cohort 2 indicated that they liked the FC-lectures, because they now better understood what they learned, and some said that they now really understood the explanatory theories. Some others indicated that these lectures were a bit intimidating and tiring because they had to answer many questions during class. Many students indicated that the assignments were a good preparation for the examination. A few students were confused and worried that their usual approach for learning might not be good enough. There was much variation in the evaluation of the TBL-tutorials, as some students were very enthusiastic, whereas others said that they learned nothing and complained about the lack of motivation of group members. Most students agreed that the symposium with presentations was too long and boring, and that the quality of many presentations was too low. Furthermore, they indicated that the time for feedback and discussion was too short.
In cohort 3 written evaluations were more positive. Now 68.2% agreed that they learned more because they worked on the assignment prior to the lecture, 79.1% indicated that the assignments made the relevance of theories for clinical practice clearer, and 63% regarded peer discussion in the (supervised) TBL-tutorials as useful. However, still only 60.6% indicated that they agree that listening to presentations of the other groups was useful. The majority (73.1%) indicated that they started to study earlier in the course because of the FC assignments.
The oral evaluation with tutorial teachers largely provided similar positive results on all issues. Some students, however, indicated that because of the large number of assignments they spent less time studying the chapters in the textbook prior to the examination.
Discussion
The findings seem to indicate that the challenging authentic assignments foster a higher level of understanding, that is, students were better able to apply the theoretical knowledge to new case studies and clinical questions. The semi-experimental findings in cohort 2 suggest that effects were stronger, if assignments were made prior to the related lecture (O’Flaherty & Phillips, 2015), and if students further discussed these assignments in small groups and presented and discussed their answer in class (Madson et al., 2020). Students indicated that they better understand what the “advanced” level comprised. It was found that small-group work improved examination performance over and above individual work on related assignments and discussing presentations of other students. Nevertheless, student evaluations about unsupervised group work in that cohort were not that positive. This was found in earlier studies if teamwork occurred outside class and students were not able to hold teammates accountable, and teacher feedback was not immediately available (Smith et al., 2009, 2011).
Mean grades as well as the student evaluations improved in cohort 3, probably because the TBL-tutorials were now supervised, the discussion in the small groups was done more seriously and more in line with instructions, answers were presented immediately in the tutorials, and feedback and discussion with peers and teacher was more extensive (Smith et al., 2011). Also, all assignments now had to be submitted prior to the related lectures and over 70% of the students indicated that they started learning earlier, suggesting a more distributing practice for learning that is known to be more effective (Dunlosky et al., 2013). Without further experimental research, we cannot be sure which factors contributed most to the better grades in this cohort. In general, however, the findings seem in line with the literature that indicate positive effects for FC and TBL procedures (Haidet et al., 2014, Madson et al., 2020; O’Flaherty & Phillips, 2015).
Of course, the design of this study is far from perfect. It is always difficult to compare cohorts, as they may differ in more aspects than the experienced teaching procedure. Also, despite our efforts to make the open questions in each cohort equally difficult, as well as the four open questions in cohort 2, we cannot be sure that we succeeded. And since we did not use an experimental design in cohort 3, we do not know which of the changes as compared to cohort 2 resulted in the higher grades. More robust research methods are recommended to further study the effectiveness of each of the teaching elements. But as a first step, our findings seem to suggest that over the years, students’ learning positively changed in that they were better prepared for the lectures, and started studying earlier in the course. The majority appreciated the teaching techniques and their level of understanding increased as indicated by the grades for the open application examination questions.
Footnotes
Acknowledgment
The author thanks Dr Karin Smit for her help and advice.
Declaration of conflicting interests
The author has no conflicts of interest to declare.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
