Sage Journals: Discover world-class research

Abstract

OBJECTIVE

Determine if a point-based attendance system combined with longitudinal gamification is feasible and improves didactic session attendance and learner perceptions at our internal medicine residency.

METHODS

A prospective before–after cohort study. Weekly attendance was tracked from June 2022 through April 2023 at our university-affiliated internal medicine residency program. We implemented a point-based longitudinal game incentivizing residents to attend didactics with positive reinforcement in July 2022 (C: carrot). We added tiered positive reinforcement and positive punishment to the game in January 2023 (CS: carrot and stick). Attendance during these periods was compared to pre (P) and postintervention (S). Perceptions were assessed during the P, C, and CS periods with Likert scale ratings.

RESULTS

CS was associated with higher attendance than other study periods (P = .002). Median attendance was P—51% (IQR 37.5-64.5), C—65% (IQR 50-74), CS—81% (IQR 78-94), and S—66% (IQR 63-71). Perceptions were similar during pre and intervention study periods, including perceptions of camaraderie (P—4.4, C—4.4, CS—4.5; P = .56), interest in attending didactic sessions (P—3.7, C—3.4, CS—3.2; P = .21), and mandate as the primary reason for attending didactics (P—3.1, C—3.1, CS—3.2; P = .96)

CONCLUSIONS

A point-based attendance system combined with a longitudinal game that included tiered positive reinforcement and positive punishment was feasible and associated with higher didactic attendance but not associated with changes in resident perceptions.

Keywords

Gamification noon conference didactic attendance internal medicine residency graduate medical education

Introduction

Gamification is the adaptation of nongame environments to game-based learning.¹ This strategy has been shown to have a positive impact on the learning environment.² In an internal medicine residency, it has been shown to improve learner perceptions of engagement and value.³ Gamification can also enhance learner motivation. The subcomponents are typically categorized as adding extrinsic motivation (doing something because it leads to a tangible result) or influencing intrinsic motivation (doing something because it is inherently satisfying).⁴ Intrinsic motivation is typically preferred because it leads to higher creativity, improved well-being, and more significant learning.⁵ Gamification can impact intrinsic motivation by creating an inherently satisfying game or potentially catalyzing intrinsic motivation through initial extrinsic motivation.⁶ One notable downside of this approach is it could theoretically lead to the over-justification effect, that is when rewards for a previously unrewarded activity shift motivation from intrinsic to extrinsic and lead to apathy when rewards are removed.⁷

Most medical education games influence motivation and learning via behaviorist pedagogy, meaning they assume learning occurs through operant conditioning.⁸ Functionally, these motivational facets of gamification have been shown to improve attendance for game sessions.⁹ The operant conditioning of gamification can be extended to behaviors outside of game sessions and could help sustain longitudinal motivation for learners to attend nongame didactics. If so, this motivation could improve attendance in internal medicine programs, which is currently suboptimal, ranging from 26% to 61%.^10–15 The purpose of our study was to determine if using a point-based attendance system combined with longitudinal gamification is feasible, improves didactic session attendance, and improves learner perceptions of engagement and value at our internal medicine residency. We hypothesized that longitudinal gamification with positive reinforcement and positive punishment would improve didactic attendance and learner perceptions.

Methods

Design and ethical considerations

A prospective before–after cohort study at Wright Patterson Medical Center, Miami Valley Hospital, and the Dayton Veterans Affairs Hospital. The study was approved as exempt by the Wright Patterson Medical Center Institutional Review Board in accordance with 32 CFR 219.104 (Protocol #FWP20220006E). The need for consent was waived.

Setting and participants

Measurement of resident didactic attendance started in June 2022 as it coincided with the term start for the chief residents leading the intervention. The intervention periods started on July 12, 2022, and concluded on March 7, 2023. Attendance was continually monitored in a postperiod until April 13, 2023.

The Wright State University/Wright-Patterson Medical Center Internal Medicine residency program is university-affiliated with 76 categorical positions. All residents were included in the study without exclusion criteria. Residents rotate through 3 hospitals: one military medical center, one private hospital, and one Veterans Affairs hospital. Chief residents lead the same didactic curriculum sessions across the 3 sites. There are between 4 and 6 didactic curriculum sessions per week. One academic half-day session is held from 1400 to 1600 on Tuesdays. Morning reports are held 1 to 2 times weekly from 0800 to 0830, which are traditional lectures given by residents on topics included in the American Board of Internal Medicine (ABIM) blueprint. Each morning report has a same-day noon conference, which entails small group, problem-based learning from 1200 to 1300, expanding on the morning session. Other educational sessions are delivered in the residency program, but the academic half-day, morning report, and noon conference sessions are considered the core curriculum. All interventions in this study were solely related to this core curriculum. Prior to this study, the sessions were deemed mandatory but did not have formal tracking or specific consequences for not attending.

Interventions

We developed a medical knowledge game (the House Cup) that favored residents with high didactic session attendance to incentivize improved attendance. The residents were split into 4 longitudinal teams (houses) meant to compete against each other. The teams were composed of 18 to 20 residents that had an even distribution of class year. Second- and third-year residents were assigned by their continuity clinic groups, and first-year residents were randomly assigned by picking their house out of a hat (the sorting hat).

The House Cup was played 4 times throughout the 2022 to 2023 academic year. Each game was preceded by a House Cup period that lasted ∼3 months. Attendance throughout each House Cup period was calculated for each house as a percentage of didactic sessions attended. This was converted to a point-based score, utilized as the starting score for each house in the House Cup, resulting in a significant advantage for houses that attended more didactic sessions.

The House Cups comprised 50 to 65 questions regarding topics discussed at didactics during the respective House Cup period. Each House Cup was designed identically: 3 questions (worth various points) were simultaneously displayed in a custom Microsoft PowerPoint presentation. The houses took turns answering a question of their choice, receiving points for correct answers but not losing points for incorrect answers. At the end of their turn, each house received a game card containing various tasks or point changes that could be played. These promoted fun but also added randomness, which helped keep teams engaged.

We implemented 2 incentive structures throughout the duration of the study. The first incentive structure was aimed to solely include the positive reinforcement described above, leveraging innate competition as a motivational factor.¹⁶ We refer to this period as the “carrot” period. At the onset, residents in the house that won the House Cup would receive one half-day of administrative time. The administrative time could be used during clinic or elective blocks to complete nonclinical work of their choosing.

The incentive structure was adjusted on January 17, 2023, to include 2 tiers of positive reinforcement and 2 tiers of positive punishment. We refer to this period as the “tiered carrot and stick” period. Residents who attended 70% or more of didactic sessions were eligible to receive 2 half-days of administrative time if their house won the House Cup. Residents who attended 50% or more of didactic sessions were eligible to receive one half-day of administrative time if their house won the house cup. Residents who attended < 50% of didactic sessions would receive an inquiry into their lack of attendance. Residents attending < 30% of didactic sessions would be discussed during a clinical competency committee regarding disciplinary actions. After the final House Cup on March 7, 2023, the punishment system was kept in place without the positive reinforcement system or gamification (the stick period). Each set of study period rules was communicated to all residents via email and group chat messaging prior to implementation.

Outcomes measured

Resident attendance was tracked from June 7, 2022 to April 13, 2023. This included preintervention (June 7, 2022 to July 11, 2022), carrot (July 12, 2022 to January 15, 2022), tiered carrot and stick (January 16, 2023 to March 6, 2023), and postintervention (March 7, 2023 to April 13, 2023) periods.

Residents signed into didactic sessions digitally with a QR code. The QR code was a single, unchanging link to a digital form utilized for the entirety of the study period. Each site created a unique number for every didactic session to ensure residents were physically present. Residents input the unique number in the digital form and then select their name in a radio button selection. We developed a Microsoft Excel file that automatically calculated the expected number of residents that should attend didactics every week based on their rotations and planned absences. The QR code sign-ins were summed by a house and then divided by this expected number to yield the attendance percentage. These data were also combined to calculate the total resident attendance.

We assessed resident perceptions during pre, carrot, and tiered carrot and stick study periods. We distributed a digital survey with REDCap, a digital management and research software tool.^17,18 The survey was developed by consensus agreement between the investigators and included 9 questions assessing resident perceptions of didactics. All answer choices were based on a 5-point modified Likert scale rating (5 being the most positive perception, 1 being the most negative perception). The full survey can be viewed in Supplemental Material 1. The survey was distributed in July for preintervention, in September (after the first House Cup) for the carrot period, and in March (after the first House Cup with the second intervention). Residents self-reported unique, anonymous identifiers to track responses.

We assess feasibility by retrospectively counting the associated costs and hours needed to complete interventions for the study.

Statistical analysis

We used Excel version 2302 to calculate and graph descriptive statistics on the total resident attendance. Outcomes by study period were analyzed using GraphPad Prism version 9.5.1 for Windows, GraphPad Software, www.graphpad.com. We compared outcomes between the study periods using the Kruskal–Wallis test.

We performed a post hoc least squares multiple linear regression to adjust for didactic attendance negatively correlating with weeks since the start of the academic year, analyze the associations between attendance and individual intervention components (positive reinforcement [carrot], tiered positive reinforcement [tiered carrot], and positive punishment [stick]), and calculate confidence intervals for their individual effect sizes.

Results

Outcomes measured

A total of 214 core didactic sessions were monitored during the study with 5688 sign-ins. The average attendance throughout the entire study was 65%. Figure 1 shows the total weekly attendance throughout the entire study duration. The median total attendance during each study period was pre—51%, carrot—65%, tiered carrot and stick—81%, and stick—66%. Attendance during the study periods significantly differed (P = .002) with 95% confidence intervals shown in Figure 2. Multiple linear regression analysis found weeks since 1st July negatively correlated with attendance, whereas tiered positive reinforcement and positive punishment positively correlated with attendance. These results are shown in Table 1.

Figure 1.

Total weekly didactic attendance of residents throughout the study period with interventions annotated.

Figure 2.

Median weekly attendance percentage during each study period. Pre: n = 5 weeks; carrot: n = 27 weeks; carrot and stick n = 7 weeks; stick: n = 6 weeks. Error bars represent the 95% confidence interval of the median.

Table 1.

Multiple linear regression results. The multiple R for the model was 0.80 and the adjusted R² = 0.60. Estimated coefficients represent the unadjusted effect size of each variable.

VARIABLE	ESTIMATED COEFFICIENT	CONFIDENCE INTERVAL	P
Weeks since July 1	−0.95%	−1.28 to −0.61%	<.0001
Positive reinforcement	−11.51%	−24.35 to 1.33%	.077
Tiered positive reinforcement	11.62%	0.25 to 23.00%	.046
Positive punishment	14.08%	1.92 to 26.24%	.024

Ninety-six survey responses of perceptions were completed: 42 responses pre, 32 carrot, and 22 carrot and stick. Based on matching identifiers, presurveys were completed by 38% of the carrot group and 50% of the tiered carrot and stick group. Response characteristics and comparisons are shown in Table 2.

Table 2.

Resident perceptions during each phase of the study.

VARIABLE	PRE (N = 42)	CARROT (N = 32)	CARROT AND STICK (N = 22)	P
Postgraduate year	1.9 (0.9)	2.1 (0.8)	1.9 (0.8)	.66
1	40.5%	31.3%	36.4%
2	26.2%	31.3%	40.9%
3	33.3%	37.5%	22.7%
I enjoy attending didactic sessions.	3.7 (0.8)	3.8 (0.8)	3.3 (0.8)	.05
Attending didactic sessions improves my medical knowledge.	4.0 (0.7)	4.2 (0.6)	3.9 (0.8)	.51
Didactic sessions have significant educational value.	4.0 (0.8)	4.1 (0.6)	3.6 (0.8)	.04
How interested are you in attending didactic sessions?	3.7 (0.9)	3.4 (0.9)	3.2 (1.1)	.21
How satisfied are you with the current didactic structure?	3.3 (0.8)	3.3 (1.0)	2.9 (1.0)	.24
How confident are you that the current didactic structure will prepare you to pass the ABIM examination?	3.3 (0.9)	3.3 (1.0)	2.9 (1.0)	.16
I feel camaraderie with my co-residents.	4.4 (0.6)	4.4 (0.7)	4.5 (0.5)	.56
My primary motivation for attending didactics is the mandatory requirement.	3.1 (1.0)	3.1 (1.3)	3.2 (1.2)	.96
My primary motivation for attending didactics is for learning.	3.9 (0.8)	3.9 (0.8)	3.8 (1.0)	.98
Structured games add educational value to regular didactic content.	4.0 (0.8)	3.7 (1.2)	3.5 (1.2)	.29
How often do you attend didactic sessions?	4.4 (0.8)	4.2 (0.8)	4.1 (0.8)	.27
How much effort was required to perform tasks related to the game (ie, marking attendance, participating in sessions)?	2.4 (0.9)	2.3 (1.2)	2.9 (1.0)	.04

Perceptions data represent results of a 5-point modified Likert scale (5 being the most positive, 1 being the most negative). Results are represented as mean (standard deviation) unless noted as percentages. The postgraduate year represents the year of categorical residency training from 1 to 3.

Feasibility

The intervention was executed by 6 chief residents. We estimate that 30 h were required to build the automated attendance tracking system. Throughout the study duration, the tracking system was updated with schedule changes, adding about 2 h every month. Before each house cup, sign-ins were downloaded, reviewed for aberrant sign-ins, and organized before placing them into the tracking system, requiring an additional 3 to 4 h every 2 to 3 months. Each house cup game took ∼5 to 6 h to create. The intervention cost $12 to print a physical playing card deck.

Discussion

These findings do not support our initial hypothesis that longitudinal gamification with only positive reinforcement would improve didactic attendance. However, our modification with tiered positive reinforcement and positive punishment was associated with significantly higher attendance. This improvement persisted despite correcting for timing later in the academic year when residents are less likely to attend didactics.¹⁵ While part of this effect may purely be due to specific expectations and positive punishment for not meeting these expectations, the statistically significant decrease in attendance after withdrawal from the game but maintenance of the expectations and positive punishment (stick period) suggests that the game and/or tiered positive reinforcement contributed to attendance. The post hoc multiple linear regression further supports the independent effect of the game and/or tiered positive reinforcement to improve attendance.

Our secondary hypothesis was that longitudinal gamification would improve the residents’ perceptions of didactics based on previously demonstrated improved perceptions via gamification.³ Our results showed no improvement, and after implementation of the stick, show an increase in the perceived effort required for the game and a borderline significant decrease in the perceived enjoyment. These results suggest that incorporating positive punishment into the game may have negatively affected resident perceptions. Though the timing of the survey distributions may have had an important impact on these results, overly competitive games in medical education risk negatively impacting the learner experience.¹⁹ Further study is needed to determine whether this effect is truly present and causal from positive punishment implementation.

Our experience informs the practical application of incentive models for didactic attendance. We learned that the structure of the rewards system and the magnitude of the reward are crucial. Our initial rewards structure rewarded the collective groups but didn’t effectively account for individual actions. Residents could attend minimal didactics but still win if they were on a strong team. Whereas the tiered positive reinforcement system encourages individual attendance as well as team effort. A bigger reward also could have amplified the impact of positive reinforcement. For example, a study rewarding research productivity with up to $2500 yielded significantly more publications.²⁰ While the interventions were feasible and cheap, they required several hours of work from multiple invested people. So, these measures to increase the effect size of the interventions would help favor the total yield from similar gamification strategies.

This study has several limitations. The before–after cohort design prohibits the ability to draw causal conclusions. It was single-centered, which limits generalizability. Several confounding variables may have impacted attendance, including the time of the academic year, variation in residents each week, variation in clinical duty load to residents, and the quality of different didactic sessions. Though multiple linear regression adjusted for timing in the academic year, it may not be a linear relationship, such as winter months resulting in less attendance. Our attendance system is an imperfect system. While correcting for resident rotations and vacations was feasible, accounting for every last-minute absence was not feasible. The attendance results are likely slightly underestimated because of these imprecisions. Unique identifiers for the perceptions survey differed, so differences between the groups may represent differences in sampling rather than perception changes. We did not perform an a priori sample size/power analysis. We also chose to analyze the entire cohort with these differences, as the cohort matched by each unique identifier would have dramatically limited the power with a sample size of 11 survey responses. Lastly, our perceptions assessment does not have validity evidence and is limited by a suboptimal response rate which may have affected the generalizability of findings. Further study is needed to determine whether tiered positive reinforcement without positive punishment would improve attendance.

Conclusion

A point-based attendance system combined with a longitudinal firm-based game that included positive reinforcement and positive punishment was associated with higher attendance than no game and a game with only positive reinforcement. Longitudinal gamification was not associated with improved resident perceptions of didactic sessions. Our approach may be useful in settings where improving attendance is vital to maintaining residency compliance requirements. Further investigation is needed to determine the causality of longitudinal gamification on attendance and perceptions, and whether such a system has meaningful impacts on learning outcomes.

Supplemental Material

sj-pdf-1-mde-10.1177_23821205231225922 - Supplemental material for The House Cup: Using Longitudinal Gamification to Improve Didactic Attendance

Supplemental material, sj-pdf-1-mde-10.1177_23821205231225922 for The House Cup: Using Longitudinal Gamification to Improve Didactic Attendance by Brian P Elliott, Joseph Glendening, Sahana Venkatesh, Ammar Ahmad, James Prister and Kathryn M Burtson in Journal of Medical Education and Curricular Development

Supplemental Material

sj-docx-2-mde-10.1177_23821205231225922 - Supplemental material for The House Cup: Using Longitudinal Gamification to Improve Didactic Attendance

Supplemental material, sj-docx-2-mde-10.1177_23821205231225922 for The House Cup: Using Longitudinal Gamification to Improve Didactic Attendance by Brian P Elliott, Joseph Glendening, Sahana Venkatesh, Ammar Ahmad, James Prister and Kathryn M Burtson in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgements

Drs Lauren Gin and Gabrielle Krstevski for their help with implementing the project.

FUNDING

The author(s) received no financial support for the research, authorship, and/or publication of this article.: The author(s) received no financial support for the research, authorship, and/or publication of this article.

DECLARATION OF CONFLICTING INTERESTS

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

ORCID iDs

Brian P Elliott

Kathryn M Burtson

Supplemental Material

Supplemental material for this article is available online.

References

Rutledge

Walsh

Swinger

, et al. Gamification in action: theoretical and practical considerations for medical educators. Acad Med. 2018;93(7):1014-1020. doi:10.1097/ACM.0000000000002183

Abdulmajed

Park

Tekian

. Assessment of educational games for health professions: a systematic review of trends and outcomes. Med Teach. 2015 Apr;37(Suppl 1):S27-S32. doi: 10.3109/0142159X.2015.1006609. PMID: 25803590.

Dakroub

Weinberger

Levine

. Gamification for the win in internal medicine residency: a longitudinal, innovative, team-based, gamified approach to internal medicine board-review. Cureus. 2022;14(3):e22822. doi:10.7759/cureus.22822. Published 2022 Mar 3.

Krishnamurthy

Selvaraj

Gupta

, et al. Benefits of gamification in medical education. Clin Anat. 2022 Sep;35(6):795-807.

Buckley

Doyle

. Gamification and student motivation. Inter Learn Environ. 2016 Aug 17;24(6):1162-1175.

Singhal

Hough

Cripps

. Twelve tips for incorporating gamification into medical education. MedEdPublish. 2019 Nov 26;8:216.

Deci

Koestner

Ryan

. Extrinsic rewards and intrinsic motivation in education: reconsidered once again. Rev Educ Res. 2001 Mar;71(1):1-27.

Gorbanev

Agudelo-Londoño

González

, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online. 2018 Jan 1;23(1):1438718.

Nevin

Westfall

Rodriguez

, et al. Gamification as a tool for enhancing graduate medical education. Postgrad Med J. 2014;90(1070):685-693. doi:10.1136/postgradmedj-2013-132486

10.

Randall

Schreiner

Clyburn

Rockey

Duckett

. Effects of an academic half day in a residency program on perceived educational value, resident satisfaction and wellness. Am J Med Sci. 2020 Oct;360(4):342-347. doi: 10.1016/j.amjms.2020.05.013. Epub 2020 May 14. PMID: 32553748.

11.

Kaye

Khan

Gudleski

Yatsynovich

Graham

Kwiatkowski

. Implementation of a longitudinal, near-peer ECG didactic curriculum in an internal medicine residency program and impact on ECG interpretation skills. BMC Med Educ. 2023 Jul 24;23(1):526.

12.

Choi

Chodoff

Brown

, et al. Preparing future medicine physicians to care for cancer survivors: project ECHO® in a novel internal medicine and family medicine residency curriculum. J Cancer Educ. 2023 Apr;38(2):608-617.

13.

Cosimini

Mackintosh

Chang

. Number needed to eat: pizza and resident conference attendance. Med Educ. 2016;50(12):1204-1207. doi:10.1111/medu.13080

14.

McDonald

Zeger

Kolars

. Associations of conference attendance with internal medicine in-training examination scores. Mayo Clin Proc. 2008 Apr 1;83(4):449-453.

15.

FitzGerald

Wenger

. Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores? Acad Med. 2003;78(1):84-89. doi:10.1097/00001888-200301000-00015

16.

Pesare

Roselli

Corriero

Rossano

. Game-based learning and gamification to promote engagement and motivation in medical learning contexts. Smart Learn Environ. 2016;3(5):1-21.

17.

Harris

Taylor

Thielke

Payne

Gonzalez

Conde

. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010

18.

Harris

Taylor

Minor

, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. doi:10.1016/j.jbi.2019.103208

19.

Kirsch

Spreckelsen

. Caution with competitive gamification in medical education: unexpected results of a randomised cross-over study. BMC Med Educ. 2023 Dec;23(1):1-4.

20.

Chang

Mills

. Effects of a reward system on resident research productivity. JAMA Otolaryngol–Head & Neck Surg. 2013 Dec 1;139(12):1285-1290.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.04 MB