Abstract
OBJECTIVES
Medical students are increasingly using a spaced repetition software called Anki to study. There are limited studies evaluating the relationship between Anki and learner outcomes. In this study, we describe the history of Anki use in medical school and assess the potential relationships between use of Anki and medical student academic, extracurricular, and wellness outcomes.
METHODS
We used cross-sectional data from a 50-item online survey and retrospective academic performance data from our institution's outcomes database. Participants were medical students. The survey assessed the frequency and timing of Anki use, student perceived stress, sleep quality, burnout risk, and involvement in extracurricular activities. Academic success was measured by USMLE Step 1 and Step 2 scores.
RESULTS
165 students responded survey. 92 (56%) identified as daily Anki users. Daily Anki use was correlated with increased Step 1 score (P = .039), but not Step 2 scores. There was an association between Anki use and increased sleep quality (P = .01), but no difference for other measurements of wellness or extracurricular involvement.
CONCLUSION
The study demonstrates potential benefits of daily use of Anki but also confirms that a variety of study methods can be used to achieve similar medical school outcomes.
Introduction
Spaced repetition is an effective method for long-term retention of knowledge. 1 In recent years, spaced repetition flashcard software, in particular the program Anki, 2 has become increasingly popular among medical students as a preparation method for classes and board exams. No large studies have assessed what percentage of medical students use this software however one online forum dedicated to its use in medical school has over 109 000 registered active users which exceeds the 89 000 active medical students in the United States.3,4 Although it is unlikely that every medical student uses Anki, these numbers speak to its popularity as a study resource.
Previous studies have demonstrated a relationship between the use of Anki spaced repetition flashcard software and performance on United States Medical Licensing Examination (USMLE) Step 1. 5 Deng et al 5 surveyed 72 medical students and found that after controlling for academic and psychological factors, for every 1700 spaced repetition flashcards a student completed, they scored one point higher on USMLE Step 1. However, little has been done to examine additional potential effects of this study method. Following the change of USMLE Step 1 to pass/fail, associations with wellbeing and extracurricular engagement may become increasingly relevant to incoming students who will rely on other metrics to become competitive residency applicants.
In this descriptive study, we conducted a cross-sectional, online survey of medical students to better understand their use of Anki as a study tool and associated outcomes including performance on Steps 1 and 2, extracurricular involvement, and wellbeing.
Methods
Overview
This descriptive, exploratory study used cross-sectional data from a 50-item online survey (Qualtrics) and retrospective data from the Medical Education Outcomes Center (MEOC) database. 6 Study exemption was obtained from the University of Minnesota Institutional Review Board Study #00012492.
Anki Background
Anki is a free, open-source spaced repetition software that was first released in 2006. 2 It combines flashcards with an algorithm that incorporates user input to predict when a user will forget a flashcard. It then shows the user the flashcard at custom intervals intended to keep the user from forgetting the flashcard. The history of the use of Anki by medical students has not previously been described in the literature. It first rose in popularity among medical students in the mid-2010s when several students created and shared Anki decks online that were comprehensive of material on USMLE Step 1.7,8 Subsequently, these decks were used widely by medical students across the country and have been refined in a collaborative manner by students over time in an attempt to encompass the entire medical school didactic curriculum.7,8 Decks now exist for topics pertinent to residency exams, such as the American Board of Surgery In-Training Exam (ABSITE).
The Medical Education Outcomes Center
MEOC is a collaborative center at our institution that seeks to evaluate the outcomes of medical education programs through the use and linkage of diverse datasets and, as such, has access to robust and complete data about student performance. 6 In addition to data, MEOC provided consulting, administrative support, and statistical services for this project.
Study Procedures
In November 2021, an email invitation to participate in the study with a unique Qualtrics survey link was sent to all enrolled medical students using the class distribution lists. Students had 2 weeks to complete the survey and received 2 email reminders. A gift card raffle was used to incentivize students to participate. University usernames of participants were captured through a survey login feature for the purpose of connecting participant survey responses to data in the MEOC database. Data was ultimately de-identified by MEOC prior to being given to the core research team. Consent was obtained through an Information Sheet at the beginning of the survey.
Measures
Anki Use
Anki use was assessed through original survey items targeting the frequency of use, timing of Anki use (when the participant began using Anki and which semesters of medical school the participant used Anki), how many Anki flashcards the participant introduced daily, and whether the participant made their own flashcards or used pre-made flashcards. Participants were defined as an “Anki user” if they reported doing Anki reviews daily during their first and/or second year of medical school.
Wellness
Wellness was assessed through original items and a combination of standardized, validated measures including the Perceived Stress Scale, 9 the Pittsburgh Sleep Quality Index, 10 and the Maslach Burnout Inventory. 11 The Perceived Stress Scale (PSS) seeks to measure how stressful individuals characterize their life to be, with scores ranging from 0 to 40, with 40 indicating the highest level of perceived stress. The Pittsburgh Sleep Quality Index (PSQI) classifies sleep quality and quantity in the last month through a global PSQI score and 7 component scores. In this study, we collected only the “sleep duration” and “subjective sleep quality” component scores to minimize survey length. The “emotional exhaustion” item from the Maslach Burnout Inventory, a single item which has been shown to predict burnout risk in medical professionals, was used to assess burnout. The item asks participants to state how frequently they “feel burned out from [their] work” on a Likert scale. Lastly, leisure time was assessed through 2 original survey items.
Academic Success
Academic success was measured by USMLE Step 1 and USMLE Step 2 board exam scores. These scores were collected from the MEOC database. In addition, students may elect to take dedicated time to study for the USMLE Step 1 exam after their second year of medical school. Participants were asked to self-report the length of their dedicated study time to determine any potential effects that Anki use might have on the length of dedicated period required before feeling prepared to take USMLE Step 1.
Extracurricular Involvement
Extracurricular involvement was measured with original survey questions targeting the number of research publications, abstracts, and presentations as well as weekly hours of volunteering, leadership, paid work, and research during participants’ first and second years of medical school. In addition, participants were asked whether they felt that they “had adequate time to devote to extracurricular activities when school was in session during first and second year.” This question was meant to assess whether Anki users or non-Anki users felt that they had more time to dedicate to extracurriculars.
Other Covariates
To explore potential covariates, we collected demographics, conscientiousness scores, pre-medical school academic achievement (including MCAT score and GPA), whether a participant planned on taking an additional year to engage in research and other activities to enhance their residency application, study methods used other than Anki (for people who did not use Anki during medical school), number of practice questions and exams used for Step 1 preparation, and the participant's perception of competitiveness for their specialty of choice. Conscientiousness scores were determined by the 9 conscientiousness items from the Big Five Inventory, with a score of 9 being the least conscientious and a score of 45 being the most conscientious. 12 MCAT scores and pre-medical GPAs were collected via the MEOC database. All other covariates were collected via original survey questions.
Development of Original Survey Items
Several survey items were developed by our team with the help of MEOC, which has expertise in survey design. The original survey items were validated by a group of 5 medical students outside the study team, who agreed that the content and language was consistent. Items that were identified as unclear were modified and re-assessed until they were interpreted as intended.
Statistical Analysis
Statistical analysis was done using chi-squared and ANOVA tests comparing Anki users and non-daily Anki users (all other participants).
Results
A total of 165/887 (18.6%) students responded to the Qualtrics questionnaire. Most students used Anki at some point in their academic careers, with 139 (84%) reporting using Anki for at least one semester. Of the 139 students who used Anki, 121 (87%) used decks made by others and 18 (13%) chose to make their own. Of the 121 who used decks made by others, 71 (58.6%) used national decks (ie, decks shared online by other medical students). Of the 26 students that did not use Anki at all, 22 (85%) reported using outlines or notes as their primary study tool and non-Anki flashcard software, lecture slides, and PowerPoint slides. Most of these students (n = 21, 81%) made their own study materials while the remaining 5 (19%) used materials made by someone else. A large portion (94.5%) of students, both those who did and did not use Anki, also reported using third party resources as study tools. Of the 156 who used third party resources, 68 (43.6%) used them daily. Most students (n = 111, 67%) of students reported using a question bank. Of these, 1 (1%) student did 0 to 1000 questions, 5 (3.5%) did 1001 to 2000, 24 (17.2%) did 2001 to 3000, 26 (18.7%) did 3001 to 4000, 20 (14.3%) did 4001 to 5000, did 16 (11.5%) did more than 5001 questions. Most students continued to study from medical school lectures in addition to Anki and third-party resources, with 118 (71%) watching 90% to 100% of lectures. Only 14 (8%) students reported watching < 25% of medical school lectures.
Of the 139 students that reported using Anki, 92 (66%) completed their reviews of previously seen cards on a daily basis, 27 (19%) did them every few days, 3 (2%) did them weekly, 14 (10%) did them on a nonregular basis, and 3 (2%) did them only immediately prior to an exam or quiz. Students varied in how many new cards they introduced in a day, with most 86 (62%) introducing < 150 new cards per day. Demographic data is described in Table 1. There were no significant differences in age or race/ethnicity between the 2 groups. Most respondents (70.3%) identified as female. Comparing male-identifying respondents with female-identifying respondents suggested that male students were more likely to be daily Anki users than their female counterparts (P = .005). There was no significant difference in reported pre-med GPA or MCAT scores between daily and nondaily Anki users. Additionally, no significant differences in perceived specialty competitiveness or plans to take a resume-building year were identified between the 2 cohorts.
Baseline demographics and pre-medical school characteristics of participants based on frequency of Anki use.
Abbreviations: IQR, interquartile range; GPA, grade point average; MCAT, Medical College Admission Test.
Number (%).
Median (IQR).
*P-value <.05.
At the time of completing the survey, 99/165 (60%) of respondents had received a Step 1 score and 52 (31.5%) had received a Step 2 score. As shown in Table 2, we found a significant positive correlation between daily Anki use and increased Step 1 score (P = .039). There was no such difference for Step 2 scores. Anki use did not have a notable association with length of self-reported dedicated study period for Step 1. Significant relationships were not observed between the cohorts in reported weekly hours spent on volunteering, research, employment, or leadership roles. There was also no identified association in quantity of research publications and presentations completed between the 2 groups. Daily Anki use did have a significant association with improved perceived sleep quality (P = .01), with no association in reported sleep quantity. Analysis of additional measures of perceived wellness, including stress, burn out, and amount of leisure time did not reveal any statistically significant correlations, as shown in Table 3.
Academic and extracurricular outcomes of participants based on daily use of spaced repetition flashcards or non-daily of spaced repetition flashcards.
Abbreviations: USMLE, United States Medical Licensing Examination; IQR, interquartile range.
Number (%).
Median (IQR).
*P-value <.05.
Differences in wellness outcomes among medical students based on frequency of Anki use.
Abbreviations: IQR, interquartile range; EC, extracurricular activities.
Number (%).
Median (IQR).
Discussion
In this descriptive, cross-sectional study, we found that daily use of Anki was associated with increased USMLE Step 1 score and higher sleep quality among medical students at our institution. We did not find, however, an association between daily Anki use and USMLE Step 2 score, extracurricular involvement, or measures of student wellness. These results suggest that there may be some benefit to using Anki spaced repetition software in medical school but that it is not crucial to achieving academic outcomes and promoting wellbeing.
We found a strong association between the use of Anki and increased USMLE Step 1 score. This is not surprising given that several studies have demonstrated a relationship between the use of Anki and USMLE Step 1 score.5,13 A potential alternative explanation for this finding is that the students that used Anki spaced repetition software daily simply studied more than the other students. We did not compare hours of daily study between the 2 groups.
In contrast, we found no relationship between the use of Anki spaced repetition software and USMLE Step 2 score. This could be explained by several factors. First, only 32% of our sample had a USMLE Step 2 score compared to 60% of the sample with a USMLE Step 1 score. It is possible that we did not have statistical power to detect differences in our population due to our small sample. Second, we did not specifically measure whether students used Anki in their third and fourth years of medical school, only in the first 2 years. Therefore, it is possible that students altered their use of Anki spaced repetition software leading up to USMLE Step 2, resulting in no association. Third, the clerkship experience, with on-the-job learning and shelf exams obscuring the importance of prolonged dedicated study period, may itself be a great equalizer for Step 2. Finally, the Anki spaced repetition decks used for USMLE Step 2 are less developed and lower quality than the decks for USMLE Step 1, which could also explain the lack of difference.
We also found an association between daily Anki use and improved sleep quality. It is not clear what may be driving this relationship, and it is possible this finding is a type I error. One potential explanation is that students using Anki spaced repetition had increased efficiency and thus more time for sleep, but the number of hours per sleep between the groups was equivalent. Another potential explanation is that students using Anki spaced repetition software had lower stress and, therefore, better sleep; however, there was no significant difference between perceived stress scales or feelings of burnout between the groups. Lastly, it is possible that students who used Anki daily had more solidified routines, leading to improved sleep. That said, we did not account for year in medical school when assessing wellness measures such as sleep. Third- and fourth-year students were more heavily represented in the non-daily user cohort and are more likely to be subject to varied sleep schedules due to clinical rotations. This may help explain the difference in sleep quality between groups. As such, the most likely explanation is that this finding does not represent a true relationship. Indeed, we found no difference between groups in daily leisure time and the frequency at which studies interfered with social events.
We did not find significant differences in extracurricular engagement based on Anki use. As there was no difference in the perceived competitiveness of the specialties sought by the 2 groups, students’ motivation to pursue extracurriculars was likely somewhat similar. Additionally, there was no significant difference in reported leisure time between the groups indicating that the groups likely had comparable amounts of time to spend on extracurriculars. There were also no significant differences in perceived stress or feelings of burnout. This suggests that study method does not necessarily impact the stress and burnout caused by attending medical school. All these findings taken together support the idea that there is no single right way to study in medical school, and that many methods of study can achieve similar academic, extracurricular, and wellness outcomes. More work should be done to determine additional ways to optimize student wellness as well as academic outcomes and extracurricular engagement.
Our study has several limitations. Most notably, our response rate was only 18%. This limits our generalizability and represents potential sampling bias. Furthermore, this response rate limited our sample size and increased the likelihood of a type II error. We attempted to get as many responses as possible by sending the survey to the school-maintained class distribution lists. We also sent multiple reminders to fill out the survey and used a gift card raffle to incentivize students. Despite this, students may not have seen the email or may have been reluctant to fill out a survey linked to their academic information. In future studies, sending the survey to a large, national sample could improve generalizability and power and avoid these issues. Another limitation of our study was the self-reported nature of many of the items. While some outcomes like USMLE scores were obtained from the MEOC database, many outcomes were self-reported by students. This may have led to social desirability bias or accidental misreporting. For example, students may have over or under-estimated the hours they put into various extracurriculars. Using more formal materials like ERAS applications could avoid this issue in the future. Nevertheless, it's not clear that bias or misreporting would favor either Anki or non-Anki groups of students.
Conclusions
We found a significant association between daily use of Anki spaced repetition software in medical school and increased USMLE Step 1 score and higher sleep quality. No other academic, extracurricular, or wellness outcomes were significantly different between students who used Anki every day and those who did not. Our study demonstrates some potential benefits of daily use of Anki spaced repetition software while also reinforcing the idea that a variety of study methods can be used to achieve similar academic, extracurricular, and wellness outcomes.
Supplemental Material
sj-pdf-1-mde-10.1177_23821205231173289 - Supplemental material for Academic and Wellness Outcomes Associated with use of Anki Spaced Repetition Software in Medical School
Supplemental material, sj-pdf-1-mde-10.1177_23821205231173289 for Academic and Wellness Outcomes Associated with use of Anki Spaced Repetition Software in Medical School by Jillian K Wothe, Lindsey J Wanberg, Rae D Hohle, Aliya A Sakher, Laura E Bosacker, Faizel Khan, Andrew PJ Olson and David J Satin in Journal of Medical Education and Curricular Development
Footnotes
Acknowledgments
The authors would like to thank Sacha Brueggemann for her assistance with this project.
FUNDING
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author(s) received funding for this study from an Academic Investment for Education Program grant at the University of Minnesota Medical School and University of Minnesota Medical School Student Council.
DECLARATION OF CONFLICTING INTERESTS
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Author Contributions
Conception and design: JKW, LJW, RDH, AAS, APJO, and DJS. Acquisition of data: JKW, LJW, RDH, AAS, LEB, APJO, and DJS. Analysis of Data: FK. Interpretation of Data: JKW, LJW, RDH, AAS, FK, APJO, and DJS. Drafting of article: JKW, LJW, RDH, AAS, and LEB. Critical Revisions: JKW, LJW, RDH, AAS, LEB, FK, APJO, and DJS. Final Approval: JKW, LJW, RDH, AAS, LEB, FK, APJO, and DJS. Agree to be accountable for all aspects of the work: JKW, LJW, RDH, AAS, LEB, FK, APJO, and DJS. JKW and LJW are co-first authors due to both having completed the work typical of a first author.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
