Abstract
Background:
The Orthopaedic In-Training Examination (OITE) is a standardized examination administered annually to orthopaedic surgery residents. The examination is designed to evaluate resident knowledge and academic performance of residency programs.
Methods:
All OITE foot and ankle questions from 2009 through 2012 and 2017 through 2020 were analyzed. Subtopics, taxonomy, references, and use of imaging modalities were recorded.
Results:
There were a total of 167 foot and ankle (F&A)–related questions across 8 years of OITE examinations. Trauma remained the most commonly tested subtopic of F&A across both subsets, followed by rehabilitation, tendon disorders, and arthritis. We found an increase in questions related to arthritis (P = .05) and a decrease of questions related to the diabetic foot (P = .02). Taxonomy 3 questions constituted 49.5% of F&A questions from 2009 through 2012 compared with 44.7% of questions from 2017 to 2020 (P = .54). Radiography was the most commonly used imaging modality in both subsets. From 2009 to 2012, 63.6% of questions included a radiograph compared with 76.5% in 2017 through 2020 (P = .13). FAI (Foot & Ankle International), JAAOS (Journal of the American Academy of Orthopaedic Surgeons), and JBJS (The Journal of Bone and Joint Surgery) were the most commonly cited journals, making up more than 50% of total citations. Citations per question increased from 2.20 to 2.42 from 2009-2012 to 2017-2020 (P = .01). The average lag time in the early subset was 8.2 years and 8.9 years in the later subset.
Conclusion:
This study provides a detailed analysis of the F&A section of the OITE. Use of this analysis can provide residents with a guide on how to better prepare for the OITE examination.
Level of Evidence:
Level IV, cross-sectional review of Orthopaedic In-Training Examination questions
Introduction
The Orthopaedic In-Training Examination (OITE) was developed by the American Academy of Orthopaedic Surgeons (AAOS) in 1963. The purpose of this examination was to determine if each residency program was maintaining minimum academic standards for orthopaedic education and achieving educational goals. 11 The OITE is taken annually by orthopaedic surgery residents and consists of 275 questions evaluating 11 different domains that include basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The examination test is now administered to more than 4000 residents across 20 countries annually. 4
After finishing residency, graduating orthopaedic surgery residents take the ABOS (American Board of Orthopaedic Surgery) Board Certification examinations. Although several studies have found a correlation between success on the OITE and passing the ABOS Board Certification examinations, these 2 examinations have been historically written by different organizations without a universal blueprint.6,9,10,16 Most recently, a comprehensive review of OITE and Part I Certifying Examination scores from 2014-2018 found an increasing correlation between examination performance with increased residency training, noting a correlation of 0.504 for senior residents (PGY4s and PGY5s). However, a minimum OITE score threshold that corresponded to passing the Part I Certifying Exam was not identified. 8 The usefulness of the OITE has been criticized because of this mismatch, outside of its ability to compare residents on baseline knowledge. The ABOS and AAOS have recently begun to collaborate on developing questions that are included on both the Part I Certifying Examination and the OITE, with the ultimate goal “to identify the score on the AAOS OITE that approximately corresponds to the minimum passing performance level on the ABOS Part I Certifying Examination.” 12 This score would be a valuable resource in guiding resident preparation for the ABOS examination. In 2020 and 2021, the 2 examinations included some shared questions that were included in a pilot linking study to determine the OITE score that were associated with a passing score on the ABOS Part I Certifying Examination. 7 This linking study found that on the 2021 AAOS OITE, the minimum OITE score that corresponded to passing the Part I Certifying Examination was 69.2%, but this score was noted to be applicable to the 2021 OITE examination only. 12 The AAOS adopted a new blueprint in 2021 that was developed by the ABOS for the purpose of test development and examination administration, to further align the OITE with the ABOS Part I Certifying Examination. 12
Therefore, it has become more important for orthopaedic residency programs to guide resident learning to prepare for annual OITE examinations. The purpose of this study was to provide an updated assessment on trends of the foot and ankle (F&A) domain of the OITE. This section constitutes 10% of OITE examination questions according to the recently adopted blueprint. 12 Our goal was to analyze changes in the examination questions over the last 10 years. To our knowledge, there has not been a recent analysis of F&A questions since Barr et al examined OITE questions from 2006 to 2010 examinations. 2 This study attempts to answer the following questions: (1) Have the most common subsections of F&A questions changed? (2) Has the taxonomy of questions changed in recent years? (3) Has there been a change in frequency of imaging modalities used in questions on recent examinations? (4) Has there been a change to the most commonly referenced sources and the number of references used to support each question? and (5) Has there been a change in lag time of reference publication to examination date over this time period? The purpose of this manuscript is to provide insight on how programs can best support resident education regarding the topic of orthopaedic F&A and prepare them for the OITE examination.
Materials and Methods
A retrospective review of all OITE examination questions from 2009 to 2012 and 2017 to 2020 was conducted by 2 independent reviewers. Any discrepancies were discussed with a third reviewer. No Institutional Review Board (IRB) approval was needed for this study. Overall, 167 of 2116 (7.9%) questions were related to F&A. Ninety-one (8.3%) of the 1100 questions from 2009 to 2012, and 76 (7.5%) of 1016 questions from 2017 to 2020, were determined to be part of the F&A section. Data from years 2009 through 2012 were obtained from past examinations as distributed by AAOS in PDF format. Questions were not delineated by content section, and reviewer judgment was made to best determine which were designed to assess knowledge of the F&A. Questions from 2017 through 2020 were obtained directly from the AAOS website via the ResStudy online portal. 1 Data from 2014 to 2016 were not able to be obtained as the AAOS website did not begin to list questions online until 2017. PDF formatting of these questions were unable to be obtained from previous test takers. Data from 2013 was available in PDF format; however, to maintain the same time frame for both early and late data sets, questions from 2013 were omitted. Questions were reviewed by first examining the F&A subsection and subsequently reviewing all other questions for those focused on F&A, which were designated as another subsection by the AAOS. All included F&A questions from the OITE examination were further categorized into subtopics based on subtopics previously used by Barr et al. 2
Question taxonomy was determined for F&A questions based on the system described by Buckwalter et al. 3 In brief, taxonomy 1 (T1) classification is described as isolated recognition and recall of specific information; taxonomy 2 (T2) classification is described as utilizing diagnostic data and imaging interpretation; taxonomy 3 (T3) classification is described as the application of knowledge or interpretation of information to solve a problem.
Other examination information including references, reference lag time, and use of imaging modality (eg, computed tomography [CT], magnetic resonance imaging [MRI], and radiography [XR]) were assessed in this study. For each OITE question, references are cited. The authors recorded the most commonly referenced journals, as well as total number of references cited per question. Reference lag time was calculated by taking the year of the OITE examination subtracted by the reference publication year. The use of imaging modalities and type of imaging modality provided (radiography, CT, etc) were recorded for each question.
Statistical analysis was performed using Excel and GraphPad. Continuous data between 2 groups were compared with independent 2-tailed t tests and 2-sample z test for proportions. Categorical data for groups compared via chi-square tests. Descriptive statistics are described means with SDs and proportions.
Results
F&A Questions
We reviewed 8 years of OITE data that assessed 167 questions related to the F&A. For examinations between 2009 and 2012, 91 questions (8.3%) were designated as part of the F&A section. Comparatively, between 2017 and 2020, 76 questions (7.5%) were considered F&A (Figure 1).
The number of foot and ankle OITE questions was assessed for each year demonstrating a decreasing trend in newer examinations. OITE, Orthopaedic In-Training Examination.
Subtopics
Each F&A question was further categorized into a subtopic. Among questions from 2009 through 2012, the most common subtopics included trauma (23.1%), rehabilitation (18.7%), tendon disorders (9.9%), and the lesser toe (9.9%). From 2017 to 2020, the most common subtopics were trauma (28.9%), arthritis (14.5%), and rehabilitation (13.2%) (Table 1).
Foot and Ankle OITE Questions a
Abbreviations: OITE, Orthopaedic In-Training Examination; MTP, metatarsophalangeal.
Foot and ankle OITE questions were further categorized into subtopics per previous categorization by Barr et al. 2 Trauma remained the most common subtopic in both subsets. Rehabilitation remained prominent in both subsets. There was an increase in questions related to F&A arthritis and a decrease in those related to the diabetic foot.
Taxonomy
Taxonomy was assigned to each question. Among the 167 total questions, 44 (26.3%) were labeled T1, 44 (26.3%) were T2, and 79 (47.3%) were T3. From 2009 to 2012, questions categorized as T1 made up 22.0%, T2 28.6%, and T3 49.5%. In comparison, from 2017 through 2020, questions categorized as T1 were 31.6%, T2 23.7%, and T3 44.7% (Figure 2).
Foot and ankle OITE questions were graded in taxonomy according to the classification system developed by Buckhalter et al. 3 Although type 2 and 3 questions decreased when comparing the early to later years, taxonomy 3 questions remained the most prevalent. OITE, Orthopaedic In-Training Examination.
Imaging
F&A questions that required the use of image interpretation included 67.7% of all questions. These questions provided at least 1 image of radiography, CT, MRI, or clinical photo/video. Of the questions from 2009 to 2012, 72.5% used at least 1 form of imaging in comparison to 61.8% of F&A questions from 2017 to 2020. From 2009 through 2012, the most commonly used imaging was radiography (63.6%) followed by clinical picture/video (16.9%), MRI (13.0%), and CT (5.2%). In 2012, one question provided an arthroscopic image (1.3%). From 2017 to 2020, the most commonly used imaging was radiography (76.5%) followed by MRI (9.8%) and CT (9.8%), whereas only 2 clinical photos (3.9%) and no arthroscopic images were provided (Figure 3).
Imaging modalities that were provided in an OITE question stem for interpretation were recorded. Radiography remained the most prominent imaging modality provided on the examination across both subsets. OITE, Orthopaedic In-Training Examination.
References
A total of 384 references were cited across 167 F&A questions. The average references cited per question were 2.20 (SD 0.50) and 2.42 (SD 0.74) in the early and later periods, respectively (Figure 4). From 2009 to 2012, the most referenced journals included Foot & Ankle International (21.5%), Journal of the American Academy of Orthopaedic Surgeons (18%), and the Journal of Bone and Joint Surgery (17%). This coincided with questions from 2017 to 2020, where Foot & Ankle International (25%), Journal of the American Academy of Orthopaedic Surgeons (22.3%), and Journal of Bone and Joint Surgery (14.1%) were the most commonly referenced sources (Figure 5).
The average number of references provided for each OITE question was compared for each examination year. When comparing the early subset to the later subset, there was an increase in the number of references provided for each question in newer examinations (P = .01).
FAI (Foot & Ankle International), JAAOS (Journal of the American Academy of Orthopaedic Surgeons), and JBJS (The Journal of Bone and Joint Surgery) were the most commonly referenced journals in both subsets, comprising more than 50% of total citations. Journal of Orthopaedic Trauma (JOT), Surgery of Foot and Ankle (SFA), Foot and Ankle Clinics (FAC), and Clinical Orthopaedics and Related Research (CORR) were also referenced multiple times across both subsets. Because of the large number of unique sources referenced by questions, all other sources were grouped as “Other.”
Lag Time
The lag time from OITE examination year to reference publication date was used to determine the novelty of research applied to the examination. For F&A questions from 2009 to 2012, references were published an average of 8.22 years (SD 7.63) before their respective OITE citation. In comparison, references in 2017 through 2020 questions were published an average of 8.94 (SD 9.01) years prior (Figure 6).
The time difference between reference publication date and OITE examination date, defined as lag time, was calculated for each reference provided. The average lag time of questions from each year were calculated. In comparing the early subset to the later subset, there was a small increase in lag time but not significant (P = .20).
Discussion
The F&A subspecialty within orthopaedic surgery and its emphasis within orthopaedic residency training has continued to grow in recent years. In 2003, only 64.9% of orthopaedic surgery residency programs had a dedicated F&A rotation, whereas 54% of programs had only 1 F&A fellowship-trained attending. 14 In 2010, programs were surveyed that revealed an increase in the proportion of programs with a dedicated F&A rotation (80%), as well as an increase in programs that had at least 1 surgeon with predominantly F&A practice (91.3%). 13 Although this demonstrates an improved F&A experience for orthopaedic residents, many programs do not have a dedicated F&A rotation. Therefore, a detailed analysis of the F&A portion of the OITE was performed to assist with resident preparation.
F&A Questions
F&A questions made up 8.3% of questions from 2009 to 2012 and 7.5% of questions from 2017 to 2020 (P = .50). Barr et al 2 examined OITE questions from 2006 through 2010 and found that nearly 14% of questions pertained to F&A surgery. Srinivasan et al 15 also found that 14% of OITE questions from 2004 to 2008 comprised the F&A section. Our results represent a downtrend in F&A-related questions compared with prior studies.
Subtopics
From 2009 to 2012, the most common subsections of F&A questions were found to be trauma (23.1%), rehabilitation (18.7%), tendon disorders (9.9%), and the lesser toe (9.9%). Although trauma (28.9%) remained the most common subsection in the 2017 through 2020 group, arthritis (14.5%) became the second most common subsection, which increased by 9.0% across year groupings, followed by rehabilitation (13.2%). Diabetic foot questions were noted to have decreased by 8.6%. The increase in arthritis questions (P = .05) and decrease in diabetic foot questions (P = .02) were found to be significant. The increase in trauma questions (P = .39) and tendon disorders (P = .68) and decrease in questions on rehabilitation (P = .33) and the lesser toe (P = .06) were not found to be significant. All other changes within remaining categories were not found to be significant.
Srinivasan et al found that the most common subtopics from 2004 through 2008 were basic anatomy, diabetes mellitus, and posterior tibial tendon insufficiency. 15 Barr et al subclassified F&A questions in their analysis of OTIE questions from 2006 to 2010 and found that 6 subtopics made up 75% of the questions, including F&A trauma (29%), the pediatric foot (12.9%), foot deformity (10.2%), the diabetic foot (8.1%), rehabilitation (7.5%), and the great toe (7%). 2 In comparing our results to the data from Barr et al, trauma has remained the most tested subtopic within this section of the examination, demonstrating its importance to resident education.
Taxonomy
From the early subset to the later subset, there was an increase in T1 questions (+9.6%). This increase corresponded with an associated decrease in T2 (–4.9%) and T3 (–4.8%) questions. However, changes of taxonomy grading were not found to be significant: T1 P = .16, T2 P = .48, T3 P = .54). Taxonomy grade 3 made up nearly half of F&A questions during both time periods. This suggests that the test makers expect residents not only to perform simple recall but to use the application and integration of information to correctly answer the question.
Srinivasan et al found that questions from 2004 through 2008 were primarily T1 (44.5%) and T3 (41.9%), whereas only 13.6% were classified as T2. 15 Barr et al found that T1 questions made up the majority of questions on the OITE from 2006 to 2009 (45%-56%). 2 Our results show an increase in taxonomy grading compared with prior studies, representing an expectation of stronger comprehension of information within the F&A domain.
Imaging
There was a decrease in F&A questions that required image interpretation (77.5% to 61.8%) in more recent years, but this decrease was not found to be significant (P = .14). The use of radiography (12.9%) and CT (4.6%) imaging increased from the early subset to the later subset, whereas the use of MRI (3.2%), clinical photo/video (13%) and arthroscopic images (1.3%) decreased in later years. The difference in radiographic imaging (P = .13), MRI (P = .58), and CT (P = .32) were not found to be significant. The decrease in use of clinical photos in the latter group was found to be significant (P = .03).
Srinivasan et al 15 found that 41% of F&A questions from 2004 through 2008 required evaluation of imaging modalities. Radiography was the primary imaging modality provided, accounting for 76% of all imaging. Barr et al 2 found that 48.4% of F&A questions used at least 1 imaging modality. The most common modality provided was radiography, present in 83.3% of all F&A questions that included imaging. Our study demonstrates an increased emphasis on the interpretation of F&A imaging for questions on the OITE, while interpretation of radiographs in particular remained a priority by test makers.
References
There was a significant increase of 0.22 references per question from 2009 through 2012 to 2017 through 2020 (P = .01). The top 3 most referenced journals remained the same between the 2 time periods, which included FAI, JAAOS, and JBJS. The changes in the use of Foot & Ankle International (FAI) (P = .42), Journal of the American Academy of Orthopaedic Surgeons (JAAOS) (P = .29), and Journal of Bone and Joint Surgery (JBJS) (P = .44) were not found to be significant.
Barr et al 2 found an average of 84 references cited per year for F&A questions in OITEs from 2006 to 2010. In their analysis, there were approximately 2 references per question. 2 Our data also found just over 2 references cited per question. Barr et al also looked at the most common journal cited. Their results coincided with ours in that FAI, JBJS, and JAAOS were the most commonly cited journals, making up 42.3% of total references. 2 Our data show an increase in the total references from the same 3 sources, indicating the increasing relevance of FAI, JBJS, and JAAOS to the F&A community and the OITE. Residents should primarily focus on articles from these 3 journals when preparing for the F&A section of the OITE.
Lag Time
To our knowledge, there has not been a previous study that has evaluated lag time of F&A OITE questions. Lag time represents the discordance between recent literature and literature that is being tested on OITE examinations. References with a lag time of 1 year to 55 years in the 2009-2012 group, and 1 year to 58 years in the 2017-2020 group. Mode, defined as the lag time length that was most prevalent, was also examined. The mode of lag time from 2009 to 2012 was 5 years, whereas the mode from 2017 through 2020 was 3 years. There was an increase in lag time of 0.72 years from the early subset to the later subset (P = .20). Although the increase was not significant, the decrease in the mode within the more recent subset of years demonstrates a shift toward more recent publications referenced by F&A questions.
Limitations
This study does not come without limitations. The study was unable to evaluate OITE questions from 2014 through 2016 because of inability to obtain these questions, and the decision was made to omit questions from 2013 in order to maintain even time frames of each data set. The exclusion of this data provides a smaller sample size and limits our evaluation of the time point at which changes within F&A questions occurred.
Questions from 2017 through 2020 were obtained through the AAOS website, which divided the questions into subsections, including F&A. It is possible that an F&A question was placed under another subsection by the AAOS. Although all questions were reviewed to include F&A questions appearing under a different subsection, the total number of F&A questions could be underestimated. Questions evaluated from 2009-2012 examinations were not located on the AAOS website, but instead were reviewed as PDF files of the examination, as the AAOS did not post examination questions prior to 2017. The PDF version did not have questions divided into subsections, and thus the reviewers determined which questions fell within the F&A category. Although questions were reviewed by 2 separate authors, it is possible that questions were labeled as F&A that should not have been, and questions that were not labeled as F&A that should have been. This could have led to a misrepresentation of the number of intended F&A questions.
The taxonomy assigned to each question was determined by reviewers the previous classification scheme described by Buckwalter et al. 3 This scheme was shown to have 85% agreement between the test makers and residents when evaluating questions. 3 Although validated, the subjective nature of assigning taxonomy grades could have skewed results.
Although this study evaluated trends of F&A questions on the OITE, its utility in assisting resident preparation for the ABOS board certification examinations remains unclear. Subsequent studies evaluating trends of OITE questions after the recent collaboration between ABOS and AAOS would be warranted to determine the changes taken by the AAOS to align their examination with the board certification examination.
Conclusion
Performance on the F&A domain of the OITE has been found to improve up to 12% after completion of a dedicated F&A rotation. 5 Although all programs may be unable to arrange a dedicated F&A rotation for their residents, our analysis of F&A questions can be used to guide residents and their residency programs to effectively design curricula to achieve success on the OITE and sets the framework for future studies to evaluate changes in this section of the OITE with the recent collaboration between AAOS and ABOS to better align the examinations.
Footnotes
Ethical Approval
Ethical approval was not sought for the present study because no patient information or clinical information was utilized for the study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. ICMJE forms for all authors are available online.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
