The American Board of Vascular Medicine Endovascular Board Examination – Raising the bar by testing the test

Over the past several decades, efforts to measure, publicly report, and reward physician performance have gained increasing importance. However, currently available metrics to assess physician quality and clinical performance are far from ideal. ¹ Board certification was designed as one such measure, to provide an overall assessment of physician competence. ² Certification by a medical specialty board is meant to indicate that a physician has the knowledge, experience, and skills for providing quality health care within a given specialty. Yet, data supporting the association between board certification status and provision of superior quality of care are limited and somewhat controversial.^3–6 Therefore, it is important to critically evaluate the content of board certification exams on an ongoing basis to ensure that it is not only current, but also directly relevant to the care to be provided by the group of physicians who will be taking the exam.

In 2002, the Society of Vascular Medicine (SVM) conducted a survey of its members to determine how interested they would be in obtaining certification in vascular medicine. ⁷ Among those who responded, 95% voted in favor of being recognized as a vascular medicine specialist through board certification. ⁷ This led to the inception of the American Board of Vascular Medicine (ABVM) in 2003 as a collaborative effort of the SVM, American College of Cardiology, and Society for Cardiovascular Angiography and Interventions. Between 2003 and 2005, volunteer subject matter experts (SMEs) in the fields of cardiology, vascular medicine, radiology, and vascular surgery convened to develop an outline for the ABVM examinations. ⁷ A comprehensive test blueprint was created by the SMEs from which test items were developed and reviewed, and ultimately 175 and 100 items were selected for the general examination and endovascular examination, respectively. The first ABVM certification exam was offered in 2005. In 2014, the number of examination questions was changed to 150 items on the general examination and 135 items (110 scored and 25 experimental items) on the endovascular examination. ⁷ The ABVM examination items and the overall exam itself have consistently demonstrated satisfactory performance characteristics (as reflected by the p-value, point-biserial correlation for each item, and Cronbach’s alpha reliability coefficient, Livingston decision consistency index for the overall exam). ⁸ Notwithstanding the above positive attributes, the proportion of items based on multidisciplinary guideline recommendations and/or evidence-based data has not been evaluated.

To address this question, in this issue of Vascular Medicine, Slovut and colleagues ⁹ examined the Class of Recommendation (COR) and Level of Evidence (LOE) for each of the 110 scored items in the 2015 ABVM endovascular examination in an effort to establish a new evidence-based metric for improving the quality of certification examinations. In addition to the traditional COR I–III and LOE A–C, items that were not eligible for assignment using the traditional evidence-based metrics were categorized as COR X and LOE X. ¹⁰ The proportion of items assigned to each COR category was: COR I=15%, COR II=40%, COR III=3%, COR X=42%. Similarly, the proportion of items assigned to each LOE was: Level A=12%, Level B=34%, Level C=32%, LOE X=22%. ⁹ The authors concluded that ‘more than half of the 2015 ABVM endovascular board examination items were supported by strong scientific evidence or fact-based knowledge’, and that ‘the use of alternate classification schema, such as ones based on evidence-based metrics (COR and LOE), may be powerful tools for improving certification exams in healthcare’. ⁹

Although the article by Slovut and colleagues ⁹ represents an important contribution to the literature, there are certain limitations that need mention. First, the use of a pooled consensus approach to categorize items can certainly be a source of significant bias. A more sound approach would be to have two or more experts categorize the questions and then measure inter-rater agreement using Cohen’s kappa coefficient. Second, the authors only studied the 2015 ABVM endovascular examination items and it remains unclear if the proportion of items in each COR and LOE have remained the same or have changed over the years, or whether the general examination items perform similarly.

Despite these limitations, Slovut and colleagues, ⁹ for the first time, have demonstrated that an ‘evidence-based’ metric can be applied to board examination items and this holds potential as a tool for improving certification exams. The question that remains is how best we can incorporate this metric into the exam construct. As pointed out by the authors, certain items on the exam require fact-based knowledge and therefore cannot be categorized using the evidence-based schema. Similarly, many important clinical questions in practice do not lend themselves to clinical trials. For example, cohort or case–control studies may be more appropriate to answer questions related to epidemiology and prognosis. Further, although randomized trials are unavailable, there may be a very clear clinical consensus that a particular test or therapy is useful or effective and is therefore considered a standard of care or best practice. Thus, from the point of view of evaluating certification exam items, COR may be a better measure than the LOE.

Perhaps a first step would be to develop a blueprint integrating the four content areas of the ABVM endovascular examination and the four CORs (I–III and X, where X would refer mainly to items requiring fact-based knowledge). Each new test item developed by the SMEs would then be assigned a content area and a COR. ABVM exam writers can then allocate the proportion of items to be included in each of the 16 categories (4 content areas × 4 COR). It would be interesting to determine if such an approach leads to superior performance characteristics of the overall exam, compared with the conventional approach. Finally, the ABVM and SVM (as well as other certifying boards and professional societies) should track the proportion of eligible physicians taking the certification exam(s), pass rates, and clinical performance of examinees to determine if improvement in certification exam standards is associated with parallel improvements in these measures. The study by Slovut and colleagues ⁹ not only lays the groundwork for future ABVM exams, but also raises the bar for other certifying boards and societies moving forward.