Endovascular medicine certification 2005–2014: Report from the American Board of Vascular Medicine

Abstract

This report describes the methods used to develop and maintain the endovascular medicine certification examination, which has been offered by the American Board of Vascular Medicine (ABVM) to practicing physicians since 2005. The report covers the methods and findings used to develop examination specifications which ensure the content and construct validity of the examination assessment such that the examination is reflective of the job tasks associated with the endovascular medicine specialty, as well as being a meaningful indicator of whether a candidate possesses the knowledge and skills necessary for competent practice. Further, this report covers the procedures used to develop, maintain, and administer the examination, including a summary of the content review process and the use of statistical information. Based on psychometric evaluation of the examination’s performance, the ABVM’s endovascular medicine examination appears to be a valid assessment of professional competency in the specialty area; a finding that supports the inference that candidates who pass the examination are qualified to practice in a manner that protects patients.

Introduction

The American Board of Vascular Medicine (ABVM) is dedicated to the certification of qualified vascular and endovascular physicians. Certification is based upon formal professional education (i.e. fellowship training) or practice in vascular medicine with experience-based requirements and examination (i.e. the practice pathway). Since 2005, the ABVM has developed and administered examinations in vascular medicine and endovascular medicine that provide evidence to the medical community of expertise in vascular medicine knowledge and skills. In this report, we describe the methods for creating and validating the endovascular examination, look at pass rates, and provide a portrait of examinees.

Methods

In 2004, an endovascular examination committee was formed that included representation by several nationally-recognized experts from the fields of cardiology, vascular medicine, radiology, and vascular surgery. Core examination content was based on the experience of committee members and included 14 major topics such as vascular anatomy and pathophysiology, arterial atherosclerotic occlusive disease, venous disease, and aneurysmal disease and aortic dissection. A 24-page endovascular blueprint was devised that provided a detailed summary of the scope of each topic area. Content balance was assigned by the committee.

Examination 2005–2013

The 2005–2013 examinations consisted of 100 items with a time limit of two hours. The format was four-option multiple choice. The initial examinations were administered on paper, but more recently, have been given via computer. After examination administration, the endovascular committee met under the direction of previous examination development vendors (ACT, Inc, 2005–2006; the American Registry for Diagnostic Medical Sonography, 2007–2012) to establish the passing score using the bookmark method. ¹ Examination questions were reviewed by a full panel of subject matter experts who represented a diversity of practice settings, geographic regions, and subspecialty practice areas. Questions that yielded statistical characteristics outside of acceptable ranges were removed from scoring, and sometimes revised for future use. Statistical characteristics considered in the review were each item’s p-value, a sample-dependent measure of item difficulty, and point-biserial correlation, a sample-dependent measure of an item’s ability to differentiate between higher- and lower-ability candidates. P-value can be interpreted as the percentage of candidates who answered the item correctly, and the acceptable range of difficulty levels for a four-option multiple choice item typically ranges from 0.25 to 0.95. Higher point-biserial correlation values indicate that higher-scoring candidates answered correctly more often than lower-scoring candidates. The acceptable range of discrimination levels for an item ranges from 0.10 and above. The endovascular writing committee provided additional questions that could be added to the item bank.

Job analysis and revision of the examination

In 2013, the ABVM Board of Directors retained PSI Services LLC, an examination development and delivery vendor, to conduct a job analysis, which is a systematic inquiry to identify the work activities, the context in which those tasks and activities are carried out, and the most crucial competencies (knowledge areas, skills, and abilities) required to perform a job role successfully. ² The aim of the job analysis process was to produce exam specifications that accurately reflected the scope of endovascular practice, allowing for the development of a fair, accurate, and realistic assessment of each candidate’s readiness for certification. Some of the principals guiding test development included:

• - Evidence of validity based on test content requires a thorough and explicit definition of the content domain of interest. Examination validity is the extent to which an examination accurately measures what it purports to measure. In this instance, the intended purpose of the endovascular medicine examination is to assess whether practicing physicians have the required competencies to perform the job tasks associated with the Endovascular Medicine specialty in a way that is not unsafe or harmful to patients.

• - When evidence of validity based on test content is presented, the rationale for defining and describing a specific job content domain in a particular way (e.g. in terms of tasks to be performed or knowledge, skills, abilities and other personal characteristics) should be stated clearly.

• - The content domain covered by a credentialing test should be defined clearly and justified in terms of the importance of content for credential-worthy performance in an occupation or profession. Rationale should be provided to support a claim that the knowledge or skills being assessed are required for credential-worthy performance in an occupation and are consistent with the purpose for which the licensing or certification program was instituted. ³

The job analysis included interviews with subject matter experts (SMEs), a draft of examination specifications, a job analysis panel meeting, a job analysis survey, and the development of examination specifications. SMEs who were interviewed were selected to represent a wide variety of work-related characteristics such as years of experience, work setting, geographical location, and areas of specialty in order to develop a scope of practice that reflects the roles and responsibilities of the job and is relatively free from bias. In order to define the scope of practice, PSI Services conducted four one-hour telephone interviews with four SMEs selected by ABVM between 18 March 2014 and 21 March 2014 to discuss tasks and competencies associated with the endovascular medicine job role. Interview questions solicited SMEs’ experiences and perspectives regarding regularly-performed job tasks, competencies required to perform job tasks, differences among practice settings, and the educational background of practitioners. SME responses were aggregated and analyzed to develop a draft content outline for future test development. Common themes and topics were identified and classified into domains and subdomains. Task and knowledge statements were written for concepts identified by the SMEs. A two-day workshop was conducted on 30–31 March 2014 in Washington, DC with eight SMEs to review the content outline, discuss the scope of practice, and make adjustments to the content outline. The SMEs refined task and knowledge statements, organized them into a domain and subdomain structure, and then estimated the relative content weighting for each subdomain.

Polling the practitioners: The job analysis survey

PSI Services LLC developed and administered a survey to validate the tasks and knowledge developed in the job analysis panel and to identify potential tasks and knowledge that were not considered central to the role. Respondents were asked three questions regarding each task and knowledge statement, and were asked to rate each task and knowledge statement on a multi-point scale:

1. Frequency – Consider whether the task is a part of your job, and if so, how often you perform the task (six-point scale ranging from ‘not relevant’ to ‘very frequently’).

2. Importance – Consider how performance of the task relates to outcomes pertaining to treatment (six-point scale ranging from ‘not relevant’ to ‘critically important’).

3. When required? – Consider whether a practitioner of endovascular medicine is expected to hold responsibility for this task upon entry to licensed practice (three-point scale consisting of: ‘not expected’, ‘some responsibility’, and ‘full responsibility’).

The survey included 11 demographic questions regarding professional characteristics relevant to the specialty. It also requested information regarding case distribution (i.e. percentage of endovascular, coronary, and structural cases). The first demographic question – ‘Are you a practicing physician in the field of endovascular medicine?’ – served as a filter to allow only those who responded in the affirmative to move forward with the rest of the survey.

The job analysis survey was sent via email using online survey software to 658 individuals who were eligible to sit for the endovascular medicine examination from 2008 to 2012. A monetary incentive was provided to those who responded to all the survey questions: four randomly-selected respondents were selected to receive a $50 gift card. The survey opened on 1 May 2014 and closed on 2 June 2014.

On 13 June 2014, six of the SME panelists met to review the results of the job analysis survey, consider potential adjustments to the content weighting based on the results of the survey, and finalize the content outline for the endovascular medicine certification examination. Table 1 shows the final weighting for the content outline. The disease states are weighted for arterial disease (64%), venous disease (23%), atypical vascular disorders (10%), and lymphatic disease (3%). The weighting reflects the judgment of the SMEs, with adjustments to reflect patterns amongst practicing endovascular physicians. The examination content outline is shown in Table 2. An endovascular examination committee, which consists of seven endovascular specialists, one vascular medicine physician, and one vascular surgeon, continues to generate content for the examination. This year (2015), the examination consists of 110 scored items with known statistical characteristics that have been validated previously and 25 ‘experimental’ items that are administered to examinees in order to evaluate their statistical performance but do not contribute to an examinee’s score. Examinees have three hours to complete the test.

Table 1.

Weights of content-sampling domains of the endovascular medicine content outline.

Knowledge	Weight (job analysis survey)	Weight (final for endovascular examination)
Pathophysiology	7.5%	4%
Epidemiology and risk factors	7.8%	4%
Clinical standards for screening	8.7%	1%
History and physical examination	12.2%	6%
Differential diagnosis, imaging, and laboratory evaluation	11.2%	10%
Endovascular interventions and surgery	11.6%	43%
Procedural complications	9.7%	12%
Medical treatment	10.5%	5%
Clinical guidelines for intervention	10.3%	4%
Prognosis, surveillance, and re-intervention	10.5%	12%

The survey weight is based on the job analysis survey, while the final weight is based on the assessment of SMEs that factors in results from the job analysis survey.

Table 2.

Examination content outline for the endovascular medicine examination, effective 2014.

	Content area	# items	% items
1	Pathophysiology, risk factors, and screening	9	8%
1.a	Pathophysiology	4	4%
1.a.1	Arterial diseases	1	1%
1.a.2	Venous diseases	1	1%
1.a.3	Atypical vascular disorders	1	1%
1.a.4	Lymphatic Diseases	1	1%
1.b	Epidemiology and risk factors	4	4%
1.b.1	Arterial diseases	3	3%
1.b.2	Venous diseases	1	1%
1.c	Clinical standards for screening	1	1%
2	Diagnosis and testing	18	16%
2.a	History and physical examination	7	6%
2.a.1	Arterial diseases	4	4%
2.a.2	Venous diseases	1	1%
2.a.3	Atypical vascular disorders	1	1%
2.a.4	Lymphatic diseases	1	1%
2.b	Differential diagnosis, imaging, and laboratory evaluation	11	10%
2.b.1	Arterial diseases	7	6%
2.b.2	Venous diseases	3	3%
2.b.3	Atypical vascular disorders	1	1%
3	Treatment and interventions	70	64%
3.a	Endovascular interventions and surgery	47	43%
3.a.1	Arterial diseases	32	29%
3.a.2	Venous diseases	10	9%
3.a.3	Atypical vascular disorders	5	5%
3.b	Procedural complications	13	12%
3.b.1	Arterial diseases	9	8%
3.b.2	Venous diseases	3	3%
3.b.3	Atypical vascular disorders	1	1%
3.c	Medical treatment	6	5%
3.c.1	Arterial diseases	2	2%
3.c.2	Venous diseases	2	2%
3.c.3	Atypical vascular disorders	1	1%
3.c.4	Lymphatic diseases	1	1%
3.d	Clinical guidelines for intervention	4	4%
4	Prognosis, surveillance, and re-intervention	13	12%
4.a	Prognosis, surveillance, and re-intervention	13	12%
4.a.1	Arterial diseases	9	8%
4.a.2	Venous diseases	3	3%
4.a.3	Atypical vascular disorders	1	1%
TOTAL		110	100%

To qualify for admission to the examination, applicants must (a) possess a valid, unrestricted license to practice medicine in the jurisdiction of practice, (b) hold primary board certification (ABIM, ABOIM, ABS, ABR) or specialty board certification in cardiology, cardiothoracic surgery, interventional radiology, vascular surgery or vascular medicine (ABVM General Examination), (c) meet the training requirements for peripheral intervention through either the practice pathway or fellowship training pathway, and (d) possess an attestation of privileges of fellowship training statement. The requirements for the formal training pathway and practice pathway are delineated on the ABVM website: http://www.vascularboard.org/cert_reqs.cfm.

Results

The secure examination has been administered annually from 2005 to the present. The total number of examinees has been 1009, as of 30 November 2014. Approximately half of the examinees were admitted under the training pathway, while the other half came via the practice pathway. The average pass rate from 2005–2013 has been 82%. The statistical performance of the examination is summarized in Table 3. In 2014, 95 people took the examination. To date, 816 examinees have passed the endovascular medicine examination and have become Endovascular Diplomates of the American Board of Vascular Medicine.

Table 3.

Statistical performance characteristics of endovascular medicine examination, from examination’s initial launch to the present.

Measure	Low range	High range	Value
Average p-value	0.00	1.00	0.67
Average difficulty level of examination questions	Very hard (none correct)	Very easy (all correct)
Average point-biserial correlation	–1.00	1.00	0.19
Average discrimination index level of questions	Poor discrimination	Perfect discrimination
Cronbach’s alpha reliability coefficient	0.00	1.00	0.73
Measure of consistency among responses to items	Completely inconsistent	Perfectly consistent
Decision consistency index	0.00	1.00	0.75
Estimate of the consistency of pass/fail decisions	Completely inconsistent	Perfectly consistent
Pass rate	0%	100%	80%
Percentage of candidates who pass	None pass	All pass

A total of 126 (19.1%) individuals responded to the job analysis survey. After excluding surveys from non-practicing physicians, responses from 108 (16.4%) Endo-vascular Medicine specialists were analyzed. Results of the complete survey are available in the online supplement.

Practitioners who took the survey represented 33 states, with the greatest number of responses being from Florida (n=11), Illinois (n=10), and Michigan (n=8). Survey respondents were primarily men (95%) between the ages of 40 and 49 years. Respondents represented a range of specialties including cardiology (76%), vascular surgery (12%), vascular medicine (6.5%), radiology (2.8%), and cardiothoracic surgery (1.9%). The majority of respondents (56%) were certified in both endovascular medicine and vascular medicine. Peripheral intervention volumes are shown in Table 4. Years in practice are shown in Figure 1, and distribution of endovascular procedures among various vascular areas is shown in Figure 2. Most respondents (91%) indicated that passing the endovascular examination was not a requirement for obtaining vascular privileges in their practice setting. Additional details regarding the job analysis survey are available in the online supplement.

Table 4.

Endovascular case volumes of physician candidates for the 2013 ABVM endovascular examination.

Number of peripheral cases per year	#	%
<100 cases/year	37	34.60%
100–200 cases/year	43	40.20%
200–400 cases/year	15	14.00%
>400 cases/year	12	11.20%
Total	107

Figure 1.

Years of experience of endovascular specialists who responded to the 2014 job analysis survey.

Figure 2.

Distribution of procedures performed by endovascular specialists who responded to the 2014 job analysis survey.

Discussion

The ABVM endovascular medicine examination is rigorous, psychometrically valid, and provides a measure of competence to practice endovascular medicine. Over the past decade, the examination has performed well, and has been continuously modified to reflect contemporary practice. From 2005–2013, the overall pass rate for first-time takers of the endovascular examination was 82%. This pass rate is at the lower end for subspecialty examinations offered by the American Board of Internal Medicine (ABIM). The subspecialty examination most analogous to the ABVM endovascular examination is the interventional cardiology examination offered by the ABIM. For that examination, the first-time taker pass rate was 92–95% from 2009–2013. ⁴

The recertification process promotes life-long learning and provides a standard for demonstrating ongoing competence to practice. The American Board of Medical Specialties (ABMS) has created a four-part construct for maintenance of certification: evidence of good professional behavior through an unrestricted medical license; life-long learning and periodic self-assessment; cognitive expertise as demonstrated by passing a secure examination; and measuring and improving performance in practice. ⁵

The general public values certification and, in a Gallup survey, indicated that when given the choice between a board-certified physician or a non-certified physician recommended by a trusted friend or family member, the majority reported that they would choose the former. ⁶ Patients expect physicians to be evaluated by patients, nurses and other physicians and pass a written test of clinical knowledge at regular intervals. ⁶ A recent review of showed than in many specialty areas, certification status was associated with better quality of care based on clinical process and outcome measures. ⁷ Thus, although 91% of practitioners in our job analysis survey responded that certification did not affect their ability to practice at their institutions, certification remains important to endovascular practitioners.

The ABVM shares the goals outlined by ABMS and has designed its recertification process to make the process as relevant as possible to practitioners, while at the same time ensuring that the process is efficient and cost effective. To qualify for recertification, practitioners must obtain a specific number of continuing medical education (CME) credits for attending meetings and courses specific to endovascular medicine. The ABVM also has devised a recertification examination that is shorter (70 items) and yields results immediately upon completing the examination.

The recently-completed job analysis is the primary mechanism by which the ABVM can continue to ensure the accuracy and defensibility of the endovascular examination. The ABVM endovascular examination continues to provide a key standard for recognizing competence to practice endovascular medicine.