Development of Vascular Surgery Skills During General Surgery Training

Abstract

Purpose: We previously have shown that performance on the National Board of Medical Examiners (NBME) part II examination does not reflect clinical skills. Many training programs use the American Board of Surgery In-Service Training Examination (ABSITE) as the only objective measure of clinical knowledge. This study evaluates the utility of the ABSITE and an objective structured clinical examination (OSCE) in measuring vascular clinical skills during general surgery residency training.

Methods: Residents' mean scores on the vascular section of an OSCE were compared with their mean overall scores on the OSCE by using a two-way analysis of variance (ANOVA). Residents' performance on each clinical section of the ABSITE (body as a whole; gastrointestinal, cardiovascular, and respiratory systems; genitourinary/head and neck/musculoskeletal, and endocrine) and a vascular subsection (VASC) were evaluated by using ANOVA.

Results: Mean vascular scores were significantly lower than mean overall scores for residents at all levels of training (P < 0.0001). Fischer's PLSD (plausible least significant difference) post hoc test revealed that significant improvement occurred between the intern and junior years (P < 0.05), but not between the junior and senior years. In contradistinction, VASC ABSITE scores were better than all other scores for both junior and senior residents, but not for interns (senior: VASC = 96%, other = 79%, P = 0.04; junior: VASC = 84%, other = 64%, P = 0.02; intern: VASC = 63%, other = 50%, P = 0.12).

Conclusions: It is assumed that residents completing residency training are competent to perform clinical vascular examinations. Our findings paradoxically showed that although residents scored highest on the clinical vascular section of the ABSITE, they scored lowest on the vascular section of the OSCE. Although both tests found evidence of improvement between the intern and junior years, neither test found a significant improvement in vascular performance between the junior and senior years. These results emphasize that ABSITE scores do not necessarily correlate with clinical competence, and they demonstrate the need for providing more objective measures of clinical performance.

Get full access to this article

View all access options for this article.

References

1. Stillman PL , Swanson DB , Smee S , et al: Assessing clinical skills of residents with standardized patients. Ann Intern Med 105:762-771, 1986.

2. Petrusa ER , Blackwell TA , Rogers LP , et al: An objective measure of clinical performance. Am J Med 83:34-42, 1987.

3. Sloan DA , Donnelly MB , Schwartz RW , et al: The objective structured clinical examination. The new gold standard for evaluating postgraduate clinical performance. Ann Surg 222:735-742, 1995.

4. Sloan DA , Donnelly MB , Johnson SB , et al: Use of an objective structured clinical examination (OSCE) to measure improvement in clinical competence during surgical internship. Surgery 114:343-351, 1993.

5. Sloan DA , Schwartz RW , Donnelly MB , et al: Faculty sensitivity in detecting medical students' clinical competence. Medical Teacher 17:335-342, 1995.

6. Hull AL , Hodder S , Berger B , et al: Validity of three clinical performance assessments of internal medicine clerks. Acad Med 70:517-522, 1995.

7. Sloan DA , Donnelly MB , Johnson SB , et al: Assessing surgical residents' and medical students interpersonal skills. J Surg Res 57:613-618, 1994.

8. Quattlebaum TG , Darden PM , Sperry JB : In-training examinations as predictors of resident clinical performance. Pediatrics 84:165-172, 1989.

9. Schwartz RW , Donnelly MB , Sloan DA , et al: Assessing senior residents knowledge and performance. Surgery 116:634-640, 1994.

10.

10. Stenchever MA , O'Toole B , Irby D : Evaluating student performance in an obstetrics and gynecology clerkship. Am J Obstet Gynecol 134:235-237, 1979.

11.

11. Kastner L , Gore E , Novack AH : Pediatric residents' attitudes and cognitive knowledge, and faculty ratings. J Pediatr 104:814-881, 1984.

12.

12. Kwolek CJ , Donnelly MB , Sloan DA , et al: Ward evaluations: Should they be abandoned? J Surg Res 69:1-6, 1997.

13.

13. Sloan DA , Donnelly MB , Schwartz RW , et al: The use of the objective structured clinical examination (OSCE) for evaluation and instruction in graduate medical education. J Surg Res 63:225-230, 1996.

14.

14. Winckel CP , Reznick RK , Cohen R , et al: Reliability and construct validity of a structured technical skills assessment form. Am J Surg 167:423-427, 1994.

15.

15. Sloan DA , Donnelly MB , Schwartz RW , et al: Assessing the clinical competence of medical students and surgery residents in problem solving in surgical oncology. Ann Surg Oncol 1:204-212, 1994.

16.

16. Cohen R , Reznick RK , Taylor BR , et al: Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg 160:302-305, 1990.

17.

17. Joorabchi B : Objective structured clinical examination in a pediatric residency program. Am J Dis Child 145:757-762, 1991.

18.

18. Levine HG , McGuire CH : The validity of multiple choice tests as measures of competence in medicine. Am Education Res J 7:69-83, 1970.

19.

19. Risucci DA , Tortolani AJ , Ward RJ : Ratings of surgical residents by self, supervisors and peers. Surg Gynecol Obstet 169:519-526, 1989.

20.

20. Haber RJ , Avins AL : Do ratings on the American board of internal medicine resident evaluation form detect differences in clinical competence? J Gen Intern Med 9:140-145, 1994.

21.

21. Schwartz RA , Donnelly MB , Sloan DA , et al: The relationship between faculty ward evaluations, OSCE, and ABSITE as measures of surgical intern performance. Am J Surg 169:414-417, 1995.

22.

22. Metheny WP : Limitations of physician ratings in the assessment of student clinical performance in an obstetrics and gynecology clerkship. Obstet Gynecol 78:136-136, 1991.

23.

23. Rhoton MF , Barnes A , Flashburg M : Influence of anesthesiology residents noncognitive skills on the occurrence of critical incidents and the residents' overall clinical performance. Acad Med 66:359-359, 1991.

24.

24. Endean ED , Sloan DA , Veldenz HC , et al: Performance of the vascular physical examination by residents and medical students. J Vasc Surg 19:149-156, 1994.

25.

25. Sloan DA , Donnelly MB , Schwartz RW : The multidisciplinary structured clinical instruction module as a vehicle for cancer education. Am J Surg 173:220-225, 1997.

26.

26. Reznick R , Regehr G , MacRae H , et al: Testing technical skill via innovative “bench station” examination. Am J Surg 173:226-230, 1997.

27.

27. Reznick R , Smee S , Rothman A , et al: An objective structured clinical examination for the licentiate: Report of the pilot project of the medical council of Canada. Acad Med 67:487-487, 1992.