Evaluation of a Resident Selection Process in Plastic Surgery

Abstract

Methods of residency selection continue to be a contentious issue, with little being known about the reliability of the various components of the selection process. By determining the reliability and validity of application reviews and interview judgements, the process of selecting candidates for a residency program can be made more systematic and more efficient. The present study examined the selection process used at the University of Toronto Division of Plastic Surgery for the 1997 selection year. Analysis revealed the initial review of the 51 applications to have high inter-rater reliability, with a panel of three judges producing a reliability of greater than 0.80, and high test-retest reliability (r=0.75). A second review of the 14 applications from candidates selected for interview was somewhat less reliable (probably due to the restriction of range in the scores) and required at least six raters to obtain a reliability of 0.80. Inter-rater reliability of the scores from two independent interviews was quite high, with four raters providing a reliability of at least 0.80 in both interviews. Test-retest reliability between the two interviews was 0.77. Finally, the overall ‘gut’ rankings of the 11-member selection committee were quite reliable and were predicted by a combination of application scores and interview scores (multiple R of 0.96). At least within this program, each component of the selection process appeared to be quite reliable and final ‘gut’ decisions were well founded in a reasonable balance of application scores and interview scores.