Objective: To explore the validity and reliability of the affective competency score (ACS), compared to a global rating measure to predict overall competency to perform a death disclosure
in a standardized patient exercise and to investigate useful thresholds of the ACS.
Methods: Thirty-seven fourth-year students underwent standardized patient training in
death disclosure during a fourth-year emergency medicine clerkship. Students were evaluated
using a checklist, an ACS, and a global rating assessment. ACS interrater reliability, interitem
reliability, item-total reliability, and split-half reliability were calculated. Area under
the curve (AUC) measurements were used to establish criterion validity.
Results: For the ACS, item-total correlations ranged from 0.76 to 0.85, 0.76 to 0.93, and 0.42 to 0.87; the split-half reliability was 0.82 (p = 0.0001), 0.86 (p = 0.0001) and 0.55 (p = 0.0007) for the standardized patient (SP), the faculty and the medical students, respectively. Interitem
correlations were adequate. A moderate interrater correlation of the ACS was observed between
the faculty observer and the SP (r = 0.47; p = 0.04); however, the medical students' selfevaluation did not correlate significantly with either the SP (r = –0.04; p = 0.79), or the faculty observer (r = 0.00; p = 0.99). The AUC for was 0.98 (95% confidence interval [CI] 0.94 to 1.00), 0.87 (95% CI 0.73 to 0.99), and 0.74 (95% CI 0.53 to 0.95) for the faculty, SP, and medical
student, respectively.
Conclusions: The ACS may be a valid, reliable, and useful measure to assess communication skills by faculty or SPs in this setting. At an ACS score of 16, 19, and 21 points for faculty,
SPs, and medical students, respectively, there is 100% specificity for the detection of
competency assessed on a global rating. However, the ACS appears to have limited reliability
and validity when used by medical students.
