An accepted gold standard for measuring change in participant behavior is third-party observation. This method is highly resource intensive, and many small-scale evaluations may not be in a position to use this approach. This study was designed to assess the validity and reliably of aggregated group self-assessments as one way to measure workshop effectiveness. In this study, participants completed a pre-, post-, and retrospective self-assessment on their perceived skill level in delivering feedback. Trained raters scored recorded role-play episodes. A statistically and practically significant difference in feedback skills was detected in both the self-assessments and observer ratings. The instruments used to assess participants' feedback skills had acceptable reliability. Those charged with workshop evaluation should have some confidence that aggregated self-assessments can be used to help determine workshop effectiveness.
Albanese, M., Dottl, S., Mejicano, G., Zakowski, L., Seibert, C., Van Eyck, S., et al. (2006). Distorted perceptions of competence and incompetence are more than regression effects. Advances in Health Sciences Education, 11, 267-278.
2.
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time, and data constraints. American Journal of Evaluation, 25, 5-37.
3.
Bardella, I.J., Janosky, J., Elicki, D.M., Ploof, D., & Kolarik, R. (2005). Observed versus reported precepting skills: Teaching behaviours in a community ambulatory clerkship. Medical Education , 39, 1036-1044.
4.
Barnsley, L., Lyon, P.M., Ralston, S.J., Hibbert, E., Cunningham, L., Gordon, F.C., et al. (2004). Medical Education, 38, 358-367.
5.
Bernthal, P.R. (1995, September). Evaluation that goes the distance. Training & Development, 49, 41-45.
6.
Brooks, L., & Gersh, T.L. (1998). Assessing the impact of diversity initiatives using the retrospective pretest design. Journal of College Student Development, 34, 383-385.
7.
Davis, D.A., Mazmanian, P.E., Fordis, M., Van Harrison, R., Thorpe, K.E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence. Journal of the American Medical Association, 296, 1094-1102.
8.
D'Eon, M.F. (1997). Strengthening faculty development in medical education through action research. Unpublished doctoral dissertation, University of Saskatchewan, Saskatoon, Canada.
9.
D'Eon, M.F. (2004). Evaluation of a teaching workshop for residents at the University of Saskatchewan: A pilot study. Academic Medicine, 79, 791-797.
10.
Eva, K.W., Cunnington, J.P.W., Reiter, H.I., Keane, D.R., & Norman, G.R. (2004). How can I know what I don't know? Poor self-assessment in a well-defined domain. Advances in Health Sciences Education: Theory to Practice, 9, 211-224.
11.
Evans, A.W., Leeson, R.M.A., Newton John, T.R.O., & Petrie, A. (2005). The influence of self-deception and impression management upon self-assessment in oral surgeryBritish Dental Journal, 198, 765-769.
12.
Evans, A.W., McKenna, C., & Oliver, M. (2002). Self-assessment in medical practice. Journal of the Royal Society of Medicine, 95, 511-513.
13.
Fitzgerald, J.T., Gruppen, L.D., & White, C.B. (2000). The influence of task formats on the accuracy of medical students' self-assessments. Academic Medicine, 75, 737-741.
14.
Fitzgerald, J.T., White, C.B., & Gruppen, L.D. (2003). A longitudinal study of self-assessment accuracy . Medical Education, 37, 645-649.
Hojat, M., & Xu, G. (2004). A visitor's guide to effect sizesAdvances in Health Sciences Education: Theory to Practice, 9:241-249.
17.
Howard, G.S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93-106.
18.
Howard, G.S., Millham, J., Slaten, S., & O'Donnel, L. (1981). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 144-150.
19.
Hyman, R.B. (1993). Evaluation of an intervention for staff in long-term care facility using a retrospective pretest design. Evaluation in the Health Professions, 16, 212-224.
20.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134.
21.
McEvoy, G.M., & Buller, P.F. (1990, August). Five uneasy pieces in the training evaluation puzzle. Training and Development Journal, 44, 39-40.
22.
Nachmias, D., & Nachmias, C. (1987). Research methods in the social sciences (3rd ed.). New York: St. Martin's.
23.
Orsmond, P., Merry, S., & Reiling, K. (1997). A study in self-assessment: Tutor and students' perceptions of performance criteria. Assessment and Evaluation in Higher Education, 22, 357-368.
24.
Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.
25.
Reid, A., Stritter, F.T., & Arndt, J.E. (1997). Assessment of faculty development program outcomes . Family Medicine, 29, 242-247.
26.
Rennie, P.R. (1996, November). Evaluating training. Management, pp. 23-24.
Skeff, K.M., Stratos, G.S., & Bergen, M.R. (1997). A regional teaching improvement program for community-based teachers. American Journal of Medicine, 106, 76-80.
29.
Sprangers, M., & Hoogstraten, J. (1991). Subject bias in three self-report measures of change. Methodika, 5, 1-13.
30.
Stone, S., Ferguson, W., Mazok, K., Wellman, S., Devaney-O'Neil, S., Jacobson, E., et al. (2003). Development and implementation of an objective structured teaching exercise (OSTE) to evaluate improvement in feedback skills following a faculty development workshop. Teaching and Learning in Medicine, 15, 7-13.
31.
Vnuk, A., Owen, H., & Plummer, J. (2006). Assessing proficiency in adult basic life support: Student and expert assessment and the impact of video recording. MedicalTeacher, 28, 429-434.
32.
Ward, M., Gruppen, L., & Regehr, G. (2002). Measuring self-assessment: Current state of the art. Advances in Health SciencesEducation, 7, 63-80.
33.
Wayne, D.B., Butter, J., Siddall, V.J., Fudala, M.J., Wade, L.D., Feinglass, J., & McGaghie, W.C. (2006). Graduating internal medicine residents' self-assessment and performance of advanced cardiac life support skills. Medical Teacher, 28, 365-369.
