In many countries, peer assessment programs based on the examination of patient charts are becoming a standard to assess physician’s clinical performance. Although data on validity of the process are acceptable, reliability issues need some improvement. This article addresses the rarely studied aspect of optimal number of patient charts for an acceptable reliable assessment. Fifteen patient charts for each of a group of 20 practicing physicians were independently reviewed by 4 professional peer assessors. Generalizability (G) and decision (D) studies were applied to the data. It appears that as few as 10 patient charts are sufficient for any assessor to obtain a G coefficient of 0.80. Results of the current study suggest the possibility of getting generalizable assessments by peer reviewer with minimal information. These results are not in accordance with the concept of case specificity in which it is claimed that performance on a case is a poor predictor of performance on a different case.
Carrière, B., Gagnon, R., Charlin, B., Downing, S., & Bordage G. ( 2008). Assessing clinical reasoning in pediatric emergency medicine: Validity evidence for a Script Concordance Test. Annals of Emergency Medicine.
2.
Gagnon, R., Charlin, B., Lambert, C., Carrière, B., & Van der Vleuten, C. (2008). Script Concordance Testing: more cases or more questions?Advances in Health Sciences Education: Theory and Practice, 15 May.
3.
Onerheim, R., Racette, P., Jacques, A., & Gagnon, R.J. ( 2008). Improving the quality of surgical pathology reports for breast cancer: A centralized audit with feedback. Archives of Pathology and Laboratory Medicine, 132, 1428-1431.
4.
Nguyen, D.-Q., Weber, F., & Gagnon, R.J. ( 2007). Adjonction de descripteurs qualitatifs à l’échelle ordinale des épreuves de format mini-CEX pour optimiser l’évaluation formative des compétences cliniques. Pédagogie médicale, 8, 2-11.
5.
Goulet, F., Jacques, A., Gagnon, R.J., Racette, P., & Sieber, W. ( 2007). Assessment of family physicians performance using patient charts: Inter-rater reliability and concordance with chart-stimulated recall . Evaluation in the Health Professions, 30, 376-392.
6.
Dumoulin, N., Gagnon, R.J., Hudon, G., Jacques, A., Lavoie, V., Leblanc P., et al. (June 2007). Évaluation de l’atteinte des objectifs de formation par des résidents finissants en médecine familiale. Le Médecin du Québec.
7.
Goulet F., Gagnon R., & Gingras M-E. ( 2007). Influence of remedial professional development programs for poorly performing physicians. Journal of Continuing Education in the Health Professions, 27, 42-48.
8.
Gagné, J.P., Billard, M., Gagnon, R.J., Laurion, M., & Jacques, A. ( 2007). Province-wide population survey of acute appendicitis in Canada. New twists to an old disease. Surgical Endoscopy , 21, 1383-1387.
9.
Meterissian, S., Zabolotny, B., Gagnon, R.J., & Charlin, B. ( 2007). Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills? The American Journal of Surgery, 193, 248-251
10.
Brennan, R. ( 2001). Generalizability theory. New York , Springer-Verlag.
11.
Colliver, J., Markwell, S., Vu, N.V., & Barrows, H. ( 1990). Case specificity of standardized-patient examinations. Consistency of performance on components of clinical competence within and between cases. Evaluation & the Health Professions, 13, 252-261.
12.
Emasson, G., Berg, L., Carlson, P., Linström, K., & Bengtsson, C. ( 1998). Facilitating quality improvement in primary health care by practice visiting. Quality in Health Care, 7, 48-54.
13.
Faulkner, D. ( 1998). Dear doctor: You have been selected for assessment by the Peer Assessment. Members’ Dialogue, 6, 8-13.
14.
Goldman, R. ( 1994). The reliability of peer assessments. A meta-analysis . Evaluation & the Health Professions, 17, 3-21.
15.
Goulet, F. ( 2007). Assessment of family physicians’ performance using patient charts. Interraters reliability and concordance with chart-stimulated recall interview. Evaluation & the Health Professions , 30, 376-392.
16.
Goulet, F., Jacques, A., Gagnon, R., Bourbeau, D., Laberge, D., Melanson, J., et al. (2002). Performance assessment. Family physicians in Montreal meet the mark!Canadian Family Physician, 48, 1337-1344.
17.
Hayward, R., McMahon, L., & Bernard, A. ( 1993). Evaluating the care of general medicine inpatients: how good is implicit review? Annals of Internal Medicine, 118, 550-556.
18.
Kreiter, C., & Bergus, G. ( 2007). Case specificity: empirical phenomenon or measurement artifact? Teaching and Learning in Medicine, 19, 378-381.
19.
McAuley, R., Paul, W., Morrison, G., Beckett, R., & Goldsmith, C. ( 1990). Five-year results of the peer assessment program of the College of Physicians and Surgeons of Ontario. Canadian Medical Association Journal, 143, 1193-1199.
20.
Mungen, B., & Reinhart, M. ( 1987). Field trial of multiple recertification methods. In J. Lloyd, & D. Geeds (Eds.), Recertification for medical specialists (pp. 71-88.). Evanston, Illinois. American Board of Medical Specialties.
21.
Norman, G. ( 1989). Comprehensive assessment of clinical competence of family/ general physicians using multiple measures. Paper presented at the Proceedings of the twenty-eight Annual Conference in conjunction with the 100th Annual Meeting of the Association of American Medical Colleges , Boston, MA.
22.
Norman, G., Bordage, G., Page, G., & Keane, D. ( 2006). How specific is case specificity? Medical Education, 40(7), 618-623.
23.
Norman, G., Davis, D., Lamb, S., Hanna, E., Caulford, P., & Kaigas, T. ( 1993). Competency assessment of primary care physicians as part of a peer review program. Journal of American Medical Association , 270, 1046-1051.
24.
Nunally, J.C., & Bernstein, I.H. (1994). Psychometric Theory. New York, McGraw-Hill, 3rd edition.
25.
Page, G., Bates, J., Dyer, S., Vincent, D., Bordage, G., & Jacques, A. ( 1995). Physician-assessment and physician-enhancement programs in Canada. Canadian Medical Association Journal, 153, 1723-1728.
26.
Parboosingh, J., Avard, D., Lockyer, J., Watson, M., Pim, C., & Yee, J. ( 1987). The rise of clinical recall interviews as a method of determining needs in Continuing Medical Education. Paper presented at the Proceedings of AAMC Research in Medical Education Conference , Washington, DC.
27.
Rethans, J., Martin, E., & Metsemakers, J. (1994). To what extent do clinical notes by general practitioners reflect actual medical performance? A study using simulated patients. British Journal of General Practice , 44, 153-156.
28.
Royal College of General Practitioners. (1985 ). What sort of doctor? Assessing quality of care in general practice. London: Royal College of General Practitioners.
29.
Shavelson, R., & Webb, N. ( 1991). Generalizability theory. A primer. Thousand Oaks: SAGE.
30.
Smith, M., Artherly, A., Kane, R., & Pacala, J. ( 1997). Peer review of the quality of care: reliability and sources of variability for outcome and process assessments. Journal of American Medical Association, 278, 1573-1578.
31.
Tugwell, P., & Dok, C. ( 1985). Medical record review. In G. Norman (Ed.), Assessing clinical competence. New York: Springer Publishing Co.
32.
Violato, C., Marini, A., Toews, J., Lockyer, J., & Fidler, H. ( 1997). Assessing physicians in practice; feasability and psychometric properties of using peers, consulting physicians, covisiters and patients to assess physicians. Academic Medicine, 72, 582-584.
33.
Wilson, A., & McDonald, P. ( 1994). Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultations . British Medical Journal, 309, 1483-1485.
34.
Wu, L., & Ashton, C. ( 1997). Chart review. A need for reappraisal. Evaluation & the Health Professions, 20(2), 146-163.
