The aim of this study is to identify the sources of construct validity evidence and determine generalizability of the scores given to medical students on pediatrics clerkship evaluation at the College of Medicine and Medical Sciences, Arabian Gulf University (CMMS-AGU). The CMMS-AGU is a problem-based medical school in Bahrain. The sample was composed of 140 students. Validity evidence and generalizability theory were sought. Results indicated that pediatrics clerkship evaluation at CMMS-AGU has content validity. Some limitations may have occurred in response process, but the students performance was observed directly by the tutors. There was evidence of range restriction and ``halo'' effect. Interrater reliability was 0.74, and the generalizability coefficient for three raters was 0.61. Students' contribution to the variance component of the global clerkship evaluation scores was 24%. Decision study indicated that seven tutors are required to achieve a generalizability coefficient of about 0.80. In conclusion, this study indicated that the pediatrics clerkship evaluation at the CMMS-AGU has, overall, some acceptable content and internal structure evidence of validity and that the raters are the major source of error. To improve the construct validity, there is a need to increase the number of raters and to train raters about the proper process of clerkship evaluation.
Abouna, G.M., & Hamdy, H. (1999). The integrated direct observation clinical encounters examination (IDOCEE)-An objective assessment of students' clinical competence in a problem-based learning curriculum. Medical Teacher, 21, 67-72.
2.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: Author.
3.
Bandiera, G.W., Morrison, L.J., & Regehr, G. (2002). Predictive validity of the global assessment form used in a final-year undergraduate rotation in emergency medicine. Academic Emergency Medicine, 9, 889-895.
4.
Boodoo, G.M., & O'Sullivan, P. (1986). Assessing pediatric clerkship evaluations using generalizability theory. Evaluation and the Health Professions, 9, 467-486.
5.
Brennan, R.L. (2001). Generalizability theory. New York: Springer-Verlag.
6.
Carline, J.D., Paauw, D.S., Thiede, K.W., & Ramsey, P.G. (1992). Factors affecting the reliability of ratings of students' clinical skills in a medicine clerkship. Journal of General Internal Medicine, 7, 506-510.
7.
Cohen, R., Rothman, A.I., Poldre, P., & Ross, J. (1991). Validity and generalizability of global ratings in an objective structured clinical examination. Academic Medicine, 66, 545-548.
8.
College of Medicine and Medical Sciences, Arabian Gulf University. (2002-2004). Prospectus. Bahrain . Retrieved from www.agu.edu.bh. Retrieved on 24th June 2004.
9.
College of Medicine and Medical Sciences, Arabian Gulf University, Bahrain. (2001). Clerkship logbook (1st ed.). Bahrain: Author.
10.
Crick, J.E., & Brennan, R.L. (1983). Manual for GENOVA, A generalized analysis of variance system. The American College Testing Program, Iowa City, IA: ACT Inc.
11.
Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002). Generalizability: A key to unlock professional assessment. Medical Education, 36, 972-978.
12.
Crossley, J., Humphris, G., & Jolly, B. (2002). Assessing health professionals. Medical Education, 36, 800-804.
13.
Cunnington, J. (2002). Evolution of student assessment in McMaster University's MD Program. Medical Teacher, 24, 354-260.
14.
Davis, M.H. (2001). Tutor reports. In J. A. Dent, & R. M. Harden (Eds.), (Eds.), A practical guide for medical teachers (pp. 336-343). London: Churchill Livingstone.
15.
de Haes, J.C., Oort, F., Oosterveld, P., & Cate, O.T. (2001). Assessment of medical students' communicative behavior and attitudes: Estimating the reliability of the use of the Amsterdam attitudes and communication scale through generalizability coefficients. Patient Education and Counseling, 45, 35-42.
16.
DeMers, J.L. (1978). Observational assessment of performance. In K. Morgan, & D. Irby (Eds.), (Eds.), Evaluating clinical competence in the health professions (pp. 89-115). St. Louis, MO: CV Mosby.
17.
Downing, S.M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830-837.
18.
Downing, S.M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38, 1006-1012.
19.
Downing, S.M. (2005). Threats to the validity of clinical teaching assessments: What about rater error? Medical Education, 39, 353-355.
20.
Gray, J.D. (1996). Global rating scales in residency education. Academic Medicine, 71, S55-S63.
21.
Haber, R.J., & Avins, A.L. (1994). Do ratings on the American Board of Internal Medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9, 140-145.
22.
Hamdy, H., Prasad, K., Williams, R., & Salih, F.A. (2003). Reliability and validity of the direct observation clinical encounter examination (DOCEE). Medical Education , 37, 205-212.
23.
Harden, R.M., & Gleeson, F.A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). ASME Medical Education , Booklet no. 8.
24.
Hasanain, M., Connell, K.J., Downing, S.M., Olthoff, A., & Yudkhowsky, R. (2004). Toward meaningful evaluation of clinical competence: The role of direct observation in clerkship ratings . Academic Medicine, 79(Suppl.), S21-4.
25.
Hebert, R., & Bravo, G. (1996). Development and validation of an evaluation instrument for medical students in tutorials. Academic Medicine, 71, 488-494.
26.
Jolly, B., & Spencer, J. (2002). The metric of medical education. Medical Education, 36, 798-799.
27.
Kassebaum, D.G., & Eaglen, R.H. (1999). Shortcomings in the evaluation of students' clinical skills and behaviors in medical school. Academic Medicine , 74, 842-849.
28.
Kreiter, C.D., & Ferguson, K. (2001). Examining the generalizability of ratings across clerkship using a clinical evaluation form. Evaluation of the Health Professions, 24, 36-46.
29.
Kreiter, C.D., Gordon, J.A., Elliott, S.T., & Ferguson, K.J. (1999). A prelude to modeling the expert: A generalizability study of expert ratings of performance on computerized clinical simulations. Advances in Health Sciences Education , 4, 261-270.
30.
Kumar, A., Gera, R., Shah, G., Godambe, S., & Kallen, D.J. (2004). Student evaluation practices in pediatric clerkships: A survey of the medical schools in the United states and Canada. Clinical Pediatrics (Phila), 43, 729-735.
31.
Levine, H.G., Yunke, R., & Bee, D. (1986). Pediatric resident performance: The reliability and validity of the rating form. Evaluation of the Health Professions, 9, 62-74.
32.
Linn, R.L., & Gronlund, N.E. (1995). Rating scales. In Kevin M. Davis, (Eds.), Measurement and assessment in teaching (pp. 246-257.). NJ: Prentice-Hall Inc. A Simon & Schuster, New Jersey.
33.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement. (3rd ed., pp. 13-103). New York: American Council on Education and Macmillan.
34.
Morton, J., Cumming, A., & Cameron, H. (2007). Performance-based assessment in undergraduate medical education. The Clinical Teacher, 4, 36-41.
35.
Nasca, T.J., Gonnella, J.S., Hojat, M., Veloski, J., Erdmann, J.B., & Robeson, M., et al. (2002). Conceptualization and measurement of clinical competence of residents: A brief rating form and its psychometric properties . Medical Teacher, 24, 299-303.
36.
Nendaz, M.R., Perrier, A., Simonet, M.L., Huber, P., Junod, A., & Vu, N.V. (2001). Appraisal of clinical competence during clerkships: How knowledgeable in curriculum and assessment development should a physician-examiner be? Academic Medicine, 76, S99-101.
37.
Pangaro, L.N. (2000). Investing in descriptive evaluation: A vision for the future of assessment. Medical Teacher, 22, 478-481.
38.
Silber, C., Nasca, T., Paskin, D., Eiger, G., Robeson, M., & Jon Velosk, J. (2004). Do global rating forms enable program directors to assess the ACGME competencies?Academic Medicine, 79, 549-556.
39.
Smith, R. (1998). All changed, changed utterly. British Medical Journal, 316, 1917-1918.
40.
Streiner, C. (1995). Clinical ratings-ward evaluation. In S. Shannon, & G. Norman (Eds.), (Eds.), Evaluation methods: A resource handbook (pp. 29-32). Hamilton, ON: Program for Educational Development, McMaster University.
41.
Thompson, W.G., Lipkin, M. Jr., Gilbert, D.A., Guzzo, R.A., & Roberson, L. (1990). Evaluating evaluation: Assessment of the American Board of Internal Medicine Resident Evaluation Form. Journal of General Internal Medicine, 5, 214-217.
42.
van der Vleuten, C.P.M., & Schuwirth, L.W.T. (2005). Assessing professional competence: From methods to programmes. Medical Education , 39, 309-317.
43.
Williams, R.G., Klamen, D.A., & McGaghie, W.C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15, 270-292.
44.
Wilson, W.G., & Howley, L.D. (2004). Direct observation of students during clerkship rotations: A multilayer descriptive study. Academic Medicine , 79, 276-280.
45.
Woolliscroft, J.O., Howell, J.D., Patel, B.P., & Swanson, D.B. (1994). Resident-patient interactions: The humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors and nurses. Academic Medicine, 69, 216-224.
