This article discusses the determination of reliability coefficients for clinical evaluation forms by use of generalizability theory. This method is contrasted to some of the commonly used measures of interobserver agreement, interobserver reliability, and classical reliability. The value of this method in terms of making recommendations, such as the number ofjudges and the number of items on a form, in order to improve reliability is illustrated by two examples. Also, the difference between norm-referenced decisions and criterion-referenced decisions in determining reliability with this method is presented.
Get full access to this article
View all access options for this article.
References
1.
ALLEN, M. J.
and W. M. YEN (1979) Introduction to Measurement Theory. Belmont, CA: Wadsworth.
2.
BERK, R. K.
(1979) "Generalizability of behavioral observations: a clarification of interobserver agreement and interobserver reliability."Amer. J. of Mental Deficiency83:460-472.
3.
BRENNAN, R. L.
and M. T. KANE (1977) "An index of dependability for mastery tests."J. of Educational Measurement14:277-289.
4.
BURT, C.
(1936) "The analysis of examination marks," in P. Hartog and E. C. Rhodes (eds.) The Marks of Examiners. London: Macmillan.
5.
CORNFIELD, J.
and J. W. TUKEY (1956) 'Average values of mean squares in factorials."Annals of Mathematical Statistics27:907-949.
6.
CRONBACH, L. J.
, G. C. GLESER, H. NANDA, and N. RAJARATNAM (1972) The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: John Wiley.
7.
CRONBACH, L. J.
, N. RAJARATNAM, and G. C. GLESER (1963) "Theory of generalizability: a liberalization of reliability theory."British J. of Statistical Psychology16:137-163.
8.
DIELMAN, T. E.
, A. L. HULL and W. K. DAVIS (1979) "Interrater reliability of clinical performance ratings."Proceedings of the Eighteenth Annual Conference on Research in Medical Education: 191-196.
9.
EBEL, R. L.
(1979) Essentials of Educational Measurement. Englewood Cliffs, NJ: Prentice-Hall.
10.
EBEL, R. L.
(1951) "Estimation of the reliability of ratings."Psychometrika16:407-424.
11.
ERVITI, V.
, L. J. FABREY, and J. V. BUNCE (1979) "The development of rating scales to assess the clinical performance of medical students."Proceedings of the Eighteenth Annual Conference on Research in Medical Education: 185-190.
12.
FLEISS, J. L.
(1975) "Measuring agreement between two judges on the presence or absence of a trait."Biometrics31:651-659.
13.
GLESER, G. C.
, L. J. CRONBACH, and N. RAJARATNAM (1965)"Generalizability of scores influenced by multiple sources of variance."Psychometrika30:395-418.
14.
GULLIKSEN, H.
(1950) Theory of Mental Tests. New York: John Wiley.
15.
KELLY, M. B.
(1977) "A review of the observational data collection and reliability procedures reported in the Journal of Applied Behavior Analysis."J. of Applied Behavior Analysis10:97-101.
16.
LANDIS, J. R.
and G. G. KOCH (1971) "The measurement of observer agreement for categorical data."Biometrics33: 159-174.
17.
MORGAN, M. K.
and D. M. IRBY (1978) Evaluating Clinical Competence in the Health Professions. St. Louis: Mosby, 1978.
18.
WEBB, N. M.
and R. J. SHAVELSON (1981) "Multivariate generalizability of general educational development ratings."J. of Educ. Measurement18:13-22.
