This article addresses the problem of accounting overall chance-corrected interobserver agreement among the multivariate ratings of several judges. Modifying an approach by Berry and Mielke, a generalization of Cohen’s kappa coefficient is proposed. The generalized statistic accounts agreement for multivariate interval or nominal observations among several judges. The statistic’s metric is conventional, and in the univariate case it is equivalent to existing extensions of the kappa coefficient to several observers, as well as to a certain intraclass correlation coefficient. Application is exemplified using hypothetical and real-life data sets.
Get full access to this article
View all access options for this article.
References
1.
Berry, K. J. , & Mielke, P. W., Jr. (1988). A generalization of Cohen’s kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921-933.
2.
Brennan, R. L. , & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
3.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
4.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
5.
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322-328.
6.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: Wiley.
7.
Efron, B. , & Tibshirani, R. J. (1993). An introduction to the bootstrap. London: Chapman & Hall.
8.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
9.
Fleiss, J. L. , & Cicchetti, D. V. (1978). Inference about weighted kappa in the non-null case. Applied Psychological Measurement, 2, 113-117.
10.
Fleiss, J. L. , & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
11.
Fleiss, J. L. , Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 323-327.
12.
Fleiss, J. L. , & Shrout, P. E. (1978). Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika, 43, 259-262.
13.
Gordon, A. D. (1981). Classification: Methods for the exploratory analysis of multivariate data. London: Chapman and Hall.
14.
Haggard, E. A. (1958). Intraclass correlation and the analysis of variance. New York: Dryden Press.
15.
Hartigan, J. (1975). Clustering algorithms. New York: Wiley-Interscience.
16.
Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84, 289-297.
17.
Koppitz, E. M. (1968). Psychological evaluation of children’s human figure drawings. New York: Grune & Stratton.
18.
Landis, J. R. , & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
19.
Magnusson, D. (1967). Test theory. Reading, MA: Addison-Wesley.
20.
McGraw, K. O. , & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 308-320, 390-390.
21.
Milligan, G. W. (1996). Clustering validation: Results and implications for applied analyses. In P. Arabie, L. J. Hubert, & G. DeSoete (Eds.), Clustering and classification (pp. 341-375). Singapore: World Scientific.
22.
Rajaratnam, N. (1960). Reliability formulas for independent decision data when reliability data are matched. Psychometrika, 25, 261-271.
23.
Shrout, P. E. , & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
