The community of preference of N judges, or raters, can be quantified with coefficients measuring the agreement of the values assigned by the judges to a set of H units ("classes"). In this paper, new coefficients of agreement are suggested for the measure of intraclass consistency between observations on two variables. The coefficients are derived from a general coefficient for measuring intraclass dependence in a bivariate analysis context. It is shown that various coefficients for the univariate agreement analysis (Krippendorff's r; Fisher's intraclass correlation coefficient; Cohen's k; Scott's π Robinson's A, and Kendall's W) are particular cases of the suggested coefficients.
Get full access to this article
View all access options for this article.
References
1.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, (1), 37-46.
2.
Fabbris, L. and Gallo, F. (1991). A coefficient for measuring bivariate intraclass dependence between quantitative variable. (Journal of Italian Statistical Society in press).
3.
Fisher, R. A. (1921). On "probable error" of a coefficient of correlation deduced from a small sample. Metron, 1 (4), 3-32.
4.
Fleiss, J. L. (1966). Assessing the accuracy of multivariate observations. Journal of American Statistical Association, 61, 403-412.
5.
Kendall, M. G. (1948). Rank correlation methods. London: Griffin.
6.
Krippendorff, K. (1970). Bivariate agreement coefficients for reliability of data. In F. Borgatta and G. W. Bornstedt (Eds.), Sociological Methodology, San Francisco: Jossey Bass139-150.
7.
Mardia, K. V. (1977). Mahalanobis distances and angles. In P. R. Krishnaiah (Ed.) Multivariate Analysis IV (pp. 495-511).
8.
Maxwell, A. E. and Pilliner, A.E.G. (1968). Deriving coefficients of reliability and agreement for ratings. The British Journal of Mathematical and Statistical Psychology, 21, 105-116.
9.
Robinson, W. S. (1957). The statistical measurement of agreement. American Sociological Review, 22, 17-24.
10.
Scott, W. A. (1955). Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
