This program computes both the interjudge reliability of individual measurements and the extent to which the judges are biased in their ratings vis-à-vis each other. The methods proposed are recommended on the basis of recent developments in statistical research.
Get full access to this article
View all access options for this article.
References
1.
Bartko, J.J.The intraclass correlation coefficient as a measure of reliability. Psychological Reports, 1966, 19, 3 -11.
2.
Bartko, J. J.Corrective note to: "the intraclass coefficient as a measure of reliability."Psychological Reports , 1974, 34, 418.
3.
Ebel, R. L.Estimation of the reliability of ratings. Psychometrika, 1951, 16, 407-424 .
4.
Fleiss, J. L.Measuring agreement between two judges on the presence or absence of a trait. Biometrics , 1975, 31, 651-659. Also, paper presented at the Joint Meetings of the American Statistical Association, New York City, December, 1973.
5.
Greene, J. F., McCook, W. M. and Archambault, F. X.A computer program to calculate adjusted and unadjusted interrater reliabilities for sets and subjects of judges. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1975, 35 , 689-691.
6.
Horst, P.A generalized expression for the reliability of measures. Psychometrika, 1949 , 14, 21 -31.
7.
Petrinovich, L. F. and Hardyck, C. D.Error rates for multiple comparison methods: some evidence concerning the frequency of erroneous conclusions. Psychological Bulletin, 1969, 71 , 43-54.
8.
Snedecor, G. W.Statistical methods (5th ed.). Ames: Iowa State College Press, 1956.
9.
Winer, B. J.Statistical principles in experimental design. New York: McGraw-Hill, 1962.
10.
Winer, B. J.Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill, 1971.
