The programs described compute rater agreement and rater bias statistics with qualitative data. They also utilize techniques for (a) selecting the most reliable from a set of raters and (b) identifying those cases which are the most diflicult for raters to classify.
Get full access to this article
View all access options for this article.
References
1.
Aivano, S. L., Cicchetti, D. V., and Levine, J.Selecting the most reliable from a set of judges: Toward a general solution . Paper presented at Joint Meetings of the American Statistical Association, Boston, Massachusetts, August 1976 ;
2.
(Also, Proceedings of the American Statistical Association (Social Statistics Section) , 1976, in press).
3.
Cicchetti, D. V.A new measure of agreement between rank ordered variables. Proceedings of the American Psychological Association, 1972, 7, 17-18.
4.
Cicchetti, D. V.Assessing inter-rater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry, 1976, 129, 452-456.
5.
Cicchetti, D. V., Aivano, S. L., and Vitale, J.A computer program for assessing the reliability and systematic bias of individual measurements. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1976, 36, 761-764.
6.
Cicchetti, D. V. and Allison, T.Assessing the reliability of scoring EEG sleep records: An improved method. Proceedings and Journal of the Electrophysiological Technologists' Association, 1973, 20, 92-102.
7.
Cohen, J.A coefficient of agreement for nominal scales. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1960, 20, 37-46.
8.
Cohen, J.Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 1968, 70, 213-220.
9.
Fleiss, J. L.Measuring agreement between two judges on the presence or absence of a trait. Biometrics , 1975, 31, 651-659.
10.
Fleiss, J. L., Cohen, J. , and Everitt, B. S.Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 1969, 72, 323-327.
11.
Hall, J. N.Interrater reliability of ward rating scales. British Journal of Psychiatry, 1974, 125, 248-255.
12.
McNemar, Q.Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 1947, 12, 153-157.
13.
Overall, J. E. and Gorham, D. R.The brief psychiatric rating scale. Psychological Reports, 1962, 10, 799-812.
14.
Overall, J. E. and Klett, C. J.Applied Multivariate Analysis. New York: McGraw-Hill, 1972 (Introductory and No. 8 chapters).
