This text describes a computer program that allows computing the interrater agreement index Scott’s π for at least two ratings per object. The program allows using weights; therefore, the user is not restricted to data on a nominal level of measurement. If wanted, it is possible to compute the agreement per category.
Campbell, D.T., & Fiske, D.W. ( 1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
2.
Cicchetti, D.V. ( 1972). A new measure of agreement between rank ordered variables . Proceedings of the 80th annual convention. American Statistical Association, 7, 17-18.
3.
Cohen, J. ( 1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
4.
Fleiss, J.L. ( 1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
5.
Galtung, J. ( 1979). Measurement of agreement. In J. Galtung (Ed.), Papers on methodology. Theory and methods of social research (Vol. II, pp. 82-135). Copenhagen: Christian Eijlers.
6.
Hayes, A.F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89.
7.
Popping, R. ( 1984). AGREE, A package for computing nominal scale agreement . Computational Statistics and Data Analysis, 2, 182-185.
8.
Popping, R. ( 1988). On agreement indices for nominal data. In W. E. Saris, & I. N. Gallhofer (Eds.), Socio-metric research: Data collection and scaling (pp. 90-105). Hampshire: McMillan .
9.
Popping, R. ( 2009). Some views on agreement to be used in content analysis studies. Quality & Quantity, , DOI 10.1007/s11135-009-9258-3.
10.
Schouten, H.J.A. ( 1982). Measuring pair wise agreement among many observers. II . Some improvements and additions. Biometrical Journal, 24, 431-435.
Scott, W.A. ( 1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
13.
Tinsley, H.E.A., & Weiss, D.J. ( 2000). Interrater reliability and agreement. In H. E. A. Tinsley, & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 95-124). San Diego, CA: Academic Press.
