The application range of Cohen's Kappa is extended to the field of sequential observation data, where omission mistakes of an observer may often occur. It is shown how the omission mistakes can be incorporated into the calculation of the Kappa coefficient without violating the statistic it is based on. The enhanced coefficient is termed Kappa Coefficient With Omission Calculation (Kwoc). In addition, the so-called paradox features of the Kappa coefficient, the base-rate problem, and the observer bias are discussed. It is shown that, in the case of observation data, these attributes are desirable features of the Kappa coefficient.
Asendorpf, J., & Wallbott, H. G. (1979). Maße der Beobachterübereinstimmung: Ein systematischer Vergleich [Measures of observer agreement: A systematic comparison].Zeitschrift für Sozialpsychologie, 10, 243-252.
2.
Bakeman, R., & Gottman, J. M. (1986).Observing interaction: An introduction to sequential analysis. London: Cambridge University Press.
3.
Bales, R. F. (1950).Interaction process analysis: A method for the study of small groups. Cambridge, MA: Addison-Wesley.
4.
Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses and misuses, and alternatives.Educational and Psychological Measurement, 41, 687-699.
5.
Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa.Journal of Clinical Epidemiology, 46(5), 423-429.
6.
Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes.Journal of Clinical Epidemiology, 43, 551-558.
7.
Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37-46.
8.
Cook, R. J. (1998). Kappa and its dependence on marginal rates. In T. P. Armitage & T. Colton (Eds.),The encyclopedia of biostatictics (pp. 2166-2168). New York: John Wiley.
9.
Cranach, M. v., & Frenz, H. G. (1969). Systematische Beobachtung [Systematic observation]. In K. Gottschaldt, P. Lersch, F. Sander, & H. Thomae (Eds.),Handbuch der Psychologie in 12 Bänden (Bd. 7,1, pp. 269-331). Göttingen, Germany: Hogrefe.
10.
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes.Journal of Clinical Epidemiology, 43, 543-549.
11.
Fleiss, J. L. (1981).Statistical methods for raters and proportions. New York: John Wiley.
12.
Grove, W. M., Andreason, N. C., & McDonald-Scott, P. (1981). Reliability studies of psychiatric diagnosis.Archives of General Psychiatry, 38, 408-413.
13.
Hollenbeck, A. R. (1978). Problems of reliability in observational research. In G. P. Sackett (Ed.),Observing behavior, Vol. 2: Data collection and analysis methods (pp. 79-98). Baltimore: University Park Press.
14.
Kauffeld, S. (2003). Gruppensitzungen unter der Lupe—Das Kasseler-Kompetenz-Raster als prozessanalytische Diagnosemethode zur Teamentwicklung [Group sessions under a magnifying glass—The Kasseler-Competence-Grid as a process-analytical method of diagnosis in team development]. In S. Stumpf & A. Thomas (Eds.),Teamarbeit und Teamentwicklung. Göttingen, Germany: Hogrefe.
15.
Kvalseth, T. O. (1991). A coefficient of agreement for nominal scales: An asymmetric version of kappa.Educational and Psychological Measurement, 51, 95-101.
16.
Lantz, C. A., & Nebenzahl, E. (1996). Behavior and interpretation of the kappa statistic: Resolution of the two paradoxes.Journal of Clinical Epidemiology, 49(4), 431-434.
17.
Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives.Psychological Bulletin, 76, 365-377.
18.
Scharpf, U. (1988).Entscheidungsfindung im Gruppenprozess[Decision-making in group processes]. Konstanz, Germany: Hartung-Gorre.
19.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding.Public Opinion Quarterly, 19, 321-325.
20.
Simon, P. (2002).Die Entwicklung eines Modells der Gruppeneffektivität und eines Analyse-Instruments zur Erfassung des Leistungspotentials von Arbeitsgruppen[The development of a group effectiveness model and an analysis instrument for measuring the output potential of work groups]. Landau, Germany: Verlag Empirische Pädagogik.
21.
Spitznagel, E. L., & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic.Archives of General Psychiatry, 42(7), 725-728.
22.
Thompson, W. D., & Walter, S. D. (1988). A reappraisal of the kappa coefficient.Journal of Clinical Epidemiology, 41, 949-958.
23.
Übersax, J. S. (1987). Diversity of decision making models and the measurement of interrater agreement.Psychological Bulletin, 101, 140-146.
24.
Zwick, R. (1988). Another look at interrater agreement.Psychological Bulletin, 103(4), 374-378.
