Whenever two or more raters evaluate a patient or student, it may be necessary to determine the degree to which they assign the same label or rating to the subject. The major problem in deciding which statistic to use is the plethora of different techniques which are available. This paper reviews some of the more commonly used techniques, such as Raw Agreement, Cohen's kappa and weighted kappa, and shows that, in most circumstances, they can all be replaced by the intraclass correlation coefficient (ICC). This paper also shows how the ICC can be used in situations where the other statistics cannot be used and how to select the best subset of raters.
References
1.
RobinsL.N., HeizerJ.E., CroughanJ.National Institute of Mental Health Diagnostic Interview Schedule: its history, characteristics, and validity.Arch Gen Psychiatry1981; 38: 381–389.
2.
SpitzerR., EndicottJ.Schedule for Affective Disorders and Schizophrenia (SADS).New York: New York State Psychiatric Institute,1982.
3.
AnthonyJ.C., FollsteinM., RomanoskiA.J.Comparison of the lay Diagnostic Interview Schedule and a standardized psychiatric diagnosis.Arch Gen Psychiatry1985; 42: 667–675.
4.
MuschD.C., LandisJ.R., Higgins ITT. An application of kappa-type analyses to interobserver variation in classifying chest radiographs for pneumoconiosis.Stat Med1984; 3: 73–83.
5.
FleissJ.L.Statistical methods for rates and proportions, second edition.New York NY: Wiley, 1981.
6.
KramerM.S., FeinsteinA.R.Clinical biostatistics: LIV, the biostatistics of concordance.Clin Pharmacol Ther1981; 29: 111–123.
7.
GroveW.M., AndreasenN.C., McDonald-ScottP.Reliability studies of psychiatric diagnosis.Arch Gen Psychiatry1981; 38: 408–413.
8.
HouseA.E., HouseB.J., CampbellM.B.Measures of interobserver agreement: calculation formulas and distribution effects.J Behav Assess1981; 3: 37–57.
9.
CohenJ.A coefficient of agreement for nominal scales.Ed Psych Msmt1960; 20: 37–46.
10.
CongerA.J.Integration and generalization of kappa for multiple raters.Psychol Bull1980; 88: 322–328.
11.
FleissJ.L.Measuring nominal scale agreement among many raters.Psychol Bull1971; 76: 378–382.
12.
StreinerD.L., NormanG.R.Health measurement scales: a practical guide to their construction and use, second edition.Oxford: Oxford University Press,1994.
13.
CohenJ.Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit.Psychol Bull1968; 70: 213–220.
14.
SoekenK.L., PrescottP.A.Issues in the use of kappa to estimate reliability.Med Care1986; 24: 733–741.
15.
FleissJ.L., CohenJ.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.Ed Psych Msmt1973; 33: 613–619.
16.
LandisJ.R., KochG.G.A one-way components of variance model for categorical data.Biometrics1977; 33: 671–679.
17.
WalterS.D.Measuring the reliability of clinical data: the case for using three observers.Rev Epidém et Santé Publ1984; 32: 206–211.
18.
SpitznagelE.L., HeizerJ.E.A proposed solution to the base rate problem in the kappa statistic.Arch Gen Psychiatry1985; 42: 725–728.
19.
YuleG.U.On the methods of measuring association between two attributes.J Roy Statist Soc1912; 75: 581–642.
20.
FinnR.H.A note on estimating the reliability of categorical data.Ed Psych Msmt1970; 30: 71–76.
21.
CopasJ.B., LoeberR.Relative improvement over chance (RIOC) for 2 × 2 tables.Br J Math Statist Psychol1990; 43: 293–307.
22.
HuiS.L., WalterS.D.Estimating the error rates of diagnostic tests.Biometrics1980; 36: 167–171.
23.
StreinerD.L., MillerH.R.Maximum likelihood estimates of the accuracy of four diagnostic techniques.Ed Psych Msmt1990; 50: 653–662.
24.
MillerH.R., StreinerD.L., ParkinsonA.Maximum likelihood estimates of the ability of the MMPI and MCMI personality disorder scales and the SIDP to identify personality disorder.J Person Assess1992; 59: 1–13.
25.
StreinerD.L.A checklist for evaluating the usefulness of rating scales.Can J Psychiatry1993; 38(2): 140–148.
