We propose adjusted inference procedures for evaluating the agreement/disagreement of two raters in a clustered setting involving twins or paired body parts. These procedures include the construction of a confidence interval for the kappa statistic, a related test of statistical significance and a formula that facilitates sample size estimation. The results of a simulation study suggest that a simple adjustment using an estimated design effect will provide valid inferences. The methods proposed are illustrated using an example from the literature.
BlochDAKraemerHC. 2×2 kappa coefficients: measures of agreement or association. Biometrics1989; 45: 269–287.
2.
DonnerAEliasziwM. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance-testing and sample size estimation. Stat Med1992; 11: 1511–1519.
3.
BartfayEDonnerA. Statistical inferences for interobserver agreement studies with nominal outcome data. Statistician2001; 50: 135–146.
4.
AltayeMDonnerAKlarN. Inference procedures for assessing interobserver agreement among multiple raters. Biometrics2001; 57: 584–588.
5.
ShoukriMM. Measures of interobserver agreement and reliability , 2nd ed. Boca Raton, FL: Chapman & Hall/CRC Press, 2010.
6.
BartkoJJ. The intraclass correlation reliability coefficient. Psychol Bull1966; 83: 762–765.
7.
EbelRL. Estimation of the reliability of ratings. Psychometrika1951; 16: 407–427.
8.
HaggardEA. Intraclass correlation and the analysis of variance , New York, NY: Dryden Press, 1958.
9.
OdenNL. Estimating kappa from binocular data. Stat Med1991; 10: 1303–1311.
10.
KirchnerHLLemkeJH. Simultaneous estimation of intrarater and interrater agreement for multiple raters under order restrictions for a binary trait. Stat Med2002; 21: 1761–1772.
PanYHaberMBarnhartHX. A new permutation-based method for assessing agreement between two observers making replicated binary readings. Stat Med2012; 31: 2249–2261.
13.
CohenJ. A coefficient of agreement for nominal scale. Educ Psycol Meas1960; 20: 37–46.
14.
DonnerAKlarN. Design and analysis of cluster randomization trials in health research , New York, NY: Arnold, 2001.
15.
RaoJNKScottAJ. A simple method for the analysis of clustered binary data. Biometrics1992; 48: 577–585.
16.
CochranWG. Sampling techniques , 3rd ed. New York, NY: John Wiley & Sons, 1977.
17.
R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna: Austria, 2009. url:http://www.R-project.org.
18.
Haynam GE, Govindarajulu Z and Leone GC. Tables of the cumulative non-central chi-square distribution. Case Statistical Laboratory 1962; Publication No. 104.
19.
PowellKAObuchowskiNAChilcoteWA. File-screen versus digitized mammography: assessments of clinical equivalence. Am J Roentgenol1999; 173: 889–894.
20.
FleissJL. Statistical methods for rates and proportions , 2nd edn. New York, NY: John Wiley & Sons, 1981.
21.
BakerSGKoCWGraubardBI. A sensitivity analysis for non randomly missing categorical data arising from a national health disability survey. Biostatistics2003; 4: 41–56.
22.
LittleRJRubinDB. Statistical analysis with missing data , 2nd ed. Hoboken, NJ: John Wiley & Sons Inc, 2002.
23.
BartfayEDonnerA. The effect of collapsing multinomial data when assessing agreement. Int J Epidemiol2000; 29: 1070–1075.
24.
ChinchilliVMMartelJKKumanyikaS. A weighted concordance correlation coefficient for repeated measurement designs. Biometrics1996; 52: 341–353.
25.
CarrascoJLJoverL. Estimating the generalized concordance correlation coefficient through variance components. Biometrics2003; 59: 849–858.
26.
KingTSChinchilliVMCarrascoJ. A repeated measures concordance correlation coefficient. Stat Med2007; 26: 3095–3113.
27.
KingTSChinchilliVMWangKL. A class of repeated measures concordance correlation coefficients. J Biopharm Stat2007; 17: 653–672.
28.
ChenCCBarnhartHB. Comparison of ICC and CCC for assessing agreement for data without and with replications. Comput Stat Data Anal2008; 53: 554–564.
29.
MaYTangWFengC. Inference for kappas for longitudinal study data: application to sexual health research. Biometrics2008; 64: 781–789.
30.
HopperJLHannahMCMacaskillGT. Twin concordance for binary trait: III. A bivariate analysis of hay fever and asthma. Gen Epidemiol1990; 7: 277–289.
