The reliability of continuous or binary outcome measures is usually assessed by estimation of the intraclass correlation coefficient (ICC). A crucial step for this purpose is the determination of the required sample size. In this review, we discuss the contributions made in this regard and derive the optimal allocation for the number of subjects k and the number of repeated measurements n that minimize the variance of the estimated ICC. Cost constraints are discussed for both normally and non-normally distributed responses, with emphasis on the case of dichotomous assessments. Tables showing optimal choices of k and n are given along with the guidelines for the efficient design of reliability studies.
Get full access to this article
View all access options for this article.
References
1.
Dunn G.Design and analysis of reliability studies . Statistical Methods in Medical Research1992; 1: 123-157 .
2.
Shoukri MM.Agreement. In Armitage P , Colton T , eds Encyclopedia of biostatistics. New York: John Wiley & Sons , 1999: 117-130.
3.
Shoukri MM.Agreement. In Gail MH , Benichou J , eds Encyclopedia of epidemiologic methods. New York: John Wiley & Sons , 2000: 43-49.
4.
Haggard ER.Intraclass correlation and the analysis of variance. New York: Dryden Press , 1958.
5.
Elston R.Response to query: estimating ‘heritability’ of a continuous trait . Biometrics1977; 33: 232-233 .
6.
Fisher RA.Statistical methods for research workers. London: Oliver & Boyd , 1925.
7.
Walter DS , Eliasziw M , Donner A.Sample size and optimal design for reliability studies . Statistics in Medicine1998; 17: 101-110 .
8.
Freedman LS , Parmar MKB , Baker SG.The design of observer agreement studies with binary assessments . Statistics in Medicine1993; 12: 165-179 .
9.
Awni WM , Skaar DJ , Schwenk MH.Interindividual and intraindividual variability in labetalol pharmacokinetics . Journal of Clinical Pharmacology1988; 28: 344-349 .
10.
Donner A , Eliasziw M.Sample size requirements for reliability studies . Statistics in Medicine1987; 6: 441-448 .
11.
Eliasziw M , Donner A.A cost-function approach to the design of reliability studies . Statistics in Medicine1987; 6: 647-655 .
12.
Giraudeau B , Mary JY.Planning a reproducibility study: how many subjects and how many replicates per subject for an expected width of the 95 per cent confidence interval of the intraclass correlation coefficient . Statistics in Medicine2001; 20: 3205-3214 .
13.
Bonett DG.Sample size requirements for estimating intraclass correlations with desired precision . Statistics in Medicine2002; 21: 1331-1335 .
14.
Donner A.Sample size requirements for interval estimation of the intraclass kappa statistic . Communications in Statistics - Simulation1999; 28: 415-429 .
15.
Shoukri MM , Asyali MH , Walter SD.Issues of cost and efficiency in the design of reliability studies . Biometrics2003; 59: 1107-1112 .
16.
Rao SS.Optimization: theory and applications. New Delhi: Wiley Eastern Limited , 1984.
17.
Flynn NT , Whitley E , Peters T.Recruitment strategy in a cluster randomized trial: cost implications . Statistics in Medicine2002; 21: 397-405 .
18.
Sukhatme PV , Sukhatme BV , Sukhatme S , Asok C.Sampling theory of surveys with applications. Ames, IA: Iowa State University Press , 1984.
19.
Scheffé H.The analysis of variance. New York: John Wiley & Sons , 1959.
20.
Tukey JW.Variance of variance components I: balanced designs . Annals of Mathematical Statistics1956; 27: 722-736 .
21.
Kendall M , Stuart A.The advanced theory of statistics, Vol 1. London: Griffin , 1986.
22.
Hemmersley IM.The unbiased estimate and standard error of the intraclass variance . Metron1949; 15: 189-205 .
23.
Kraemer CH.Ramification of a population model for k as a coefficient of reliability . Psychometrika1979; 44: 461-472 .
24.
Frederic R , Gaspoz JM , Fortune IC , Minini P , Pichot V , Duverney D , Lacow JR and Barthelemy JC.Screening of obstructive sleep apnea syndrome by heart rate variability analysis . Circulation1999; 100: 1411-1415 .
25.
Donner A , Eliasziw M.Statistical implications of the choice between a dichotomous or continuous trait in studies of interobserver agreement . Biometrics1994; 50: 550-555 .
Haseman JK , Kupper LL.Analysis of dichotomous response data from certain toxicological experiments . Biometrics1979; 35: 281-293 .
28.
Bloch DA , Kraemer HC.2 2 Kappa coefficients: measures of agreement or association . Biometrics1989; 45: 269-287 .
29.
Donner A , Eliasziw M.The goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance testing, and sample size estimation . Statistics in Medicine1992; 11: 1511-1519 .
30.
Landis JR , Koch GG.The measurement of observer agreement for categorical data . Biometrics1977; 33: 159-174 .
31.
Cohen JA.A coefficient of agreement for nominal scales . Educational and Psychological Measurement1960; 20: 37-46 .
32.
Fleiss J , Cohen JA.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability . Educational and Psychological Measurement1973; 33: 613-619 .
