This review covers the design and analysis of essentially two types of reliability study: method comparison studies and generalizability (including inter-rater reliability) experiments. Likelihood-based methods of inference (confirmatory factor analysis and REML estimation of variance components, for example) are advocated, partly because of their ease of use but, primarily as a stimulus to the use of more ambitious designs for the investigation of the quality of measurements. The more sophisticated approaches are not intended to replace the simpler traditional methods, however, but are expected to be used to supplement them.
Get full access to this article
View all access options for this article.
References
1.
Bland M., Altman DGStatistical methods for assessing agreement between two methods of clinical measurementLancet1986; i: 307-10.
2.
Dunn G.The design and analysis of reliability studies. The statistical evaluation of measurement errorsLondon: Edward Arnold, 1989.
3.
Fletcher CM, Oldham PDBibliography on observer error and variation In: Witts LJ ed, Medical surveys and clinical trials2ndedition. New York: Oxford University Press, 1964: 39-44.
4.
Koran LMThe reliability of clinical methods, data and judgementsThe New England Journal of Medicine1975; 293: 642-46and 695-701.
5.
Feinstein ARA bibliography of publications on observer variabilityJournal of Chronic Diseases1985; 38: 619-32.
6.
Airy GBOn the algebraic and numerical theory of errors of observations and the combinations of observationsLondon: Macmillan , 1861.
7.
Chauvenet WAA manual of spherical and practical astronomy, 2: theory and use of astronomical instrumentsPhiladelphia: Lippincott , 1863.
8.
Adcock RJA problem in least squaresThe Analyst1878; 5: 53-55.
9.
Dunlop CBDementia praecox; some preliminary observations on brains from carefully selected cases, and a consideration of certain sources of errorAmerican Journal of Psychiatry1924; 3: 403-21.
10.
Berkson J., Magath TB, Hurn M.The error of estimate of the blood cell count as made with the hemocytometerAmerican Journal of Physiology1940 ; 128: 309-23.
11.
MacFarlane RG, King EJ, Wootton Idp, Gilchrist M.Determination of haemoglobin III. Reliability of clinical and other methodsLancet1948; i: 282-86.
12.
Fisher RAThe correlation between relations on the supposition of Mendelion inheritanceTransactions of the Royal Society of Edinburgh1918; 52: 399-433.
13.
Fisher RAStatistical methods for research workers 1st editionEdinburgh & London: Oliver & Boyd, 1929.
14.
Tippett Lhc.The methods of statistics 1st editionLondon: Williams and Norgate, 1931.
15.
Khuri AI, Sahai H.Variance components analysis: a selective literature surveyInternational Statistical Review1985; 53: 279-300.
16.
Nichols Pjr , Bailey Ntj.The accuracy of leg-length differences. An 'Observer Error' experimentBritish Medical Journal1955; ii: 1247-48.
17.
Jerushalmy J.The Statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadowsRadiol Clin North Am1969; 7: 381-92.
18.
Birkelo CC, Chamberlain WE, Phelps PS, Schools PE, Zacks D., Yerushalmy J.Tuberculosis case-finding - a comparison of effectiveness of various roentgenographic and photofluorographic methodsJAMA1947; 133:359-66.
19.
Yerushalomy J., Harkness JT, Cope JH, Kennedy BRThe role of dual reading in mass radiographyAmerican Review of Tuberculosis1950 ; 61: 443-64.
20.
Cohen J.A coefficient of agreement for nominal scalesEducational and Psychology Measurement1960; 20: 37-46.
21.
Cohen J.Weighted Kappa: nominal scale agreement with provisions for scaled disagreement of partial creditPsychol Bull1968 ; 70: 213-20.
22.
Spitzer RL, Cohen J., Fleiss JL, Endicott J.Quantification of agreement in psychiatric diagnosisArch Gen Psychiatry1967; 17: 83-87.
23.
Grove WM, Andreason NC, McDonald-Scott P., Keller MB, Shapiro RWReliability studies of psychiatric diagnosisArch Gen Psychiatry1981; 38 : 408-13.
24.
Shrout PE, Spitzer RL, Fleiss JLQuantification of agreement in psychiatric diagnosis revisitedArch Gen Psychiatry1987; 44: 172-77.
Brown W.Some experimental results in the correlation of mental abilitiesBr J Psychol1910; 3 : 296-322.
27.
Brown W.The essentials of mental measurementCambridge: Cambridge University Press, 1911.
28.
Hamilton M.Development of a rating scale for primary depressive illnessBr J Soc Psychol1967; 6: 278-96.
29.
Beck AT, Ward CH, Mendelson M., Mock JE, Erbaugh JKAn inventory for measuring depressionArch Gen Psychiatry1961; 4: 561-71.
30.
Dunn G., Hand DJ, Sham PKStatistics and the nature of depressionJournal of the Royal Statistical Society, Series A 1993 (in press).
31.
Feldt LS, Brennan RLReliability In Linn, R. L. ed, Educational measurement 3rd edition. Washington DC: American Council on Education/Macmillan, 1989: 105-46.
32.
British Standards Institute.British Standard: Precision of Test Methods. Part 1: guide for the determination of repeatability and reproducibility for a standard test method by interlaboratory tests (BS 5487 Part 1; International Standard ISO 5725), 1987 .
33.
Caulcott R., Boddy R.Statistics for analytical chemistsLondon: Chapman and Hall, 1983.
34.
Grubbs FEOn estimating precision of measuring instruments and product variabilityJournal of the American Statistical Association1948; 43: 243-64.
35.
Grubbs FEErrors of measurement, precision, accuracy and the statistical comparison of measuring instrumentsTechnometrics1973;15:53-66.
36.
Jaech JLStatistical analysis of measurement errorsNew York: Wiley, 1985.
37.
Goldberg DPThe detection of psychiatric illness by questionnaireLondon: Oxford University Press, 1972.
38.
Robinson DLREML user manualEdinburgh: Scottish Agricultural Statistical Service, 1987.
39.
Fleiss JLStatistical methods for rates and proportions, secondedition. New York: Wiley, 1981.
40.
Joreskog KGStatistical analysis of sets of congeneric testsPsychometrika1971; 36: 729-34.
41.
Lawley DN, Maxwell AEFactor analysis as a statistical method2ndedition. London: Butterworth, 1971.
42.
Cochran WGErrors of measurement in statisticsTechnometrics1968; 10: 637-65.
43.
Shyr JY, Gleser LJInference about comparative precision in linear structural relationshipsJournal of Statistical Planning and Inference1986; 14: 339-58.
44.
Theobald CM , Mallinson JRComparative calibration, linear structural relationships and congeneric measurementsBiometrics1978; 34: 39-45.
45.
Fairfield Smith H.Estimating precision of measuring instrumentsJournal of the American Statistical Association1950; 45: 447-51.
46.
Mandel J.The statistical analysis of experimental dataNew York : Wiley, 1964.
47.
Lawton WH, Sylvestre EA, Young-Ferraro BJStatistical comparison of multiple analytical procedures: application to clinical chemistryTechnometrics1979 ; 21: 397-409.
48.
Strike PWStatistical methods in laboratory medicine2ndedition. London: Butterworth-Heinemann , 1991.
49.
Barnett VDSimultaneous pairwise linear structural relationshipsBiometrics1969; 25 : 129-42.
50.
Barnett VDStraight consulting In Hand DJ, Everitt BS eds, The statistical consultant in action.Cambridge: Cambridge University Press, 1987: 26-41.
51.
Theobald CM , Mallinson JRComparative calibration, linear structural relationships and congeneric measurementsBiometrics1978; 34: 39-45.
52.
Jansen Aam.Comparative calibration and congeneric measurementsBiometrics1980; 36: 729-34.
53.
Besag Fmc, Hills M., Wardale F., Andrew CM, Craggs MDThe validation of a new ambulatory spike and wave monitorElectroencephalography and Clinical Neurophysiology1989 ; 73: 157-61.
54.
Fleiss JLEstimating the reliability of interview dataPsychometrika1970; 35: 143-62.
Bentler PMEQS structural equations program, Version 3.0Los Angeles : BMDP Statistical Softward Inc., 1989.
66.
Lewis G., Pelosi AJ, Araya R., Dunn G.Measuring psychiatric disorder in the community: a standardised assessment for use by lay interviewersPsychol Med1992: in press.
67.
Donner A., Eliasziw M.Sample size requirements for reliability studiesStat Med1987; 6: 441-48.
68.
Eliasziw M. , Donner A.A cost-function approach to the design of reliability studiesStatMed1987; 6: 647-55.
69.
David FNTables of the correlation coefficientLondon: Cambridge University Press, 1954.
70.
Hahn GJ, Nelson W.A problem in the statistical comparison of measuring devicesTechnometrics1970; 12 : 95-102.
71.
Duncan-Jones P., Fergusson DM, Ormel J., Harwood LJA model of stability and change in minor psychiatric symptoms: results from three longitudinal studiesPsychological Medicine Monograph Supplement No. 18, 1990.
72.
Henderson AS, Byrne DG, Duncan-Jones P.Neurosis and the social environmentSidney: Academic Press, 1981.
73.
Horwood LJ, Fergusson DMNeuroticism, depression and life events: a structural equation modelSocial Psychiatry1986; 21 : 63-71.
74.
Ormel J.Neuroticism and well-being inventries: measuring traits or statesPsychol Med1983; 13 : 165-76.
75.
Wheaton B., Muthen B., Alwin DF, Summers GFAssessing reliability and stability in panel models In: Heise DR ed. Sociological methodology, 1979. San Francisco: Jossey-Bass, 1977 ; 84-136.
76.
Jagodzinski KG, Kuhnel SM, Schmidt P.Is there a Socratic Effect in non-experimental panel studies?Sociological Methods and Research1987; 15: 219-58.
77.
Raffalovich LE, Bohrnstedt GWCommon, specific and error variance components of factor models: estimation with longitudinal dataSociological Methods and Research1987; 15: 385-405.
Schilling Rsf , Hughes Jpw, Digwall-Fordice L.Disagreement between observers in an epidemiological study of respiratory diseaseBritish Medical Journal1955; 1: 65-68.
80.
Bassein L., Borghi C., Costa FV, Strocchi E., Mussi A., Ambrossioni E.Comparison of three devices for measuring blood pressureStat Med1985; 4: 361-68.
Clare AW, Cairns VEDesign development and the use of the standardized interview to assess social maladjustment and dysfunction in community studiesPsychol Med1978; 8: 589-604.
83.
Cronbach LJ , Rajaratnam N., Gleser GCTheory of Generalizability: a liberalization of reliability theoryBritish Journal of Statistical Psychology1963; 16 : 137-63.
84.
Gleser GC, Cronbach LJ, Rajaratnam N.Generalizability of scores influenced by multiple sources of variancePsychometrika1965; 30: 395-418.
85.
Cronbach LJ, Gleser GL, Nanda H., Rajaratnam N.The dependability of behavioural measurements. New York: Wiley, 1972.
86.
Gleser GC, Green BL, Winget CNQuantifying interview data on psychic impairment of disaster survivorsJ Nerv Ment Dis1978; 166 : 209-16.
87.
Streiner DL, Norman GRHealth measurement scales: a practical guide to their development and useOxford: Oxford University Press , 1989.
88.
Shavelson RJ, Webb NMGeneralizability theory: a primerNewbury Park California : Sage, 1991.
89.
Shavelson RJ, Webb NMGeneralizability theory 1973-1980Br J Math Stat Psychol1981; 34: 133-66.
90.
Searle SRLinear models for unbalanced dataNew York: Wiley, 1987.
91.
Henderson CREstimation of variance and covariance componentsBiometrics1953; 9: 226-52.
92.
LaMott LRQuadratic estimation of variance componentsBiometrics1973; 29: 311-30.
93.
Rao CREstimation of variance and covariance components in linear modelsJournal of the American Statistical Association1972; 67: 112-15.
94.
Harville DAMaximum likelihood approaches to variance components estimation and to related problemsJournal of the American Statistical Association1977; 72: 320-40.
95.
Patterson HD , Thompson R.The recovery of inter-block information when block sizes are unequalBiometrika1971; 58: 545-54.
96.
Patterson HD, Thompson R.Maximum likelihood estimation of components of varianceProceedings of the 8th International Biometric Conference1975: 197-207.
97.
Robinson DLEstimation and use of variance componentsThe Statistician1987; 36 : 3-14.
Goldstein H.Multilevel mixed linear model analysis using iterative generalized least squaresBiometrika1986; 73: 43-56.
100.
Goldstein H.Multilevel models in educational and social researchLondon: Griffin, 1987.
101.
Swallow WH , Monahan JFMonte Carlo comparison of ANOVA, MIVQUE, REML and ML estimation of variance componentsTechnometrics1984; 26: 47-57.
102.
Lee TDAssessment of inter- and intra-laboratory variances: a Bayesian alternative to B5 5497The Statistician1987; 36: 161-70.
103.
Fleiss JLDesign and analysis of clinical experimentsNew York: Wiley, 1986.
104.
Bartko JJOn various intraclass correlation reliability coefficientsPsychological Bulletin1976; 83: 762-65.
105.
Mandel J.Evaluation and control of measurementsNew York: Dekker, 1991.
106.
Jaech JLEstimating within laboratory variability from interlaboratory test dataJournal of Quality Technology1979; 11: 185-91.
107.
Anderson RLDesigns and estimators for variance components In Srivastava JN ed, Statistical design and linear models.Amsterdam: North Holland, 1975: 1-30.
108.
Thompson WO , Anderson RLA comparison of designs and estimators for the two-stage nested random modelTechnometrics1975; 17: 37-44.
109.
Smith PLSampling errors of variance components in small sample multifacet generalizability studiesJournal of Educational Statistics1978; 3: 319-46.
110.
Smith PLGaining accuracy in generalizability theory: using multiple designsJournal of Educational Measurement1981; 18: 147-54.
111.
Muse HD, Anderson RLComparison of designs to estimate variance components in a two-way classification modelTechnometrics1978;20:159-66.
112.
Muse HD, Anderson RL, Thitakamol B.Additional comparisons of designs to estimate variance components in a two-way classification modelCommunications in Statistics A1982; 11: 1403-25.
113.
Haile A., Webster JTA comparison of four designs for estimating the variances of random effects in a two-way classificationTechnometrics1975; 17: 495-500.
114.
Mostafa MGDesigns for the simultaneous estimation of functions of variance components for a two-way crossed classificationBiometrika1967; 54: 127-31.
115.
Gaylor DWThe construction and evaluation of some designs for the estimation of parameters in random models. Unpublished PhD thesisNorth Carolina State College, Raleigh. Institute of Statistics, Mimeo Series No. 256, 1960.
116.
Bush N., Anderson RLA comparison of three different procedures for estimating variance componentsTechnometrics1963; 5: 421-40.
117.
Mead R.The design of experimentsCambridge: Cambridge University Press, 1988.
118.
Kendell RE , Cooper JE, Gourlay AJ, Copeland Jrm, Sharp L., Gurland BJDiagnostic criteria of American and British psychiatristsArch Gen Psych1971; 25: 122-30.
119.
Brennan RLThe calculation of reliability from a split-plot factorial designEducational and Psychological Measurement1975; 35: 779-88.
Prairie RROptimal designs to estimate variance components and to reduce product variability for nested classifications. Unpublished PhD thesis, Department of Experimental Statistics, North Carolina State University, Raleigh. Institute of Statistics, Mimeo Series No. 313, 1962.
125.
Leone FC, Nelson LS, Johnson NL, Eisenstat S.Sampling distributions of variance components. II Empirical studies of unbalanced nested designsTechnometrics1968; 10: 719-37.
126.
Walter SD, Irwig LMEstimation of test error rates, disease prevalence and relative risk from misclassified data: a reviewJ Clin Epidemiol1988; 41: 923-37.
127.
Hui SL, Walter SDEstimating the error rates of diagnostic testsBiometrics1980; 36: 167-71.
128.
Rindskopf D., Rindskopf W.The value of latent class analysis in medical diagnosisStatistics in Medicine1986 ; 5: 21-27.
129.
Espeland MA , Handelman SLUsing latent class models to characterize and assess relative error in discrete measurementsBiometrics1989; 45: 589-99.
130.
Clogg CC, Goodman LASimultaneous latent structural analysis in several groups In: Tuma NB ed. Sociological methodology, 1985 . San Francisco: Jossey-Bass, 1985: 81-110.
131.
Espeland M. , Murphy WC, Leverett DHAssessing diagnostic reliability and estimating incidence rates associated with a strictly progressive disease: dental cariesStatistics in Medicine1988; 7: 403-16.
132.
Fleiss JL, Cohen J.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliabilityEducational and Psychological Measurement1973; 33 : 613-19.
133.
Davies M., Fleiss JLMeasuring agreement for multinomial dataBiometrics1982; 38: 1047-51.
134.
Lin L I-K.A concordance correlation coefficient to evaluate reproducibilityBiometrics1989; 45: 255-68.
135.
Krippendorff K.Bivariate agreement coefficients for reliability of data In: Borgatta F, Bohrnstedt GW ed. Sociological methodology, 1970. San Francisco : Jossey-Bass, 1970: 139-50.
136.
Stavig GRMonotonic measures of agreement for ranked dataBr J Math Stat Psychol1984; 37 : 283-87.
