The problem of measuring reliability of categorical measurements, particularly diagnostic categorizations, is addressed. The approach is based on classical measurement theory and requires interpretability of the reliability coefficients in terms of loss of precision in estimation or power in statistical tests. A general model is proposed, leading to definition of reliability indices. Design and estimation approaches are discussed. Issues and approaches found in the research literature that either lead to confusing or misleading results are presented. The signs and symptoms of unreliable diagnoses are identified, and strategies for improving the reliability of such diagnoses are discussed.
Get full access to this article
View all access options for this article.
References
1.
Kraemer HCRamifications of a population model for k as a coefficient of reliabilityPsychometrika1979; 44: 461-72.
2.
Fleiss JLStatistical methods for rates of proportionsNew York : John Wiley & Sons, 1981.
3.
Bloch DA, Kraemer HC2 x 2 kappa coefficients: Measures of agreement or associationBiometrics1989; 45 : 269-87.
4.
Lord FM, Novick MRStatistical theories of mental test scoresReading, MA : Addison-Wesley Publishing Company, Inc., 1968.
5.
Cronbach LJ, Gleser G., Nanda H., Rajaratnam J.The dependability of behavioral measurementsNew York : John Wiley & Sons, 1968.
6.
Agresti A.A model for agreement between ratings on an ordinal scalePsycho Rep1988; 44 : 539-48.
7.
Becker M.Association models to analyse agreement data: two examplesStatistics in Medicine1989; 8: 1199-208.
8.
Landis JR, Koch GGThe measurement of observer agreement for categorical dataBiometrics1977; 33 : 159-74.
Algina J.Comment on Bartko's On various intraclass correlation reliability coefficientsPsychol Bull1978; 85: 135-38.
14.
Lahey MA, Downey RG, Saal FEIntraclass correlations: There's more there than meets the eyePsychol Bull1983; 93 : 586-95.
15.
Cacoullos T.A relation between t-0 and F-distributionsJournal of the American Statistical Association1965; 60: 528-31.
16.
Kraemer HCOn estimation and hypothesis testing problems for correlation coefficientsPsychometrika1975; 40: 473-85.
17.
Kraemer HC, Bloch DAKappa coefficients in epidemiology: An appraisal of a reappraisalJournal of Clinical Epidemiology1988; 41: 959-68.
18.
Fleiss JLMeasuring nominal scale agreement among many ratersPsychol Bull1971; 76: 378-82.
19.
Davies M., Fleiss JLA measure of agreement for multinomial data arrayed in a two-way layout1981.
20.
Goodman LA, Kruskal WHMeasures of association for cross classifications III: Approximate sampling theoryJournal of the American Statistical Association1963; 58: 310-64.
21.
Goodman LA, Kruskal WHMeasures of association for cross classifications, IV: Simplification of asymptotic variancesJournal of the American Statistical Association1972; 67 : 415-24.
22.
Goodman LA, Kruskal WHMeasures of association for cross classificationsJournal of the American Statistical Association1954; 49: 732-64.
23.
Goodman LA, Kruskal WHMeasures of association for cross classifications. II. Further discussion and referencesJournal of the American Statistical Association1959; 54: 123-63.
24.
Kraemer HCAssessment of 2 x 2 associations: Generalization of signal detection methodologyThe American Statistician1988; 42: 37-49.
25.
Kraemer HCA study of reliability and its hierarchical structure in observed chimpanzee behaviorPrimates1979; 20: 553-61.
26.
Kraemer HCHow many raters? Toward the most reliable diagnostic categorizationStatistics in Medicine1991 (in press).
