Percent agreement and Pearson's correlation coefficient are frequently used to represent inter-examiner reliability, but these measures can be misleading. The use of percent agreement to measure inter-examiner agreement should be discouraged, because it does not take into account the agreement due solely to chance. Caution must be used in the interpretation of Pearson's correlation, because it is unaffected by the presence of any systematic biases. Analyses of data from a reliability study show that even though percent agreement and kappa were consistently high among three examiners, the reliability measured by Pearson's correlation was inconsistent. This study shows that correlation and kappa can be used together to uncover non-random examiner error.
Get full access to this article
View all access options for this article.
References
1.
Badersten, A.; Nilveus, R.; and Egelberg, J. (1984): Reproducibility of Probing Attachment Level Measurements , J Clin Periodontol11:475-485.
2.
Bartko, J.J. (1976): On Various Intraclass Correlation Reliability Coefficients, Psychol Bull83:762-765.
3.
Bell, R.M. and Klein, S.P. (1984): Management and Evaluation of the Effects of Misclassification in a Controlled Clinical Trial, J Dent Res63 (Spec Iss):731-734.
4.
Burt, B.A. (1983): Methods for Assessing the Distribution of Oral Diseases. In: Dentistry, Dental Practice, and the Community , 3rd ed., D.F. Striffler, W.W. Young, and B.A. Burt, Eds., Philadelphia : W.B. Saunders Co., pp. 107-108.
5.
Chilton, N.W. (1982): Reliability Studies. In: Design and Analysis in Dental and Oral Research, 2nd ed., New York: Praeger Publishers, pp. 415-421.
6.
Clemmer, B.A. and Barbano, J.P. (1974): Reproducibility of Periodontal Scores in Clinical Trials, J Periodont Res9 (Suppl 14):118-128.
7.
Everitt, B.S. (1977): The Analysis of Contingency Tables. London : Chapman and Hall Ltd., pp. 5-6.
8.
Feldman, R.S.; Douglass, C.W.; Loftus, E.R.; Kapur, K.K.; and Chauncey, H.H. (1982): Inter-examiner Agreement in the Measurement of Periodontal Disease, J Periodont Res17 :80-89.
9.
Fleiss, J.L. (1981): Statistical Methods for Rates and Proportions , 2nd ed., New York: John Wiley and Sons, pp. 216, 218.
10.
Fleiss, J.L. and Chilton, N.W. (1983): The Measurement of Interexaminer Agreement on Periodontal Disease, J Periodont Res18:601-606.
11.
Fleiss, J.L. and Shrout, P.E. (1977): The Effects of Measurement Errors on Some Multivariate Procedures, Am J Public Health67:1188-1191.
12.
Fleiss, J.L.; Slakter, M.J.; Fischman, S.L.; Park, M.H.; and Chilton, N.W. (1979): Inter-examiner Reliability in Caries Trials, J Dent Res58:604-609.
13.
Fleiss, J.L.; Chilton, N.W.; and Park, M.H. (1980): Inter-and Intra-examiner Variability in Scoring Supragingival Plaque: II. Statistical Analysis, Pharm Therapeut Dent5:5-9.
14.
Horowitz, H.S. (1972): Inter- and Intra-examiner Variability. In: Proceedings of the Conference on the Clinical Testing of Cariostatic Agents, Chicago: American Dental Association, pp. 96-97.
15.
Landis, J.R. and Koch, G.G. (1977): The Measurement of Observer Agreement for Categorical Data, Biometrics33:159-174.
16.
Lemasney, J.; O'Mullane, D.; and Coleman, M. (1984): Effect of Fluoridation on Dental Health in 5- and 11-year-old Irish Schoolchildren, Community Dent Oral Epidemiol12:218-222.
17.
Leverett, D.H.; Sveen, O.B.; and Jensen, O.E. (1985): Weekly Rinsing with a Fluoride Mouthrinse in an Unfluoridated Community : Results after Seven Years, J Public Health Dent45 :95-100.
18.
Mann, J.; Goultschin, J.; and Call, R.L. (1985): Assessment of Inter-examiner Agreement in Scoring Periodontal Disease, J Periodont Res20:86-90.
19.
Pliskin, J.S.; Shwartz, M.; Grondahl, H.G.; and Boffa, J. (1984): Reliability of Coding Depth of Approximal Carious Lesions from Non-independent Interpretations of Serial Bitewing Radiographs , Community Dent Oral Epidemiol12:366-370.
20.
Ran, F. and Anaise, J.Z. (1985): Prevalence of Dental Decay in Jewish and Arab Schoolchildren in the Jerusalem Area, Community Dent Oral Epidemiol13:97-99.
21.
Vandervelden, V. (1980): Influence of Probing Force on the Reproducibility of Bleeding Tendency Measurements, J Clin Periodontol7:421-427.
22.
Vandervelden, V. and Devries, J.H. (1980): The Influence of Probing Force on the Reproducibility of Pocket Depth Measurements, J Clin Periodontol7:414-420.
23.
Vigild, M. (1985): Prevalence of Malocclusion in Mentally Retarded Young Adults, Community Dent Oral Epidemiol13 :183-184.
24.
World Health Organization (1977): Oral Health Surveys, Basic Methods, 2nd ed. Geneva: WHO, pp. 18-20.
