Estimating with a Latent Class Model the Reliability of Nominal Judgments Upon Which Two Raters Agree

Abstract

Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single rater's classifications rather than to improve reliability. When raters agree, the aggregation problem does not arise. Nevertheless, a proportion of this agreement is likely to have occurred by chance, so reliability issues remain even if the focus is on nominal judgments upon which raters concur. In this article, the reliability of nominal judgments upon which raters agree is addressed in the framework of a latent class model that includes a systematic agreement parameter. It is shown that if attention is limited to agreement cases, the value of this parameter increases as expected. Circumstances under which agreement case reliability is important are discussed.

Keywords

rater agreement rater reliability nominal judgments latent class analysis

Get full access to this article

View all access options for this article.

References

Aickin, M. (1990). Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's kappa.Biometrics,46, 293-302.

Fennig, S. , Craig, T. J. , Tanenberg-Karant, M. , & Bromet, E. J. (1994). Comparison of facility and research diagnosesin first-admissionpsychoticpatients.American Journalof Psychiatry,151(10),1423-1429.

Guggenmoos-Holzmann, I. , & Vonk, R. (1998). Kappa-like indices of observer agreement viewed from a latent class perspective.Statistics in Medicine,17, 797-812.

Hooley, J. M. , & Gotlib, I. H. (2000). A diathesis-stress conceptualization of expressed emotion and clinical outcome.Applied & Preventive Psychology,9(3), 135-151.

Lazarsfeld, P. F. , & Henry, N. W. (1968).Latent structure analysis. Boston: Houghton Mifflin.

Maxwell, A. E. (1977). Coefficients of agreement between observers and their interpretation.British Journal of Psychiatry,130, 79-83.

Schuster, C. (2004). A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales.Educational and Psychological Measurement,64(2), 243-253.

Schuster, C. , & Smith, D. A. (2002). Indexing systematic rater agreement with a latent class model.Psychological Methods,7(3), 384-395.

Schuster, C. , & Smith, D. A. (2005). Dispersion weighted kappa: An integrative framework for metric and nominal scale agreement coefficients.Psychometrika,70(1), 135-146.