The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study

Abstract

Assessing inter-rater reliability, whereby data are independently coded and the codings compared for agreements, is a recognised process in quantitative research. However, its applicability to qualitative research is less clear: should researchers be expected to identify the same codes or themes in a transcript or should they be expected to produce different accounts? Some qualitative researchers argue that assessing inter-rater reliability is an important method for ensuring rigour, others that it is unimportant; and yet it has never been formally examined in an empirical qualitative study. Accordingly, to explore the degree of inter-rater reliability that might be expected, six researchers were asked to identify themes in the same focus group transcript. The results showed close agreement on the basic themes but each analyst `packaged' the themes differently.

Keywords

inter-rater reliability qualitative research research methods.

Get full access to this article

View all access options for this article.

References

Bryman, A. and Burgess, R. G. (eds.) 1994. Analyzing qualitative data. London: Routledge.

Burgess, R. G. 1984. In the Field: An Introduction to Field Research. London: Allen and Unwin.

Daly, J. , MacDonald, I. and Willis, E. 1992. `Why Don't You Ask Them? A Qualitative Research Framework for Investigating the Diagnosis of Cardiac Normality', pp. 189-206 in J. Daly , I. MacDonald and E. Willis (eds.) Researching Health Care: Designs, Dilemmas, Disciplines. London: Tavistock.

Denzin, N. K. 1978. The Research Act: A Theoretical Introduction to Sociological Methods. New York: McGraw Hill.

Denzin, N. K. and Lincoln, Y. S. 1994. `Introduction: Entering the Field of Qualitative Research', pp.1-18 in N. K. Denzin and Y. S. Lincoln (eds.) Handbook of Qualitative Research. London: SAGE Publications.

Glaser, B. and Strauss, A. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.

Hammersely, M. 1991. Reading Ethnographic Research: A Critical Guide. London: Longman.

Marshall, C. and Rossman, G. B. 1989. Designing Qualitative Research. London: SAGE Publications.

Mays, N. and Pope, C. 1995. `Rigour and Qualitative Research'. British Medical Journal 311:109-112.

10.

Morse, J. M. 1994. `Designing Funded Qualitative Research', pp. 220-235 in N. K. Denzin and Y. S. Lincoln (eds.) Handbook of Qualitative Research. London: SAGE Publications.

11.

Moscovici, S. 1981. `On Social Representation', in J. P. Forgas (ed.) Social Cognition: Perspectives on Everyday Life. London: Academic Press.

12.

Oleson, V. , Droes, N. , Hatton, D. , Chico, N. and Schatzman, L. 1994. `Analyzing together: recollections of a team approach', in A. Bryman and R. G. Burgess (eds.) Analyzing qualitative data. London: Routledge.

13.

Tyler, S. A. 1986. `Post-modern Ethnography: From Document of the Occult to Occult Document', pp. 122-140 in J. Clifford and G. E. Marcus (eds.) Writing Culture: The Poetics and Politics of Ethnography. Berkeley: SAGE Publications.

14.

Vidich, A. J. and Lyman, S. M. 1994. `Qualitative Methods: Their History in Sociology and Anthropology', pp. 23-59 in N. K. Denzin and Y. S. Lincoln (eds.) Handbook of Qualitative Research. London: SAGE Publications.

15.

Waitzkin, H. 1991. The Politics of Medical Encounters. New Haven: Yale.