Guidelines for Establishing Reliability When Coding Narrative Data

Abstract

The use of quantitative, qualitative, and mixed methods approaches has been foundational to research on emerging adulthood, yet there remain many unresolved methodological issues pertaining to how to handle qualitative data. The purpose of this article is to review best practices for coding and establishing reliability when working with narrative data. In doing so, we highlight how establishing reliability must be seen as an evolving process, rather than simply a focus on the end product. The review is divided into three broad sections. In the first section, we discuss relatively more quantitatively focused methods of coding and establishing reliability, whereas in the second section we discuss relatively more qualitatively focused methods. In the final section, we provide recommendations for researchers interested in coding narrative and other types of open-ended data. This article is intended to serve as an essential resource for researchers working on a variety of topics related to emerging adulthood and beyond.

Keywords

interrater reliability coding kappa narrative

Get full access to this article

View all access options for this article.

References

Adler

J. M.

(2012). Living into the story: Agency and coherence in a longitudinal study of narrative identity development and mental health over the course of psychotherapy. Journal of Personality and Social Psychology, 102, 367–389.

Adler

J. M.

Chin

E. D.

Kolisetty

A. P.

Oltmanns

T. F.

(2012). The distinguishing characteristics of narrative identity in adults with features of borderline personality disorder: An empirical investigation. Journal of Personality Disorders, 26, 498–512.

Alea

Bluck

(2003). Why are you telling me that? A conceptual model of the social function of autobiographical memory. Memory, 11, 165–178.

Arnett

J. J.

(2005). The vitality criterion: A new standard of publication for journal of adolescent research. Journal of Adolescent Research, 20, 3–7.

Arnett

J. J.

(2015). Identity development from adolescence to emerging adulthood: What we know and (especially) don’t know. In McLean

K. C.

Syed

(Eds), The Oxford handbook of identity development (pp. 53–64). New York, NY: Oxford University Press.

Bakeman

Gottman

J. M.

(1986). Observing behavior: An introduction to sequential analysis. Cambridge, England: Cambridge University.

Bamberg

(2004). Talk, small stories, and adolescent identities. Human Development, 47, 366–369.

Bauer

J. J.

McAdams

D. P.

(2010). Eudaimonic growth: Narrative growth goals predict increases in ego development and subjective well-being 3 years later. Developmental Psychology, 46, 761–772.

Benish-Weisman

(2009). Between trauma and redemption story form differences in immigrant narratives of successful and nonsuccessful immigration. Journal of Cross-Cultural Psychology, 40, 953–968.

10.

Bennett

E. M.

Alpert

Goldstein

A. C.

(1954). Communications through limited-response questioning. Public Opinion Quarterly, 18, 303–308.

11.

Bernard

H. R.

(2000). Social research methods: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

12.

Bradley

E. H.

Curry

L. A.

Devers

K. J.

(2007). Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Services Research, 42, 1758–1772.

13.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101.

14.

Brennan

R. L.

Prediger

D. J.

(1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687–699.

15.

Campbell

J. L.

Quincy

Osserman

Pedersen

O. K.

(2013). Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42, 294–320.

16.

Chi

M. T. H.

(1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6, 271–315.

17.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284.

18.

Cohen

(1960). A coefficient of reliability for nominal scales. Educational and Psychological Measurement, 20, 37–46.

19.

Cook

T. D.

Campbell

D. T.

(1979). Quasi-experimentation: Design and analysis for field setting. New York, NY: Rand-McNally.

20.

Denzin

N. K.

Lincoln

Y. S.

(Eds.). (2011). The SAGE handbook of qualitative research. Thousand Oaks, CA: Sage.

21.

Dunlop

W. L.

Tracy

J. L.

(2013). Sobering stories: Narratives of self-redemption predict behavioral change and improved health among recovering alcoholics. Journal of Personality and Social Psychology, 104, 576.

22.

Feinstein

A. R.

Cicchetti

D. V.

(1990). High agreement but low kappa: The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543–549.

23.

Fiese

B. H.

Spagnola

(2005). Narratives in and about families: An examination of coding schemes and a guide for family researchers. Journal of Family Psychology, 19, 51–61.

24.

Fleiss

J. L.

(1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105–112.

25.

Fleiss

J. L.

Cohen

(1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619.

26.

Gergen

K. J.

(2009). Realities and relationships: Soundings in social construction. Cambridge, MA: Harvard University Press.

27.

Guba

E. G.

Lincoln

Y. S.

(1986). The countenances of fourth-generation evaluation: Description, judgment, and negotiation. Evaluation Studies Review Annual, 11, 70–88.

28.

Hammack

P. L.

(2006). Identity, conflict, and coexistence life stories of Israeli and Palestinian adolescents. Journal of Adolescent Research, 21, 323–369.

29.

Hammack

P. L.

(2010). Narrative and the politics of identity: The cultural psychology of Israeli and Palestinian youth. New York, NY: Oxford University Press.

30.

Hammack

P. L.

Thompson

E. M.

Pilecki

(2009). Configurations of identity among sexual minority youth: Context, desire, and narrative. Journal of Youth and Adolescence, 38, 867–883.

31.

Hammer

Berland

L. K.

(2014). Confusing claims for data: A critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences, 23, 37–46.

32.

Hill

C. E.

Knox

Thompson

B. J.

Williams

E. N.

Hess

S. A.

Ladany

(2005). Consensual qualitative research: An update. Journal of Counseling Psychology, 52, 196.

33.

Hill

C. E.

Thompson

B. J.

Williams

E. N.

(1997). A guide to conducting consensual qualitative research. The Counseling Psychologist, 25, 517–572.

34.

Hruschka

Schwartz

John

D. C.

Picone-Decaro

Jenkins

Carey

(2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods, 16, 307–331.

35.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed methods research: A research paradigm whose time has come. American Educational Research Association, 33, 14–26.

36.

Josselson

(2000). Stability and change in early memories over 22 years: Themes, variations, and cadenzas. Bulletin of the Menninger Clinic, 64, 462–481.

37.

Josselson

(2009). The present of the past: Dialogues with memory over time. Journal of personality, 77, 647–668.

38.

Korobov

(2009). “He’s got no game”: Young men’s stories about failed romantic and sexual experiences. Journal of Gender Studies, 18, 99–114.

39.

Korobov

Bamberg

(2004). Positioning a “mature” self in interactive practices: How adolescent males negotiate “physical attraction” in group talk. British Journal of Developmental Psychology, 22, 471–492.

40.

Korobov

Thorne

(2006). Intimacy and distancing young men’s conversations about romantic relationships. Journal of Adolescent Research, 21, 27–55.

41.

Korobov

Thorne

(2007). How late-adolescent friends share stories about relationships: The importance of mitigating the seriousness of romantic problems. Journal of Social and Personal Relationships, 24, 971–992.

42.

Krippendorff

(2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage.

43.

Krippendorff

(2011). Agreement and information in the reliability of coding. Communication Methods and Measures, 5, 93–112.

44.

Lieblich

Tuval-Mashiach

Zilber

(1998). Narrative research. Reading, analysis and interpretation (Vol. 47). Thousand Oaks, CA: Sage.

45.

Lilgendahl

J. P.

McAdams

D. P.

(2011). Constructing stories of self-growth: How individual differences in patterns of autobiographical reasoning relate to well-being in midlife. Journal of Personality, 79, 391–428.

46.

Lilgendahl

J. P.

McLean

K. C.

Mansfield

C. D.

(2013). When is meaning making unhealthy for the self? The roles of neuroticism, implicit theories, and memory telling in trauma and transgression memories. Memory, 21, 79–96.

47.

Lincoln

Guba

(2000). The qualitative researcher’s companion. Journal of Management, 23, 409–473.

48.

Lodi-Smith

Geise

A. C.

Roberts

B. W.

Robins

R. W.

(2009). Narrating personality change. Journal of Personality and Social Psychology, 96, 679–689.

49.

Lyons

(2009). Is the American psychological association ready to accept a division for qualitative inquiry? The Humanistic Psychologist, 37, 101–107.

50.

Main

Goldwyn

(1994). Adult attachment classification system. Unpublished manuscript, University of California, Berkeley.

51.

Marcia

J. E.

(1966). Development and validation of ego-identity status. Journal of Personality and Social Psychology, 5, 551–558.

52.

Marcia

J. E.

(1993). The ego identity status approach to ego identity. In Marcia

J. E.

Waterman

A. S.

Matteson

D. R.

Archer

S. L.

Orlofsky

J. L.

(Eds.), Ego identity: A handbook for psychosocial research (pp. 1–21). New York, NY: Springer-Verlag.

53.

Marcia

J. E.

Waterman

A. S.

Matteson

D. R.

Archer

S. L.

Orlosky

J. L.

(Eds.). (1993). Ego identity: A handbook for psychosocial research. New York, NY: Springer.

54.

Martin Andrés

Femia Marzo

(2004). Delta: A new measure of agreement between two raters. British Journal of Mathematical and Statistical Psychology, 57, 1–19.

55.

McAdams

D. P.

Anyidoho

N. A.

Brown

Huang

Y. T.

Kaplan

Machado

M. A.

(2004). Traits and stories: Links between dispositional and narrative features of personality. Journal of Personality, 72, 761–784.

56.

McAdams

D. P.

Bauer

J. J.

Sakaeda

A. R.

Anyidoho

N. A.

Machado

M. A.

Magrino -Failla

… Pals

J. L.

(2006). Continuity and change in the life story: A longitudinal study of autobiographical memories in emerging adulthood. Journal of Personality, 74, 1371–1400.

57.

McAdams

D. P.

McLean

K. C.

(2013). Narrative identity. Current Directions in Psychological Science, 22, 233–238.

58.

McLean

K. C.

Fournier

M. A.

(2008). The content and processes of autobiographical reasoning in narrative identity. Journal of Research in Personality, 42, 527–545.

59.

McLean

K. C.

Pratt

M. W.

(2006). Life’s little (and big) lessons: Identity statuses and meaning-making in the turning point narratives of emerging adults. Developmental Psychology, 42, 714–722.

60.

McLean

K. C.

Syed

(2015). The field of identity development needs an identity: An introduction to the Oxford handbook of identity development. In McLean

K. C.

Syed

(Eds.), The Oxford handbook of identity development (pp. 1–10). New York, NY: Oxford University Press.

61.

McLean

K. C.

Syed

Yoder

Greenhoot

A. F.

(2014). The role of domain content in understanding identity development processes. Journal of Research on Adolescence.

62.

Merriam

S. B.

(2009). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass.

63.

Morgan

E. M.

Korobov

(2012). Interpersonal identity formation in conversations with close friends about dating relationships. Journal of Adolescence, 35, 1471–1483.

64.

Morse

J. M.

Barrett

Mayan

Olson

Spiers

(2008). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1, 13–22.

65.

Orwin

R. G.

(1994). Evaluating coding decisions. In Cooper

Hedges

L. V.

(Eds.), The handbook of research synthesis (pp. 139–162). New York, NY: Russell Sage.

66.

Pasupathi

Mansour

Brubaker

J. R.

(2007). Developing a life story: Constructing relations between self and experience in autobiographical narratives. Human Development, 50, 85–110.

67.

Pasupathi

Rich

(2005). Inattentive listening undermines self-verification in personal storytelling. Journal of Personality, 73, 1051–1086.

68.

Pasupathi

Wainryb

(2010). On telling the whole story: Facts and interpretations in autobiographical memory narratives from childhood through midadolescence. Developmental Psychology, 46, 735–746.

69.

Patton

M. Q.

(2002). Two decades of developments in qualitative inquiry a personal, experiential perspective. Qualitative Social Work, 1, 261–283.

70.

Schachter

E. P.

(2004). Identity configurations: A new perspective on identity formation in contemporary society. Journal of Personality, 72, 167–200.

71.

Schachter

E. P.

(2005). Context and Identity formation a theoretical analysis and a case study. Journal of Adolescent Research, 20, 375–395.

72.

Scott

W. A.

(1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321–325.

73.

Shallcross

S. L.

(2013). Social support mediates the relation between attachment and responses to potentially traumatic events. Unpublished doctoral dissertation, University of Minnesota, Minneapolis, USA. Retrieved from http://purl.umn.edu/159917

74.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

75.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

76.

Syed

(2010). Memorable everyday events in college: Narratives of the intersection of ethnicity and academia. Journal of Diversity in Higher Education, 3, 56–69.

77.

Syed

Azmitia

(2008). A narrative approach to ethnic identity in emerging adulthood: Bringing life to the identity status model. Developmental Psychology, 44, 1012–1027.

78.

Syed

Juan

M. J. D.

Juang

L. P.

(2011). Might the survey be the intervention? Participating in ethnicity-related research as a consciousness-raising experience. Identity, 11, 289–310.

79.

Tappan

M. B.

(1997). Interpretive psychology: Stories, circles, and understanding lived experience. The Journal of Social Issues, 53, 645–656.

80.

Tashakkori

Creswell

J. W.

(2007). Editorial: The new era of mixed methods. The Journal of Mixed Methods Research, 1, 3–7.

81.

Taylor

J. M.

Gilligan

Sullivan

A. M.

(1996). Missing voices, changing meanings: Developing a voice-centered, relational method and creating an interpretative community. In Wilkinson

(Ed.), Feminist social psychologies: International perspectives (pp. 233–257). Buckingham, England: Open University Press.

82.

van Dulmen

M. H.

(2013). Emerging adulthood—the journal. Emerging Adulthood, 1, 3–4.

83.

Waters

T. E.

Fivush

(2014). Relations between narrative coherence, identity, and psychological well-being in emerging adulthood. Journal of Personality.

84.

Way

Santos

Niwa

E. Y.

Kim-Gervey

(2008). To be or not to be: An exploration of ethnic identity development in context. New Directions for Child and Adolescent Development, 120, 61–79.

85.

Whitehurst

G. J.

(1984). Interrater agreement for journal manuscript reviews. American Psychologist, 39, 22–28.

86.

Whittemore

Chase

S. K.

Mandle

C. L.

(2001). Validity in qualitative research. Qualitative Health Research, 11, 522–537.

87.

Wolfe

E. W.

Moulder

B. C.

Myford

C. M.

(2000). Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model. Journal of Applied Measurement, 2, 256–280.

88.

Zwick

(1988). Another look at interrater agreement. Psychological Bulletin, 103, 374–401.