Though qualitative research has become more prevalent in practice over the last 30 years, there is still considerable uncertainty among researchers regarding how to ensure inter-rater consistency when teams are tasked with coding qualitative data. In this article, we offer an explanation of a methodology that our qualitative team used to achieve systematic coding of our dataset in a way that preserved the contextual, subjective nature of the data, lent itself to the deductive and inductive creation of a layered codebook, and ensured consistent application of the codebook to varied types of data. This methodology prepared us to draw logical and substantiated conclusions during subsequent analyses; hence, the process serves as a welcome addition to the literature on consistently coding qualitative data in a manner that honors its defining characteristics.
AguinisHHillNBaileyJ (2019) Best practices in data collection and preparation: recommendations for reviewers, editors, and authors. Organizational Research Methods. Epub ahead of print 13 March. DOI: 10.1177/1094428119836485
2.
ArmstrongDGoslingAWeinmanJ, et al. (1997) The place of inter-rater reliability in qualitative research: an empirical study. Sociology31(3): 597–606.
3.
BernardHR (2000) Social Research Methods: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage Publication.
4.
BraunVClarkeV (2006) Using thematic analysis in psychology. Qualitative Research in Psychology3(2): 77–101.
5.
BurlaLKnierimBBarthJ, et al. (2008) From text to codings: intercoder reliability assessment in qualitative content analysis. Nursing Research57(2): 113–117.
6.
CampbellJLQuincyCOssermanJ, et al. (2013) Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociological Methods & Research42(3): 294–320.
7.
CareyJWMorganMOxtobyMJ (1996) Intercoder agreement in analysis of responses to open-ended interview questions: examples from tuberculosis research. CAM Journal8(3): 1–5.
8.
CharmazK (2006) Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Thousand Oaks, CA: Sage Publications.
9.
ChurchSPDunnMProkopyL (2019) Benefits to qualitative data quality with multiple coders: two case studies in multi-coder data analysis. Journal of Rural Social Science34(1/2): 1–14.
10.
EisnerE (1998) The Enlightened Eye: Qualitative Inquiry and the Enhancement of Educational Practice. Upper Saddle River, NJ: Prentice Hall.
11.
ElliottRFischerCTRennieDL (1999) Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology38(3): 215–229.
12.
ElmorePBWoehlkePL (1998) Twenty years of research methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research. In: American Educational Research Association Annual Meeting, San Diego, CA.
13.
FeredayJMuir-CochraneE (2006) Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods5(1): 80–92.
14.
GarrisonDRCleveland-InnesMKooleM, et al. (2006) Revisiting methodological issues in transcript analysis: negotiated coding and reliability. The Internet and Higher Education9(1): 1–8.
15.
GlazerJLEganC (2018) The ties that bind: building civic capacity for the Tennessee Achievement School District. American Educational Research Journal55(5): 928–964.
16.
GolafshaniN (2003) Understanding reliability and validity in qualitative research. The Qualitative Report8(4): 597–606.
17.
GubaELincolnY (1994) Competing paradigms in qualitative research. In: DenzinNKLincolnY (eds) Handbook of Qualitative Research. Thousand Oaks, CA: Sage Publications, 105–117.
HammerDBerlandLK (2014) Confusing claims for data: a critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences23(1): 37–46.
20.
HayesAFKrippendorffK (2007) Answering the call for a standard reliability measure for coding data. Communication Methods and Measures1(1): 77–89.
21.
HillCThompsonBWilliamsE (1997) A guide to conducting consensual qualitative research. The Counseling Psychologist25(4): 517–572.
22.
HillCKnoxSThompsonB, et al. (2005) Consensual qualitative research: an update. Journal of Counseling Psychology52(2): 196–205.
23.
HodsonR (1998) Organizational ethnographies: an underutilized resource in the sociology of work. Social Forces76(4): 1173–1208.
24.
HruschkaDJSchwartzDSt. JohnDC, et al. (2004) Reliability in coding open-ended data: lessons learned from HIV behavioral research. Field Methods16(3): 307–331.
25.
KirkJMillerM (1986) Reliability and Validity in Qualitative Research. Newbury Park, CA: Sage Publications.
26.
KurasakiK (2000) Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods12(3): 179–194.
27.
LevittHBambergMCreswellJ, et al. (2018) Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: the APA publications and communications board task force report. American Psychologist73(1): 26–46.
LombardMSnyder-DuchJBrackenCC (2002) Content analysis in mass communication: assessment and reporting of intercoder reliability. Human Communication Research28(4): 587–604.
30.
MacPhailCKhozaNAblerL, et al. (2016) Process guidelines for establishing intercoder reliability in qualitative studies. Qualitative Research16(2): 198–212.
31.
MacQueenKMMcLellanEKayK, et al. (1998) Codebook development for team-based qualitative analysis. CAM Journal10(2): 31–36.
32.
MarshallCRossmanG (1989) Designing Qualitative Research. Newbury Park, CA: Sage Publications.
33.
MaysNPopeC (2000) Assessing quality in qualitative research. The BMJ320(7226): 50–52.
34.
MilesMHubermanA (1994) Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications.
35.
MorseJ (1999) Myth #93: reliability and validity are not relevant to qualitative inquiry. Qualitative Health Research9(6): 717–718.
36.
MorseJBarrettMMayanM, et al. (2002) Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods1(2): 13–22.
37.
NobleHSmithJ (2015) Issues of validity and reliability in qualitative research. Evidence Based Nursing18(2): 34–35.
38.
NowellLNorrisJWhiteD, et al. (2017) Thematic analysis: striving to meet the trustworthiness criteria. International Journal of Qualitative Methods16(1): 1–13.
39.
O’ConnorCOJoffeH (2020) Intercoder reliability in qualitative research: debates and practical guidelines. International Journal of Qualitative Research19(January): 1–13.
40.
PetersonJ (2019) Presenting a qualitative study: a reviewer’s perspective. Gifted Child Quarterly63(3): 147–158.
41.
RobertsKDowellANieJ-B (2019) Attempting rigour and replicability in thematic analysis of qualitative research data: a case study of codebook development. BMC Medical Research Methodology19(66): 1–8.
42.
SaldañaJ (2016) The Coding Manual for Qualitative Researchers (3rd edn.). London, UK: Sage Publications.
43.
SandelowskiM (1986) The problem of rigor in qualitative research. Advances in Nursing Science8(3): 27–37.
44.
SchoenfeldAH (1992) On paradigms and methods: what do you do when the ones you know don’t do what you want them to? Issues in the analysis of data in the form of videotapes. The Journal of the Learning Sciences2(2): 179–214.
SmithML (1987) Publishing qualitative research. American Educational Research Journal24(2): 173–183.
47.
SmithBMcGannonK (2018) Developing rigor in qualitative research: problems and opportunities within sport and exercise psychology. International Review of Sport and Exercise Psychology11(1): 101–121.
48.
StearnsEBottiaMCGierschJ, et al. (2019) Do relative advantages in STEM grades explain the gender gap in selection of a STEM major in college? A multimethod answer. American Educational Research Journal57(1): 218–257.
49.
SyedMNelsonS (2015) Guidelines for establishing reliability when coding narrative data. Emerging Adulthood3(6): 375–387.
50.
TobinGBegleyC (2004) Methodological rigour within a qualitative framework. Journal of Advanced Nursing48(4): 388–396.
WestonCGandellTBeauchampJ, et al. (2001) Analyzing interview data: the development and evolution of a coding system. Qualitative Sociology24(3): 381–400.
