Measuring and Visualizing Coders’ Reliability: New Approaches and Guidelines From Experimental Data

Abstract

This study investigates inter- and intracoder reliability, proposing a new approach based on social network analysis (SNA) and exponential random graph models (ERGM). During a recent exit poll, the responses of voters to two open-ended questions were recorded. A coding experiment was conducted where a group of coders coded a sample of text segments. Analyzing the data, we show that the proposed SNA/ERGM method extends significantly our analytical leverage, beyond what popular tools such as Krippendorff’s α and Fleiss’s κ have to offer. The reliability of coding for individual coders differed significantly for the two questions although they were very similar and the same codebook was used. We conclude that the main advantages of the proposed SNA/ERGM method are the intuitive visualizations and the nuanced measurements. Detailed guidelines are provided for practitioners who would like to use the proposed method in operational settings.

Keywords

coding reliability SNA ERGM qualitative

Get full access to this article

View all access options for this article.

References

Almquist

Z. W.

Bagozzi

B. E.

. 2017. “Using Radical Environmentalist Texts to Uncover Network Structure and Network Features.” Sociological Methods & Research. doi:10.1177/0049124117729696.

Baird

J. A.

Meadows

Leckie

Caro

. 2017. “Rater Accuracy and Training Group Effects in Expert- and Supervisor-based Monitoring Systems, Assessment in Education.” Principles, Policy & Practice 24(1):44–59.

Bais

Frank

Schouten

Barry

Lugtig

Peter

Toepoel

Vera

Arends-Tòth

Judit

Douhou

Salima

Kieruj

Natalia

Morren

Mattijn

Vis

Corrie

. 2019. “Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably? A Case Study on 11 Dutch General Population Surveys.” Sociological Methods & Research 48(2):263–95.

Benoit

Kenneth

Conway

Drew

Benjamin Lauderdale

Laver

Michael

Mikhaylov

Slava

. 2016. “Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review 110(2):278–95.

Benoit

Kenneth

Munger

Spirling

. 2019. “Measuring and Explaining Political Sophistication through Textual Complexity.” American Journal of Political Science 63(2):491–508.

Berry

J. K.

Mielke

P. W.

. 1988. “A Generalization of Cohen’s Kappa Agreement Measure to Interval Measurement and Multiple Raters.” Educational and Psychological Measurement 48(4):921–33.

Block

2015. “Reciprocity, Transitivity, and the Mysterious Three-cycle.” Social Networks 40:163–73. doi:10.1016/j.socnet.2014.10.005.

Cabrera

L. Y.

Reiner

P. B.

. 2018. “A Novel Sequential Mixed-method Technique for Contrastive Analysis of Unscripted Qualitative Data: Contrastive Quantitized Content Analysis.” Sociological Methods & Research 47(3):532–48. doi:10.1177/0049124116661575.

Campbell

J. L.

Quincy

Osserman

Pedersen

O. K.

. 2013. “Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement.” Sociological Methods & Research 42(3):294–320. http://smr.sagepub.com/content/42/3/294.abstract?etoc.

10.

Compton

Love

T. P.

Sell

. 2012. “Developing and Assessing Intercoder Reliability in Studies of Group Interaction.” Sociological Methodology 42(1):348–64.

11.

Crittenden

K. S.

Hill

R. J.

1971. “Coding Reliability and Validity of Interview Data.” American Sociological Review 36:1073–80.

12.

Ennser-Jedenastik

Meyer

. n.d. “The Impact of Party Cues on Manual Coding of Political Texts.” Political Science Research and Methods, 1–9. doi:10.1017/psrm.2017.29.

13.

Feng

G. C.

2014. “Estimating Intercoder Reliability: A Structural Equation Modelling Approach.” Quality & Quantity 48:2355–69.

14.

Fleiss

Joseph L.

1971. “Measuring Nominal Scale Agreement among Many Raters.” Psychological Bulletin 76:378–82.

15.

Friedkin

N. E.

1982. “Information Flow through Stron and Weak Ties in Intraorganizational Social Networks. Social Networks 3:273–85.

16.

Girard

J. M.

Cohn

J. F.

. 2016. “A Primer on Observational Measurement.” Assessment 23(4):404–13. doi:10.1177/1073191116635807.

17.

Grimmer

Stewart

B. M.

. 2011. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3):267–97.

18.

Gummer

Blumenberg

M. S.

Roßmannm

. 2019. Learning Effects in Coders and Their Implications for Managing Content Analyses. International Journal of Social Research Methodology 22(2):139–52.

19.

Haselmayer

Jenny

. 2017. “Sentiment Analysis of Political Communication: Combining a Dictionary Approach with Crowdcoding.” Quality & Quantity 51:2623–46.

20.

Hayes

A. F.

Krippendorff

. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measurement 1(1):77–89.

21.

Hooge

I. T. C.

Niehorster

D. C.

Nyström

Andersson

Hessels

R. S.

. 2018. “Is Human Classification by Experienced Untrained Observers a Gold Standard in Fixation Detection?” Behavior Research Methods 50(5):1864–81.

22.

Horn

2018. “Can the Online Crowd Match Real Expert Judgments? How Task Complexity and Coder Location Affect the Validity of Crowd-coded Data.” European Journal of Political Research 58(1):236–47.

23.

Hruschka

D. J.

Schwartz

John

D. C.

Picone-Decaro

Jekins

R. A.

Carey

J. W.

. 2004. “Reliability in Coding Open-ended Data: Lessons Learned from HIV Behavioral Research.” Field Methods 16(3):307–31.

24.

Jones

P. J.

Mair

Mcnally

R. J.

. 2018. “Visualizing Psychological Networks: A Tutorial in R.” Frontiers in Psychology 9:1–12. doi:10.3389/fpsyg.2018.01742.

25.

King

2003. “The Future of Replication.” International Studies Perspectives 4:443–99.

26.

Kolaczyk

E. D.

Csárdi

. 2014. Statistical Analysis of Network Data with R. New York: Springer.

27.

Langfelder

Horvath

. 2008. “WGCNA: An R Package for Weighted Correlation Network Analysis.” BMC Bioinformatics 9:599.

28.

Larsson

Nyström

Andersson

Stridh

. 2015. “Detection of Fixations and Smooth Pursuit Movements in High-speed Eye-tracking Data.” Biomedical Signal Processing and Control 18:145–52.

29.

Lamprianou

2018. “Investigation of Rater Effects Using Social Network Analysis and Exponential Random Graph Models.” Educational and Psychological Measurement 78(3):430–59.

30.

Lamprianou

Symeou

Theodorou

2019. “All We Need is Love (and Money)! What do Higher Education Students Want from their Families?” Research Papers in Education 34(3):352–72.

31.

Lehmann

Zobel

. 2018. “Positions and Saliency of Immigration in Party Manifestos: A Novel Dataset Using Crowd Coding.” European Journal of Political Research 57(4):1056–83.

32.

Leiva

F. M.

Rios

F. J. M.

Martinez

T. L.

. 2006. “Assessment of Interjudge Reliability in the Open-ended Questions Coding Process.” Quality & Quantity 40:519–37.

33.

Cornelius

S. P.

Liu

Y.-Y.

Wang

Barabási

A.-L.

. 2017. “The Fundamental Advantages of Temporal Networks.” Science 1042:1042–46. doi:10.1126/science.aai7488.

34.

Lind

Gruber

Boomgaarden

H. G.

. 2017. “Content Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent Constructs.” Communication Methods and Measures 11(3):191–209.

35.

Lusher

Koskinen

Robins

. 2013. Exponential Random Graph Models for Social Networks. Cambridge, England: Cambridge University Press.

36.

Marquardt

Pemstein

Petrarca

Seim

Wilson

Bernhard

Coppedge

Lindberg

. 2017. “Experts, Coders and Crowds: An Analysis of Substitutability.” VDem Working Paper No. 53. https://ssrn.com/abstract=3046462 or http://dx.doi.org/10.2139/ssrn.3046462.

37.

Mikhaylov

Laver

Benoit

K. R.

. 2011. “Coder Reliability and Misclassification in the Human Coding of Party Manifestos.” Political Analysis 20(1):78–91.

38.

Mizunoa

Takayasub

Takayasua

. 2006. “Correlation Networks among Currencies.” Physica A: Statistical Mechanics and Its Applications 364:S336–42.

39.

Morris

Peter A.

1977. “Combining Expert Judgments: A Bayesian Approach.” Management Science 23(7):679–93.

40.

Mozetič

Grčar

Smailović

. 2016. “Multilingual Twitter Sentiment Classification: The Role of Human Annotators.” PLoS One 11(5):e0155036. doi:10.1371/journal.pone.0155036.

41.

Nelson

L. K.

Burk

Knudsen

McCall

. 2018. “The Future of Coding: A Comparison of Hand-coding and Three Types of Computer-assisted Text Analysis Methods.” Sociological Methods & Research. https://doi.org/10.1177/0049124118769114.

42.

Oleinik

Popova

Kirdina

Shatalova

. 2014. “On the Choice of Measures of Reliability and Validity in the Content-analysis of Texts.” Quality and Quantity 48:2703–18.

43.

Padilla

L. M.

Creem-Regehr

S. H.

Hegarty

Stefanucci

J. K.

. 2018. “Decision Making with Visualizations: A Cognitive Framework across Disciplines.” Cognitive Research: Principles and Implications 3:29.

44.

Robins

Pattison

Kalish

Lusher

. 2007. “An Introduction to Exponential Random Graph (p*) Models for Social Networks.” Social Networks 29:173–91.

45.

Scott

1988. “Social Network Analysis.” Sociology 22(1):109–27.

46.

Sgammato

Donoghue

J. R.

. 2018. “On the Performance of the Marginal Homogeneity Test to Detect Rater Drift.” Applied Psychological Measurement 42(4):307–20.

47.

Shrout

P. E.

Fleiss

J. L.

. 1979. “Intraclass Correlations: Uses in Assessing Rater Reliability.” Psychological Bulletin 86:420–28.

48.

Snijders

T. A. B.

van de Bunt

G. G.

Steglich

C. E. G.

. 2010. “Introduction to Stochastic Actor-based Models for Network Dynamics.” Social Networks 32(1):44–60.

49.

Trampe

Quoidbach

Taquet

. 2015. “Emotions in Everyday Life.” PLoS One 10(12). doi:10.1371/journal.pone.0145450.

50.

van den Hoonaard

2008. “Inter- and Intracoder Reliability.” Pp. 446–47 in The Sage Encyclopedia of Qualitative Research Methods, edited by Given

. Thousand Oaks, CA: Sage Publications. doi:10.4135/9781412963909.n223

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB