This study investigates inter- and intracoder reliability, proposing a new approach based on social network analysis (SNA) and exponential random graph models (ERGM). During a recent exit poll, the responses of voters to two open-ended questions were recorded. A coding experiment was conducted where a group of coders coded a sample of text segments. Analyzing the data, we show that the proposed SNA/ERGM method extends significantly our analytical leverage, beyond what popular tools such as Krippendorff’s α and Fleiss’s κ have to offer. The reliability of coding for individual coders differed significantly for the two questions although they were very similar and the same codebook was used. We conclude that the main advantages of the proposed SNA/ERGM method are the intuitive visualizations and the nuanced measurements. Detailed guidelines are provided for practitioners who would like to use the proposed method in operational settings.
AlmquistZ. W.BagozziB. E.. 2017. “Using Radical Environmentalist Texts to Uncover Network Structure and Network Features.” Sociological Methods & Research. doi:10.1177/0049124117729696.
2.
BairdJ. A.MeadowsM.LeckieG.CaroD.. 2017. “Rater Accuracy and Training Group Effects in Expert- and Supervisor-based Monitoring Systems, Assessment in Education.” Principles, Policy & Practice24(1):44–59.
3.
BaisFrankSchoutenBarryLugtigPeterToepoelVeraArends-TòthJuditDouhouSalimaKierujNataliaMorrenMattijnVisCorrie. 2019. “Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably? A Case Study on 11 Dutch General Population Surveys.” Sociological Methods & Research48(2):263–95.
4.
BenoitKennethConwayDrewBenjamin LauderdaleE.LaverMichaelMikhaylovSlava. 2016. “Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review110(2):278–95.
5.
BenoitKennethMungerK.SpirlingA.. 2019. “Measuring and Explaining Political Sophistication through Textual Complexity.” American Journal of Political Science63(2):491–508.
6.
BerryJ. K.MielkeP. W.. 1988. “A Generalization of Cohen’s Kappa Agreement Measure to Interval Measurement and Multiple Raters.” Educational and Psychological Measurement48(4):921–33.
7.
BlockP.2015. “Reciprocity, Transitivity, and the Mysterious Three-cycle.” Social Networks40:163–73. doi:10.1016/j.socnet.2014.10.005.
CampbellJ. L.QuincyC.OssermanJ.PedersenO. K.. 2013. “Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement.” Sociological Methods & Research42(3):294–320. http://smr.sagepub.com/content/42/3/294.abstract?etoc.
10.
ComptonD.LoveT. P.SellJ.. 2012. “Developing and Assessing Intercoder Reliability in Studies of Group Interaction.” Sociological Methodology42(1):348–64.
11.
CrittendenK. S.HillR. J.1971. “Coding Reliability and Validity of Interview Data.” American Sociological Review36:1073–80.
12.
Ennser-JedenastikL.MeyerJ.. n.d. “The Impact of Party Cues on Manual Coding of Political Texts.” Political Science Research and Methods, 1–9. doi:10.1017/psrm.2017.29.
FleissJoseph L.1971. “Measuring Nominal Scale Agreement among Many Raters.” Psychological Bulletin76:378–82.
15.
FriedkinN. E.1982. “Information Flow through Stron and Weak Ties in Intraorganizational Social Networks. Social Networks3:273–85.
16.
GirardJ. M.CohnJ. F.. 2016. “A Primer on Observational Measurement.” Assessment23(4):404–13. doi:10.1177/1073191116635807.
17.
GrimmerJStewartB. M.. 2011. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis21(3):267–97.
18.
GummerT.BlumenbergM. S.RoßmannmJ.. 2019. Learning Effects in Coders and Their Implications for Managing Content Analyses. International Journal of Social Research Methodology22(2):139–52.
19.
HaselmayerM.JennyM.. 2017. “Sentiment Analysis of Political Communication: Combining a Dictionary Approach with Crowdcoding.” Quality & Quantity51:2623–46.
20.
HayesA. F.KrippendorffK.. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measurement1(1):77–89.
21.
HoogeI. T. C.NiehorsterD. C.NyströmM.AnderssonR.HesselsR. S.. 2018. “Is Human Classification by Experienced Untrained Observers a Gold Standard in Fixation Detection?” Behavior Research Methods50(5):1864–81.
22.
HornA.2018. “Can the Online Crowd Match Real Expert Judgments? How Task Complexity and Coder Location Affect the Validity of Crowd-coded Data.” European Journal of Political Research58(1):236–47.
23.
HruschkaD. J.SchwartzD.JohnD. C.Picone-DecaroE.JekinsR. A.CareyJ. W.. 2004. “Reliability in Coding Open-ended Data: Lessons Learned from HIV Behavioral Research.” Field Methods16(3):307–31.
24.
JonesP. J.MairP.McnallyR. J.. 2018. “Visualizing Psychological Networks: A Tutorial in R.” Frontiers in Psychology9:1–12. doi:10.3389/fpsyg.2018.01742.
25.
KingG.2003. “The Future of Replication.” International Studies Perspectives4:443–99.
26.
KolaczykE. D.CsárdiG.. 2014. Statistical Analysis of Network Data with R. New York: Springer.
27.
LangfelderP.HorvathS.. 2008. “WGCNA: An R Package for Weighted Correlation Network Analysis.” BMC Bioinformatics9:599.
28.
LarssonL.NyströmM.AnderssonR.StridhM.. 2015. “Detection of Fixations and Smooth Pursuit Movements in High-speed Eye-tracking Data.” Biomedical Signal Processing and Control18:145–52.
29.
LamprianouI.2018. “Investigation of Rater Effects Using Social Network Analysis and Exponential Random Graph Models.” Educational and Psychological Measurement78(3):430–59.
30.
LamprianouI.SymeouL.TheodorouE.2019. “All We Need is Love (and Money)! What do Higher Education Students Want from their Families?” Research Papers in Education34(3):352–72.
31.
LehmannP.ZobelM.. 2018. “Positions and Saliency of Immigration in Party Manifestos: A Novel Dataset Using Crowd Coding.” European Journal of Political Research57(4):1056–83.
32.
LeivaF. M.RiosF. J. M.MartinezT. L.. 2006. “Assessment of Interjudge Reliability in the Open-ended Questions Coding Process.” Quality & Quantity40:519–37.
33.
LiA.CorneliusS. P.LiuY.-Y.WangL.BarabásiA.-L.. 2017. “The Fundamental Advantages of Temporal Networks.” Science1042:1042–46. doi:10.1126/science.aai7488.
34.
LindF.GruberM.BoomgaardenH. G.. 2017. “Content Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent Constructs.” Communication Methods and Measures11(3):191–209.
35.
LusherD.KoskinenJ.RobinsG.. 2013. Exponential Random Graph Models for Social Networks. Cambridge, England: Cambridge University Press.
MikhaylovS.LaverM.BenoitK. R.. 2011. “Coder Reliability and Misclassification in the Human Coding of Party Manifestos.” Political Analysis20(1):78–91.
38.
MizunoaT.TakayasubH.TakayasuaM.. 2006. “Correlation Networks among Currencies.” Physica A: Statistical Mechanics and Its Applications364:S336–42.
39.
MorrisPeter A.1977. “Combining Expert Judgments: A Bayesian Approach.” Management Science23(7):679–93.
40.
MozetičI.GrčarM.SmailovićJ.. 2016. “Multilingual Twitter Sentiment Classification: The Role of Human Annotators.” PLoS One11(5):e0155036. doi:10.1371/journal.pone.0155036.
41.
NelsonL. K.BurkD.KnudsenM.McCallL.. 2018. “The Future of Coding: A Comparison of Hand-coding and Three Types of Computer-assisted Text Analysis Methods.” Sociological Methods & Research. https://doi.org/10.1177/0049124118769114.
42.
OleinikA.PopovaI.KirdinaS.ShatalovaT.. 2014. “On the Choice of Measures of Reliability and Validity in the Content-analysis of Texts.” Quality and Quantity48:2703–18.
43.
PadillaL. M.Creem-RegehrS. H.HegartyM.StefanucciJ. K.. 2018. “Decision Making with Visualizations: A Cognitive Framework across Disciplines.” Cognitive Research: Principles and Implications3:29.
44.
RobinsG.PattisonP.KalishY.LusherD.. 2007. “An Introduction to Exponential Random Graph (p*) Models for Social Networks.” Social Networks29:173–91.
SgammatoA.DonoghueJ. R.. 2018. “On the Performance of the Marginal Homogeneity Test to Detect Rater Drift.” Applied Psychological Measurement42(4):307–20.
SnijdersT. A. B.van de BuntG. G.SteglichC. E. G.. 2010. “Introduction to Stochastic Actor-based Models for Network Dynamics.” Social Networks32(1):44–60.
49.
TrampeD.QuoidbachJ.TaquetM.. 2015. “Emotions in Everyday Life.” PLoS One10(12). doi:10.1371/journal.pone.0145450.
50.
van den HoonaardW.2008. “Inter- and Intracoder Reliability.” Pp. 446–47in The Sage Encyclopedia of Qualitative Research Methods, edited by GivenL.. Thousand Oaks, CA: Sage Publications. doi:10.4135/9781412963909.n223
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
