Human coding remains an important part of the data-generating process for many political scientists. Yet, we lack a systematic understanding of how researchers approach and describe the human coding process. I analyze published articles in major political science journals from 2010 to 2024 that mention human coders (N = 258). While articles largely state some form of intercoder reliability measure, a substantial percentage of articles lack minimally descriptive information on coder qualifications and replicable coding procedures—components that, respectively, are a best practice and are important for ensuring research transparency. The results suggest that some researchers emphasize the product of human coding without fully addressing how human coding is used as a process. I conclude with suggestions for better describing human coders’ work.
AguinisHSolarinoAM (2019) Transparency and Replicability in Qualitative Research: The Case of Interviews with Elite Informants. Strategic Management Journal40 (8): 1291–1315.
2.
AhnSAmesAJMyersND (2012) A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses. Review of Educational Research82 (4): 436–476.
3.
AlvarezRMKeyEMNúñezL (2018) Research Replication: Practical Considerations. PS: Political Science & Politics51 (2): 422–426.
4.
Anani SarabMRAmini FarsaniM (2024) Second-Order Synthesis of Meta-Analytic Studies in Applied Linguistics (1998–2021). Quality & Quantity58 (2): 1517–1543.
Arel-BundockVBriggsRCDoucouliagosH, et al. (2024) Quantitative Political Science Research Is Greatly Underpowered. The Journal of Politics86: 765–778.
7.
BagozziBEBrandtPTFreemanJR, et al. (2019) The Prevalence and Severity of Underreporting Bias in Machine- and Human-Coded Data. Political Science Research and Methods7 (3): 641–649.
8.
BenoitKConwayDLauderdaleBE, et al. (2016) Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review110 (2): 278–295.
9.
BudakCGarrettRKSudeD (2021) Better Crowdcoding: Strategies for Promoting Accuracy in Crowdsourced Content Analysis. Communication Methods and Measures15 (2): 141–155.
10.
CarlsonDMontgomeryJM (2017) A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political Texts. American Political Science Review111 (4): 835–843.
DeaneKStevanoS (2016) Towards a Political Economy of the Use of Research Assistants: Reflections from Fieldwork in Tanzania and Mozambique. Qualitative Research16 (2): 213–228.
13.
DeBellM (2013) Harder Than It Looks: Coding Political Knowledge on the ANES. Political Analysis21 (4): 393–406.
14.
DumendenI (2012) The Research Assistant: Invisible and Silenced, Exploited and Disposable. In: VicarsMMcKennaTWhiteJ (eds) Discourse, Power, and Resistance Down Under. Rotterdam: Sense, pp.9–20.
15.
DyrstadKMosesJW (2023) Big Data Meets Open Political Science: An Empirical Assessment of Transparency Standards 2008–2019. European Political Science22 (2): 182–201.
16.
EdgellABLachapelleJMaerzSF (2025) Achieving Transparency, Traceability, and Readability with Human-Coded Data. PS: Political Science & Politics58 (2): 346–351.
17.
ElmanCKapiszewskiD (2014) Data Access and Research Transparency in the Qualitative Tradition. PS: Political Science & Politics47 (1): 43–47.
18.
FreeseJPetersonD (2017) Replication in Social Science. Annual Review of Sociology43: 147–165.
GoehringB (2024) Improving Content Analysis: Tools for Working with Undergraduate Research Assistants. PS: Political Science & Politics57 (1): 57–63.
21.
GoldenMA (1995) Replication and Non-Quantitative Research. PS: Political Science & Politics28 (3): 481–483.
22.
GrimmerJRobertsMEStewartBM (2021) Machine Learning for Social Science: An Agnostic Approach. Annual Review of Political Science24 (1): 395–419.
23.
HakTBerntsT (1996) Coder Training: Theoretical Training or Practical Socialization?Qualitative Sociology19 (2): 235–257.
24.
HayesAFKrippendorffK (2007) Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures1 (1): 77–89.
25.
HeseltineMClemm Von HohenbergB (2024) Large Language Models as a Substitute for Human Experts in Annotating Political Text. Research & Politics11 (1): 20531680241236239.
26.
HjorthFKlemmensenRHoboltS, et al. (2015) Computers, Coders, and Voters: Comparing Automated Methods for Estimating Party Positions. Research & Politics2 (2): 205316801558047.
27.
HornA (2019) Can the Online Crowd Match Real Expert Judgments? How Task Complexity and Coder Location Affect the Validity of Crowd-coded Data. European Journal of Political Research58 (1): 236–247.
28.
KaufmanAR (2024) Selecting More Informative Training Sets with Fewer Observations. Political Analysis32 (1): 133–139.
29.
KeyEM (2016) How Are We Doing? Data Access and Replication in Political Science. PS: Political Science & Politics49 (2): 268–272.
30.
KingG (2007) An Introduction to the Dataverse Network as an Infrastructure for Data Sharing. Sociological Methods & Research36 (2): 173–199.
31.
KnutsenCHMarquardtKLSeimB, et al. (2024) Conceptual and Measurement Issues in Assessing Democratic Backsliding. PS: Political Science & Politics57 (2): 162–177.
32.
KrippendorffK (2018) Content Analysis: An Introduction to Its Methodology, 4th edn. Los Angeles, CA: Sage.
33.
LacySWatsonBRRiffeD, et al. (2015) Issues and Best Practices in Content Analysis. Journalism & Mass Communication Quarterly92 (4): 791–811.
34.
LevickLOlavarria-GambiM (2020) Hindsight Bias in Expert Surveys: How Democratic Crises Influence Retrospective Evaluations. Politics40 (4): 494–509.
35.
LindstädtRProkschSOSlapinJB (2020) When Experts Disagree: Response Aggregation and Its Consequences in Expert Surveys. Political Science Research and Methods8 (3): 580–588.
36.
LovejoyJWatsonBRLacyS, et al. (2016) Three Decades of Reliability in Communication Content Analyses: Reporting of Reliability Statistics and Coefficient Levels in Three Top Journals. Journalism & Mass Communication Quarterly93 (4): 1135–1159.
37.
MartínezI Coma FVan HamC (2015) Can Experts Judge Elections? Testing the Validity of Expert Judgments for Measuring Election Integrity. European Journal of Political Research54 (2): 305–325.
38.
MarquardtKLPemsteinDSeimB, et al. (2019) What Makes Experts Reliable? Expert Reliability and the Estimation of Latent Traits. Research & Politics6 (4): 205316801987956.
39.
McDermottR (2010) Data Collection and Collaboration: Editor’s Introduction. PS: Political Science & Politics43 (1): 15–16.
40.
McDermottR (2022) Breaking Free: How Preregistration Hurts Scholars and Science. Politics and the Life Sciences41 (1): 55–59.
41.
McMannKPemsteinDSeimB, et al. (2022) Assessing Data Quality: An Approach and An Application. Political Analysis30 (3): 426–449.
42.
MikhaylovSLaverMBenoitKR (2012) Coder Reliability and Misclassification in the Human Coding of Party Manifestos. Political Analysis20 (1): 78–91.
43.
MonroeKR (ed.) (2005) Perestroika! The Raucous Rebellion in Political Science. New Haven, CT: Yale University Press.
44.
Niemann-LenzJDittrichAScheperJ (2023) Coding Quality in Manual Content Analysis: An Exploration of Coder Characteristics and Category Types for Crowdworkers and Student Coders. Studies in Communication and Media12 (4): 327–353.
45.
NosekBMummeryCScarabelliL, et al. (2025) Reproducibility and Transparency: What’s Going on and How Can We Help. Nature Communications16 (1): 1082.
46.
O’BrienTLHawkinsSLLoeschA (2022) Scientific Disciplines and the Admissibility of Expert Evidence in Courts. Socius: Sociological Research for a Dynamic World8: 237802312211080.
47.
O’BrochtaWParikhS (2021) Anomalous Responses on Amazon Mechanical Turk: An Indian Perspective. Research & Politics8 (2): 1–4.
48.
OfosuGKPosnerDN (2023) Pre-Analysis Plans: An Early Stocktaking. Perspectives on Politics21 (1): 174–190.
49.
ParitoshP (2012) Human Computation Must Be Reproducible. CrowdSearch Workshop, Lyon, 17 April.
50.
PickelSStarkTBreustedtW (2015) Assessing the Quality of Quality Measures of Democracy: A Theoretical Framework and Its Empirical Application. European Political Science14 (4): 496–520.
51.
ReiterD (2020) Measurement Replication in Qualitative and Quantitative Studies. In: ElmanCGerringJMahoneyJ (eds) The Production of Knowledge: Enhancing Progress in Social Science. Cambridge: Cambridge University Press, pp.284–300.
52.
RubensonD (2021) Tie My Hands Loosely: Pre-Analysis Plans in Political Science. Politics and the Life Sciences40 (2): 142–151.
53.
SchedlerA (2012) Judgment and Measurement in Political Science. Perspectives on Politics10 (1): 21–36.
54.
SongHTolochkoPEberlJM, et al. (2020) In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis. Political Communication37 (4): 550–572.
55.
SteltenpohlCNLustickHMeyerMS, et al. (2023) Rethinking Transparency and Rigor from a Qualitative Open Science Perspective. Journal of Trial and Error4 (1): 47–59.
56.
StockemerDKoehlerSLentzT (2018) Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature. PS: Political Science & Politics51 (4): 799–803.
57.
TrippAMDionML (2023) American Political Science Review Editorial Report: Executive Summary (Fall 2022). Political Science Today3 (1): 24–28.
58.
WinterNJGHughesAGSandersLM (2020) Online Coders, Open Codebooks: New Opportunities for Content Analysis of Political Communication. Political Science Research and Methods8 (4): 731–746.
59.
YingLMontgomeryJMStewartBM (2022) Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures. Political Analysis30 (4): 570–589.
60.
ZamithRLewisSC (2015) Content Analysis and the Algorithmic Coder: What Computational Social Science Means for Traditional Modes of Media Analysis. The Annals of the American Academy of Political and Social Science659 (1): 307–318.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
