Crowdcoding, a method that outsources “coding” tasks to numerous people on the internet, has emerged as a popular approach for annotating texts and visuals. However, the performance of this approach for analyzing social media data in the context of journalism and mass communication research has not been systematically assessed. This study evaluated the validity and efficiency of crowdcoding based on the analysis of 4,000 tweets about the 2016 U.S. presidential election. The results show that compared with the traditional quantitative content analysis, crowdcoding yielded comparably valid results and was superior in efficiency, but was more expensive under most circumstances.
BenoitK.ConwayD.LauderdaleB. E.LaverM.MikhaylovS. (2016). Crowd-sourced text analysis: Reproducible and agile production of political data. American Political Science Review, 110(2), 278–295.
3.
BergJ. (2016). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy Journal, 37(3), 543–576.
4.
BeyerM. A.LaneyD. (2012). The importance of “big data”: A definition. Gartner.
5.
CollingwoodL.WilkersonJ. (2012). Tradeoffs in accuracy and efficiency in supervised learning methods. Journal of Information Technology & Politics, 9(3), 298–318.
6.
DrapeauR.ChiltonL. B.BraggJ.WeldD. S. (2016, September). Microtalk: Using argumentation to improve crowdsourcing accuracy [Paper presentation]. Fourth AAAI Conference on Human Computation and Crowdsourcing, Austin, TX, United States.
7.
FininT.MurnaneW.KarandikarA.KellerN.MartineauJ.DredzeM., (2010, June). Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 80-88). Association for Computational Linguistics.
8.
FromreideH.HovyD.SøgaardA., (2014, May). Crowdsourcing and annotating NER for Twitter# drift. In LREC proceedings (pp. 2544-2547).
9.
GuoL.VargoC.PanZ.DingW.IshwarP. (2016). Big social data analytics in journalism and mass communication: Comparing dictionary-based text analysis and unsupervised topic modeling. Journalism & Mass Communication Quarterly, 93(2), 332–359.
10.
GurariD.LiQ.StanglA. J.GuoA.LinC.GraumanK.LuoJ.BighamJ. P. (2018, February). Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3608–3617). IEEE.
11.
GwetK. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.
12.
HaraK.LeV.FroehlichJ. (2013, April). Combining crowdsourcing and Google Street view to identify streetlevel accessibility problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13) (pp. 631–640). IEEE.
13.
HaselmayerM.JennyM. (2017). Sentiment analysis of political communication: Combining a dictionary approach with crowdcoding. Quality & Quantity, 51(6), 2623–2646.
14.
HoweJ. (2008). Crowdsourcing: How the power of the crowd is driving the future of business. Crown.
15.
HungN. Q. V.TamN. T.TranL. N.AbererK. (2013). An evaluation of aggregation techniques in crowdsourcing. In LinX.ManolopoulosY.SrivastavaD.HuangG. (Eds.), Web information systems engineering—WISE (pp. 1–15). Springer.
16.
IrshadH.Montaser-KouhsariL.WaltzG.BucurO.NowakJ. A.DongF.BeckA. H. (2015). Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: Evaluating experts, automated methods, and the crowd. Pacific Symposium on Biocomputing, 2015, 294–305.
17.
JungherrA.TheocharisY. (2017). The empiricist’s challenge: Asking meaningful questions in political science in the age of big data. Journal of Information Technology & Politics, 14(2), 97–109.
18.
KimA. E.HansenH. M.MurphyJ.RichardsA. K.DukeJ.AllenJ. A. (2013). Methodological considerations in analyzing Twitter data. Journal of the National Cancer Institute Monographs, 2013(47), 140–146.
19.
KrippendorffK. (2004). Content analysis: An introduction to its methodology. SAGE.
20.
LacyS.WatsonB. R.RiffeD.LovejoyJ. (2015). Issues and best practices in content analysis. Journalism & Mass Communication Quarterly, 92(4), 791–811.
21.
LindF.GruberM.BoomgaardenH. G. (2017). Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures, 11(3), 191–209.
22.
LovejoyJ.WatsonB. R.LacyS.RiffeD. (2016). Three decades of reliability in communication content analyses reporting of reliability statistics and coefficient levels in three top journals. Journalism & Mass Communication Quarterly, 93(4), 1135–1159.
23.
MellebeekB.BenaventF.GrivollaJ.CodinaJ.Costa-JussaM. R.BanchsR. (2010, June). Opinion mining of Spanish customer comments with non-expert annotations on Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 114–121). Association for Computational Linguistics.
24.
NeuendorfK. A. (2017). The content analysis guidebook. SAGE.
25.
PotterW. J.Levine-DonnersteinD. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27(3), 258–284.
26.
RiffeD.LacyS.FicoF. (2014). Analyzing media messages: Using quantitative content analysis in research. Routledge.
27.
RossJ.IraniL.SilbermanM. S.ZaldivarA.TomlinsonB. (2010, April). Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. In CHI’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). Association for Computing Machinery.
28.
SamekiM.GentilM.MaysK. K.GuoL.BetkeM. (2016, August). Dynamic allocation of crowd contributions for sentiment analysis during the 2016 US presidential election [Paper presentation]. Association for the Advancement of Artificial Intelligence, Austin, TX, United States.
SilbermanM. S.TomlinsonB.LaPlanteR.RossJ.IraniL.ZaldivarA. (2018). Responsible research with crowds: Pay crowdworkers at least minimum wage. Communications of the ACM, 61(3), 39–41.
31.
SnowR.O’ConnorB.JurafskyD.NgA. Y. (2008, October). Cheap and fast—But is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254–263). Association for Computing Machinery.
32.
SorokinA.ForsythD. (2008, June). Utility data annotation with Amazon Mechanical Turk. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (pp. 1–8). IEEE.
33.
ThelwallM.BuckleyK.PaltoglouG. (2012). Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, 63(1), 163–173.
34.
VargasS.McCreadieR.MacDonladC.OunisI. (2016, March). Comparing overall and targeted sentiments in social media during crises. In Proceedings of Tenth International AAAI Conference on Web and Social Media (pp.695-698). Association for the Advancement of Artificial Intelligence.
35.
ZhaiH.LingrenT.DelegerL.LiQ.KaiserM.StoutenboroughL.SoltiI. (2013). Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of Medical Internet Research, 15(4), Article e73.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
