Accurate,Fast,But Not Always Cheap: Evaluating “Crowdcoding” as an Alternative Approach to Analyze Social Media Data

Abstract

Crowdcoding, a method that outsources “coding” tasks to numerous people on the internet, has emerged as a popular approach for annotating texts and visuals. However, the performance of this approach for analyzing social media data in the context of journalism and mass communication research has not been systematically assessed. This study evaluated the validity and efficiency of crowdcoding based on the analysis of 4,000 tweets about the 2016 U.S. presidential election. The results show that compared with the traditional quantitative content analysis, crowdcoding yielded comparably valid results and was superior in efficiency, but was more expensive under most circumstances.

Keywords

content analysis crowdcoding crowdsourcing Twitter sentiment analysis

Get full access to this article

View all access options for this article.

References

Alonso

Marshall

Najork

(2014). Crowdsourcing a subjective labeling task: A human-centered framework to ensure reliable results (Microsoft Research, MSR-TR-2014–91). https://www.microsoft.com/en-us/research/publication/crowdsourcing-a-subjective-labeling-task-a-human-centered-framework-to-ensure-reliable-results/

Benoit

Conway

Lauderdale

B. E.

Laver

Mikhaylov

(2016). Crowd-sourced text analysis: Reproducible and agile production of political data. American Political Science Review, 110(2), 278–295.

Berg

(2016). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy Journal, 37(3), 543–576.

Beyer

M. A.

Laney

(2012). The importance of “big data”: A definition. Gartner.

Collingwood

Wilkerson

(2012). Tradeoffs in accuracy and efficiency in supervised learning methods. Journal of Information Technology & Politics, 9(3), 298–318.

Drapeau

Chilton

L. B.

Bragg

Weld

D. S.

(2016, September). Microtalk: Using argumentation to improve crowdsourcing accuracy [Paper presentation]. Fourth AAAI Conference on Human Computation and Crowdsourcing, Austin, TX, United States.

Finin

Murnane

Karandikar

Keller

Martineau

Dredze

, (2010, June). Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 80-88). Association for Computational Linguistics.

Fromreide

Hovy

Søgaard

, (2014, May). Crowdsourcing and annotating NER for Twitter# drift. In LREC proceedings (pp. 2544-2547).

Guo

Vargo

Pan

Ding

Ishwar

(2016). Big social data analytics in journalism and mass communication: Comparing dictionary-based text analysis and unsupervised topic modeling. Journalism & Mass Communication Quarterly, 93(2), 332–359.

10.

Gurari

Stangl

A. J.

Guo

Lin

Grauman

Luo

Bigham

J. P.

(2018, February). Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3608–3617). IEEE.

11.

Gwet

K. L.

(2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.

12.

Hara

Froehlich

(2013, April). Combining crowdsourcing and Google Street view to identify streetlevel accessibility problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13) (pp. 631–640). IEEE.

13.

Haselmayer

Jenny

(2017). Sentiment analysis of political communication: Combining a dictionary approach with crowdcoding. Quality & Quantity, 51(6), 2623–2646.

14.

Howe

(2008). Crowdsourcing: How the power of the crowd is driving the future of business. Crown.

15.

Hung

N. Q. V.

Tam

N. T.

Tran

L. N.

Aberer

(2013). An evaluation of aggregation techniques in crowdsourcing. In Lin

Manolopoulos

Srivastava

Huang

(Eds.), Web information systems engineering—WISE (pp. 1–15). Springer.

16.

Irshad

Montaser-Kouhsari

Waltz

Bucur

Nowak

J. A.

Dong

Beck

A. H.

(2015). Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: Evaluating experts, automated methods, and the crowd. Pacific Symposium on Biocomputing, 2015, 294–305.

17.

Jungherr

Theocharis

(2017). The empiricist’s challenge: Asking meaningful questions in political science in the age of big data. Journal of Information Technology & Politics, 14(2), 97–109.

18.

Kim

A. E.

Hansen

H. M.

Murphy

Richards

A. K.

Duke

Allen

J. A.

(2013). Methodological considerations in analyzing Twitter data. Journal of the National Cancer Institute Monographs, 2013(47), 140–146.

19.

Krippendorff

(2004). Content analysis: An introduction to its methodology. SAGE.

20.

Lacy

Watson

B. R.

Riffe

Lovejoy

(2015). Issues and best practices in content analysis. Journalism & Mass Communication Quarterly, 92(4), 791–811.

21.

Lind

Gruber

Boomgaarden

H. G.

(2017). Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication Methods and Measures, 11(3), 191–209.

22.

Lovejoy

Watson

B. R.

Lacy

Riffe

(2016). Three decades of reliability in communication content analyses reporting of reliability statistics and coefficient levels in three top journals. Journalism & Mass Communication Quarterly, 93(4), 1135–1159.

23.

Mellebeek

Benavent

Grivolla

Codina

Costa-Jussa

M. R.

Banchs

(2010, June). Opinion mining of Spanish customer comments with non-expert annotations on Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk (pp. 114–121). Association for Computational Linguistics.

24.

Neuendorf

K. A.

(2017). The content analysis guidebook. SAGE.

25.

Potter

W. J.

Levine-Donnerstein

(1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27(3), 258–284.

26.

Riffe

Lacy

Fico

(2014). Analyzing media messages: Using quantitative content analysis in research. Routledge.

27.

Ross

Irani

Silberman

M. S.

Zaldivar

Tomlinson

(2010, April). Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. In CHI’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). Association for Computing Machinery.

28.

Sameki

Gentil

Mays

K. K.

Guo

Betke

(2016, August). Dynamic allocation of crowd contributions for sentiment analysis during the 2016 US presidential election [Paper presentation]. Association for the Advancement of Artificial Intelligence, Austin, TX, United States.

29.

Sameki

Lai

Mays

K. K.

Guo

Ishwar

Betke

(2019). BUOCA: Budget-optimized crowd worker allocation. https://arxiv.org/abs/1901.06237

30.

Silberman

M. S.

Tomlinson

LaPlante

Ross

Irani

Zaldivar

(2018). Responsible research with crowds: Pay crowdworkers at least minimum wage. Communications of the ACM, 61(3), 39–41.

31.

Snow

O’Connor

Jurafsky

A. Y.

(2008, October). Cheap and fast—But is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254–263). Association for Computing Machinery.

32.

Sorokin

Forsyth

(2008, June). Utility data annotation with Amazon Mechanical Turk. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (pp. 1–8). IEEE.

33.

Thelwall

Buckley

Paltoglou

(2012). Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, 63(1), 163–173.

34.

Vargas

McCreadie

MacDonlad

Ounis

(2016, March). Comparing overall and targeted sentiments in social media during crises. In Proceedings of Tenth International AAAI Conference on Web and Social Media (pp.695-698). Association for the Advancement of Artificial Intelligence.

35.

Zhai

Lingren

Deleger

Kaiser

Stoutenborough

Solti

(2013). Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of Medical Internet Research, 15(4), Article e73.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.18 MB