Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System

Abstract

Crowdsourcing recently became a popular approach to substitute time consuming and expensive human subject studies, but its application is generally limited to simple and short-term experimental tasks, such as testing visual perception. The goal of this study is to test if crowdsourcing is applicable to a more complicated user study. Thus, we replicated a controlled lab study of decision-making tasks with different sorting techniques using crowdsourcing. A total of 98 participants were recruited via the Amazon Mechanical Turk service, and they participated in the study remotely through web interfaces. Experiment results indicate that performance measures of our crowdsourcing experiment was not exactly equivalent to lab experiments. However, we found potential sources of problems that we can improve to make the crowdsourcing experiment more viable.

Get full access to this article

View all access options for this article.

References

Akkaya

Conrad

Wiebe

Mihalcea

(2010). Amazon mechanical turk for subjectivity word sense disambiguation. In Proceedings of the NAACL HLT 2010. Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles. 195–203.

Bloodgood

Callison-Burch

(2010). Using mechanical turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles. 208–211.

Caplin

Dean

Martin

(2009). Search and Satisficing! Working Paper.

Heer

Bostock

(2010). Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Proceedings of the 28th international conference on Human factors in computing systems. 203–212.

Hur

(2009). SimulSort: Multivariate Data Exploration through an Enhanced Sorting Technique. Presented at the Human-Computer Interaction International. 684-693.

Hur

Kim

Savikhin

(n.d.) A Comparative Study of Three Sorting Techniques in Performing Cognitive Tasks on a Tabular Representation. Submitted to International Journal of Human-Computer Interaction.

Kittur

Chi

E. H.

Suh

(2008). Crowdsourcing user studies with Mechanical Turk. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. 453-456.

Kosara

Ziemkiewicz

(2010). Do Mechanical Turks Dream of Square Pie Charts? In BELIV ’10: Beyond Time and Errors: Novel Evaluation Methods for Information Visualization. ACM. 373–382.

Marge

Banerjee

Rudnicky

(2010). Using the Amazon Mechanical Turk for transcription of spoken language. ICASSP, March.

10.

Mason

W. A.

Watts

D. J.

(2009). Financial incentives and the “performance of crowds.” Proceedings of the Human Computation Workshop. Paris: ACM.

11.

Paolacci

Chandler

Ipeirotis

(2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5).

12.

Reips

(2002). Standards for Internet-Based Experimenting. Experimental Psychology (formerly "Zeitschrift für Experimentelle Psychologie"), 49(4), 243-256.

13.

Ross

Irani

Silberman

M. S.

Zaldivar

Tomlinson

(2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, CHI EA ’10 (pp. 2863–2872). New York, NY, USA: ACM.

14.

Sala

Partridge

Jacobson

Begole

(2007). An exploration into activity-informed physical advertising using pest. Pervasive Computing, 73–90.

15.

Snow

O’Connor

Jurafsky

A. Y.

(2008). Cheap and fast---but is it good?: evaluating nonexpert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 254-263. Honolulu, Hawaii: Association for Computational Linguistics.

16.

Sorokin

Forsyth

(2008). Utility data annotation with Amazon Mechanical Turk. In Computer Vision and Pattern Recognition Workshops.

17.

Stolee

K. T.

Elbaum

(2010). Exploring the use of crowdsourcing to support empirical studies in software engineering. In Proceedings of the 2010 ACMIEEE International Symposium on Empirical Software Engineering and Measurement. 1–4.