Effects of Payment Rate and Country’s Income Level on Attitude Toward aCrowdsourcing Task

Abstract

Using online crowdsourcing platforms has become an option for researchers to rapidly recruit participants for completing tasks that require human ingenuity. However, a growing concern is that workers on online crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), receive unfair compensation for tasks completed. In this article, we explored the eﬀects of the income level of participant’s country and the rate of payment on perceived payment fairness, task quality, and subjective experience. We tested our hypothesis using 3-way ANOVA and chi-square test of independence. The results showed that lower compensation increased the number of participants whose data might be unusable for research. Participants in the lower compensation rate group reported better perceived performance compared to those who received higher compensation. We found that high income country participants report less perceived eﬀort and frustration than lower-middle income country participants.

Get full access to this article

View all access options for this article.

References

Antin

Shaw

(2012). Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the us and india. In Proceedings of the sigchi conference on human factors in computing systems (pp. 2925–2934).

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s mechanical turk: A new source of inexpensive, yet high-quality data? In Kazdin

A. E.

(Ed.), Methodological issues and strategies in clinical research (p. 133–139). American Psychological Association.

d’Eon

Goh

Larson

Law

(2019). Paying crowd workers for collaborative work. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–24.

Difallah

Filatova

Ipeirotis

(2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the eleventh acm international conference on web search and data mining (pp. 135–143).

Fantom

Serajuddin

(2016). The world bank’s classiﬁcation of countries by income. The World Bank.

Ford

J. B.

(2017). Amazon’s mechanical turk: a comment. Journal of Advertising, 46(1), 156–158.

Hara

Adams

Milland

Savage

Callison-Burch

Bigham

J. P.

(2018). A data-driven analysis of workers’ earnings on amazon mechanical turk. In Proceedings of the 2018 chi conference on human factors in computing systems (pp. 1–14).

Hart

S. G.

Staveland

L. E.

(1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier.

Howe

(2006). The rise of crowdsourcing. Wired magazine, 14(6), 1–4.

10.

Irani

L. C.

Silberman

M. S.

(2013). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the sigchi conference on human factors in computing systems (pp. 611–620).

11.

Kaufmann

Schulze

Veit

(2011). More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk. In Amcis (Vol. 11, pp. 1–11).

12.

Marge

Banerjee

Rudnicky

A. I.

(2010). Using the amazon mechanical turk for transcription of spoken language. In 2010 ieee international conference on acoustics, speech and signal processing (pp. 5270–5273).

13.

Martin

Hanrahan

B. V.

O’Neill

Gupta

(2014). Being a turker. In Proceedings of the 17th acm conference on computer supported cooperative work & social computing (p. 224-235).

14.

Mason

Suri

(2012). Conducting behavioral research on amazon’s mechanical turk. Behavior research methods, 44(1), 1–23.

15.

Mellis

A. M.

Bickel

W. K.

(2020). Mechanical turk data collection in addiction research: Utility, concerns and best practices. Addiction, 115(10), 1960–1968.

16.

Otey v. Crowdflower, Inc., Case No. 12-cv-05524-JST (N.D. Cal. Aug. 27, 2013

17.

Reese

Heath

(2016). Inside amazon’s clickworker platform: How half a million people are being paid pennies to train ai. TechRepublic. com.

18.

Salehi

Irani

L. C.

Bernstein

M. S.

Alkhatib

Ogbe

Milland

(2015). We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd annual acm conference on human factors in computing systems (pp. 1621–1630).

19.

Whiting

M. E.

Hugh

Bernstein

M. S.

(2019). Fair work: Crowd work minimum wage with one line of code. In Proceedings of the aaai conference on human computation and crowdsourcing (Vol. 7, pp. 197–206).