Effect of Payment Methods in Crowdsourcing Platforms

Abstract

The use of an online crowdsourcing platform makes it easier and quicker for researchers to recruit participants online. It also allows researchers to adopt another payment method traditionally used on crowdsourcing platforms: the piece-rate payment method. This paper investigated the effects of the participants’ location, the payment method, and the compensation rate on subjective experience, motivation, and perceived payment fairness. This paper extends last year’s paper, in which the participants’ location and the payment method were the focus of the investigation. Two additional studies have been conducted using piece-rate payment method. The data were analyzed using a 3-way ANOVA and a chi-square test of independence. The results showed that the piece-rate payment method was less likely to attract workers who were motivated to work for money than the quota payment method. There was no difference in perceived payment fairness between using a quota payment method and a piece-rate payment method.

Get full access to this article

View all access options for this article.

References

Antin

Shaw

(2012). Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2925–2934).

Bartneck

Croft

Kulic

(2008). Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. (Publisher: University of Hertfordshire)

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality data? In Kazdin

A. E.

(Ed.), Methodological issues and strategies in clinical research (p. 133–139). American Psychological Association. (Publisher: American Psychological Association)

Callison-Burch

(2014). Crowd-workers: Aggregating information across turkers to help them ﬁnd higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing.

d’Eon

Goh

Larson

Law

(2019). Paying Crowd Workers for Collaborative Work. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–24. (Publisher: ACM New York, NY, USA)

Goodman

J. K.

Cryder

C. E.

Cheema

(2013). Data collection in a ﬂat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.

Hanrahan

B. V.

Martin

Willamowski

Carroll

J. M.

(2019). Investigating the Amazon Mechanical Turk market through tool design. Computer Supported Cooperative Work (CSCW), 28(5), 795–814. (Publisher: Springer)

Hara

Adams

Milland

Savage

Callison-Burch

Bigham

J. P.

(2018). A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–14).

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). Elsevier.

10.

Henrich

Heine

S. J.

Norenzayan

(2010). Most people are not weird. Nature, 466(7302), 29–29.

11.

Howe

(2006). The rise of crowdsourcing. Wired magazine, 14(6), 1–4.

12.

Ikeda

Bernstein

M. S.

(2016). Pay it backward: Per-task payments on crowdsourcing platforms reduce productivity. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 4111–4121).

13.

Mason

Watts

D. J.

(2009). Financial incentives and the” performance of crowds”. In Proceedings of the ACM SIGKDD workshop on human computation (pp. 77–85).

14.

Otey v. Crowdflower, Inc., Case No. 12-cv-05524-JST (N.D. Cal. Aug. 27, 2013

15.

Soratana

Liu

Yang

X. J.

(2021). Effects of Payment Rate and Country’s Income Level on Attitude Toward a Crowdsourcing Task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2021.

16.

Whiting

M. E.

Hugh

Bernstein

M. S.

(2019). Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 7, pp. 197–206). (Issue: 1)