Design Considerations for the Development of Crowdsourcing Systems

Abstract

We present a case study on factors that affect the usability and User Experience (UX) of a crowdsourcing platform for argumentation. While this particular system is focused on argumentation, we have aimed to abstract our findings such that they are generalizable to other use cases so that others may consider these lessons learned and recommendations for the development of more effective crowdsourcing and collaborative work platforms. Several themes were identified in participant responses about the usability and UX of the crowdsourcing system, including a desire for less structure, the need for additional training, the desire for a streamlined workflow and UI, improved navigation, and the enjoyment of interactions with other users.

Get full access to this article

View all access options for this article.

References

Brabham

D. C.

(2008). Crowdsourcing as a model for problem solving: An introduction and cases. Convergence, 14(1), 75–90. doi: 10.1177/1354856507084420

Brabham

D. C.

(2013). Crowdsourcing. Cambridge: MIT Press.

Brooke

(1996). SUS: A “quick and dirty” usability scale. In Jordan

P.W.

Thomas

Werdmeester

B.A.

McClelland

L.L.

(Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor & Francis.

Brooke

(2013). SUS: A retrospective. Journal of Usability Studies 8(2), 29–40.

Bucheler

Sieg

J.H.

Fuchslin

R. M.

Pfeifer

(2010). Crowdsourcing, open innovation and collective intelligence in the scientific method: A research agenda and operational framework. The 12th International Conference on the Synthesis and Simulation of Living Systems, 679–686. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability.

Dorton

S.L.

Harper

Creed

G.A.

Banta

H.G.

(In press). Up for debate: Effects of formal structure on argumentation quality in a crowdsourcing platform. 13^th International Conference on Social Computing and Social Media.

Drapeau

Chilton

L.B.

Bragg

Weld

D.S.

(2016). MicroTalk: Using argumentation to improve crowdsourcing accuracy. Fourth AAAI Conference on Human Computation and Crowdsourcing, AAAI Press.

Garrett

(2010). The elements of user experience: user-centered design for the web and beyond. Retrieved from https://ptgmedia.pearsoncmg.com/images/9780321683687/samplepages/0321683684.pdf

Hogg

Lerman

(2015). Disentangling the effects of social signals. Human Computation, 2(2), 189–208.

10.

Kim

Sterman

Cohen

A.A.B

Bernstein

(2017). Mechanical novel: Crowdsourcing complex work through reflection and revision. ACM Conference on Computer-Supported Cooperative Work and Social Computing.

11.

McLellan

Muddimer

Peres

S. C.

(2012). The effect of experience on System Usability Scale ratings. Journal of Usability Studies, 7(2), 56–67.

12.

Metzger

M. J.

Flanagin

A. J.

(2013). Credibility and trust of information in online environments: The use of cognitive heuristics. Journal of Pragmatics, 59, 210–220.

13.

Moreville

(2004). User Experience Design. Semantic Studios. Retrieved from: http://semanticstudios.com/user_experience_design/

14.

Surowiecki

2004. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations (1st ed.).New York: Doubleday.