Crowdsourcing arose as a problem-solving strategy that uses a large number of workers to achieve tasks and solve specific problems. Although there are many studies that explore crowdsourcing platforms and systems, little attention has been paid to define what a crowd-powered project is. To address this issue, this article introduces a general-purpose conceptual model that represents the essential elements involved in this kind of project and how they relate to each other. We consider that the workflow in crowdsourcing projects is context-oriented and should represent the planning and coordination by the crowdsourcer in the project, instead of only facilitating decomposing a complex task into subtask sets. Since structural models are limited to cannot properly represent the execution flow, we also introduce the use of behavioural conceptual models, specifically Unified Modeling Language (UML) activity diagrams, to represent the user, tasks, assets, control activities and products involved in a specific project.
HortonJJ. Online labor markets. In: SaberiA (ed.) Internet and network economics. Berlin; Heidelberg: Springer, 2010, pp. 515–522.
2.
MasonWSuriS. Conducting behavioral research on Amazon’s Mechanical Turk. Behav Res Methods2012; 44(1): 1–23.
3.
MourelatosEGiannakopoulosNTzagarakisM. Personality traits and performance in online labour markets. Behav Inform Technol. Epub ahead of print 7 September 2020. DOI: 10.1080/0144929X.2020.1815840.
4.
AmorimMNNetoFRASantosCAS. Achieving complex media annotation through collective wisdom and effort from the crowd. In: Proceedings of the 2018 25th international conference on systems, signals and image processing (IWSSIP), Maribor, 20–22 June 2018, pp. 1–5. New York: IEEE.
5.
DwarakanathAChintalaUShrikanthN, et al. Crowd build: a methodology for enterprise software development using crowdsourcing. In: Proceedings of the 2015 IEEE/ACM 2nd international workshop on crowdsourcing in software engineering (CSI-SE), Florence, 19 May 2015, pp. 8–14. Piscataway, NJ: IEEE.
6.
MaoKCapraLHarmanM, et al. A survey of the use of crowdsourcing in software engineering. J Syst Software2017; 126: 57–84.
7.
LiGZhengYFanJ, et al. Crowdsourced data management: overview and challenges. In: Proceedings of the 2017 ACM international conference on management of data, Chicago, IL, 14–19 May 2017, pp. 1711–1716. New York: ACM.
8.
SutherlinG. A voice in the crowd: broader implications for crowdsourcing translation during crisis. J Inf Sci2013; 39(3): 397–409.
9.
HosseiniMShahriAPhalpK, et al. Crowdsourcing: a taxonomy and systematic mapping study. Comput Sci Rev2015; 17: 43–69.
10.
LuzNSilvaNNovaisP. A survey of task-oriented crowdsourcing. Artif Intell Rev2015; 44(2): 187–213.
11.
GhezziAGabelloniDMartiniA, et al. Crowdsourcing: a review and suggestions for future research. Int J Manag Rev2018; 20(2): 343–363.
12.
DoanARamakrishnanRHalevyAY. Crowdsourcing systems on the World-Wide Web. Commun ACM2011; 54(4): 86–96.
13.
Estellés-ArolasEDe GuevaraFGL. Towards an integrated crowdsourcing definition. J Inf Sci2012; 38(2): 189–200.
14.
ZhaoYZhuQ. Evaluation on crowdsourcing research: current status and future direction. Inform Syst Front2014; 16(3): 417–434.
15.
Assis NetoFRSantosCAS. Understanding crowdsourcing projects: a systematic review of tendencies, workflow, and quality management. Inform Process Manag2018; 54(4): 490–506.
16.
OlivéA. Conceptual modeling of information systems. 1st ed. Berlin; Heidelberg: Springer, 2007.
17.
LukyanenkoRParsonsJ. Lightweight conceptual modeling for crowdsourcing. In: NgW. N.StoreyVCC. S. C. TrujilloJC, (eds) Conceptual modeling: 32nd international conference (ER 2013), Hong Kong, China, November 11–13, 2013. Berlin; Heidelberg: Springer, 2013, pp. 508–511.
18.
AriaMCuccurulloC. Bibliometrix: an R-tool for comprehensive science mapping analysis. J Informetr2017; 11(4): 959–975.
19.
CaiCWGippelJZhuY, et al. The power of crowds: grand challenges in the Asia-Pacific region. Aust J Manage2019; 44(4): 551–570.
20.
CupidoKOphoffJ. A conceptual model of critical success factors for an e-Government crowdsourcing solution. In: Proceedings of the 14th European conference on e-Government (ECEG’2014; ed IonasA), Brasov, 12–13 June 2014, pp. 77–84. Reading: Academic Conferences and Publishing International Limited.
21.
HassanHArisSArshadN, et al. Conceptual model for a sustainable crowdsourcing ecosystem. Pertanika J Sci Technol2017; 25(110): 89–98.
22.
Lenart-GansiniecR. Crowdsourcing and organisational learning in public organisations: a conceptual model. In: Proceedings of the 18th European conference on knowledge management (ECKM’2017; ed MarimonFMas-MachucaMBerbegal-MirabentJ, et al.), Barcelona, 7–8 September 2017, vol. 1, pp. 577–584. Sonning Common: Academic Conferences and Publishing International Limited.
23.
PedersenJKocsisDTripathiA, et al. Conceptual foundations of crowdsourcing: a review of IS research. In: Proceedings of the annual Hawaii international conference on system sciences, Wailea, HI, 7–10 January 2013, pp. 579–588. New York: IEEE.
24.
ZakariahZJanomNArshadDDNH. Crowdsourcing business model for low income community: conceptual model. In: Proceedings of the 2016 SAI computing conference (SAI), London, 13–15 July 2016, pp. 1173–1177. New York: IEEE.
25.
GeminoAWandY. Complexity and clarity in conceptual modeling: comparison of mandatory and optional properties. Data Knowl Eng2005; 55(3): 301–326.
26.
BarbosaCEEpelbaumVJAntelioM, et al. Conceptual crowdsourcing models for e-learning. In: Proceedings of the 2014 IEEE international conference on systems, man, and cybernetics (SMC), San Diego, CA, 5–8 October 2014, pp. 1121–1127. New York: IEEE.
27.
BozzonAFraternaliPGalliL, et al. Modeling crowdsourcing scenarios in socially-enabled human computation applications. J Data Semant2014; 3(3): 169–188.
28.
BranzovTIvanovaK. Conceptual modeling of crowdsourcing variations. In: Proceedings of the 12th International Conference Information Systems and Grid Technologies (ISGT’2018), Sofia, Bulgaria, 16–17 November 2018, vol. 2464, pp. 7–14. CEUR-WS (CEUR Workshop Proceedings).
29.
HetmankL. Developing an ontology for enterprise crowdsourcing. In: Multikonferenz Wirtschaftsinformatik (MKWI’2014), Paderborn, 26–28 February 2014.
30.
LuzNPobletMSilvaN, et al. Defining human-machine micro-task workflows for constitution making. In: KamińskiBKerstenGESzapiroT (eds) Outlooks and insights on group decision and negotiation. Cham: Springer International Publishing, 2015, pp. 333–344.
31.
ThuanNHAntunesPJohnstoneD. A decision tool for business process crowdsourcing: ontology, design, and evaluation. Group Decis Negot2018; 27(2): 285–312.
32.
HetmankL. Components and functions of crowdsourcing systems – a systematic literature review. In: Proceedings of the 11th international conference on Wirtschaftsinformatik, Leipzig, Germany, 27 February–1 March 2013, pp. 55–69. Association for Information Systems/AIS Electronic Library.
33.
HoweJ. Crowdsourcing: why the power of the crowd is driving the future of business. 1st ed. New York: Crown Business, 2008.
34.
AmorimMNSalemeEBDe Assis NetoFR, et al. Crowdsourcing authoring of sensory effects on videos. Multimed Tools Appl2019; 78: 19201–19227.
35.
De AmorimMNSantosCASTavaresOL. A crowdsourcing method for sign segmentation in Brazilian sign language videos. In: Proceedings of the Brazilian symposium on multimedia and the web, São Luís, Brazil, 30 November–4 December 2020, pp. 105–112. New York: ACM.
36.
AfuahATucciCL. Crowdsourcing as a solution to distant search. Acad Manage Rev2012; 37(3): 355–375.
37.
LiuXLuMOoiBC, et al. CDAS: a crowdsourcing data analytics system. Proc VLDB Endow2012; 5: 1040–1051.
BrabhamDC. Moving the crowd at Threadless: motivations for participation in a crowdsourcing application. Inform Commun Soc2010; 13(8): 1122–1145.
41.
BrabhamDC. Crowdsourcing: a model for leveraging online communities. In: DelwicheAHendersonJJ (eds) The participatory cultures handbook (Routledge edition). Abingdon: Taylor & Francis, 2012, pp. 120–129.
42.
SchenkEGuittardC. Towards a characterization of crowdsourcing practices. J Innov Econ Manage2011; 7(1): 93–107.
43.
QuinnAJBedersonBB. Human computation: a survey and taxonomy of a growing field. In: Proceedings of the SIGCHI conference on human factors in computing systems, Vancouver, BC, Canada, 7–12 May 2011, pp. 1403–1412. New York: Association for Computing Machinery (ACM).
44.
LarsonMMelenhorstMMenéndezM, et al. Chapter 10: using crowdsourcing to capture complexity in human interpretations of multimedia content. In: IonescuBBenois-PineauJPiatrikT, et al. (eds) Fusion in computer vision. Cham: Springer, 2014, pp. 229–269.
45.
BartekMATruittARWidmer-RodriguezS, et al. The promise and pitfalls of using crowdsourcing in research prioritization for back pain: cross-sectional surveys. J Med Internet Res2017; 19(10): e341.
46.
MinderPSeukenSBernsteinA, et al. CrowdManager – combinatorial allocation and pricing of crowdsourcing tasks with time constraints. In: Proceedings of the workshop on social computing and user generated content in conjunction with ACM conference on electronic commerce (ACM-EC’2012), Valencia, 7 June 2012, pp. 1–18. New York: ACM.
47.
BernsteinMSLittleGMillerRC, et al. Soylent: a word processor with a crowd inside. In: Proceedings of the 23rd annual ACM symposium on user interface software and technology, New York, 3–6 October 2010, pp. 313–322. New York: ACM.
48.
EricksonT. Geocentric crowdsourcing and smarter cities: enabling urban intelligence in cities and regions. In: Proceedings of the 1st international workshop on ubiquitous crowdsourcing/12th ACM International Conference on Ubiquitous Computing. (UbiComp’10), 26–29 September 2010, Copenhagen, Denmark, pp. 1–4. New York: ACM.
49.
GeigerDRosemannMFieltE, et al. Crowdsourcing information systems– definition, typology, and design. In: Proceedings of the 33rd annual international conference on information systems (ICIS), Orlando, FL, 16–19 December 2012, pp. 1–11. Atlanta, GA: Association for Information Systems/AIS Electronic Library.
50.
LeePTYLuiRWCChauM. Do achievement goals and work nature affect contributor performance in gamified crowdsourcing? An exploratory study in an academic setting. In: Proceedings of the workshop on e-Business, Santa Clara, CA, 12 December 2018, pp. 132–140. Berlin; Heidelberg: Springer.
51.
ChittilappillyAIChenLAmer-YahiaS. A survey of general-purpose crowdsourcing techniques. IEEE T Knowl Data En2016; 28(9): 2246–2266.
52.
KulkarniACanMHartmannB. Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 conference on computer supported cooperative work (CSCW’12), Seattle, WA, 11–15 February 2012, pp. 1003–1012. New York: Association for Computing Machinery (ACM).
53.
ModaresnezhadMIyerLPalviaP, et al. Information Technology (IT) enabled crowdsourcing: a conceptual framework. Inform Process Manag2020; 57(2): 102135.
54.
OgieRI. Adopting incentive mechanisms for large-scale participation in mobile crowdsensing: from literature review to a conceptual framework. Hum Centric Comput Inf Sci2016; 6(1): 24.
ChengJTeevanJIqbalST, et al. Break it down: a comparison of macro- and microtasks. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems (CHI’2015), Seoul, Republic of Korea, 18–23 April 2015, pp. 4061–4064. New York: ACM.
60.
MourelatosETzagarakisMDimaraE. A review of online crowdsourcing platforms. South-East Eur J Econ2016; 14(1): 59–73.
61.
MajchrzakAMalhotraA. Towards an information systems perspective and research agenda on crowdsourcing for innovation. J Strategic Inf Syst2013; 22(4): 257–268.
62.
LitmanLRobinsonJAbberbockT. TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res Methods2017; 49(2): 433–442.
63.
BrunoBFagginiMParzialeA. Motivation, incentives and performance: an interdisciplinary review. Int J Bus Manag2017; 12(12): 29–52.
64.
PilourdaultJAmer-YahiaSLeeD, et al. Motivation-aware task assignment in crowdsourcing. In: MitschangBMarklVBressS, et al. (eds) Advances in database technology: 20th international conference on extending database technology (EDBT’2017), Venice, 21–23 March 2017. Konstanz: OpenProceedings.org, 2017, pp. 246–257.
65.
PilourdaultJAmer-YahiaSRoySB, et al. Task relevance and diversity as worker motivation in crowdsourcing. In: Proceedings of the 2018 IEEE 34th international conference on data engineering (ICDE), Paris, 16–19 April 2018, pp. 365–376. New York: IEEE.
66.
SunYWangNYinC, et al. Understanding the relationships between motivators and effort in crowdsourcing marketplaces: a nonlinear analysis. Int J Inform Manag2015; 35(3): 267–276.
67.
PeeLGKohEGohM. Trait motivations of crowdsourcing and task choice: a distal-proximal perspective. Int J Inform Manag2018; 40: 28–41.
68.
GonzalezALIzidoroDWillrichR, et al. OurMap: representing crowdsourced annotations on geospatial coordinates as linked open data. In: AntunesPGerosaMASylvesterA, et al. (eds) Collaboration and technology. Berlin; Heidelberg: Springer, 2013, pp. 77–93.
69.
BrankJGrobelnikMMladenićD. A survey of ontology evaluation techniques. In: Proceedings of the conference on data mining and data warehouses (SiKDD’2005), Ljubljana, Slovenia, 17 October 2005, pp. 166–170. International Multiconference Information Society. Jožef Stefan Institute.
70.
Freelancer. Hire freelancers and find freelance jobs online – freelancer, 2021, http://freelancer.com
71.
Upwork® Global, Inc. ODesk: hire freelancers make things happen, 2021, http://upwork.com
72.
Amazon Mechanical Turk (AMT), Inc. Amazon Mechanical Turk and its affiliates, 2021, https://www.mturk.com/
LuzNPereiraCSilvaN, et al. An ontology for human-machine computation workflow specification. In: PolycarpouMDe CarvalhoACPLFPanJ-S, et al. (eds) Hybrid artificial intelligence systems. Cham: Springer International Publishing, 2014, pp. 49–60.
75.
ByunTMHalpinPFSzerediD. Online crowdsourcing for efficient rating of speech: a validation study. J Commun Disord2015; 53: 70–83.
76.
Nguyen-DinhL-VWaldburgerCRoggenD, et al. Tagging human activities in video by crowdsourcing. In: Proceedings of the 3rd ACM conference on international conference on multimedia retrieval (ICMR’13), Dallas, TX, 16–20 April 2013, pp. 263–270. New York: ACM Press.
77.
AltmeyerMLesselPKrügerA. Expense control: a gamified, semi-automated, crowd-based approach for receipt capturing. In: Proceedings of the 21st international conference on intelligent user interfaces (IUI’16), Sonoma, CA, 7–10 March 2016, pp. 31–42. New York: ACM Press.
78.
CaoCCChenLJagadishHV. From labor to trader: opinion elicitation via online crowds as a market. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’14), New York, 24–27 August 2014, pp. 1067–1076. New York: ACM Press.
79.
KimS-HLiSKwonBC, et al. Investigating the efficacy of crowdsourcing on evaluating visual decision supporting system. In: Proceedings of the 55th annual meeting human factors and ergonomics society, Las Vegas, NV, 19–23 September 2011, pp. 1090–1094. SAGE Publishing. https://doi.org/10.1177%2F1071181311551228
