Crowdsourcing for Research: Perspectives From a Delphi Panel

Abstract

Crowdsourcing, an open call for the public to collaborate and participate in problem solving, has been increasingly employed as a method in health-related research studies. Various reviews of the literature across different disciplines found crowdsourcing being used for data collection, processing, and analysis as well as tasks such as problem solving, data processing, surveillance/monitoring, and surveying. Studies on crowdsourcing tend to focus on its use of software, technology and online platforms, or its application for the purposes previously noted. There is need for further exploration to understand how best to use crowdsourcing for research, as there is limited guidance for researchers who are undertaking crowdsourcing for the purposes of scientific study. Numerous authors have identified gaps in research related to crowdsourcing, including a lack of decision aids to assist researchers using crowdsourcing, and best-practice guidelines. This exploratory study looks at crowdsourcing as a research method by understanding how and why it is being used, through application of a modified Delphi technique. It begins to articulate how crowdsourcing is applied in practice by researchers, and its alignment with existing research methods. The result is a conceptual framework for crowdsourcing, developed within traditional and existing research approaches as a first step toward its use in research.

Keywords

research methodology crowdsourcing Delphi method

Public and patient engagement, alongside activities such as knowledge translation and mobilization, are becoming standard requirements for health sciences and services research funding (Domecq et al., 2014; Frank et al., 2015; Tetroe et al., 2008). While some mature methodologies, such as participatory research, embrace non-researcher involvement in research, new methods are also emerging to encourage public involvement in research. Crowdsourcing, “an online, distributed problem solving and production model” (Brabham, 2008), is one method through which researchers are engaging the public. The term citizen science, defined as “a form of research collaboration involving members of the public in scientific research projects to address real-world problems” (Wiggins & Crowston, 2011), is frequently used synonymously with crowdsourcing and aims to address the same notion of engaging the public in research.

Various reviews of the literature across different disciplines found crowdsourcing being used for data collection, processing, and analysis as well as tasks such as problem solving, data processing, surveillance/monitoring, and surveying (Crequit et al., 2018; Hossain & Kauranen, 2015; Ranard et al., 2014). Websites such as TurkPrime, Profilic.co, and Crowdcrafting.org engage the crowd for the purposes of recruitment, data collection, or data analysis for their studies (Bassi et al., 2020; Crequit et al., 2018; Litman et al., 2017; Peer et al., 2017). Studies on crowdsourcing tend to focus on its use of software, technology, online platforms, or its application for the purposes previously noted (Hossain & Kauranen, 2015). There is need for further exploration to understand how best to use crowdsourcing for research, as there is limited guidance for researchers who undertake crowdsourcing for the purpose of research studies (Buettner, 2015; Law et al., 2017). Numerous authors have identified gaps in the research related to crowdsourcing, including a lack of decision aids to assist researchers using crowdsourcing, and best practice guidelines (Buettner, 2015; Law et al., 2017; Sheehan, 2018).

In this study, we sought to explore crowdsourcing as a research method by understanding how and why it is being used, through the use of a modified Delphi technique to identify how crowdsourcing is being applied in practice by researchers and how it aligns with any specific research methods. We aimed to develop a conceptual framework for crowdsourcing within traditional and existing research approaches, as a first step toward supporting researchers using this method.

Method

The Delphi Technique

The Delphi technique, developed in the 1950s by the RAND Corporation, is a method used to achieve consensus among experts (Okoli & Pawlowski, 2005) and to summarize the array of positions that these experts have taken on a topic (Mullen, 2003). It is also frequently used where little evidence exists, or where the knowledge base is limited. Linstone and Turoff (2002) note that it facilitates group communication to enable collective problem solving. Furthermore, according to the Encyclopedia of Research Design, the Delphi method attempts to understand what could or should be (Salkind, 2010), unlike surveys, which aim to address what is.

Given that we are looking to identify the perspectives that exist among experts on the use of crowdsourcing for research, and establish consensus around preliminary definitional characteristics, we considered the Delphi technique to be particularly appropriate to our design. Furthermore, the Delphi technique had the potential to provide insights within this exploratory study, which will scaffold the induction of a general model or theory (Steinert, 2009). The exploratory data collected allow for the development of a conceptual model. In addition, a panel study (as opposed to the responses of any individual expert) may provide the most relevant “answers” to our research questions, given the limited numbers of experts in this area.

Identifying the expert panel

According to Rowe and Wright (2001), the composition of a Delphi panel should be heterogeneous, to ensure members’ combined experience and knowledge represents the full research domain. The long-standing debate of who qualifies as an expert for the purpose of a Delphi has resulted in very broad inclusion criteria such as “informed individuals” as well as more narrowly defined criteria, such as “specialists in a field” (Baker et al., 2006). The nascent nature of crowdsourcing in research required the term “expert” to be interpreted broadly, and we elected to include individuals with experience in the application of crowdsourcing for research, as well as individuals who had studied the topic of crowdsourcing itself. Given that the purpose of this study was to identify salient characteristics of crowdsourcing within research settings, we conducted a literature review to create a list of potential expert panelists, comprising researchers and/or academics who either use crowdsourcing in their research methods, or research the topic of crowdsourcing. Figure 1 presents a graphical depiction of how panel members were selected.

Figure 1.

Participant identification and recruitment process.

First, a list of published studies was assembled by using the keyword terms “crowdsourc*” and (“medical” or “health”), with the filters “English” and “peer-reviewed.” This search resulted in 275 articles identified in PubMed, and 126 articles in ProQuest, for a total of 401 articles – 15 of which were duplicates. The titles of these articles were then reviewed for relevance, and 154 articles were removed that neither discussed the use of crowdsourcing, nor employed crowdsourcing as a primary research methodology. An additional 99 articles that included editorials and commentaries, articles that only referenced the term crowdsourcing in a non-substantive manner (primarily in a broader social media context), focused on crowdfunding (which is not considered to be crowdsourcing for the purposes of this study), and/or did not deploy crowdsourcing for their research, were removed post abstract review. This resulted in a total of 133 articles.

From those articles, email addresses for the first and corresponding authors’ were located (where publicly available). Although a total of 203 researchers were solicited to participate in this research study, 20 emails “bounced back,” meaning that a maximum of 183 emails were delivered. From these 183 emails, 18 individuals agreed to participate in the study.

Crowdsourcing Framework

Working from the more than 40 different definitions for the term crowdsourcing, Estellés-Arolas and González-Ladrón-De-Guevara (2012) developed an integrated definition of crowdsourcing which consists of eight discrete characteristics (p. 197):

there is a clearly defined crowd (size and typology – skills/knowledge of crowd);

there exists a task with a clear goal (task-based, what the participant has to do);

the recompense received by the crowd is clear (what do they get in return – material or not);

the crowdsourcer is clearly identified (any entity or individual);

the compensation to be received by the crowdsourcer is clearly defined (what is the benefit to the crowdsourcers);

it is an online assigned process of participative type (type of process);

it uses an open call of variable extent (type of call); and

it uses the internet (medium).

These characteristics serve as a starting point for constructing a framework for understanding crowdsourcing within a research context. In the absence of a commonly agreed-upon definition for crowdsourcing, these characteristics provide a common language to help facilitate an understanding of its application. Despite the information science undertone, the application of these characteristics within a research context was deemed appropriate, given that they were informed by a non-discipline-specific review of the literature. Furthermore, the characteristics were identified as a result of the comprehensive and integrative process by which the authors developed them (Estellés-Arolas & González-Ladrón-De-Guevara, 2012).

Procedure

The Delphi process includes a minimum of two rounds of questionnaires with each subsequent round building on the previous responses (Fefer et al., 2016). All forms and letters were also reviewed and pre-tested to ensure clarity, prior to data collection. We completed two rounds of questionnaires and content analysis to identify domains of consensus across the experts solicited for their opinion on the use of crowdsourcing for research. For both rounds, a mix of questions were used, including open-ended, edit, rank, and rate. The questions for both rounds can be found in the supplemental appendix. For each Delphi round, data were generated and analyzed by the authors of this article. The first round of questions focused on general questions to gain a broad understanding of how crowdsourcing is used in research. Round 1 questions focused on the ways in which crowdsourcing is being used in research, as well as the key characteristics of crowdsourcing. Participants were asked to identify key characteristics of crowdsourcing for research, and to rate the importance of characteristics identified by Estellés-Arolas and González-Ladrón-De-Guevara (2012). The data from the first round were summarized and shared with panel members to use in the next round. Round 2 questions aimed to further understand why researchers were using crowdsourcing and move toward a framework for using crowdsourcing in research, by trying to improve upon and adapt the characteristics identified by Estellés-Arolas and González-Ladrón-De-Guevara (2012). The threshold for consensus on positions was set at 70% for rating-based questions. A third round was not undertaken as it was determined that further consensus was unlikely, based on the responses in the first two rounds.

This study utilized Qualtrics Survey Software to distribute the questionnaires to the expert panelists via email. The use of the software enabled rapid analysis of responses and allowed for the Delphi process to be conducted in an efficient manner. Procedures employed within this study (including both recruitment and informed consent) were approved by the non-medical research ethics board at the University of Western Ontario (protocol # 108655).

Results

Characterizing the Expert Panelists

The characterization of panel expertise is important in the context of the Delphi method, given its reliance on expert opinion. The researchers identified as experts in crowdsourcing were deemed as such based on the nascent nature of the subject matter. Of the 18 respondents who agreed to participate, 15 completed the Round 1 survey and 12 completed the Round 2 survey. Survey participants were a mix of researchers who had used crowdsourcing in their research (n = 15, 83%) and those who studied the topic of crowdsourcing (n = 3, 16%). Panelists reported published studies using both quantitative and qualitative methods.

When asked to rate level of experience in the application of crowdsourcing on a scale from 0 to 100, participants’ scores ranged from 21 to 100 (M = 66, SD = 23). When asked to explain their ratings, the relative newness of crowdsourcing in research was frequently referenced, along with notations of having employed the approach only once, or a very limited number of times. Some panelists qualified their expertise in crowdsourcing:

I employ crowdsourcing in multiple ways across many platforms, I am an expert in citizen science (a form of crowdsourcing) and regularly review papers on the topic. I am an invited speaker on crowdsourcing across many disciplines.

I spend a significant amount of my professional work on crowdsourced technologies for health but certainly have room to grow in my knowledge in this area.

I have been involved in ethics approvals, developing web sites, recruiting citizens, supporting them, and generating research results based on their research and presenting these at conferences. However, there are many aspects of crowdsourcing that I have yet to experience.

I am regarded as an expert in using crowdsourcing as a source of convenience samples. I have also used crowds to code sentiment. However, I have very limited experience in other human computation applications and almost all of my experience is confined to Amazon Mechanical Turk.

In the last four years, I have been actively engaged in investigating what would motivate people to participate in social responsible crowdsourcing projects.

As demonstrated through these quotes from the panelists, their self-rated experience ranged from applying it for research purposes to focusing on a specific aspect of crowdsourcing.

The Use of Crowdsourcing for Research

Panelists identified numerous uses of crowdsourcing in research, based on the literature and their own experience, including recruiting research participants, data collection, data analysis, and developing interventions. Individually, panelists used crowdsourcing for participant recruitment, data collection, and data analysis. In some instances, the purpose of crowdsourcing in their research studies was tied to the fulfillment of traditional participant or subject roles, such as recruitment and the provision of data:

I have used crowdsourcing to recruit convenience samples of research participants . . .

My project recruited citizens via the web from across the world to contribute data . . .

This type of role includes inviting the crowd to complete tasks such as questionnaires, providing personal information, and undertaking other online activities to generate data for research purposes. For example,

I have used crowdsourcing to get participants to take surveys.

Panel members who undertook clinical or medical quantitative research studies tended to identify these types of uses for crowdsourcing. In this case, where the primary purpose is to access participants, crowdsourcing differs little from other recruitment methods, and thus requires similar considerations with regard to methodological rigor and appropriateness.

Researchers are also using crowdsourcing to engage the crowd in activities such as data collection and analysis, activities that have been more traditionally the role of researchers. Panelists noted,

As a form of data collection from human participants.

. . . to annotate histopathological images.

Used crowdsourcing to develop intervention tools . . .

Other panelists identified similar uses identified in the literature such as,

. . . particularly in public health and infectious disease, there are studies that crowdsource information from the public on things like the flu . . .

In these instances, the crowd supports the research study through the provision of their knowledge, experience, and skills. There is a deeper level of engagement and perhaps an underlying trust factor that the crowd has the capability to undertake such tasks. Leveraging the data collection and analytical capabilities of the crowd are, however, contingent upon the nature of the research ranging from simple tasks such as tracking and monitoring to more complex types of problem solving.

In limited instances, researchers are building capacity through the engagement of the crowd to undertake co-researcher type activities, and providing education and training to the crowd:

. . . citizen scientists volunteer their time towards the scientific process in an active research study. They go through extensive training (ethics, enrollment procedures, data prep and some analysis).

While this type of research capacity building is common practice with qualitative research methods such as participatory action research, it was referenced by only one panelist.

The least frequently identified uses of crowdsourcing in research were study and instrument design with expert panel members citing concerns with lack of expertise and knowledge of the crowd to undertake such work. Most of the expert panelists noted the distinction between the roles of the crowd versus those of the researchers. This underscores the fact that specific research expertise and skills are required for many studies, and so areas such as study and instrument design, or even data analysis in some instances, are areas that may extend beyond the capabilities of the crowd. However, this blurring of roles is common in non-research crowdsourcing activities (Howe, 2009) and was acknowledged by panelists:

There is a small literature that uses crowds to provide other services traditionally performed by experts (e.g., psychological therapy for subclinical issues, or screening medical images).

In addition, panelists distinguished between the crowd as general members of the public and a crowd of experts. As one panelist noted,

Sometimes you need a special crowd, other times any crowd will do.

I also sometime see crowdsourced views of experts.

The Benefits and Challenges of Using Crowdsourcing for Research

Panelists were asked why they used crowdsourcing and to identify some of the benefits and challenges associated with its use. Members of the Delphi panel tended to view the crowd as a supplement to the capacity and capabilities of professional researchers—In other words, participants were seen to be an on-demand pool of resources. Benefits and challenges were categorized into five broad themes: process, people, knowledge, data, and experience. Table 1 summarizes panelist responses within those categories.

Table 1.

Benefits and Challenges of Using Crowdsourcing in Research.

Theme	Benefits	Challenges
Process	• Low cost • Fast	• Need to carefully consider ethical implications
People	• Access to large numbers of people • Diverse population	• Self-selected • Lack of representativeness
Knowledge	• Outsider perspective • Knowledge mobilization	• “Colloquial” knowledge of subject matter
Data	• Large volumes that would not otherwise be possible	• Quality, validity, and reliability issues
Experience	• Innovation spurred by the diversity of ideas • The crowd benefits from their access to researchers	• Lack of research experience and understanding of research practices

Based on feedback from the panel, the use of crowdsourcing for research is an effective and efficient process to overcome barriers such as time limitation, volume of data, and costs, regardless of how the crowd is being leveraged.

The Characteristics of Crowdsourcing for Research

In an effort to identify a potential framework for crowdsourcing in research, panelists were asked to indicate the importance of each of the characteristics of crowdsourcing as identified by Estellés-Arolas and González-Ladrón-De-Guevara (2012) in the research context by rating it on a scale of 0 to 100, with 0 being the lowest rating and 100 being the highest. Table 2 summarizes the rating scores and provides the average for each characteristic.

Table 2.

Importance of Characteristics of Crowdsourcing in Research.

Characteristic	Minimum	Maximum	M	SD
there is a clearly defined crowd	12	100	65.4	30.3
there exists a task with a clear goal	20	100	83.6	20.4
the recompense received by the crowd is clear	19	90	62.3	25.5
the crowdsourcer is clearly identified	13	100	57.1	28.0
the compensation to be received by the crowdsourcer is clearly defined	10	98	55.6	25.1
it is an online assigned process of participative type	10	82	50.2	22.8
it uses an open call to a variable extent	26	100	58.8	25.7
it uses the internet	7	100	58.2	34.6

The only characteristic that panelists agreed was important within the research context was “there exists a task with a clear goal” which had an average rating of 83.62. When asked to explain the lack of consensus in what characteristic of crowdsourcing are important for research, three common themes emerged across the responses from the panelists, including issues related to the definitions of terms, the specific task being assigned to the crowd, and the nature of the study in which crowdsourcing is being applied. On issues related to the definitions of terms and the lack of clarity around language, panelists noted,

We all have different assumptions of what crowdsourcing is. . . . Not sure what definitions others are using.

It largely depends on how you interpret this. When Estelles Arolas & Gonzales Ladron-Guevara talk about a “clearly defined crowd,” I interpret that to mean . . .

The issue of compensation in crowdsourcing is always murky, because some scholars interpret the word “compensation” (or in this case “recompense”) to mean strictly monetary reward. Of course, many crowdsourcing efforts involve no monetary reward at all . . .

Panelists also noted the disagreement in what characteristics of crowdsourcing are important for research that could result from the specific task being assigned to the crowd:

So depending on the task we assumed, the rating can vary.

The variability of the response may depend on how people leverage the crowd in their work.

I interpret that to mean that a task is designed for and targets a particular kind of person . . . I don’t think a clearly defined crowd is nearly as important as a clearly defined problem and solution parameters.

Finally, the variation in responses from panelists was also attributed to the nature of the study in which the crowdsourcing was being undertaken:

It really depends on the study design and the background of the researcher . . .

It will depend on your research question and goals how much you need the crowd accurately defined.

Different study fields may have different ideas on this. The requirements of different studies may be varying . . .

Given the lack of consensus around the characteristics of crowdsourcing as defined by Estellés-Arolas and González-Ladrón-De-Guevara (2012), expert panel members were provided with supplementary descriptors and statements aimed at clarifying each of the characteristics for the research context and asked to rate its importance in relation to the description of the original characteristic on a scale of 1 to 10 (1 being not important and 10 being very important). Table 3 provides a summary of the ratings.

Table 3.

Importance of Supplementary Statements.

Crowdsourcing characteristic	Minimum	Maximum	M	SD
“there is a clearly defined crowd”
The crowd should be defined in terms of skills and/or experience and/or knowledge required	1.0	10.0	6.2	3.6
The crowd should include anyone who chooses to participate	1.0	10.0	6.0	3.0
“there exists a task with a clear goal”
The overarching purpose of the study is defined	1.0	10.0	7.8	3.0
The task to be completed by the participant is explicitly defined	1.0	10.0	7.9	2.9
“the recompense received by the crowd is clear”
If participants are to be compensated, the compensation is explicitly defined	6.0	10.0	8.5	1.4
“the crowdsourcer is clearly identified”
The crowd should know who is conducting the research	1.0	10.0	6.5	3.1
The crowdsourcer’s contact information should be available	1.0	10.0	6.1	3.7
“the compensation to be received by the crowdsourcer is clearly defined”
The crowdsourcer should disclose any compensation to be received as a result of the research	2.0	10.0	7.5	2.7
The crowdsourcer should declare any conflict of interest	1.0	10.0	7.9	2.6
“it is an online assigned process of participative type” and “it uses an open call to a variable extent”
Crowdsourcing is an open online participatory process	0.0	10.0	5.0	3.7

The only characteristics where the panelists thought the supplementary statements improved and clarified the original statements were “there exists a task with a clear goal,” “the recompense received by the crowd is clear,” and “the compensation to be received by the crowdsourcer is clearly defined.” When asked to provide comments and/or edits to each of the supplementary statements, the majority of the comments suggested the supplementary statements did not add to or improve the characteristics.

For the characteristics related to an open call, or using online tools (e.g., the internet), panelists suggested other channels could be used to facilitate crowdsourcing in research, including texting, audience response in a live setting, in person events, public spaces, traditional media, sensor systems, community meetings, and recruitment from public places. As panelists noted,

Ornithologists have been doing crowdsourcing of bird observations since before the internet and are/were organized in birders clubs. If that’s one idea of crowdsourcing you have then it’s clear that it doesn’t need to be online.

Also, some people may see plenty of great crowdsourcing examples that use SMS text messages . . . which isn’t technically the internet.

However, panelists did appear to support the idea of an open call, with comments such as

Being open to a large number of relevant people

Crowdsourcing places no particular requirements on the people that comprise the crowd.

Again, this raises the matter of composition of the crowd.

Finally, when asked whether the same ethical standards apply when using crowdsourcing in research studies, 67% of the panelists agreed, 8% were uncertain, and 25% disagreed. The panelists who disagreed noted that sometimes crowdsourcing is used because it is easier from a requirements perspective, as it is not considered human subjects research.

Discussion

This modified Delphi study demonstrates a broad range of research applications for crowdsourcing, alongside the various benefits and challenges associated with its use. While no general consensus was achieved on the characteristics of crowdsourcing for research purposes, the findings revealed gaps in knowledge regarding the application for crowdsourcing in research from a methodological and methods perspective. This led us to develop a conceptual framework for researchers to consider when deploying crowdsourcing in their studies based on whether the research study is quantitative or qualitative and how the use of crowdsourcing would align with existing research paradigms. Recognizing that we are in the early stages of exploring crowdsourcing as a research tool, this framework aims to contextualize crowdsourcing as a research method based on how it is currently being used by researchers within existing forms of inquiry and research paradigms. As such, the discussion also presents considerations related to other methodical questions that arose from the modified Delphi process.

The way in which crowdsourcing is used in research is contingent upon the task that is assigned, and this is fundamentally driven by the needs of the research study. These uses of crowdsourcing can be mapped along a continuum (Figure 2). At one end of the continuum, crowdsourcing is used for basic research purposes such as subject or participant recruitment, while at the other end, crowdsourcing serves as a mechanism for capacity building and co-researcher type activities. Moving from left to right, the level of expertise, skill, and experience required of the crowd increases. Considering the research task with the level of crowd expertise, skill, and experience allows for the role of the crowd to be defined as one of subject/participant, citizen scientist, or co-researchers. Furthermore, these research tasks and roles must be considered in the context of the research methodology—quantitative or qualitative—as each has a different set of implications. The application of crowdsourcing in research should align philosophically and methodologically with the research paradigm in which it is being deployed and therefore should align with the standards of those methods.

Figure 2.

Continuum of crowdsourcing in research.

This spectrum aligns with the positivist to critical theory paradigms continuum originally created by Lincoln and Guba (2011), and later modified by Heron and Reason (1997), with the addition of the participatory paradigm. The continuum allows for fluidity between the categories where the complexity of the task dictates where it rests along the continuum. Furthermore, the role of the researcher also evolves along the continuum, from sole conductor of a research study to a more collaborative model that may entail activities such as educating and training the crowd to facilitate their participation.

The task, therefore, will also dictate the composition and size of the crowd. Where the task is complex, for example, developing algorithms for the prediction of disease progression for Amyotrophic Lateral Sclerosis (Kuffner et al., 2015), the task is likely to draw experts in the field who are qualified to address the challenge and have an interest in doing so, thus limiting the size of the crowd. At the opposite end of the spectrum, where the task is more general, such as rating food choices (Turner-McGrievy et al., 2015), the crowd is likely to be larger, with a range of skills and backgrounds. Therefore, it is important for researchers to clearly articulate the goal of the study, the task that is being assigned to the crowd, and how the task relates to the study, so participants can self-select based on what they perceive they can contribute. Furthermore, researchers’ need to invest in crowd capacity building will be determined by the complexity of the task, or a requirement for specific skills.

When cross-referencing panelist use of crowdsourcing for research, and its benefits to the published literature on the topic, the crowd was rarely (if ever) engaged in a fully participatory, collaborative, co-research approach among the experts solicited to participate in this study. Concepts related to building public capacity and training the crowd, knowledge mobilization, and two-way engagement between professional scientists and citizen scientists appeared to be tertiary objectives. Thus, leveraging the crowd to build capacity for research in the community, and to mobilize knowledge, appears to be underutilized opportunities—particularly given research that suggests that user-driven research can accelerate and improve the innovation adoption process of a solution or new knowledge (Celi et al., 2014).

Definitions of Crowdsourcing for Research

One possible way to consider crowdsourcing is in the context of the research paradigm in which the crowd will be engaged. The paradigm thus defines the roles of the crowd. If the role of the crowd can be defined in generally acceptable research terms (i.e., participant, data collection, analysis, study design, etc.), it makes it possible to develop a lexicon or terminology that aligns with the roles and paradigms from research subject or participant, to citizen scientist, to co-researcher.

One particular characteristic of crowdsourcing, its online nature and use of the internet, warrants mentioning in the context of defining crowdsourcing for research. Despite the vast majority of definitions referencing the online and internet aspects of crowdsourcing, panelists expanded the scope to include other mechanisms and channels, while maintaining the open call component to enable almost anyone who wishes to participate to be able to do so. This expansion aligns with inclusivity and equity principles of this type of research.

Issues of Integrity and Quality

Ideally, the use of crowdsourcing in research studies should have the same demands for integrity and quality as do studies that deploy other methods. When used for the purposes of recruitment, researchers should acknowledge and recognize issues related to sample representativeness, self-selection, and generalizability, where these are important factors based on the research study design. As quantitative and qualitative research methodologies and approaches have differing views on participant recruitment, the way in which each researcher addresses this will be contingent upon his or her area of study. Similarly, issues related to quality of data will likely be addressed according to research methodology or approach. Various methods to ensure quality have, however, been identified, including bringing reported data together with diagnostic or other clinical measures (Chunara et al., 2013), in-house calculations and physician verification (Swan, 2012), and reputation metrics for evaluating user-generated content (McCoy et al., 2014). While research suggests the quality of crowdsourced data is similar to that of non-crowdsourced data (Behrend et al., 2011; Swan, 2012), researchers should build mechanisms to ensure quality into their study design where appropriate.

Adherence to Research Standards

When undertaken for research purposes, crowdsourcing should be held to the same ethical standards as other approaches. The question remains, however, whether the task being assigned positions the crowd as participants, citizen scientists, or co-researchers, and whether this positioning informs how and which ethical practices apply? Ideally, the defined role of the crowd would dictate which ethical practices apply. Where the crowd is actively involved in complex areas of the study, are they research participants and/or researchers? And do the standards of human ethics still apply?

One less murky area is the need for transparency around the benefits for both the participants and researchers. The expert panel identified the need to be explicit in identifying the compensation, monetary or otherwise, for the crowd’s participation as well as for the researchers. Thus, regardless of the methodological reasons for using crowdsourcing, any application of the method within a research study should be done within a context of maintaining appropriate ethical and professional standards.

Limitations

There are numerous definitional challenges when considering the use of crowdsourcing in research. Estellés-Arolas and González-Ladrón-De-Guevara (2012) provided a common definition and framework which, despite being framed using an information technology context, can be adapted to other research domains (Bassi et al., 2019). A key limitation of this definition is, however, within the general domain of applicability to crowdsourcing activities possible beyond online and internet activities. Thus, additional research is required on the application of non-internet-based crowdsourcing for research.

The Delphi panel experts may have interpreted the questions differently based on their own experiences. In some instances, the responses provided reflected a lack of certainty in terms of what the survey questions were asking and how it specifically pertained to their work. There was also a range of knowledge and experience in using crowdsourcing for research among the panelists, making it difficult to come to consensus. This problem was exacerbated by the relative novelty of crowdsourcing and the limited body of literature on its use in health-related research.

Directions for Future Research

As crowdsourcing and other methods are explored for the purposes of research, the research context risks becoming lost in the novelty and attractiveness of possibilities presented by information technologies and new communications channels. While these new opportunities should be embraced, this should be done in a way that maintains the integrity of research paradigms. The ease with which researchers have access to the data and capabilities beyond their institutions and communities, through the crowd, should be leveraged in a responsible manner.

Future research should supplement the information uncovered in this study with case studies and interviews of researchers using crowdsourcing. This may provide an opportunity to further explore and examine the implementation of crowdsourcing in specific settings and implementations. This additional research could also highlight contextual differences that may depend on the research area in which crowdsourcing is deployed.

Conclusion

As evidenced by the Delphi panelists and the current body of literature, the multitude of purposes for which crowdsourcing is being used for research across various disciplines presents significant opportunities for researchers. Researchers from different fields are using crowdsourcing for everything from quantitative surveys to more qualitative participatory purposes. Furthermore, as a nascent approach, the concept of crowdsourcing is frequently interpreted as being simply an online platform (Bassi et al., 2020). The challenge for researchers, however, is to consider all of the characteristics of crowdsourcing, not limited to the online, technological, or platform considerations, and more holistically consider alignment between theoretical perspective and research methods (Finlay & Ballinger, 2006) to ensure its appropriate use for research. The conceptual framework presented within this article provides researchers with a first step toward considering how they will use crowdsourcing within traditional and existing research approaches.

Supplemental Material

sj-pdf-1-sgo-10.1177_2158244020980751 – Supplemental material for Crowdsourcing for Research: Perspectives From a Delphi Panel

Supplemental material, sj-pdf-1-sgo-10.1177_2158244020980751 for Crowdsourcing for Research: Perspectives From a Delphi Panel by H. Bassi, L. Misener and A. M. Johnson in SAGE Open

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

A. M. Johnson

Supplemental Material

Supplemental material for this article is available online.

References

Baker

Lovell

Harris

(2006). How expert are the experts? An exploration of the concept of “expert” within Delphi panel techniques. Nurse Researcher, 14(1), 59–70. https://doi.org/10.7748/nr2006.10.14.1.59.c6010

Bassi

Lee

C. J.

Misener

Johnson

A. M.

(2020). Exploring the characteristics of crowdsourcing: An online observational study. Journal of Information Sciences, 46(3), 291–312. https://doi.org/10.1177/0165551519828626

Behrend

T. S.

Sharek

D. J.

Meade

A. W.

Wiebe

E. N.

(2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813. https://doi.org/10.3758/s13428-011-0081-0

Brabham

D. C.

(2008). Crowdsourcing as a model for problem solving: An introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90. https://doi.org/10.1177/1354856507084420

Buettner

(2015). A systematic literature review of crowdsourcing research from a human resource management perspective. In 48th Hawaii international conference on system sciences (pp. 4609–4618). https://doi.org/10.1109/HICSS.2015.549

Celi

L. A.

Ippolito

Montgomery

R. A.

Moses

Stone

D. J.

(2014). Crowdsourcing knowledge discovery and innovations in medicine. Journal of Medical Internet Research, 16(9), e216. https://doi.org/10.2196/jmir.3761

Chunara

Smolinski

M. S.

Brownstein

J. S.

(2013). Why we need crowdsourced data in infectious disease surveillance. Current Infectious Disease Reports, 15(4), 316–319. https://doi.org/10.1007/s11908-013-0341-5

Crequit

Mansouri

Benchoufi

Vivot

Ravaud

(2018). Mapping of crowdsourcing in health: Systematic review. Journal of Medical Internet Research, 20(5), e187. https://doi.org/10.2196/jmir.9330

Domecq

J. P.

Prutsky

Elraiyah

Wang

Nabhan

Shippee

Brito

J. P.

Boehmer

Hasan

Firwana

Erwin

Eton

Sloan

Montori

Asi

Abu Dabrh

A. M.

Murad

M. H.

(2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14(1), Article 89. https://doi.org/10.1186/1472-6963-14-89

10.

Estellés-Arolas

González-Ladrón-de-Guevara

(2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38(2), 189–200. https://doi.org/10.1177/0165551512437638

11.

Fefer

J. P.

De-Urioste Stone

Daigle

Silka

(2016, April). Using the Delphi technique to identify key elements for effective and sustainable visitor use planning frameworks. SAGE Open. https://doi.org/10.1177/2158244016643141

12.

Finlay

Ballinger

(Eds.). (2006). Qualitative research for allied health professionals: Challenging choices. John Wiley.

13.

Frank

Forsythe

Ellis

Schrandt

Sheridan

Gerson

Konopka

Daugherty

(2015). Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Quality of Life Research, 24(5), 1033–1041. https://doi.org/10.1007/s11136-014-0893-3

14.

Heron

Reason

(1997). A participatory inquiry paradigm. Qualitative Inquiry, 3(3), 274–294. https://doi.org/10.1177/107780049700300302

15.

Hossain

Kauranen

(2015). Crowdsourcing: A comprehensive literature review. Strategic Outsourcing: An International Journal, 8(1), 2–22. https://doi.org/10.1108/SO-12-2014-0029

16.

Howe

(2009). Crowdsourcing: Why the power of the crowd is driving the future of business. Three Rivers Press.

17.

Kuffner

Zach

Norel

Hawe

Schoenfeld

Wang

Fang

Mackey

Hardiman

Cudkowicz

Sherman

Ertaylan

Grosse-Wentrup

Hothorn

van Ligtenberg

Macke

Meyer

Scholkopf

. . .Leitner

(2015). Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression. Nature Biotechnology, 33(1), 51–57. https://doi.org/10.1038/nbt.3051

18.

Law

Gajos

K. Z.

Wiggins

Gray

M. L.

Williams

(2017). Crowdsourcing as a tool for research: Implications of uncertainty. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (pp. 1544–1561). https://doi.org/10.1145/2998181.2998197

19.

Lincoln

Y. S.

Guba

E. G.

(2011). Paradigmatic controversies, contradictions, and emerging confluences, revisited. In Denzin

Lincoln

Y. S.

(Eds.), The SAGE handbook of qualitative research (pp. 97–128). Sage.

20.

Linstone

H. A.

Turoff

(2002). The Delphi method ( Linstone

H. A.

Turoff

, Eds.). Addison-Wesley. https://web.njit.edu/~turoff/pubs/delphibook/index.html

21.

Litman

Robinson

Abberbock

(2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433–442. https://doi.org/10.3758/s13428-016-0727-z

22.

McCoy

A. B.

Wright

Rogith

Fathiamini

Ottenbacher

A. J.

Sittig

D. F.

(2014). Development of a clinician reputation metric to identify appropriate problem-medication pairs in a crowdsourced knowledge base. Journal of Biomedical Informatics, 48, 66–72. https://doi.org/10.1016/j.jbi.2013.11.010

23.

Mullen

P. M.

(2003). Delphi: Myths and reality. Journal of Health Organization and Management, 17(1), 37–52. https://doi.org/10.1108/14777260310469319

24.

Okoli

Pawlowski

S. D.

(2005). The Delphi method as a research tool: An example, design considerations and applications. Information and Management, 42(1), 15–29. https://doi.org/10.1016/j.im.2003.11.002

25.

Peer

Brandimarte

Samat

Acquisti

(2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. https://doi.org/10.1016/j.jesp.2017.01.006

26.

Ranard

B. L.

Y. P.

Meisel

Z. F.

Asch

D. A.

Hill

S. S.

Becker

L. B.

Seymour

A. K.

Merchant

R. M.

(2014). Crowdsourcing: Harnessing the masses to advance health and medicine, a systematic review. Journal of General Internal Medicine, 29(1), 187–203. https://doi.org/10.1007/s11606-013-2536-8

27.

Rowe

Wright

(2001). Expert opinions in forecasting: The role of the Delphi technique. In Armstrong

J. S.

(Ed.), Principles of forecasting (Vol. 30, pp. 125–144). Springer. https://doi.org/10.1007/978-0-306-47630-3_7

28.

Salkind

(2010). Encyclopedia of research design. Sage. https://doi.org/10.4135/9781412961288

29.

Sheehan

K. B.

(2018). Crowdsourcing research: Data collection with Amazon’s Mechanical Turk. Communication Monographs, 85(1), 140–156. https://doi.org/10.1080/03637751.2017.1342043

30.

Steinert

(2009). A dissensus based online Delphi approach: An explorative research tool. Technological Forecasting and Social Change, 76(3), 291–300. https://doi.org/10.1016/j.techfore.2008.10.006

31.

Swan

(2012). Crowdsourced health research studies: An important emerging complement to clinical trials in the public health research ecosystem. Journal of Medical Internet Research, 14(2), e46. https://doi.org/10.2196/jmir.1988

32.

Tetroe

J. M.

Graham

I. D.

Foy

Robinson

Eccles

M. P.

Wensing

Durieux

Légaré

Nielson

C. P.

Adily

Ward

J. E.

Porter

Shea

Grimshaw

J. M.

(2008). Health research funding agencies’ support and promotion of knowledge translation: An international study. The Milbank Quarterly, 86(1), 125–155. https://doi.org/10.1111/j.1468-0009.2007.00515.x

33.

Turner-McGrievy

G. M.

Helander

E. E.

Kaipainen

Perez-Macias

J. M.

Korhonen

(2015). The use of crowdsourcing for dietary self-monitoring: Crowdsourced ratings of food pictures are comparable to ratings by trained observers. Journal of the American Medical Informatics Association, 22(e1), e112–119. https://doi.org/10.1136/amiajnl-2014-002636

34.

Wiggins

Crowston

(2011). From conservation to crowdsourcing: A typology of citizen science. In 2011 44th Hawaii international conference on system sciences (pp. 1–10). https://doi.org/10.1109/HICSS.2011.207

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB