Evaluators can work with brief units of text-based data, such as open-ended survey responses, text messages, and social media postings. Online crowdsourcing is a promising method for quantifying large amounts of text-based data by engaging hundreds of people to categorize the data. To further develop and test this method, individuals were recruited through online crowdsourcing to code open-ended survey responses, using a predetermined list of thematic codes that were derived from the responses. The study compared the coding results obtained from online crowdsourcing with coding results obtained from researcher coders. Additionally, the study examined feedback from the crowdsourced coders about their experiences with the task. The results suggested that online crowdsourcing can produce comparable results to researcher coding, but that the comparability of the results may differ across codes. This method may increase the efficiency of quantifying text-based data and provide evaluators with valuable feedback on their coding schemes.
AtkissonC.MonaghanC.BrentE. (2010). Using computational techniques to fill the gap between qualitative data analysis and text analytics. Kwalon, 15, 6–20.
2.
AzzamT.HarmanE. (2016). Crowdsourcing for quantifying transcripts: An exploratory study. Evaluation and Program Planning, 54, 63–73. doi:10.1016/j.evalprogplan.2015.09.002
3.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N. (2011). The viability of crowdsourcing for survey research. Behavioral Research Methods, 43, 800–813. doi:10.3758/s13428-011-0081-0
4.
BenoitK.ConwayD.LauderdaleB. E.LaverM.MikhaylovS. (2016). Crowd-sourced text analysis: Reproducible and agile production of political data. American Political Science Review, 110, 278–295. doi:10.1017/S0003055416000058
5.
ByunT. M.HalpinP. F.SzerediD. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53, 70–83. doi:10.1016/j.jcomdis.2014.11.003
FitzpatrickJ. L. (2012). An introduction to context and its role in evaluation practice. New Directions for Evaluation, 2012, 7–24. doi:10.1002/ev.20024
8.
HarmanE. H. (2016). Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards (Unpublished Dissertation). Claremont Graduate University, Claremont, CA.
9.
HarmanE. H.AzzamT. (in press). Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences. Evaluation and Program Planning.
10.
HarrisJ. K.MartA.Moreland-RussellS.CaburnayC. A. (2015). Diabetes topics associated with engagement on twitter. Preventing Chronic Disease, 12, E62. doi:10.5888/pcd12.140402
11.
KirkhartK. E. (2010). Eyes on the prize: Multicultural validity and evaluation theory. American Journal of Evaluation, 31, 400–413. doi:10.1177/1098214010373645
12.
McAllister ByunT.HalpinP.SzerediD. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53, 70–83. doi:10.1016/j.jcomdis.2014.11.003
13.
NealJ. W.NealZ. P.VanDykeE.KornbluhM. (2014). Expediting the analysis of qualitative data in evaluation: A procedure for the rapid identification of themes from audio recordings (RITA). American Journal of Evaluation, 36, 118–132. doi:10.1177/1098214014536601
14.
PaolacciG.ChandlerJ. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188. doi:10.1177/0963721414531598
15.
PeerE.VosgerauJ.AcquistiA. (2013). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y
16.
RogD. J. (2012). When background becomes foreground: Toward context-sensitive evaluation practice. New Directions for Evaluation, 2012, 25–40. doi:10.1002/ev.20025
17.
SandelowskiM.VoilsC.KnaflG. (2009). On quantitizing. Journal of Mixed Methods Research, 3, 208–222. doi:10.1177/1558689809334210
18.
SmithS. M.RosterC. A.GoldenL. L.AlbaumG. S. (2016). A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal of Business Research, 69, 3139–3148. doi:10.1016/j.jbusres.2015.12.002
19.
SurowieckiJ. (2005). The wisdom of crowds. New York, NY: Anchor Books.
20.
ThomasD. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27, 237–246. doi:10.1177/1098214005283748
21.
Tosti-KharasJ.ConleyC. (2016). Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in Psychology, 7, 1–9.
22.
WeinbergJ.FreeseJ.McElhattanD. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310. doi:10.15195/v1.a19
23.
WhiteD. E.OelkeN. D.FriesenS. (2012). Management of a large qualitative data set: Establishing trustworthiness of the data. International Journal of Qualitative Methods, 11, 244–258.
24.
YinR. K. (2011). Qualitative research from start to finish. New York, NY: The Guilford Press.
