Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim to build evaluator capacity to integrate big data analytics into their studies. We focus our efforts on a specific topic modeling technique referred to as latent Dirichlet allocation (LDA) because of the ubiquitousness of qualitative textual data in evaluation. Given current equity debates, both within evaluation and the communities in which we practice, we analyze 1,796 tweets that use the hashtag #equity with the R packages topicmodels and ldatuning to illustrate the use of LDA. Furthermore, a freely available workbook for implementing LDA topic modeling is provided as Supplemental Material Online.
AroraS.GeR.MoitraA. (2012, October). Learning topic models—Going beyond SVD. In ShmoysD.RoughgardenT. (Eds.), 2012 IEEE 53rd annual symposium on foundations of computer science (pp. 1–10). IEEE.
5.
ArunR.SureshV.MadhavanC. V.MurthyM. N. (2010). On finding the natural number of topics with latent Dirichlet allocation: Some observations. In ZakiM. J.YuJ. X.RavindranB.PudiV. (Eds.), Advances in knowledge discovery and data mining (pp. 391–402). Springer.
6.
BambergerM. (2017). Integrating big data into the monitoring and evaluation of development programmes. UN Global Pulse.
7.
BleiD. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84.
BleiD. M.LaffertyJ. D. (2009). Topic models. In SrivastavaA.SahamiM. (Eds.), Text mining (pp. 71–93). Chapman and Hall/CRC.
10.
BleiD. M.NgA. Y.JordanM. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3(January), 993–1022.
11.
BurscherB. (2016). Machine learning-based content analysis automating the analysis of frames and agendas in political communication research[Unpublished doctoral dissertation]. Department of Social and Behavioral Sciences, University of Amsterdam.
12.
CaoJ.XiaT.LiJ.ZhangY.TangS. (2009). A density-based method for adaptive LDA model selection. Neurocomputing, 72(7–9), 1775–1781.
13.
ChandraY.ShangL. (2017). An RQDA-based constructivist methodology for qualitative research. Qualitative Market Research: An International Journal, 20(1), 90–112.
14.
ChoS. W. (2016). The tension between access and success: Challenges and opportunities for community colleges in the United States. In ShahM.BennettA.SouthgateE. (Eds.), Widening higher education participation (pp. 181–194). Chandos Publishing.
15.
DeveaudR.San JuanE.BellotP. (2014). Accurate and effective latent concept modeling for ad hoc information retrieval. Document Numérique, 17(1), 61–84.
16.
DicleM. F.DicleB. (2018). Content analysis: Frequency distribution of words. The Stata Journal, 18(2), 379–386.
17.
EisensteinJ.O’ConnorB.SmithN. A.XingE. (2010, October). A latent variable model for geographic lexical variation. In Proceedings of the 2010 conference on empirical methods in natural language processing (pp. 1277–1287). https://aclanthology.org/D10-1124.pdf
18.
Fabra-MataJ.MygindJ. (2019). Big data in evaluation: Experiences from using Twitter analysis to evaluate Norway’s contribution to the peace process in Colombia. Evaluation, 25(1), 6–22.
19.
FieldA.MilesJ.FieldZ. (2012). Discovering statistics using R. SAGE.
20.
FiggouL.PavlopoulosV. (2015). Social psychology: Research methods. International Encyclopedia of the Social & Behavioral Sciences, 22, 544–552.
21.
FinucaneM. M.MartinezI.CodyS. (2018). What works for whom? A Bayesian approach to channeling big data streams for public program evaluation. American Journal of Evaluation, 39, 109–122.
22.
FoxJ.WeisbergS. (2011). An R companion to applied regression (2nd ed.). SAGE.
23.
GolinoH.ChristensenA. P.MoulderR. G.JrKimS.BokerS. M. (2020, April). Modeling latent topics in social media using dynamic exploratory graph analysis: The case of the right-wing and left-wing trolls in the 2016 US elections. PsyArXic Preprints. https://psyarxiv.com/tfs7c
24.
GriffithsT. L.SteyversM. (2004). Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl 1), 5228–5235.
25.
HollibaughG. E. (2019). The use of text as data methods in public administration: A review and an application to agency priorities. Journal of Public Administration Research and Theory, 29(3), 474–490.
26.
HornJ. L. (1967). On subjectivity in factor analysis. Educational and Psychological Measurement, 27(4), 811–820.
27.
HornikK.GrünB. (2011). topicmodels: An R package for fitting topic models. Journal of Statistical Software, 40(13), 1–30.
JordanM. I.MitchellT. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260.
30.
KaislerS.ArmourF.EspinosaJ. A.MoneyW. (2013, January). Big data: Issues and challenges moving forward. In SpragueR. H.Jr. (Ed.), 2013 46th Hawaii international conference on system sciences (pp. 995–1004). IEEE.
31.
KatalA.WazidM.GoudarR. H. (2013, August). Big data: Issues, challenges, tools and good practices. In 2013 Sixth international conference on contemporary computing (pp. 404–409). IEEE.
32.
KrippendorffK. (2004). Content analysis: An introduction to its methodology (2nd ed.). SAGE.
33.
LemireS.PeckL. R.PorowskiA. (2020). The growth of the evaluation tree in the policy analysis forest: Recent developments in evaluation. Policy Studies Journal, 48, S47S70.
34.
LetouzéE. (2012). Big data for development: Opportunities and challenges. United Nations Global Pulse.
35.
LetouzéE.AreiasA.JacksonS. (2016). The evaluation of complex development interventions in the age of big data. In BamgergerM.VaessenJ.RaimondoE. (Eds.), Dealing with complexity in development evaluation: A practical approach (pp. 221–250). SAGE Publications.
36.
McGroryC. A. (2012). Variational Bayesian inference for mixture models. In MengersenK. L.RobertC. P.TitteringtonD. M. (Eds.), Case studies in Bayesian statistical modelling and analysis (pp. 388–402). John Wiley & Sons.
37.
MervisJ. (2012). Agencies rally to tackle big data. Science, 336(6077), 22.
38.
MilsteinB. L.WetterhallS. F., & CDC Evaluation Working Group (2000). A framework featuring steps and standards for program evaluation. Health Promotion Practice, 1(3), 221–228.
39.
MurphyK. P. (2012). Machine learning: A probabilistic perspective. MIT.
40.
NeuendorfK. A. (2017). The content analysis guidebook (2nd ed.). SAGE.
41.
NeuendorfK. A. (2019). Content analysis and thematic analysis. In BroughP. (Ed.), Research methods for applied psychologists: Design, analysis and reporting (pp. 211–223). Routledge.
ObeidJ. S.WeedaE. R.MatuskowitzA. J.GagnonK.CrawfordT.CarrC. M.FreyL. J. (2019). Automated detection of altered mental status in emergency department clinical notes: A deep learning approach. BMC Medical Informatics and Decision Making, 19(1), 164.
44.
OussousA.BenjellounF. Z.LahcenA. A.BelfkihS. (2018). Big data technologies: A survey. Journal of King Saud University—Computer and Information Sciences, 30, 431–448.
45.
PapadimitriouC. H.RaghavanP.TamakiH.VempalaS. (2000). Latent semantic indexing: A probabilistic analysis. Journal of Computer and System Sciences, 61(2), 217–235.
46.
PattonM. (2015). Qualitative research & evaluation methods: Integrating theory and practice. SAGE.
47.
PaucaV. P.ShahnazF.BerryM. W.PlemmonsR. J. (2004, April). Text mining using non-negative matrix factorizations. In BerryM. W.DayalU.KamathC.SkillicornD. (Eds.), Proceedings of the 2004 SIAM international conference on data mining (pp. 452–456). Society for Industrial and Applied Mathematics.
48.
PoppingR. (2015). Analyzing open-ended questions by means of text analysis procedures. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 128(1), 2339.
49.
PritchardJ. K.StephensM.DonnellyP. (2000). Inference of population structure using multilocus genotype data. Genetics, 155, 945–959.
50.
R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
51.
RatnerC. (2002). Subjectivity and objectivity in qualitative methodology. Forum: Qualitative Social Research, 3(3), 18.
52.
ReioT. G.JrShuckB. (2015). Exploratory factor analysis: Implications for theory, research, and practice. Advances in Developing Human Resources, 17(1), 1225.
53.
RichardsL. (1999). Using NVivo in qualitative research. SAGE.
54.
RobertsM. E.StewartB. M.AiroldiE. M. (2016). A model of text for experimentation in the social sciences. Journal of the American Statistical Association, 111(515), 988–1003.
55.
RobertsM. E.StewartB. M.NielsenR. A. (2020). Adjusting for confounding with text matching. American Journal of Political Science, 64, 887–903.
56.
Rosen-ZviM.GriffithsT.SteyversM.SmithP. (2004). The author-topic model for authors and documents. In MeekC. (Ed.), Proceedings of the 20th conference on Uncertainty in artificial intelligence (pp 487–494). AUAI Press.
RussellM. A. (2011). Mining the social web: Analyzing data from Facebook, Twitter, LinkedIn, and other social media sites. O’Reilly Media.
59.
SalganikM. J. (2018). Bit by bit: Social research in the digital age. Princeton University Press.
60.
SchoberM. F.ConradF. G. (2015). Improving social measurement by understanding interaction in survey interviews. Policy Insights from the Behavioral and Brain Sciences, 2(1), 211–219.
61.
SchwartzH. A.EichstaedtJ. C.KernM. L.DziurzynskiL.RamonesS. M.AgrawalM.ShahA.KosinskiM.StillwellD.SeligmanM. E. P.UngarL. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS One, 8(9), e73791.
62.
SilgeJ.RobinsonD. (2016). Tidytext: Text mining and analysis using tidy data principles in R. Journal Open Source Software, 1(3), 37.
63.
SilgeJ.RobinsonD. (2017). Text mining with R: A tidy approach. O’Reilly Media.
StevensK.KegelmeyerP.AndrzejewskiD.ButtlerD. (2012, July). Exploring topic coherence over many models and many topics. In Proceedings of the 2012 Joint Conference on empirical methods in natural language processing and computational natural language learning (pp. 952–961). Association for Computational Linguistics.
66.
SteyversM.GriffithsT. (2007). Probabilistic topic models. In LandauerT.McNamaraD.DennisS.KintschW. (Eds.), Latent semantic analysis: A road to meaning. Laurence Erlbaum.
67.
TangJ.MengZ.NguyenX.MeiQ.ZhangM. (2014, January). Understanding the limiting factors of topic modeling via posterior contraction analysis. In International conference on machine learning (pp. 190–198). http://proceedings.mlr.press/v32/tang14
68.
ThomasD. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27, 237–246.
69.
TiptonE.MatlenB. J. (2019). Improved generalizability through improved recruitment: Lessons learned from a large-scale randomized trial. American Journal of Evaluation, 40(3), 414–430.
VaessenJ.LemireS.BefaniB. (2020). Evaluation of international development interventions: An overview of approaches and methods. Independent Evaluation Group, World Bank.
72.
WestlundE.StuartE. A. (2017). The nonuse, misuse, and proper use of pilot studies in experimental evaluation research. American Journal of Evaluation, 38(2), 246–261.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
