Validation is at the heart of methodological discussions about topic modeling. The authors argue that validation based on human reading hinges on distinctive words and readers’ labeling of a topic, and it overlooks the probability of conflicting results from semantically similar models, such as regressions or other methods. This runs counter to the presumption that topic modeling can reveal features of documents that have some measurable association with social aspects outside the text. The authors develop a similar topic identifying procedure to verify that semantically similar solutions yield similar results in further analysis. The authors argue that future validations of topic modeling must consider such procedures.
BadenChristianPipalChristianSchoonveldeMartijnvan der VeldenMariken A. C. G.2022. “Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda.” Communication Methods and Measures16(1):1–18.
2.
BailChristopher A.2016. “Cultural Carrying Capacity: Organ Donation Advocacy, Discursive Framing, and Social Media Engagement.” Social Science & Medicine165:280–88.
3.
BelfordMarkGreeneDerek. 2020. “Ensemble Topic Modeling Using Weighted Term Co-associations.” Expert Systems with Applications161:113709.
4.
BleiDavid M.2012. “Probabilistic Topic Models.” Communications of the ACM55(4):77–84.
5.
BuurmaRachel Sagner. 2015. “The Fictionality of Topic Modeling: Machine Reading Anthony Trollope’s Barsetshire Series.” Big Data & Society2(2).
6.
ChakrabartiParijatFryeMargaret. 2017. “A Mixed-Methods Framework for Analyzing Text Data: Integrating Computational Techniques with Qualitative Methods in Demography.” Demographic Research37(42):1351–82.
7.
ChangJonathanGerrishSeanWangChongBoyd-GraberJordan L.BleiDavid M.2009. “Reading Tea Leaves: How Humans Interpret Topic Models.” Pp. 288–96 in Advances in Neural Information Processing Systems, Vol. 22, edited by BengioY.SchuurmansD.LaffertyJ. D.WilliamsC. K. I.CulottaA.Red Hook, NY: Curran Associates.
8.
ChuangJasonRobertsMargaret E.StewartBrandon M.WeissRebeccaTingleyDustinGrimmerJustinHeerJeffrey. 2015. “TopicCheck: Interactive Alignment for Assessing Topic Model Stability.” Pp. 175–84 in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Denver, CO: Association for Computational Linguistics.
9.
DennyMatthew J.SpirlingArthur. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis26(2):168–89.
10.
DiMaggioPaul. 2015. “Adapting Computational Text Analysis to Social Science (and Vice Versa).” Big Data & Society2(2):1–5.
11.
DiMaggioPaulNagManishBleiDavid. 2013. “Exploiting Affinities between Topic Modeling and the Sociological Perspective on Culture: Application to Newspaper Coverage of U.S. Government Arts Funding.” Poetics41(6):570–606.
12.
EdelmannAchimMohrJohn W.2018. “Formal Studies of Culture: Issues, Challenges, and Current Trends.” Poetics68:1–9.
13.
EdelmannAchimWolffTomMontagneDanielleBailChristopher A.2020. “Computational Social Science and Sociology.” Annual Review of Sociology46(1):61–81.
14.
FligsteinNeilBrundageJonah StuartSchultzMichael. 2017. “Seeing Like the Fed: Culture, Cognition, and Framing in the Failure to Anticipate the Financial Crisis of 2008.” American Sociological Review82(5):879–909.
15.
GoldstoneAndrewUnderwoodTed. 2014. “The Quiet Transformations of Literary Studies: What Thirteen Thousand Scholars Could Tell Us.” New Literary History45(3):359–84.
16.
GrimmerJustinKingGary. 2011. “General Purpose Computer-Assisted Clustering and Conceptualization.” Proceedings of the National Academy of Sciences108(7):2643–50.
17.
GrimmerJustinRobertsMargaret E.StewartBrandon M.2021. “Machine Learning for Social Science: An Agnostic Approach.” Annual Review of Political Science24:395–419. https://doi.org/10.1146/annurev-polisci-053119-015921
18.
GrimmerJustinRobertsMargaret E.StewartBrandon M.2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton, NJ: Princeton University Press.
19.
GrimmerJustinStewartBrandon M.2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis21(3):267–97. https://doi.org/10.1093/pan/mps028
20.
KarellDanielFreedmanMichael. 2019. “Rhetorics of Radicalism.” American Sociological Review84(4):726–53.
21.
KnoxDeanLucasChristopherTam ChoWendy K.2022. “Testing Causal Theories with Learned Proxies.” Annual Review of Political Science25(1):419–41.
22.
KoltcovSergeiKoltsovaOlessiaNikolenkoSergey. 2014. “Latent Dirichlet Allocation: Stability and Applications to Studies of User-Generated Content.” Pp. 161–65 in Proceedings of the 2014 ACM Conference on Web Science, WebSci ’14. New York: Association for Computing Machinery.
23.
MarshallEmily A.2013. “Defining Population Problems: Using Topic Models for Cross-National Comparison of Disciplinary Development.” Poetics41(6):701–24.
24.
MohrJohn W.BailChristopher A.FryeMargaretLenaJennifer C.LizardoOmarMcDonnellTerence E.MischeAnnTavoryIddoWherryFrederick F.2020. Measuring Culture. New York: Columbia University Press.
25.
MohrJohn W.BogdanovPetko. 2013. “Introduction Models: What They Are and Why They Matter.” Poetics41(6):545–69.
26.
MohrJohn W.RobinWagner-PacificiBreigerRonald L.BogdanovPetko. 2013. “Graphing the Grammar of Motives in National Security Strategies: Cultural Interpretation, Automated Text Analysis and the Drama of Global Politics.” Poetics41(6):670–700.
27.
MuñozJohnYoungCristobal. 2018. “We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness.” Sociological Methodology48(1):1–33.
NelsonLaura K.2021. “Cycles of Conflict, a Century of Continuity: The Impact of Persistent Place-Based Political Logics on Social Movement Strategy.” American Journal of Sociology127(1):1–59.
30.
NelsonLaura K.BurkDerekKnudsenMarcelMcCallLeslie. 2021. “The Future of Coding: A Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods.” Sociological Methods & Research50(1):202–37.
31.
Pardo-GuerraJuan Pablo. 2020. “Where Are the Market Devices? Exploring the Links among Regulation, Markets, and Technology at the Securities and Exchange Commission, 1935.” Theory and Society49(2):245–76.
32.
Pardo-GuerraJuan PabloPahwaPrithviraj. 2022. “The Extended Computational Case Method: A Framework for Research Design.” Sociological Methods & Research51(4):1826–67.
33.
QuinnKevin M.MonroeBurt L.ColaresiMichaelCrespinMichael H.RadevDragomir R.2010. “How to Analyze Political Attention with Minimal Assumptions and Costs.” American Journal of Political Science54(1):209–28.
34.
RiegerJonas. 2020. “ldaPrototype: A Method in R to Get a Prototype of Multiple Latent Dirichlet Allocations.” Journal of Open Source Software5(51):2181.
35.
RobertsMargaret E.StewartBrandon M.TingleyDustin. 2016. “Navigating the Local Modes of Big Data: The Case of Topic Models.” Pp. 51–97 in Computational Social Science, edited by AlvarezR. M.Cambridge, UK: Cambridge University Press.
36.
SteyversMarkGriffithsTom. 2007. “Probabilistic Topic Models.” Handbook of Latent Semantic Analysis427(7):424–40.
37.
WilkersonJohnCasasAndreu. 2017. “Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges.” Annual Review of Political Science20:529–44.
38.
YingLuweiMontgomeryJacob M.StewartBrandon M.2022. “Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures.” Political Analysis30(4):570–89.
39.
YoungCristobalHolsteenKatherine. 2017. “Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis.” Sociological Methods & Research46(1):3–40.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
