Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into “promptbooks.” We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE while also underscoring risks like non-random error patterns that can bias downstream analyses.
AherGati V.ArriagaRosa I.KalaiAdam Tauman. 2023. “Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies.” Pp. 337–71 in Proceedings of the 40th International Conference on Machine Learning. Honolulu, Hawaii, USA: PMLR.
2.
AlveroA. J.LeeJinsookRegla-VargasAlejandraKizilcecRenéJoachimsThorstenAntonioAnthony Lising. 2024. “Large Language Models, Social Demography, and Hegemony: Comparing Authorship in Human and Synthetic Text.” SocArxiv. doi: 10.31235/osf.io/qfx4a.
3.
ArgyleLisa P.BusbyEthan C.FuldaNancyGublerJoshua R.RyttingChristopherWingateDavid. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis31(3):337–51. doi: 10.1017/pan.2023.2.
4.
BailChristopher A. 2024. “Can Generative AI Improve Social Science?” Proceedings of the National Academy of Sciences121(21):e2314021121. doi: 10.1073/pnas.2314021121.
5.
BaranickAlanaSheelerJimMillerStephen. 2005. Life on the Death Beat: A Handbook for Obituary Writers. Oak Park, IL: Marion Street Press.
6.
BearmanPeterFarisRobertMoodyJames. 1999. “Blocking the Future: New Solutions for Old Problems in Historical Social Science.” Social Science History23(4):501–33. doi: 10.1017/S0145553200021854.
7.
BearmanPeter S.StovelKatherine. 2000. “Becoming a Nazi: A Model for Narrative Networks.” Poetics27(2):69–90. doi: 10.1016/S0304-422X(99)00022-4.
8.
BeltagyIzPetersMatthew E.CohanArman. 2020. “Longformer: The long-document transformer.” arXiv preprint arXiv:2004.05150.
9.
BonikowskiBartLuoYuchenStuhlerOscar. 2022. “Politics as Usual? Measuring Populism, Nationalism, and Authoritarianism in U.S. Presidential Campaigns (1952–2020) with Neural Language Models.” Sociological Methods & Research51(4):1721–87.
10.
BourdieuPierre. 1988. Homo Academicus. Cambridge, UK: Polity Press.
11.
BraunRobertKoopmansRuud. 2010. “The Diffusion of Ethnic Violence in Germany: the Role of Social Similarity.” European Sociological Review26(1):111–23. doi: 10.1093/esr/jcp056.
12.
BraunRobertKoopmansRuud. 2014. “Watch the Crowd: Bystander Responses, Trickle-Down Politics, and Xenophobic Mobilization.” Comparative Political Studies47(4):631–58. doi: 10.1177/0010414013488544.
13.
BroskaDavidHowesMichaelvan LoonAustin. 2024. The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations. OSF preprint: https://doi.org/10.31235/osf.io/j3bnt_v3.
14.
BrownTom B.MannBenjaminRyderNickSubbiahMelanieKaplanJaredDhariwalPrafullaNeelakantanArvindShyamPranavSastryGirishAskellAmandaAgarwalSandhiniHerbert-VossArielKruegerGretchenHenighanTomChildRewonRameshAdityaZieglerDaniel M.WuJeffreyWinterClemensHesseChristopherChenMarkSiglerEricLitwinMateuszGrayScottChessBenjaminClarkJackBernerChristopherMcCandlishSamRadfordAlecSutskeverIlyaAmodeiDario. 2020. “Language Models Are Few-Shot Learners.” arXiv:2005.14165.
15.
BurnhamMichael. 2024. “Stance Detection: a Practical Guide to Classifying Political Beliefs in Text.” Political Science Research and Methods: 1–18. doi: https://doi.org/10.1017/psrm.2024.35.
16.
BythewayBillJohnsonJulia. 1996. “Valuing Lives? Obituaries and the Life Course.” Mortality1(2):219–34. doi: 10.1080/713685833.
17.
CarleyKathleen. 1993. “Coding Choices for Textual Analysis: A Comparison of Content Analysis and Map Analysis.” Sociological Methodology23:75–126. doi: 10.2307/271007.
18.
CarleyKathleen. 1994. “Extracting Culture Through Textual Analysis.” Poetics22(4):291–312. doi: 10.1016/0304-422X(94)90011-6.
19.
CDC. 2001. “HIV and AIDS — United States, 1981–2000.” Morbidity and Mortality Weekly Report50(21):429–56.
20.
ChaeYoungjinDavidsonThomas. 2024. “Large Language Models for Text Classification. From Zero-Shot Learning to Instruction-Tuning”. Working paper.
21.
DagdelenJohnDunnAlexanderLeeSanghoonWalkerNicholasRosenAndrew S.CederGerbrandPerssonKristin A.JainAnubhav. 2024. “Structured Information Extraction from Scientific Text with Large Language Models.” Nature Communications15(1):1418.
22.
DavidsonThomas. 2024. “Start Generating: Harnessing Generative Artificial Intelligence for Sociological Research.” Socius: Sociological Research for a Dynamic World10:23780231241259651. doi: 10.1177/23780231241259651.
23.
DoSaloméOllionÉtienneShenRubing. 2022. “The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy.” Sociological Methods & Research53(3):1167–200.
24.
DubeyAbhimanyuJauhriAbhinavPandeyAbhinavKadianAbhishekAl-DahleAhmadLetmanAieshaMathurAkhil, et al.2024. “The Llama 3 Herd of Models.” arXiv. arXiv:2407.21783.
25.
DunivinZackary. 2024. “Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks.” arXiv. arXiv:2401.15170.
26.
EgamiNaokiHinckMusashiStewartBrandonWeiHanying. 2023. “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models.” Proceedings of the 37th International Conference on Neural Information Processing Systems, Article 3000, 68589-68601. arXiv:2306.04746.
27.
EgamiNaokiHinckMusashiStewartBrandon M.WeiHanying. 2024. “Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses.” Working paper. https://naokiegami.com/paper/dsl_ss.pdf.
28.
FowlerBridgetBielsaEsperança. 2007. “The Lives We Choose to Remember: A Quantitative Analysis of Newspaper Obituaries.” The Sociological Review55(2):203–26. doi: 10.1111/j.1467-954X.2007.00702.x.
29.
FoySteven L.RayRashawn. 2019. “Skin in the Game: Colorism and the Subtle Operation of Stereotypes in Men’s College Basketball.” American Journal of Sociology125(3):730–85.
30.
FranzosiRoberto. 1989. “From Words to Numbers: A Generalized and Linguistics-Based Coding Procedure for Collecting Textual Data.” Sociological Methodology19(1):263–98.
31.
FranzosiRoberto. 1990. “Computer-assisted Coding of Textual Data: An Application to Semantic Grammars.” Sociological Methods & Research19(2):225–57.
32.
FranzosiRoberto. 1994. “From Words to Numbers: a set Theory Framework for the Collection, Organization, and Analysis of Narrative Data.” Sociological Methodology24:105–36.
33.
FranzosiRoberto. 1998. “Narrative as Data: Linguistic and Statistical Tools for the Quantitative Study of Historical Events.” International Review of Social History43:81–104.
34.
GaoChenLanXiaochongLuZhihongMaoJinzhuPiaoJinghuaWangHuandongJinDepengLiYong. 2023. “S3: Social-Network Simulation System with Large Language Model-Empowered Agents.” arXiv. arXiv:2307.14984.
GilardiFabrizioAlizadehMeysamKubliMaël. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences120(30):e2305016120. doi:10.1073/pnas.2305016120.
37.
GoldensteinJanPoschmannPhilipp. 2019. “Analyzing Meaning in Big Data: Performing a Map Analysis Using Grammatical Parsing and Topic Modeling.” Sociological Methodology49(1):83–131. doi: 10.1177/0081175019852762.
38.
GötzFriedrich M.MaertensRakoenLoombaSahilvan der LindenSander. 2024. “Let the Algorithm Speak: How to Use Neural Networks for Automatic Item Generation in Psychological Scale Development.” Psychological Methods29(3):494–518.doi: https://doi.org/10.1037/met0000540.
39.
GrayRobertNeuhoffDavid L.. 1998. “Quantization.” IEEE Transactions on Information Theory44(6):2325–83.
40.
GrimmerJustinRobertsMargaret E.StewartBrandon M.. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton: Princeton University Press.
41.
GrimmerJustinStewartBrandon M.. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis21(3):267–97.
42.
GrishmanRalph. 2019. “Twenty-Five Years of Information Extraction.” Natural Language Engineering25(6):677–92. doi: 10.1017/S1351324919000512.
43.
GrishmanRalph. 2022 “Information Extraction.” Pp. 934–50 in The Oxford Handbook of Computational Linguistics, edited by MitkovR.. London: Oxford University Press.
44.
HortonJohn J. 2023. “Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?” National Bureau of Economic Research. NEBR Working Papers. Working Paper 31122. doi: 10.3386/w31122.
45.
HumeJanice. 2000. Obituaries in American Culture. Jackson: University Press of Mississippi.
46.
HwangJackelynNaikNikhil. 2023. “Systematic Social Observation at Scale: Using Crowdsourcing and Computer Vision to Measure Visible Neighborhood Conditions.” Sociological Methodology53(2):183–216. doi: 10.1177/00811750231160781.
47.
JacobsGillesHosteVéronique. 2022. “SENTiVENT: Enabling Supervised Information Extraction of Company-Specific Events in Economic and Financial News.” Language Resources and Evaluation56(1):225–57. doi: 10.1007/s10579-021-09562-4.
48.
JallinojaRiitta. 2011 “Obituaries as Family Assemblages.” Pp. 78–91 in Families and Kinship in Contemporary Europe: Rules and Practices of Relatedness, edited by JallinojaR.WidmerE. D.. London: Palgrave Macmillan UK.
49.
JiangAlbertSablayrollesAlexandreRouxAntoineMenschArthurSavaryBlancheBamfordChrisChaplotDevendra Singh, et al.2024. “Mixtral of experts.” arXiv. arXiv:2401.04088.
50.
JoyceKellySmith-DoerrLaurelAlegriaSharlaBellSusanCruzTaylorHoffmanSteve G.NobleSafiya UmojaShestakofskyBenjamin. 2021. “Toward a Sociology of Artificial Intelligence: A Call for Research on Inequalities and Structural Change.” Socius: Sociological Research for a Dynamic World7:2378023121999581. doi: 10.1177/2378023121999581.
51.
JurafskyDanielMartinJames H.. 2025. “Information Extraction: Relations, Events, and Time.” Pp. 435–60 in Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, third edition. Edited by D.JurafskyJ. H.Martinhttps://web.stanford.edu/∼jurafsky/slp3/20.pdf.
52.
KarellDanielSachsJeffreyBarrettRyan. 2023. “Synthetic Duality: A Framework for Analyzing Natural Language Generation's Representation of Social Reality.” SocArXiv. https://osf.io/htxa4.
53.
KimJunsolLeeByungkyu. 2024. “AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction.” arXiv. arXiv:2305.09620.
54.
KnightCarly. 2022. “When Corporations Are People: Agent Talk and the Development of Organizational Actorhood, 1890–1934.” Sociological Methods & Research51(4):1634–80. doi: 10.1177/00491241221122528.
55.
KoleckTheresa A.DreisbachCaitlinBournePhilip E.BakkenSuzanne. 2019. “Natural Language Processing of Symptoms Documented in Free-Text Narratives of Electronic Health Records: A Systematic Review.” Journal of the American Medical Informatics Association26(4):364–79. doi: 10.1093/jamia/ocy173.
56.
KozlowskiAustinEvansJames A.. 2024. “Simulating Subjects: The Promise and Peril of AI Stand-Ins for Social Agents and Interactions.” OSF preprint: https://doi.org/10.31235/osf.io/vp3j2.
57.
KozlowskiAustinKwonHyunkuEvansJames A.. 2024. “In Silico Sociology: Forecasting COVID-19 Polarization with Large Language Models.” ArXiv preprint: arXiv:2407.11190.
58.
KrippendorffKlaus. 2019. Content Analysis: An Introduction to Its Methodology. CA: Sage Publications, Inc.
59.
LawTinaRobertoElizabeth. 2024. “Generative Multimodal Models for Social Science: an Application with Satellite and Streetscape Imagery.” SocArXiv preprint: https://doi.org/10.31235/osf.io/6jq32.
60.
LesterBrianAl-RfouRamiConstantNoah. 2021. “The power of scale for parameter-efficient prompt tuning.” arXiv preprint arXiv:2104.08691.
LongGary L. 1987. “Organizations and Identity: Obituaries 1856-1972.” Social Forces65(4):964. doi: 10.2307/2579019.
65.
MaYuboCaoYixinHongYong ChingSunAixin. 2023. “Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 10572–10601.
66.
MakoviKingaHagenRyanBearmanPeter. 2016. “The Course of Law: State Intervention in Southern Lynch Mob Violence 1882–1930.” Sociological Science3:860–88. doi: 10.15195/v3.a37.
67.
MartinJohn Levi. 2000. “What Do Animals Do All Day?: The Division of Labor, Class Bodies, and Totemic Thinking in the Popular Imagination.” Poetics27(2–3):195–231. doi: 10.1016/S0304-422X(99)00025-X.
68.
McFarlandDaniel A.WolffTom. 2022. “Writing into Relationships.” Social Networks71:96–114.
69.
MischeAnnPattisonPhilippa. 2000. “Composing a Civic Arena: Publics, Projects, and Social Settings.” Poetics27(2):163–94. doi:10.1016/S0304-422X(99)00024-8.
70.
MohrJohn W. 1994. “Soldiers, Mothers, Tramps and Others: Discourse Roles in the 1907 New York City Charity Directory.” Poetics22(4):327–57. doi: 10.1016/0304-422X(94)90013-2.
71.
MohrJohn W.DuquenneVincent. 1997. “The Duality of Culture and Practice: Poverty Relief in New York City, 1888-1917.” Theory and Society26(2/3):305–56.
72.
MohrJohn W.LeeHelene K.. 2000. “From Affirmative Action to Outreach: Discourse Shifts at the University of California.” Poetics28(1):47–71. doi: 10.1016/S0304-422X(00)00024-3.
73.
MohrJohn W.Wagner-PacificiRobinBreigerRonald L.BogdanovPetko. 2013. “Graphing the Grammar of Motives in National Security Strategies: Cultural Interpretation, Automated Text Analysis and the Drama of Global Politics.” Poetics41(6):670–700. doi: 10.1016/j.poetic.2013.08.003.
74.
NaikNikhilKominersScott DukeRaskarRameshGlaeserEdward L.HidalgoCésar A.. 2017. “Computer Vision Uncovers Predictors of Physical Urban Change.” Proceedings of the National Academy of Sciences114(29):7571–76. doi: 10.1073/pnas.1619003114.
OllionÉtienneShenRubingMacanovicAnaChatelainArnault. 2024. “The Dangers of Using Proprietary LLMs for Research.” Nature Machine Intelligence6(1):4–5. doi: 10.1038/s42256-023-00783-6.
77.
PalmerAlexisSmithNoah A.SpirlingArthur. 2024. “Using Proprietary Language Models in Academic Research Requires Explicit Justification.” Nature Computational Science4(1):2–3. doi: 10.1038/s43588-023-00585-1
78.
ParkJoon SungO’BrienJosephCaiCarrie JunMorrisMeredith RingelLiangPercyBernsteinMichael S.. 2023 “Generative Agents: Interactive Simulacra of Human Behavior.” Pp. 1–22 in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ‘23. New York, NY, USA: Association for Computing Machinery.
79.
PlauthBenjaminNguyenKhanh XuanTrinhTu. 2024. “Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A.” arXiv:2402.13213v2.
80.
RenzeMatthewGuvenErhan. 2024. “The effect of sampling temperature on problem solving in large language models.” arXiv preprint arXiv:2402.05201.
81.
SmithTammy. 2007. “Narrative Boundaries and the Dynamics of Ethnic Conflict and Conciliation.” Poetics35(1):22–46. doi: 10.1016/j.poetic.2006.11.001.
82.
SpirlingArthur. 2023. “Why Open-Source Generative AI Models Are an Ethical Way Forward for Science.” Nature616(7957):413–413. doi:10.1038/d41586-023-01295-4.
83.
StuhlerOscar. 2021. “What’s in a Category? A New Approach to Discourse Role Analysis.” Poetics88:101568. doi:10.1016/j.poetic.2021.101568.
84.
StuhlerOscar. 2022. “Who Does What to Whom? Making Text Parsers Work for Sociological Inquiry.” Sociological Methods & Research51(4):1580–633.
85.
StuhlerOscar. 2024. “The Gender Agency Gap in Fiction Writing (1850 to 2010).” Proceedings of the National Academy of Sciences121(29):e2319514121. doi: 10.1073/pnas.2319514121.
86.
StuhlerOscarOllionEtienneSchultzÉmilienTonCat Dang. 2025. “Replication Materials for: From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models.” SocArXiv. doi: 10.17605/OSF.IO/HWUVS.
87.
TillyCharles. 1995. Popular Contention in Great Britain, 1758-1834. Cambridge, MA: Harvard University Press.
88.
TillyCharles. 1997. “Parliamentarization of Popular Contention in Great Britain, 1758-1834.” Theory and Society26(2/3):245–73.
89.
TillyCharles. 2002. “Event Catalogs as Theories.” Sociological Theory20(2):248–54. doi: 10.1111/1467-9558.00161.
90.
TörnbergPetter. 2023. “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.” arXiv. arXiv:2304.06588.
WangDan J.SouleSarah A.. 2012. “Social Movement Organizational Collaboration: Networks of Learning and the Diffusion of Protest Tactics, 1960-1995.” American Journal of Sociology117(6):1674–1722. doi:https://doi.org/10.1086/664685.
93.
WangDan J.SouleSarah A.. 2016. “Tactical Innovation in Social Movements: The Effects of Peripheral and Multi-Issue Protest.” American Sociological Review81(3):517–48.
94.
WangYanshanWangLiweiRastegar-MojaradMajidMoonSungrimShenFeichenAfzalNaveedLiuSijiaZengYuqunMehrabiSaeedSohnSunghwanLiuHongfang. 2018. “Clinical Information Extraction Applications: A Literature Review.” Journal of Biomedical Informatics77:34–49. doi: 10.1016/j.jbi.2017.11.011.
95.
WeiJasonWangXuezhiSchuurmansDaleBosmaMaartenXiaFeiChiEdLeQuoc V.ZhouDenny. 2022. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Advances in Neural Information Processing Systems35:24824–37. arXiv:2201.11903.
96.
WhiteJulesFuQuchenHaysSamSandbornMichaelOleaCarlosGilbertHenryElnasharAshrafSpencer-SmithJesseSchmidtDouglas C.. 2023. “A prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.” arXiv:2302.11382.
97.
ZhuXunyuLiJianLiuYongMaCanWangWeiping. 2024. “A Survey on Model Compression for Large Language Models.” arXiv:2308.07633.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
