The growing popularity of generative artificial intelligence (AI) tools presents new challenges for data quality in online surveys and experiments. This study examines participants’ use of large language models to answer open-ended survey questions and describes empirical tendencies in human versus large language model (LLM)-generated text responses. In an original survey of research participants recruited from a popular online platform for sourcing social science research subjects, 34 percent reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants’ LLM use in online research.
AgnewWilliamBergmanA. StevieChienJenniferDazMarkEl-SayedSeliemPittmanJaylenMohamedShakirMcKeeKevin R.. 2024. “The Illusion of Artificial Inclusion.” Pp. 1-12 in Proceedings of the CHI Conference on Human Factors in Computing Systems, arXiv:2401.08572 [cs].
2.
AlvarezR. MichaelLonna RaeAtkesonLevinInesLiYimeng. 2019. “Paying Attention to Inattentive Survey Respondents.” Political Analysis27(2):145-162.
3.
AlveroAJJinsookLeeAlejandraRegla-VargasRenKizilcecThorstenJoachimsAntonioAnthony Lising. 2024.“Large Language Models, Social Demography, and Hegemony: Comparing Authorship in Human and Synthetic Text.” Journal of Big Data11(1):138.
4.
AlveroAJPeñaCourtney. 2023. “AI Sentience and Socioculture.” Journal of Social Computing4(3):205-220.
5.
AndersonBarrett R.ShahJash HemantKreminskiMax. 2024. “Homogenization Effects of Large Language Models on Human Creative Ideation.” Pp. 413-425 in Proceedings of the 16th Conference on Creativity & Cognition, C’C ’24, New York, NY, USA. Association for Computing Machinery.
6.
AngelinaWangMorgensternJamieDickersonJohn P.. 2025. “Large language models that replace human participants can harmfully misportray and flatten identity groups.” Nature Machine Intelligence 7:400-411.
7.
ArgyleLisa P.BusbyEthan C.FuldaNancyGublerJoshua R.RyttingChristopherWingateDavid. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis31(3):337-351.
8.
Arseniev-KoehlerAlina. 2022. “Theoretical Foundations and Limits of Word Embeddings: What Types of Meaning Can They Capture?” Sociological Methods & Research 53(4):1753-1793.
9.
AshokkumarAshwiniHewittLukeGhezaeIsaiasWillerRobb. 2024. “Predicting Results of Social Science Experiments Using Large Language Models.” Available at: https://docsend.com/view/ity6yf2dansesucf
10.
BailChristopher A. 2024. “Can Generative AI Improve Social Science?” Proceedings of the National Academy of Sciences121(21):e2314021121.
11.
BellAndrew M.GiftThomas. 2023. “Fraud in Online Surveys: Evidence From a Nonprobability, Subpopulation Sample.” Journal of Experimental Political Science10(1):148-153.
12.
BenderEmily M.GebruTimnitMcMillan-MajorAngelinaShmitchellShmargaret. 2021. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” Pp. 610-623 in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event Canada. ACM.
13.
BerinskyAdam J.MargolisMichele F.SancesMichael W.. 2014. “Separating the Shirkers From the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science58(3):739-753.
14.
BickAlexanderBlandinAdamDemingDavid J.. 2024. “The Rapid Adoption of Generative AI.” NBER Working Paper 32966.
15.
BisbeeJamesClintonJoshua D.DorffCassyKenkelBrentonLarsonJennifer M.. 2024. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis32(4):401-416.
16.
BoelaertJulienCoavouxSamuelOllionEtiennePetevIvaylo D.PrgPatrick. 2024. “Machine Bias: Generative Large Language Models Have a View of Their Own.” Available at: https://osf.io/r2pnb
17.
BrynjolfssonErikLiDanielleRaymondLindsey. 2023. “Generative AI at Work.” arXiv:2304.11771 [econ, q-fin].
18.
CaoYongZhouLiSEPLeeSeolhwaCabelloLauraChenMinHershcovichDaniel. 2023. “Assessing Cross-Cultural Alignment Between ChatGPT and Human Societies: An Empirical Study.” arXiv:2303.17466 [cs].
19.
ChengMyraPiccardiTizianoYangDiyi. 2023. “CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations.” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing:10853-10875.
20.
ChmielewskiMichaelKuckerSarah C.. 2020. “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results.” Social Psychological and Personality Science11(4):464-473.
21.
ChristoforouEvgeniaDemartiniGianlucaOtterbacherJahna. 2024. “Generative AI in Crowdwork for Web and Social Media Research: A Survey of Workers at Three Platforms.” Proceedings of the International AAAI Conference on Web and Social Media18: 2097-2103.
22.
CliffordScottJeritJennifer. 2016. “Cheating on Political Knowledge Questions in Online Surveys: An Assessment of the Problem and Solutions.” Public Opinion Quarterly80(4):858-887.
23.
CoppockAlexanderMcClellanOliver A.. 2019. “Validating the Demographic, Political, Psychological, and Experimental Results Obtained From a New Source of Online Survey Respondents.” Research & Politics6(1):2053168018822174.
24.
CuskleyChristineSulikJustin. 2024. “The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms.” Perspectives on Psychological Science19(6):891-899.
Dell’AcquaFabrizioMcFowland IIIEdwardMollickEthan R.Lifshitz-AssafHilaKelloggKatherineRajendranSaranKrayerLisaCandelonFrancoisLakhaniKarim R.. 2023. “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality.” Harvard Business School Working Paper 24-013.
27.
DhillonParamveer S.MolaeiSomayehLiJiaqiGolubMaximilianZhengShaochunRobertLionel Peter. 2024. “Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models.” Pp. 1-18 in Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.
28.
DoshiAnil R.HauserOliver P.. 2024. “Generative AI Enhances Individual Creativity but Reduces the Collective Diversity of Novel Content.” Science Advances10(28):eadn5290.
29.
DouglasBenjamin D.EwellPatrick J.BrauerMarkus. 2023. “Data Quality in Online Human-Subjects Research: Comparisons Between MTurk, Prolific, CloudResearch, Qualtrics, and SONA.” PloS one18(3):e0279720.
30.
DurmusEsinNyugenKarinaLiaoThomas I.SchieferNicholasAskellAmandaBakhtinAntonChenCarolHatfield-DoddsZacHernandezDannyJosephNicholasLovittLianeMcCandlishSamSikderOrowaTamkinAlexThamkulJanelKaplanJaredClarkJackGanguliDeep. 2023. “Towards Measuring the Representation of Subjective Global Opinions in Language Models.” arXiv:2306.16388 [cs].
31.
EgamiNaokiFongChristian J.GrimmerJustinRobertsMargaret E.StewartBrandon M.. 2022. “How to Make Causal Inferences Using Texts.” Science Advances8(42):eabg2652.
32.
EpsteinZivHertzmannAaronCreativitythe Investigators of HumanAktenMemoFaridHanyFjeldJessicaFrankMorgan R.GrohMatthewHermanLauraLeachNeilMahariRobertPentlandAlex “Sandy”RussakovskyOlgaSchroederHopeSmithAmy. 2023. “Art and the Science of Generative AI.” Science380(6650):1110-1111.
GrahamMatthew H. 2024. “Detecting and Deterring Information Search in Online Surveys.” American Journal of Political Science68(4):1315-1334.
41.
GrigoropoulouNikolitsaSmallMario L.. 2022. “The Data Revolution in Social Science Needs Qualitative Research.” Nature Human Behaviour6(7):904-906.
42.
GroenendykEricKrupnikovYannaRyanJohn BarryConnorsElizabeth C.. 2025. “Selecting Out of “Politics”: The Self-Fulfilling Role of Conflict Expectation.” American Political Science Review119(1):40-55.
43.
GummerTobiasKunzTanja. 2022. “Relying on External Information Sources When Answering Knowledge Questions in Web Surveys.” Sociological Methods & Research51(2):816-836.
44.
HadlerPatricia. 2023. “The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys.” Sociological Methods & Research54(1):106-139.
45.
HaganJohnRymond-RichmondWenona. 2008. “The Collective Dynamics of Racial Dehumanization and Genocidal Victimization in Darfur.” American Sociological Review73(6):875-902.
46.
HaidtJonathanGrahamJesse. 2007. “When Morality Opposes Justice: Conservatives Have Moral Intuitions that Liberals May Not Recognize.” Social Justice Research20(1):98-116.
47.
HartmannJochenSchwenzowJasperWitteMaximilian. 2023. “The Political Ideology of Conversational AI: Converging Evidence on ChatGPT’s Pro-Environmental, Left-Libertarian Orientation.” arXiv:2301.01768.
48.
HaslamNickLoughnanSteve. 2014. “Dehumanization and Infrahumanization.” Annual Review of Psychology65(1):399-423.
49.
HuttoC.GilbertEric. 2014. “VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text.” Proceedings of the International AAAI Conference on Web and Social Media8(1):216-225.
50.
IyengarShantoLelkesYphtachLevenduskyMatthewMalhotraNeilWestwoodSean J.. 2019. “The Origins and Consequences of Affective Polarization in the United States.” Annual Review of Political Science22: 129-146.
51.
JakeschMauriceBhatAdvaitBuschekDanielZalmansonLiorNaamanMor. 2023. “Co-Writing with Opinionated Language Models Affects Users’ Views.” Pp. 1-15 in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery.
52.
JardinaAshleyPistonSpencer. 2019. “Racial Prejudice, Racial Identity, and Attitudes in Political Decision Making.” in Oxford Research Encyclopedia of Politics.
53.
JardinaAshleyPistonSpencerStephens-DouganLaFleur. n.d.“The influence of the terms we use on measures of racial attitudes.” (Working Paper).
54.
JensenJeffrey L.KarellDanielTanigawa-LauColeHabashNizarOudahMaiFairus Shofia FaniDhia. 2022. “Language Models in Sociological Research: An Application to Classifying Large Administrative Data and Measuring Religiosity.” Sociological Methodology52(1):30-52.
55.
KennedyRyanCliffordScottBurleighTylerWaggonerPhilip D.JewellRyanWinterNicholas J.G.. 2020. “The Shape of and Solutions to the MTurk Quality Crisis.” Political Science Research and Methods8(4):614-629.
56.
KimJunsolLeeByungkyu. 2023. “AI-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys.” arXiv:2305.09620 [cs].
57.
KirkHannah RoseVidgenBertieRottgerPaulHaleScott A.. 2024. “The Benefits, Risks and Bounds of Personalizing the Alignment of Large Language Models to Individuals.” Nature Machine Intelligence6(4):383-392.
58.
KirkHannah RoseYennieJunFilippoVolpinHaiderIqbalEliasBenussiFredericDreyerAleksandarShtedritskiYukiAsano. 2021. “Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models.” Pp. 2611-2624 in Advances in Neural Information Processing Systems, volume 34.
59.
KitturAniketNickersonJeffrey V.BernsteinMichaelGerberElizabethShawAaronZimmermanJohnLeaseMattHortonJohn. 2013. “The Future of Crowd Work.” Pp. 1301-1318 in Proceedings of the 2013 conference on Computer supported cooperative work, San Antonio Texas USA. ACM.
60.
KoeneckeAllisonChoiAnna Seo GyeongMeiKatelyn X.SchellmannHilkeSloaneMona. 2024. “Careless Whisper: Speech-to-Text Hallucination Harms.” Pp. 1672-1681 in The 2024 ACM Conference on Fairness, Accountability, and Transparency, Rio de Janeiro Brazil. ACM.
61.
KoeneckeAllisonNamAndrewLakeEmilyNudellJoeQuarteyMinnieMengeshaZionToupsConnorRickfordJohn R.JurafskyDanGoelSharad. 2020. “Racial Disparities in Automated Speech Recognition.” Proceedings of the National Academy of Sciences117(14):7684-7689.
62.
KozlowskiAustin C.KwonHyunkuEvansJames A.. 2024. “In Silico Sociology: Forecasting COVID-19 Polarization with Large Language Models.” arXiv:2407.11190 [cs].
63.
KozlowskiAustin C.TaddyMattEvansJames A.. 2019. “The Geometry of Culture: Analyzing the Meanings of Class Through Word Embeddings.” American Sociological Review84(5):905-949.
64.
LandesvatterCamilleBauerPaul C.. 2024. “How Valid Are Trust Survey Measures? New Insights From Open-Ended Probing Data and Supervised Machine Learning.” Sociological Methods & Research [Advanced online publication] 00491241241234871.
65.
LeeMinaGeroKaty IlonkaChungJohn Joon YoungShumSimon BuckinghamRahejaVipulShenHuaVenugopalanSubhashiniWambsganssThiemoZhouDavidAlghamdiEmad A.AugustTalBhatAvinashChoksiMadiha ZahrahDuttaSenjutiGuoJin L. C.HoqueMdNaimulKimYewonKnightSimonNeshaeiSeyedParsaShibaniAntonetteShrivastavaDishaShroffLilaSergeyukAgniaStarkJessiStermanSarahWangSitongBosselutAntoineBuschekDanielChangJoseph CheeChenSherolKreminskiMaxParkJoonsukPeaRoyRhoEugenia Ha RimShenZejiangSiangliuluePao. 2024. “A Design Space for Intelligent and Interactive Writing Assistants.” Pp. 1-35 in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.
66.
LeeMinaLiangPercyYangQian. 2022. “CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities.” Pp. 1-19 in CHI Conference on Human Factors in Computing Systems, arXiv:2201.06796 [cs].
67.
LiangWeixinIzzoZacharyZhangYaohuiLeppHaleyCaoHanchengZhaoXuandongChenLingjiaoYeHaotianLiuShengHuangZhiMc FarlandDaniel A.ZouJames Y.. 2024. “Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews.” arXiv:2403.07183 [cs].
68.
ManzaJeffCrowleyNed. 2018. “Class Divisions and Political Attitudes in the 21st Century.” In Handbook of Attitudes, Volume 2: Applications. Routledge.
69.
MartinJohn Levi. 2023. “The Ethico-Political Universe of ChatGPT.” Journal of Social Computing4(1):1-11.
70.
MendelsohnJuliaTsvetkovYuliaJurafskyDan. 2020. “A Framework for the Computational Linguistic Analysis of Dehumanization.” Frontiers in Artificial Intelligence3(55):1-24.
71.
MollickEthan. 2023. “Everyone is Above Average.” Available at: https://www.oneusefulthing.org/p/everyone-is-above-average
72.
MotokiFabioNetoValdemar PinhoRodriguesVictor. 2024. “More Human Than Human: Measuring ChatGPT Political Bias.” Public Choice198(1):3-23.
73.
MuszyskiMarek. 2023. “Attention Checks and How to Use Them: Review and Practical Recommendations.” Ask: Research and Methods32(1):3-38.
74.
NaousTarekRyanMichael J.RitterAlanWXu. 2024. “Having Beer After Prayer? Measuring Cultural Bias in Large Language Models.” arXiv:2305.14456 [cs].
75.
NeumannTerrenceLeeSooyongDe-ArteagaMariaFazelpourSinaLeaseMatthew. 2024. “Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation.” arXiv:2401.16558 [cs].
76.
NoyShakkedZhangWhitney. 2023.“Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science381(6654):187-192.
77.
OllionEtienneShenRubingMacanovicAnaChatelainArnault. 2023. “ChatGPT for Text Annotation? Mind the Hype!” Available at: https://osf.io/preprints/socarxiv/x58kn
78.
OuyangLongWuJeffJiangXuAlmeidaDiogoWainwrightCarroll L.MishkinPamelaZhangChongAgarwalSandhiniSlamaKatarinaRayAlexSchulmanJohnHiltonJacobKeltonFraserMillerLukeSimensMaddieAskellAmandaWelinderPeterChristianoPaulLeikeJanLoweRyan. 2022. “Training Language Models to Follow Instructions with Human Feedback.” arXiv:2203.02155 [cs].
79.
PadmakumarVishakhHeHe. 2023. “Does Writing With Language Models Reduce Content Diversity?” arXiv:2309.05196 [cs].
80.
ParkPeter S.SchoeneggerPhilippZhuChongyang. 2024. “Diminished Diversity-of-Thought in a Standard Large Language Model.” Behavior Research Methods56(6):5754-5770.
81.
PenningtonJeffreySocherRichardManningChristopher. 2014. “Glove: Global Vectors for Word Representation.” Pp. 1532-1543 in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. Association for Computational Linguistics.
82.
PeytchevAndy. 2009. “Survey Breakoff.” Public Opinion Quarterly73(1):74-97.
83.
PotterYujinLaiShiyangKimJunsolEvansJamesSongDawn. 2024. “Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters.” arXiv:2410.24190.
84.
ReynoldsJeremyKincaidReilly. 2024. “It Depends: Platform Dependence, Schedule Control, and Satisfaction Among Gig Workers on Amazon’s Mechanical Turk Platform.” Sociological Forum40(1):65-78.
85.
RobertsMargaret E.StewartBrandon M.TingleyDustinLucasChristopherLuisJetson Leder’GadarianShana KushnerAlbertsonBethanyRandDavid G.. 2014. “Structural Topic Models for Open-Ended Survey Responses.” American Journal of Political Science58(4):1064-1082.
86.
RozadoDavid. 2023. “The Political Biases of ChatGPT.” Social Sciences12(3):148.
87.
SanturkarShibaniDurmusEsinLadhakFaisalLeeCinooLiangPercyHashimotoTatsunori. 2023. “Whose Opinions Do Language Models Reflect?” Pp. 29971-30004 in Proceedings of the 40th International Conference on Machine Learning. PMLR. ISSN: 2640-3498.
88.
SharmaAshishLinInna W.MinerAdam S.AtkinsDavid C.AlthoffTim. 2023. “Human–AI Collaboration Enables More Empathic Conversations in Text-Based Peer-to-Peer Mental Health Support.” Nature Machine Intelligence5(1):46-57.
89.
ShumailovIliaShumaylovZakharZhaoYirenPapernotNicolasAndersonRossGalYarin. 2024. “AI Models Collapse When Trained on Recursively Generated Data.” Nature631(8022):755-759.
90.
Slack. 2024. “The Fall 2024 Workforce Index Shows Executives and Employees Investing in AI, But Uncertainty Holding Back Adoption.” Slack.
91.
SmallMario L. 2024. “The Value of Qualitative Data in Understanding Failure in Prediction.” Proceedings of the National Academy of Sciences121(28):e2409327121.
92.
SmolanskyAdeleCramAndrewRaduescuCorinaZeivotsSandrisHuberElaineKizilcecRene F.. 2023. “Educator and Student Perspectives on the Impact of Generative AI on Assessments in Higher Education.” Pp. 378-382 in Proceedings of the Tenth ACM Conference on Learning Scale, L@S ’23, New York, NY, USA. Association for Computing Machinery.
93.
StoltzDustin S.TaylorMarshall A.. 2021. “Cultural Cartography With Word Embeddings.” Poetics88: 101567.
94.
Tai-SealeMingBaxterSally L.VaidaFlorinWalkerAmandaSitapatiAmy M.OsborneChadDiazJosephDesaiNimitWebbSophiePolstonGregoryHelstenTeresaGrossErinThackaberryJessicaMandviAmmarLillieDustinLiSteveGinGeneenAcharSurajHofflichHeatherSharpChristopherMillenMarleneLonghurstChristopher A.. 2024. “AI-Generated Draft Replies Integrated Into Health Records and Physicians’ Electronic Communication.” JAMA Network Open7(4):e246565.
95.
TaoYanVibergOlgaBakerRyan S.KizilcecRene F.. 2024. “Cultural Bias and Cultural Alignment of Large Language Models.” arXiv:2311.14096 [cs].
96.
TernovskiJohnOrrLilla. 2022. “A Note on Increases in Inattentive Online Survey-Takers Since 2020.” Journal of Quantitative Description: Digital Media2:1-35.
97.
van LoonAustinGoldbergAmirSrivastavaSameer B.. 2024. “Imagined Otherness Fuels Blatant Dehumanization of Outgroups.” Communications Psychology2(1):1-14.
98.
VeselovskyVeniaminRibeiroManoel HortaCozzolinoPhilipGordonAndrewRothschildDavidWestRobert. 2025. Prevalence and Prevention of Large Language Model Use in Crowd Work.” Communications of the ACM 68(2): 42-47.
99.
VeselovskyVeniaminRibeiroManoel HortaWestRobert. 2023. “Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks.” arXiv:2306.07899 [cs].
100.
WeinbergJillFreeseJeremyMcElhattanDavid. 2014. “Comparing Data Characteristics and Results of an Online Factorial Survey Between a Population-Based and a Crowdsource-Recruited Sample.” Sociological Science1: 292-310.
101.
WrightDustinAroraArnavBorensteinNadavYadavSrishtiBelongieSergeAugensteinIsabelle. 2024. “Revealing Fine-Grained Values and Opinions in Large Language Models.” arXiv:2406.19238.
102.
YangKemingBanamahAhmad. 2014. “Quota Sampling as an Alternative to Probability Sampling? An Experimental Study.” Sociological Research Online19(1):56-66.
103.
ZhangSimoneXuJanetAlveroAJ. 2025. Generative AI Meets Open-Ended Survey Responses - Replication Data. Available at: https://osf.io/f7na8/
104.
ZieglerJeffrey. 2022. “A Text-as-Data Approach for Using Open-Ended Responses as Manipulation Checks.” Political Analysis30(2):289-297.
105.
ZuellCorneliaMenoldNataljaKorberSabine. 2015. “The Influence of the Answer Box Size on Item Nonresponse to Open-Ended Questions in a Web Survey.” Social Science Computer Review33(1):115-122.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
