This study attempts to advance automated content analysis from consensus-oriented to coordination-oriented practices, thereby embracing diverse coding outputs and exploring the dynamics among differential perspectives. As an exploratory investigation, we evaluate six GPT-4o configurations to analyze sentiment in Fox News and MSNBC transcripts on Biden and Trump during the 2020 U.S. presidential campaign. By assessing each model’s alignment with partisan perspectives, we explore how partisan selective processing can be identified in LLM-Assisted Content Analysis (LACA). The findings indicate that LLM-based partisan perspective simulations reflect politically polarized standpoints across partisan groups, revealing a pronounced divergence in sentiment analysis between Democrat-aligned and Republican-aligned persona models. This pattern is evident in intercoder-reliability metrics, which are higher among same-partisan than cross-partisan persona model pairs. Results also suggest that LLM partisan simulations exhibit stronger ideological biases when analyzing politically congruent content. This approach enhances the nuanced understanding of LLM outputs and advances the integrity of AI-driven social science research and may also enable simulations of real-world implications.
ArgyleL. P.BusbyE. C.FuldaN.GublerJ. R.RyttingC.WingateD. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337–351. https://doi.org/10.1017/pan.2023.2
5.
BabbieE. (2015). Observing ourselves: Essays in social research. Waveland Press.
6.
BailC. A. (2024). Can generative AI improve social science?Proceedings of the National Academy of Sciences of the United States of America, 121(21), Article e2314021121. https://doi.org/10.1073/pnas.2314021121
7.
BenderE. M.GebruT.McMillan-MajorA.ShmitchellS. (2021, March). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). Virtual Event Canada. Publisher: Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922
8.
BenoitW. L. (2014). Content analysis in political communication. In Sourcebook for political communication research (pp. 268–279). Routledge.
9.
BerelsonB. (1952). Content analysis in communication research. Free Press.
10.
CalderonN.ReichartR.DrorR. (2025). The alternative annotator test for LLM-as-a-Judge: How to statistically justify replacing human annotators with LLMs. arXiv preprint arXiv:2501.10970. https://arxiv.org/abs/2501.10970
11.
ChewR.BollenbacherJ.WengerM.SpeerJ.KimA. (2023). LLM-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924. https://arxiv.org/abs/2306.14924
12.
D’AlessioD.AllenM. (2000). Media bias in presidential elections: A meta‐analysis. Journal of Communication, 50(4), 133–156. https://doi.org/10.1093/joc/50.4.133
13.
DemszkyD.YangD.YeagerD. S.BryanC. J.ClapperM.ChandhokS.PennebakerJ. W.HechtC.JamiesonJ.JohnsonM.JonesM.Krettek-CobbD.LaiL.JonesMitchellN.OngD. C.DweckC. S.GrossJ. J. (2023). Using large language models in psychology. Nature Reviews Psychology, 2(11), 688–701. https://doi.org/10.1038/s44159-023-00241-5
14.
GaoC.LanX.LuZ.MaoJ.PiaoJ.WangH.LiY. (2023). S3: Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.1498. https://arxiv.org/abs/2307.14984
15.
GielensE.SowulaJ.LeifeldP. (2025). Goodbye human annotators? Content analysis of social policy debates using ChatGPT. Journal of Social Policy, 1–20. https://doi.org/10.1017/s0047279424000382
16.
GilardiF.AlizadehM.KubliM. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America, 120(30), Article e2305016120. https://doi.org/10.1073/pnas.2305016120
GiorgiT.CimaL.FagniT.AvvenutiM.CresciS. (2024). Human and LLM biases in hate speech annotations: A socio-demographic analysis of annotators and targets. arXiv preprint arXiv:2410.07991. https://arxiv.org/abs/2410.07991.
19.
GlickmanM.SharotT. (2025). How human–AI feedback loops alter human perceptual, emotional, and social judgements. Nature Human Behaviour, 9(2), 345–359. https://doi.org/10.1038/s41562-024-02077-2
20.
GöncüA. (1993). Development of intersubjectivity in social pretend play. Human Development, 36(4), 185–198. https://doi.org/10.1159/000278206
21.
HallS. (1980). Encoding/decoding. In HallS.HobsonD.LoweA.WillisP. (Eds.), Culture, media, language (pp. 128–138). Hutchinson.
22.
HancockJ. T.NaamanM.LevyK. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89–100. https://doi.org/10.1093/jcmc/zmz022
23.
HartmannJ.SchwenzowJ.WitteM. (2023). The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768. https://arxiv.org/abs/2301.01768
24.
HayesA. F.KrippendorffK. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. https://doi.org/10.1080/19312450709336664
25.
HeseltineM.Clemm von HohenbergB. (2024). Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1), 1–10. https://doi.org/10.1177/20531680241236239
26.
HolstiO. R. (1969). Content analysis for the social sciences and humanities. Addison-Wesley.
JanisI. (1965). The problem of validating content analysis. In LasswellH. D.LeitesN. (Eds.), Language of politics (pp. 55–82). MIT.
29.
KathirgamalingamA.LindF.Bernhard-HarrerJ.BoomgaardenH. G. (2024, October 8). Agree to disagree? Human and LLM coder bias for constructs of marginalization. OSF Preprints. https://doi.org/10.31235/osf.io/agpyr
30.
KlingerU.KreissD.MutsvairoB. (2023). Platforms, power, and politics: An introduction to political communication in the digital age. John Wiley & Sons.
31.
KolbeR. H.BurnettM. S. (1991). Content-analysis research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research, 18(2), 243–250. https://doi.org/10.1086/209256
32.
KrippendorffK. (1980). Validity in content analysis. In MochmannE. (Ed.), Computerstrategien für die Kommunikationsanalyse (pp. 69–112). Campus.
33.
KrippendorffK. (1999). Content analysis: An introduction to its methodology (2nd ed). Sage.
34.
KrippendorffK. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. https://doi.org/10.1093/hcr/30.3.411
35.
KrippendorffK. (2018). Content analysis: An introduction to its methodology (4th ed). Sage.
36.
KroonA.WelbersK.TrillingD.van AtteveldtW. (2024). Advancing automated content analysis for a new era of media effects research: The key role of transfer learning. Communication Methods and Measures, 18(2), 142–162. https://doi.org/10.1080/19312458.2023.2261372
37.
LaingR. D.PhillipsonH.LeeA. R. (1966). Interpersonal perception: A theory and method of research. Tavistock Publications.
38.
LatsisJ. (2006). Convention and intersubjectivity: New developments in French economics. Journal for the Theory of Social Behaviour, 36(3), 255–277. https://doi.org/10.1111/j.1468-5914.2006.00307.x
39.
LombardM.Snyder‐DuchJ.BrackenC. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4), 587–604. https://doi.org/10.1093/hcr/28.4.587
ManningB. S.ZhuK.HortonJ. J. (2024). Automated social science: Language models as scientist and subjects (no. 32381). National Bureau of Economic Research.
42.
MatthesJ.KohringM. (2008). The content analysis of media frames: Toward improving reliability and validity. Journal of Communication, 58(2), 258–279. https://doi.org/10.1111/j.1460-2466.2008.00384.x
MesseriL.CrockettM. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002), 49–58. https://doi.org/10.1038/s41586-024-07146-0
45.
MishraA. (2023). AI alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048https://arxiv.org/abs/2310.16048.
46.
MitchellM. (2019). Artificial intelligence: A guide for thinking humans. Pelican.
47.
NeuendorfK. A. (2004). Content analysis: A contrast and complement to discourse analysis. Qualitative Methods, 2(1), 33–36.
OreskesN. (2019). Why trust science?Princeton University Press.
53.
OsgoodC. E. (1959). The representation model and relevant research methods. In de Sola PoolI. (Ed.), Trends in content analysis (pp. 33–88). University of Illinois Press.
54.
ParkJ. S.O’BrienJ.CaiC. J.MorrisM. R.LiangP.BernsteinM. S. (2023, October). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on UserInterface Software and Technology (pp. 1–22). https://doi.org/10.1145/3586183.3606763
PeyréG.CuturiM. (2019). Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning, 11(5–6), 355–607. https://doi.org/10.1561/2200000073
58.
PlattM.PlattD. (2024, October). Effectiveness of generative artificial intelligence for scientific content analysis. In 2023 IEEE 17th International Conference on Application of Information and Communication Technologies (AICT) (pp. 1–4). IEEE.
59.
PopperK. (1978). Three worlds. University of Michigan.
60.
PotterY.LaiS.KimJ.EvansJ.SongD. (2024). Hidden persuaders: LLMs' political leaning and their influence on voters. ArXiv preprint arXiv:2410.24190. https://arxiv.org/abs/2410.24190
61.
PotterW. J.Levine-DonnersteinD. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27(3), 258–284. https://doi.org/10.1080/00909889909365539
RiffeD.LacyS.WatsonB. R.LovejoyJ. (2023). Analyzing media messages: Using quantitative content analysis in research. Routledge.
64.
RommetveitR. (1979). On the architecture of intersubjectivity. In RommetveitR.BlakarR. M. (Eds.), Studies of language, thought, and verbal communication (pp. 93–107). Academic.
65.
SachdevaP. S.BarretoR.von VacanoC.KennedyC. J. (2022, June). Assessing annotator identity sensitivity via item response theory: A case study in a hate speech corpus. In (1585–1603), Seoul, Republic of Korea. Publisher: Association for Computing Machinery. https://doi.org/10.1145/3531146.3533216
66.
SanturkarS.DurmusE.LadhakF.LeeC.LiangP.HashimotoT. (2023, July). Whose opinions do language models reflect? In International Conference on Machine Learning (pp. 29971–30004). PMLR.
67.
SchifferA. J. (2006). Assessing partisan bias in political news: The case (s) of local senate election coverage. Political Communication, 23(1), 23–39. https://doi.org/10.1080/10584600500476981
68.
ShaniC.SofferL.JurafskyD.LeCunY.Shwartz-ZivR. (2025). From tokens to thoughts: How LLMs and humans trade compression for meaning. arXiv preprint arXiv:2505.17117. https://arxiv.org/abs/2505.17117
69.
ShapiroG.MarkoffJ. (1997). A matter of definition. In RobertsC. W. (Ed.), Text analysis for the social sciences: Methods for drawing statistical inferences from texts and transcripts (pp. 9–31). Lawrence Erlbaum.
70.
SmolkaA. L. B.De GoesM. C.PinoA. (1995). The constitution of the subject: A persistent question. In WertschJ. V.Del RioP.AlvarezA. (Eds.), Sociocultural studies of mind (pp. 165–184). Cambridge University Press.
71.
SongH.TolochkoP.EberlJ. M.EiseleO.GreussingE.HeidenreichT.BoomgaardenH. G.GalygaS. (2020). In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis. Political Communication, 37(4), 550–572. https://doi.org/10.1080/10584609.2020.1723752
72.
SorensenT.MooreJ.FisherJ.GordonM.MireshghallahN.RyttingC. M.ChoiY. (2024). A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070. https://arxiv.org/abs/2402.05070
73.
SorokaS.WlezienC. (2022). Information and Democracy. Cambridge University Press.
TörnbergP. (2022). How digital media drive affective polarization through partisan sorting. Proceedings of the National Academy of Sciences of the United States of America, 119(42), Article e2207159119. https://doi.org/10.1073/pnas.2207159119
TrillingD. (2025). LLM-based agents in simulation. In 75th International Communication Association Annual Conference,Denver (CO), US.
78.
VillaniC. (2009). Optimal transport: Old and new (Vol. 338, p. 23). Springer.
79.
WenZ.YounesR. (2024). ChatGPT vs media bias: A comparative study of GPT-3.5 and fine-tuned language models. arXiv preprint arXiv:2403.20158. https://arxiv.org/abs/2403.20158
80.
ZiemsC.HeldW.ShaikhO.ChenJ.ZhangZ.YangD. (2024). Can large language models transform computational social science?Computational Linguistics, 50(1), 237–291. https://doi.org/10.1162/coli_a_00502
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
