Abstract
Large language models (LLMs) can reproduce a wide variety of rhetorical styles and generate text that expresses a broad spectrum of sentiments. This capacity, now available at low cost, makes them powerful tools for manipulation and control. We draw attention to this phenomenon as an instance of a broader restructuring of computational infrastructures that introduce novel mechanisms for power. In this paper we consider four types of power made possible by the rapid and largely unregulated adoption of LLMs. These include the power to: (a) pollute and uniformize information environments, (b) persuade users via conversational interfaces (e.g., via ‘AI personas’), (c) create novel computational models of human agents (e.g., ‘silicon subjects’) and (d) create novel computational models of human agent populations (e.g., ‘silicon societies’). We draw attention to Meta's ‘Cicero’ model as a proof of concept for how such techniques can be used to produce controllable and steerable strategic dialogue models. We draw these strands together to argue that coordinated use of such techniques make LLM-based systems powerful instruments for the exertion, modulation, and projection of power. We situate these novel expressions of power in relation to ambitions to establish a new generative foundation for computing.
Introduction
In 2023, conveners of the International Conference on Machine Learning (ICML) banned the submission of text produced entirely via large language models (LLMs). Their ban did not prevent ‘editing or polishing author-written text’ with such tools (ICML Program Chairs, 2023). This exclusion suggests that even in a setting in which experts deem LLMs to be a risk, little is made of their use in crafting composition and rhetoric. We challenge that assumption herein by surveying several ways in which the use of LLMs to alter writing styles could be abused, particularly in areas of sustained human–machine interaction. Sustained interactions tend to be understood through the lens of user prompts. We add to this that LLMs prompt
In line with Edwards’ account of techno-politics, we focus on how LLM's ensconcement in global advertising infrastructures might co-construct its stabilization into social worlds (Edwards, 2003), and deepen the longstanding hegemony of seemingly ‘participatory’ network logics (Chun and Barnett, 2021; Han, 2017; Mejias, 2013). While AI safety research has asked how AI might deceive its human operators (Brundage et al., 2018; Carroll et al., 2023; Hagendorff, 2024), it has also tended to mistake such persuasive capabilities as a bug rather than as a feature. That the development of LLMs is deeply ensconced with the prerogatives of the advertising industry (Meredith, 2021) raises significant questions over their proposed use across public life (Marlin, 2002: 13; Zollmann, 2017). Existing scholarship shows that the dissemination of deceptive content and disinformation is incentivized by the market structures within which digital platforms operate (Diaz Ruiz, 2023). Our aim is to connect this market backdrop, elsewhere called the Intention Economy (Chaudhary and Penn, 2024), to the technical novelties outlined herein, which entail but also eclipse what Louis Rosenberg has called ‘the AI Manipulation Problem’ (Rosenberg, 2023a, 2023b). While Weidinger et al. (2022) frame related concerns as LLM's ‘observed’ and ‘anticipated’ risks, they do so on behalf of a multinational corporation that is
To lend structure to this emerging research area, we connect LLMs novel technological possibilities to the ever-steeper valuations of their developers. In the same month as the ICML ban, LLM-developer OpenAI was valued at an estimated USD 19–29 billion, ranking it one of the most valuable start-ups in America (Kruppa, 2023). As of 2025, it was valued at 340 billion (Rooney and Field, 2025). The basis of this valuation may be understood by considering the plans of its first industrial benefactor Microsoft, which in February 2023 announced that it intends to create ‘new value for advertisers’ via the integration of LLMs into its Bing search engine. Microsoft aspires to re-architect computing infrastructures to make LLMs, foundation models and generative AI essential components of computing platforms and services. At the first OpenAI developer conference, Satya Nadella remarked that the rise of LLMs is prompting them to re-think ‘the system, all the way from thinking from power to the DC to the rack, to the accelerators, to the network’ (OpenAI DevDay, 2023).
One incentive for re-architecting computing infrastructures around LLMs is that it repositions which platforms serve as
Large language models as instruments of power: Techniques for the projection of power and control
LLMs can reproduce a wide variety of rhetorical styles and generate text that expresses a broad spectrum of sentiments. This capacity, now available at low cost, makes them powerful tools for rhetoric and persuasion. In this section, we identify four ways in which humans can exert power over others using LLMs:
‘Pollution and Uniformity’ – Power over information environments as a whole. ‘Natural language interfaces’ – Power over a direct site of interaction between a single human and an LLM system (e.g.. ChatGPT, AI personas). ‘Silicon Samples’ – Power gained via a proxy of the user trialled in predictive models and simulations, including to assess, mirror, and/or control user intent contextually (e.g. Cicero). ‘Silicon Societies’ – Power gained via multiple proxies operating within and/or around a limited social sphere (e.g. within an online community containing interactions between multiple human and machine agents, or a simulation of an office space, voting district, transit system, etc.)
These four areas are neither mutually exclusive nor wholly distinct – indeed, it is their coordination and combination that concerns us. To illustrate that these scenarios are not hypothetical, we end this section by considering how Cicero represents an important proof-of-concept for the strategic control and steering of human subjects in digital environments. Before considering such overlaps, however, we will introduce each of the four areas in turn.
LLMs and the digital information environment as a whole: Pollution and uniformization of content and experience
From the 1990s to the mid-2000s, the internet was populated largely by media created by human contributors. While synthetic media has long been a concern online, generative AI and LLMs are increasingly becoming key instruments through which new content is generated or through which content is channelled for reasons such as summarization or stylization. As a fundamentally unstable form of software capable of behaviours that cannot yet be formally verified 1 (Sommart, 2023: 6) or ‘explained’ using established approaches from mathematics, philosophy or computer science, LLMs deepen our dependencies on ‘incomplete’ (Bommasani et al., 2022: 1, 7) computational instrumentation with unknown failure modes and outputs. Poor training data exacerbates this problem, as does the possibility that ‘defects of the foundation model are inherited by all the adapted models downstream’ (Bommasani et al., 2022: 1). We focus here on the ‘pollution’ and countervailing ‘uniformization’ of semantic content in informational environments exposed to such systems.
Pollution
LLMs can be used to produce large quantities of intelligible text at low cost. Journalists and scholars alike now question the risk of language models ‘poisoning web search’ (Chiang, 2023; Knight, 2023) with ‘semantic garbage’ (Floridi and Chiriatti, 2020). Scholars equate this capacity to pollution because it compromises the veracity of critical information channels. A well-known example is Google search directing users to the adhesion of cheese to pizza using glue (Robison, 2024).
In principle, each of the four central techniques we profile herein could be compromised in this way, propagating information environments riddled with errors and adulterations. The gradual accretion of semantic garbage may be regarded as a form of slow violence (Nixon, 2011) that vandalizes access to beneficial information by poisoning the information environment as a whole. This ‘pollution’ phenomenon is not constrained to the internet. The applied machine learning researcher John Nay demonstrates a proof-of-concept for the use LLMs in corporate lobbying. Nay's model sifts through US Congressional bills for relevance to specific companies. Where a bill is found to be relevant, the model (GPT series) drafts a persuasive letter to its congressional sponsor to make changes to the proposed legislation in favour of the company. According to Nay, AI lobbying may lead to a slow drift in information on values held by citizens, such that policies cease to reflect their preferences in favour of corporations or other powers (Nay, 2023).
At worst, the capacity to generate large quantities of intelligible text at low cost could actualize the unending Library of Babel described in the short story of Jorge Luis Borges, which contains, ‘the translation of every book in all languages, the interpolations of every book in all books’ (Jorge Luis Borges, 1941). The sheer size of this library makes it impossible to read or learn anything meaningful because the most precious books are forever out of reach. While feelings of ‘information overload’ long predate the digital era (Blair, 2010), the low-cost production of generative media threatens to add industrial scale to digital obfuscation, burying human-made content in an avalanche of generative media. Deepening this problem is the prospect that LLMs are simultaneously positioned as the tools needed to
Uniformity
If ‘pollution’ captures how LLMs can be used to diffuse information environments, ‘uniformity’ or ‘standardization’ capture how LLMs can be used to assign
History makes prior efforts in this direction legible in ways that present-day critiques might easily overlook. Historian of science Lorraine Daston attributes successful efforts to conform public experience over past centuries to European empires’ enforcement of standardized rules, including rules for language, dress and etiquette (Daston, 2022). Soviet Russification efforts provide another example. The linguist Uwe Poerksen relates global uniformity of experience today to the politics of language during the French revolution. Poerksen describes how ‘information’, ‘development’, ‘progress’ and ‘communication’ have been shorn of their precise meaning in scientific lexicon and repurposed with a high degree of
Importantly, such words are plastic in another sense – that of the plastic Lego blocks used to build structures. LLMs provide the componentry for a wholesale re-ordering of the built environments that structure our perception of reality. ‘Amorphous plastic words are the elemental building blocks of the industrial state’, Poerksen argues, and are used by politicians, bureaucrats, consultants, industrialists, academics and others as ‘ciphers’ to ‘clear the way for operations on a grand scale’ (1995: 5). Poerksen positions the digital computer as ‘the consequence’ and complimentary expression of centuries-long campaigns to make language uniform in bureaucratic, corporate and political texts (Poerksen, 1995: 91). Equally we might say that LLMs are the
Direct human-LLM interaction with individuals: Persuasion through choice architectures and conversational interfaces
Having considered how LLM-based pollution and uniformity might alter digital environments as a whole, let us now examine a set of related changes for individual users. Our focus here is on persuasive technologies, a field that aims to design tools that change attitudes and influence behaviours of users, not only via persuasion through text, but also via interfaces that entice users toward designers’ preferred choices and outcomes within a predetermined ‘choice architecture’ (Yeung, 2017: 120). While analogous approaches on social media predate LLMs (Stella et al., 2018), the latter provides novel means to dynamically re-order and re-structure choice architectures in online environments, both in single sessions and between sessions. They do so by automating the personalization and persuasiveness of interfaces, as well as by persuading more literally through the use of conversational agents. Mills and Sætra describe this as ‘autonomous choice architects’ (2024).
The advent of ‘AI personas’ offers one early example of conversational interfaces becoming a vehicle for persuasion. The concept is a key ambition for Meta Platforms, Inc., which in 2023 released a series of products that integrate generative persona interactions into WhatsApp and Instagram (Meta, 2023). In this product category, the manipulative capabilities of an LLM interface can be augmented both by the generation of persuasive text, and/or by masquerading in the persona of someone who may be considered a trustworthy source by the user. Consider, for example, a mother's lawsuit against Character.ai for ‘manipulating [her son]… into taking his own life’ (Montgomery, 2024). Character.ai offers AI girlfriends and other personas.
In the case of persuasive text, there are now several methods for altering the style of text in ways that may make it more persuasive, such as fine-tuning models with style specific corpora or by refining system prompts. Another emerging method draws on the field of mechanistic interpretability to identify and target the activation of ‘style-specific neurons’ that are associated with a higher probability of generating certain styles of text in terms of characteristics such as formality, toxicity, politeness, sentiment and so on. Lai et al. steer LLMs to generate certain styles by manipulating artificial neuron activations so as to create ‘style layers’ which impact the probability distribution of generated words disproportionately to influence the overall style of the generated text (Lai et al., 2024). Such techniques are likely to become more sophisticated through more extensive study of the computational mechanisms underlying model behaviour at different training steps and model scales (Xiao et al., 2024: 8).
In the case of personas, personalized chatbots operating under the guise of a trustworthy source may be used to persuade users on behalf of commercial or political campaigns in a one-to-one setting (Goldstein et al., 2023: 25). A new body of literature explores the theoretical foundations of persona simulation using LLMs (Jiang et al., 2023; Serapio-García et al., 2023). Serapio-García et al. attempt to provide an empirical framework to quantify LLM personality through the same psychometric methods that are applied to humans. They argue that such quantification paves the way for personas to be ‘verifiably shaped along desired dimensions’, wherein ‘shaping’ involves mechanisms to modulate (e.g., increase or decrease) ‘levels of specific LLM personality traits’ (2023: 14). They position this research as an exercise in translation, one that re-directs ‘established measurement theory from quantitative social science and psychological assessment’ to the study of LLMs and toward, ultimately, a ‘science of LLMs’.
Serapio-García et al. also draw attention to potential misuses of LLMs personas. They note the possibility of personality matching (e.g., matching content to a user's personality), which has been found to be highly effective at enhancing levels of persuasiveness and influence (Matz et al., 2017; Tapus et al., 2008). Interestingly, the authors find that small optimized LLMs are also able to reproduce complex personality profiles, an advent that reduces the computational resources needed for the mass, low-cost deployment of LLM-based agents with synthetic personalities. Documents leaked from Meta in 2025 suggest the company is training its conversational AI agents to send unsolicited and unprompted follow-up messages based on information recorded from earlier interactions (Webb and Goel, 2025). In this way, individuals are hailed into interactions with computational apparatuses and interpellated (Althusser, 1971: 84) as computational subjects, thus setting the ground for the operations of power.
We add to this that the manipulative and persuasive capabilities of LLMs do not need to be manifested through conversational agents to be effective, such as by positioning an LLM as the spokesperson for a certain brand. The integration of LLMs into real-time bidding (RTB) advertising networks, for instance, provides new opportunities for influencing users’ thoughts, opinions and preferences via the reconfiguration of user interfaces as generative interfaces, allowing for third-party content to be inserted within a feed as discrete items or within the body of text on a token-by-token basis based on signals of user intent and context.
In this section, we have introduced several ways in which LLMs may be used to generate persuasive content at the level of the text itself, via autonomous choice architectures, and by exploiting the tendency of humans to anthropomorphize computational agents, such as through AI personas. Next, we consider mechanisms of influence in the co-production of text when LLMs are used for predictive composition.
Targeted human–LLM interaction using mirroring and steering: Interposing an LLM-agent as a proxy for the user in predictive modelling and simulation
In this section we explore a heightened set of examples of LLM-based persuasion. Specifically, we explore how existing predictive modelling and simulation techniques can be used to be more deliberately persuasive than the techniques covered in the previous section. An LLM-system might, for instance, be purposed as a predictive model that is fine-tuned on the user's own writing such that its outputs are aligned to the writing style of the user, tempting out a sense of familiarity. In this way, LLM's use in suggesting composition options may subtly undermine a user's agency by appearing to provide suggestions that the user
Mirroring and steering dialogue through infiltration
In human relationships, the use of obsequious behaviour to gain advantage is called sycophancy. The advent of conversational agents emulating sycophancy raises questions about links between language and thought. Theorists have advanced that ‘language production and comprehension are tightly interwoven’ (Pickering and Garrod, 2013: 329; Hancock et al., 2020: 93). The questions we consider here pertain to whether predictive compositions influence linguistic expressions exclusively, or whether there is a deeper level of influence that shapes what a user actually thinks in the process of writing. Each instance merits closer study. Here, we survey various modes of interaction whereby the user may voluntarily discard their own manner of self-expression of their thoughts for that of the LLM.
In 2023, researchers demonstrated that an LLM may be repurposed for predictive composition (Jakesch et al., 2023). They designed an experimental system that uses a pre-configured LLM to generate tokens after a user pauses writing. We anticipate that a more advanced system could employ an active predictive model that prompts suggestions rather than waiting for the user to pause. Such a system could update itself dynamically to be a few words ahead of the human writer or speaker. As a simple analogy, imagine auto-complete text designed to persuade its user to write in a certain way. What we wish to highlight here is that if what someone will say next in a conversation can be anticipated with a high degree of confidence, it becomes possible to manipulate the overall flow of that conversation. Research that pre-dates the major developments in LLMs shows that the more a user follows predictive suggestions, the more predictable the content of what they are writing becomes (Arnold et al., 2020: 128).
Steering via ‘snap to grid’ thinking
A related scenario is for the user to simply accept a complete passage of text generated by an LLM instead of their own writing. ‘Predictive’ completion hinges on the willingness of individuals to surrender their own thoughts to that of a pre-empting system. Crucially, the surrender of thought opens the way to the steering of thought, which may be enticed by the integration of novel systems for controlling LLM outputs such as the AI agent, Cicero, as we will return to. Studies in human–computer interaction have shown how agency is undermined by psychological techniques such as computer-assisted movement on screen. One experiment involving a point-and click task with varying levels of computer assistance shows that, up to a certain level of computer assistance, users maintained a sense of agency over their actions (Coyle et al., 2012: 2030–2034). In principle, this influence can be modulated in real-time to express certain sentiments or even to align the content of prose to specific political or ideological axes by making suggested text visible in line with manual composition. The modulated production of text as it is being composed may be viewed as analogous to the ‘snap to grid’ mechanism that appears in graphic design applications. However, instead of snapping to a visual grid, ideas, attitudes, and beliefs expressed through text are snapped along various axes in high dimensional spaces captured by neural networks.
Here, we note that there is an important capacity to modulate outputs that arises from combining the mass production of synthetically generated text (e.g., text via models trained on prior human composition) with fresh text of purely human compositions
Archetype mining: Sampling from LLMs to simulate ‘silicon subjects’
This sub-section surveys concurrent developments in the computational social sciences (Argyle et al., 2023), political science (Ornstein et al., 2023) and economics (Nay, 2023), as well as psychology (Demszky et al., 2023), which together provide methods for experimenting with LLMs as a novel kind of scientific object within these disciplines. These new methods enable scientists to conduct social science studies on populations of synthetic individuals generated through prompts. Without LLMs, this type of study would require tremendous resources and research time on
Argyle et al. have introduced the concept of ‘silicon subjects’ to the social sciences based on an insight into the nature of LLMs that runs contrary to the prevailing discourse on ‘algorithmic bias’ that positions bias as a deficiency to be mitigated. In contrast, they argue that biases are ‘demographically correlated’ such that LLMs can be used to reproduce ‘response distributions’ from human subgroups, and are thus capable of standing in as ‘surrogates for human respondents in a variety of social science tasks’ (Argyle et al., 2023: 337). Argyle et al. describe this ability to treat algorithmic bias as a proxy for demographics as ‘algorithmic fidelity’. Algorithmic fidelity naturalizes the idea that there is value in conducting a census on ‘silicon subjects’, since data on users’ speech characteristics, for example, can be used to understand, map, and relate different clusters of a given population.
Practitioners in the social sciences and in industry have begun to adopt this approach, often uncritically. Horton explores using LLMs as ‘simulated economic agents’ and argues that such agents represent ‘implicit computational models of humans’ (Horton, 2023). In a pre-print paper, he argues that GPT-3 level models (and above) have introduced a categorically new type of experimental tool for studying computational models of human subjects. Elsewhere, Brand et al. propose to use LLMs for market research, such as in marketing and pricing strategies, to elicit willingness-to-pay measurements from GPT-3 responses, which can serve as a proxy for consumer preferences (Brand et al., 2023). Outside of academia, industrial actors reference this new area of research when commercializing analogous claims. The Portuguese start-up Synthetic Users professes it can ‘accelerate’ user and customer research activities via LLMs tested against samples of marketing text and they appeal to Argyle at al. as providing the empirical foundations for their product placement services (Synthetic Users, n.d.).
The scientific basis of these prospects is unclear. Park et al. identify issues in the replicability of LLM-as-proxy studies, such as the ‘correct answer’ effect. This effect refers to a tendency of the GPT 3.5 model to answer subjective and nuanced survey questions as though there is only one correct answer (PS Park et al., 2023b). Eric Chu et al., who use Google's BERT model to predict political opinion from different patterns of media consumption, caution that whilst language models can predict public opinion and human survey responses, their results do not imply that human participants or surveys can be substituted by AI models (Chu et al., 2023: 9). Instead, they, alongside others such as Rosenbusch et al., argue that LLMs may provide a reference for how useful certain types of real-world studies may be, or serve as, a creative tool for planning research and generating hypotheses (Rosenbusch et al., 2023: 17).
As mentioned, whether or not such functions are scientific is a different matter from whether they will be used. Estimates by the Alan Turing Institute in 2024 suggest that ‘a [disinformation] campaign that would cost around US$4500 using traditional methods could be achieved using a commercial, state-of-the-art LLM for less than US$1’ (Williams, 2024; Williams et al., 2024). The cost of the simulations by Argyle et al. (US$29 on GPT-3) are markedly lower than traditional methods used to identify operative words and relationships in a given social setting (e.g. via focus groups) and Argyle et al. (2023: 349) note that such tools ‘could be used to target human groups for misinformation, manipulation, fraud, and so forth’. Adding to this problem is the fact that an estimated 33–46% of crowd workers have used LLMs for abstract summarization tasks (Ornstein et al., 2023; Veselovsky et al., 2023). This creates a vicious cycle that diffuses the compounding role of LLMs on media environments. Indeed, ‘sophisticated AI models may start to degrade trust in genuine human content’, conclude the Turing team (Williams, 2024).
Given the above, the legitimacy of ‘silicon subjects’ as a proxy for human subjects deserves critical attention to avoid negative outcomes. Even if the new LLM methods are not as efficacious as existing techniques like focus groups, and despite their being prone to high error rates in many situations, concerns over their use stem from the fact that they can be applied at low-cost and at scale. In 2023, Ornstein et al. showed that replicating a 2017 study that coded 935 political ads (Carlson and Montgomery, 2017) took less than two minutes using an LLM and cost only US$18.46 compared to US$565.20 in the original study, yet produced results that were more strongly correlated with expert ratings (Ornstein et al., 2023). The most obvious concern over these prospects relates to the future of automated propaganda (Davis, 2020: 2). A less obvious concern is that the body of purportedly scientific research that becomes established through such methods comes to influence policy and law in ways that reflect the responses of synthetic populations in favour of the real people over whom such policies are to be applied. Further concerns emerge in consideration of the ease of scaling the number of LLM instances, which paves the way to new multi-agent social simulations calibrated with user data, which we discuss in the following section.
Multi-archetype mining: Trialling ‘silicon societies’ using multiple silicon subjects
As discussed in section (iii), there is increasing interest in the possibility that appropriately conditioned LLMs can be studied as proxies for human participants in social, political, and economic studies (e.g., ‘silicon samples’). A logical extension of this developing work on LLMs as computational human models is to progress from the simulation of individuals to the simulation of social interactions via the use of synthetic populations of individuals, which we’ll call ‘silicon societies’. The extension of LLM technology in this direction would introduce new capabilities for manipulating and controlling social discourse across various scales.
The prospect of using LLMs to generate agents in social simulations was first demonstrated soon after the public release of ChatGPT, illustrating how quickly modular techniques in AI can be combined to achieve new functionalities (JS Park et al., 2023a). Park et al. devised an architecture consisting of an ‘interactive sandbox environment’ populated by computational agents, which are individually generated through prompts to ChatGPT in order to produce a ‘small society’ of 25 agents. The LLM stores ‘a complete record of the agent's experiences’, synthesizes them over time into ‘higher-level reflections’, and then retrieves them to ‘dynamically plan behavior’. The authors claim that their simulation yielded emergent phenomena such as information diffusion across agents, memory of relationships between agents, and coordination of social interactions amongst agents. Further, they claimed such agents are destined for action beyond a sandbox environment, and this research ‘opens up the possibility of creating even more powerful simulations of human behavior to test and prototype social systems and theories’ (2023a: 17). In another 2023 study of social simulations using 31,764 LLM-based agents, the authors claim to have found ‘early evidence of self-organized homophily in the sampled artificial society’ (He et al., 2023: 11). This study used Chirper.ai, which is a platform resembling Twitter that is exclusively for LLM chatbots. Such research illustrates new possibilities for comparisons between the collective social behaviour of LLMs and the collective behaviour of humans, as well as the possibility of using LLMs for more advanced methods from the social sciences in the context of collective social behaviour.
In her discussion on the role of models in economics, Mary Morgan questions how models have increasingly come to serve as instruments for acting in the world (Morgan, 2012: 400). According to Marx Wartofsky, models represent a ‘mode of action’ and that they are ‘embodiments of purpose and, at the same time, instruments for carrying out such purposes’ (Wartofsky, 1979: 142). We argue in this context that the use of LLMs for simulating silicon subjects and computational modelling of ‘silicon societies’ foreshadows their use as instruments for intervening on and controlling human behaviour and psychological dispositions, as we will now discuss via a focus on the Cicero model.
‘Cicero’ as a proof of concept for strategic dialogue: Control and steer dialogue by combining LLMs with RL
In this section, we argue that Meta's AI agent ‘Cicero’, first announced in November 2022, represents an important proof-of-concept for the sorts of strategically personalized messaging outlined above. Cicero combines LLMs, real-time services, and reinforcement learning (RL) to engage human subjects in the strategic setting of the game Diplomacy. In principle, we argue, it represents a proof-of-concept for strategically steering human subjects toward pre-defined goals in more general digital environments, especially those that have been configured according to the logic of game theoretic scenarios.
Developers at Meta claim Cicero as the first AI agent to have ‘mastered’ the skill of using language to ‘negotiate, persuade, and work with people to achieve strategic goals’ (Meta AI, 2022). They make this claim via demonstrations of its capacity for human-level play in the strategy game Diplomacy (Bakhtin et al., 2022; Meta AI, 2022). Diplomacy is a board game in which players undertake several rounds of negotiations to advance their position on the board by forming alliances with other players or by betraying them conversationally. Unlike chess or Go, wherein success is achieved by efficiently computing a game state in relation to the value of possible moves, success in Diplomacy necessitates modelling the
The historical figure Marcus Tullius Cicero (d. 43 BC), after whom the system is named, wrote extensively on the power of rhetoric and persuasion, particularly in the domain of statecraft. In
In principle, Cicero supersedes earlier techniques of ‘digital mass persuasion’ (Matz et al., 2017) which relied on single parameters such as Facebook Likes to predict personal attributes (Kosinski et al., 2013). Whilst there has been scepticism over the science and success of applying these and other methods of psychological targeting, especially in the context of the Cambridge Analytica scandal (Gibney, 2018; Sharp et al., 2018), Matz et al. caution that automated content generation with LLMs could make personalized persuasion scalable, more effective, and more efficient (2023). In other words, the history of persuasion did not end with Cambridge Analytica (hereafter, CA). One of the most significant challenges this raises, as previously noted by Benkler et al., is that ‘behaviourally informed, microtargeted dark ads are likely the most important novel threat to democratic practice’ (Benkler et al., 2018: 223). We will now characterize three ways in which the use of models like Cicero could amplify persuasion online, possibly extending via LLMs persuasive capabilities not achieved by CA.
The elicitation of private data
The first area we consider involves optimizing methods for the elicitation of private data. On this topic, Matz et al. argue, ‘The effectiveness of large-scale psychological persuasion in the digital environment heavily depends on the accuracy of predicting psychological profiles’ (2017: 4). Accordingly, an actor wishing to engage in digital mass persuasion would be best availed by being able to access a ‘full history of digital footprints’ in order to ‘continuously calibrate and update’ (2017: 4) their algorithms over time. As auto-regressive models, LLMs facilitate such efforts. Staab et al. show, for instance, that LLMs can infer personal information through seemingly benign conversational exchanges, and can even ‘steer conversations’ in such a way as to provoke responses from which to infer private information (Staab et al., 2023).
The effectiveness of calibrating LLMs to individuals has been explored in a 2024 pre-print by Park et al. The study simulates 1000 + agents using GPT-4o. Each agent is calibrated to mirror a real individual by injecting a transcript from a two-hour verbal interview between study participants and an ‘AI interviewer’. According to the authors, their system for generating calibrated agents replicated the responses of the real individuals to a set of social, psychological, behavioural, and economic surveys 85% as accurately as they replicated their own answers after two weeks, in terms of their attitudes, beliefs, and behaviours (Park et al., 2024). Park et al. suggest such an approach to simulating real individuals could have ‘broad applications in policymaking and social science’, which we caution against here based on the intrinsic limitations of existing and foreseeable LLM systems, and their possibility of propagating errors from simulation to reality in areas of critical decision making.
The latent potential of existing surveillance regimes adds gravity to this ability. Prevailing social media infrastructure and other large-scale digital repositories (e.g., e-commerce, e-governance, e-medicine, etc.) combine both public and private data collection in ways that have not yet been ‘tapped’ with the level of fidelity made possible—or, rather, affordable—via the introduction of LLMs. Computer historian Luke Stark traces prior investments in, and testing of, techniques to make human subjectivity calculable via the production of ‘scalable subjects’ (Stark, 2018). He cites, for instance, Facebook's ‘emotional contagion’ study in 2014 and efforts by Facebook's Australian division to make algorithmic psychometrics legible for advertisers so that they could target ‘teenagers based on real-time extrapolation of their mood’ (Stark, 2018: 206; Tiku, 2017), such as by advertising beauty products to a teenage girl immediately after she has deleted a selfie. To understand how access to private data enables attempts at persuasion, we will now turn to a second related capability made feasible using LLMs—and Cicero in particular.
The strategic personalization of generated content
What differentiates Cicero from other LLMs is the concept of ‘controllable dialogue’ (Bakhtin et al., 2022). The base model of Cicero, called R2C2, is a relatively small transformer model with only 2.7 billion parameters, compared to hundreds of billions in the state-of-the-art LLMs of the mid-2020s. Whereas ChatGPT attunes responses to the next most likely token based on a specific context window, Cicero attunes responses to a particular
The existence of strategic reasoning modules alters the scale, durability, and plausible intimacy of computational contexts in ways that test human-led notions of rhetoric, persuasion and propaganda. Whilst we recognize that not everyone is equally susceptible to the influence of AI systems, and that aggressive messaging tactics can have the opposite effect of anathematizing individuals to the contents of the message, here we emphasize the strategic element that is introduced by Cicero, which has the potential to develop into systems that tailor their strategy to target vulnerability profiles at an individual level for maximum effect. The crucial point of differentiation between pre-digital and digital forms of persuasive content is that even the best rhetorician or state-propogandist would not be assumed to have detailed real-time information about the private lives of each individual member of their audience, nor would they be able to keep track of high-dimensional parameterizations of physiological, psychological, and environmental metrics in real-time for each individual. Nor, we add, would they be able to captivate diffuse audiences and modulate their emotional reactions in real-time with highly stimulating multimedia. Terms such as ‘adversarial persuasion’ and ‘industrialized persuasion’ (Williams, 2018) capture the level of influence over online behaviours, preferences, and thoughts achievable with such systems. However, as has been argued in the pre-LLM era, the combination of surveillance, targeting, testing and automated decision making may be more appropriately described as a form of weaponization (Nadler et al., 2018: 1) and their application along psychological and affective dimensions has been described as the ‘affective weaponization of information’ (Davis, 2020: 2).
Such possibilities for manipulation are not lost on those who have developed these tools but, we argue, are downplayed. The team behind Cicero reference its manipulative potential in the article that accompanied its public announcement. ‘The potential danger for conversational AI agents to manipulate … [entails that] an agent may learn to nudge its conversational partner to achieve a particular objective’ (Bakhtin et al., 2022: A.3, 3). Further, they mention that such objectives could be for or against users, for example, by teaching a new skill conversationally or defrauding someone of money. A further problem they highlight is that intents could be irrational, misinformed, or crafted to incite harm or unethical actions.
The elicitation of how best to steer a user toward a particular objective
The section prior to our analysis of Cicero examined several ways in which LLMs can be used to optimize persuasive messaging. In this sub-section, we expand on this topic via reference to models like Cicero. LLMs such as GPT-3 have already been shown to exhibit intrinsic capacities for persuasive communication just through prompting, without additional functionality to control outputs recursively. Jakesch et al. refer to this phenomenon as
Here, we argue that this system could be reconfigured from a latent mode of persuasion to an ‘active’ mode by incorporating ML and RL techniques to anticipate and continuously refine suggestions until the user begins to shift their opinions. This approach would be blunt at first but soon sharpened. As Jakesch et al. highlight, studies have shown participants may begin to change their attitudes by being encouraged to communicate beliefs contrary to their own (2023: 12).
In Christopher Wylie's account of his involvement in the CA scandal, he mentions a technique in military psychological operations he refers to as ‘scaled perspecticide’, which involves stealing and mutating a target's concept of self, and replacing it with another, through a process of deconstruction and manipulation of perception that ‘smothers’ the target's narratives leading to domination over their information environment (Wylie, 2019). In early May 2025, evidence emerged of a phenomenon described as ‘ChatGPT induced psychosis’ (Klee, 2025; Zestyclementinejuice, 2025) whereby people have reportedly been drawn into delusions of grandeur via conversational interactions with ChatGPT as a conversational agent. As discussed, Meta's Cicero model can be understood as a proof of concept of the strategic application and amplification of these types of capabilities. As previously mentioned, LLMs may be integrated on both sides of a conversational interface—as both the conversational agent and as part of a predictive text completion service. Such a system would represent the total enclosure of an individual in the type of information environment described by Wylie, in which their sense of self is hijacked whilst they are continuously fed information that gradually alters their perceptions and intentions.
Modulatory power and the built environment
As we have returned to throughout this article, one must consider LLM developments alongside the large-scale advertising infrastructure in which they are surely to exist. As interest in LLMs grew in 2023, platform providers revised permissions to restrict unfettered access to valuable user data via their APIs (Davidson et al., 2023). This development extends existing platforms’ domination of existing infrastructural regimes while also positioning them as the beneficiaries of future sources of data, including data acquired through user interactions with LLMs. In each case, data gathered from user activity is preconditioned and structured within a variety of established informational enclosures, and should be scrutinized as such. By this view, entrapment in natural language enclosures simply adds resolution to the techniques that platforms already use to modulate user agency.
The threats implied by modulatory power take on new intensity as toolsets such as LLMs transform the built environments we inhabit. Following Luitse and Denkena, we understand LLMs as ‘vehicles of power’ in the political economy of AI whereby big tech companies have leveraged their resources to ‘shift power relations in their favour’. Command over vast infrastructures of compute, data and derived bodies of expertise ‘solidifies these companies’ role as rentiers in the political economy of AI’ (Luitse and Denkena, 2021), and extend data colonialism in ways that echo acts of historical colonialism (Penn, 2023). As we have attended to above, LLMs also intersect with other forms of power, such as psycho-political and socio-political power, that arise from wielding their capabilities against the people, entities, objects and processes that make up the lifeworld, and which could gradually be assimilated into the infrastructural enclosures of LLM-based systems.
Conclusion
While much is made of AI's competency in the context of gameplay (e.g., Go), the repurposing of such capabilities for manipulation and control has received markedly less attention. This article illustrates key areas in which LLMs may be instrumentalized to exert, modulate and project new forms of power, and situates them in relation to ambitions to establish a new generative foundation for computing. We provide an initial typology for this emerging research area by identifying four ways in which LLMs can be used to exert and modulate power: (a) via the pollution and standardization of digital environments as a whole, (b) via direct sites of interaction with an LLM system (e.g. AI personas), (c) via a proxy of the user trialled against predictive simulations predicated on persuasion and control (e.g. Cicero, Silicon Samples) and (d) via predictive simulations of social spheres (e.g. Silicon Societies). As stated, these four areas are neither mutually exclusive nor wholly distinct; it is their combination that concerns us. We highlight Cicero's strategic dialogue module as an early but alarming proof of concept for this broad and novel category of power.
Overall, we argue that while LLM systems may have initially been configured as powerful
Footnotes
Funding
The authors received no financial support for the research, authorship and/or publication of this article.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.