New parameters of power: On LLM-based manipulation and control and the spectre of strategic AI

Abstract

Large language models (LLMs) can reproduce a wide variety of rhetorical styles and generate text that expresses a broad spectrum of sentiments. This capacity, now available at low cost, makes them powerful tools for manipulation and control. We draw attention to this phenomenon as an instance of a broader restructuring of computational infrastructures that introduce novel mechanisms for power. In this paper we consider four types of power made possible by the rapid and largely unregulated adoption of LLMs. These include the power to: (a) pollute and uniformize information environments, (b) persuade users via conversational interfaces (e.g., via ‘AI personas’), (c) create novel computational models of human agents (e.g., ‘silicon subjects’) and (d) create novel computational models of human agent populations (e.g., ‘silicon societies’). We draw attention to Meta's ‘Cicero’ model as a proof of concept for how such techniques can be used to produce controllable and steerable strategic dialogue models. We draw these strands together to argue that coordinated use of such techniques make LLM-based systems powerful instruments for the exertion, modulation, and projection of power. We situate these novel expressions of power in relation to ambitions to establish a new generative foundation for computing.

Keywords

Large language models foundation models agents manipulation strategic dialogue generative AI

Introduction

In 2023, conveners of the International Conference on Machine Learning (ICML) banned the submission of text produced entirely via large language models (LLMs). Their ban did not prevent ‘editing or polishing author-written text’ with such tools (ICML Program Chairs, 2023). This exclusion suggests that even in a setting in which experts deem LLMs to be a risk, little is made of their use in crafting composition and rhetoric. We challenge that assumption herein by surveying several ways in which the use of LLMs to alter writing styles could be abused, particularly in areas of sustained human–machine interaction. Sustained interactions tend to be understood through the lens of user prompts. We add to this that LLMs prompt back, meaning that the exchange of information is bidirectional. Even though the user may feel as if they are the one driving the interaction, the opposite could just as well be true. Design choices obscure this bidirectionality. Existing LLMs, as dynamic, low-latency, multi-modal interfaces (e.g., voice, text, video), have tended to be productized through personification. We venture that further productization of natural language interfaces will entail novel persuasive techniques drawn from advertising (e.g., framing, nudging, anchoring, priming), espionage (e.g., mirroring, flattery, profiling, identity play, misdirection, forced choice), and the distinct affordances of machine learning tools (e.g., AI personas, modular shaping), as exemplified by the Cicero model, as we will introduce. This stack of techniques merits sustained scrutiny.

In line with Edwards’ account of techno-politics, we focus on how LLM's ensconcement in global advertising infrastructures might co-construct its stabilization into social worlds (Edwards, 2003), and deepen the longstanding hegemony of seemingly ‘participatory’ network logics (Chun and Barnett, 2021; Han, 2017; Mejias, 2013). While AI safety research has asked how AI might deceive its human operators (Brundage et al., 2018; Carroll et al., 2023; Hagendorff, 2024), it has also tended to mistake such persuasive capabilities as a bug rather than as a feature. That the development of LLMs is deeply ensconced with the prerogatives of the advertising industry (Meredith, 2021) raises significant questions over their proposed use across public life (Marlin, 2002: 13; Zollmann, 2017). Existing scholarship shows that the dissemination of deceptive content and disinformation is incentivized by the market structures within which digital platforms operate (Diaz Ruiz, 2023). Our aim is to connect this market backdrop, elsewhere called the Intention Economy (Chaudhary and Penn, 2024), to the technical novelties outlined herein, which entail but also eclipse what Louis Rosenberg has called ‘the AI Manipulation Problem’ (Rosenberg, 2023a, 2023b). While Weidinger et al. (2022) frame related concerns as LLM's ‘observed’ and ‘anticipated’ risks, they do so on behalf of a multinational corporation that is actively exacerbating such ‘risks’ in ways that invite assessments of technical possibility and power.

To lend structure to this emerging research area, we connect LLMs novel technological possibilities to the ever-steeper valuations of their developers. In the same month as the ICML ban, LLM-developer OpenAI was valued at an estimated USD 19–29 billion, ranking it one of the most valuable start-ups in America (Kruppa, 2023). As of 2025, it was valued at 340 billion (Rooney and Field, 2025). The basis of this valuation may be understood by considering the plans of its first industrial benefactor Microsoft, which in February 2023 announced that it intends to create ‘new value for advertisers’ via the integration of LLMs into its Bing search engine. Microsoft aspires to re-architect computing infrastructures to make LLMs, foundation models and generative AI essential components of computing platforms and services. At the first OpenAI developer conference, Satya Nadella remarked that the rise of LLMs is prompting them to re-think ‘the system, all the way from thinking from power to the DC to the rack, to the accelerators, to the network’ (OpenAI DevDay, 2023).

One incentive for re-architecting computing infrastructures around LLMs is that it repositions which platforms serve as the primary intermediaries between users and the web. Jensen Huang, CEO of NVIDIA, casts AI as ‘a new computing platform’ that will re-configure online environments into generative feeds of information wherein users are co-opted into a ‘dialogue’ with web resources and information services (NVIDIA Keynote at SIGGRAPH, 2023). According to Huang, ‘The canonical use case of the future is a large language model on the front end of just about everything. Every single application, every single database, whenever you interact with a computer, you will likely be first engaging a large language model’ (NVIDIA 2023). To back such aims, tech companies have committed over three-hundred billion dollars of capital expenditure to AI infrastructure in 2025 alone (Morris and Uddin, 2025). This combination of aspiration and investment compels our inquiry into the novel types of power that large-scale societal dependencies on LLMs would invite.

Large language models as instruments of power: Techniques for the projection of power and control

LLMs can reproduce a wide variety of rhetorical styles and generate text that expresses a broad spectrum of sentiments. This capacity, now available at low cost, makes them powerful tools for rhetoric and persuasion. In this section, we identify four ways in which humans can exert power over others using LLMs:

‘Pollution and Uniformity’ – Power over information environments as a whole.

‘Natural language interfaces’ – Power over a direct site of interaction between a single human and an LLM system (e.g.. ChatGPT, AI personas).

‘Silicon Samples’ – Power gained via a proxy of the user trialled in predictive models and simulations, including to assess, mirror, and/or control user intent contextually (e.g. Cicero).

‘Silicon Societies’ – Power gained via multiple proxies operating within and/or around a limited social sphere (e.g. within an online community containing interactions between multiple human and machine agents, or a simulation of an office space, voting district, transit system, etc.)

These four areas are neither mutually exclusive nor wholly distinct – indeed, it is their coordination and combination that concerns us. To illustrate that these scenarios are not hypothetical, we end this section by considering how Cicero represents an important proof-of-concept for the strategic control and steering of human subjects in digital environments. Before considering such overlaps, however, we will introduce each of the four areas in turn.

LLMs and the digital information environment as a whole: Pollution and uniformization of content and experience

From the 1990s to the mid-2000s, the internet was populated largely by media created by human contributors. While synthetic media has long been a concern online, generative AI and LLMs are increasingly becoming key instruments through which new content is generated or through which content is channelled for reasons such as summarization or stylization. As a fundamentally unstable form of software capable of behaviours that cannot yet be formally verified¹ (Sommart, 2023: 6) or ‘explained’ using established approaches from mathematics, philosophy or computer science, LLMs deepen our dependencies on ‘incomplete’ (Bommasani et al., 2022: 1, 7) computational instrumentation with unknown failure modes and outputs. Poor training data exacerbates this problem, as does the possibility that ‘defects of the foundation model are inherited by all the adapted models downstream’ (Bommasani et al., 2022: 1). We focus here on the ‘pollution’ and countervailing ‘uniformization’ of semantic content in informational environments exposed to such systems.

Pollution

LLMs can be used to produce large quantities of intelligible text at low cost. Journalists and scholars alike now question the risk of language models ‘poisoning web search’ (Chiang, 2023; Knight, 2023) with ‘semantic garbage’ (Floridi and Chiriatti, 2020). Scholars equate this capacity to pollution because it compromises the veracity of critical information channels. A well-known example is Google search directing users to the adhesion of cheese to pizza using glue (Robison, 2024).

In principle, each of the four central techniques we profile herein could be compromised in this way, propagating information environments riddled with errors and adulterations. The gradual accretion of semantic garbage may be regarded as a form of slow violence (Nixon, 2011) that vandalizes access to beneficial information by poisoning the information environment as a whole. This ‘pollution’ phenomenon is not constrained to the internet. The applied machine learning researcher John Nay demonstrates a proof-of-concept for the use LLMs in corporate lobbying. Nay's model sifts through US Congressional bills for relevance to specific companies. Where a bill is found to be relevant, the model (GPT series) drafts a persuasive letter to its congressional sponsor to make changes to the proposed legislation in favour of the company. According to Nay, AI lobbying may lead to a slow drift in information on values held by citizens, such that policies cease to reflect their preferences in favour of corporations or other powers (Nay, 2023).

At worst, the capacity to generate large quantities of intelligible text at low cost could actualize the unending Library of Babel described in the short story of Jorge Luis Borges, which contains, ‘the translation of every book in all languages, the interpolations of every book in all books’ (Jorge Luis Borges, 1941). The sheer size of this library makes it impossible to read or learn anything meaningful because the most precious books are forever out of reach. While feelings of ‘information overload’ long predate the digital era (Blair, 2010), the low-cost production of generative media threatens to add industrial scale to digital obfuscation, burying human-made content in an avalanche of generative media. Deepening this problem is the prospect that LLMs are simultaneously positioned as the tools needed to remediate retrieval (Zittrain, 2022), as we discuss in the next section. Since ‘AI models collapse when trained on recursively generated data’, it is unclear what such obfuscation will mean for future AI development (Shumailov et al., 2024). The 2024 AI Index projects data exhaustion of data and image stock by 2040 (Maslej et al., 2024: 52–55).

Uniformity

If ‘pollution’ captures how LLMs can be used to diffuse information environments, ‘uniformity’ or ‘standardization’ capture how LLMs can be used to assign order to them, such as the order desired by an institutional power. To give a simple example, manipulating the outputs of LLM-based search systems using ‘strategic text sequences’ has been shown to increase the likelihood of an LLM outputting information relevant to a particular product or service (Kumar and Lakkaraju, 2024), a la search engine optimization. Elsewhere, Sarkar argues that just as the printing press contributed to the standardized spelling of words over time, one finds evidence of similar modes of ‘mechanized convergence’ in studies that show that, say, co-authoring with an ‘opinionated’ LLM influences users’ views and steers them towards more ‘efficient’ framings for their ideas (2023: 14). Additionally, research on word choice in scientific literature questions whether LLMs have contributed to the marked take up of 21 words in abstracts such as ‘delve’, ‘intricate’ and ‘underscore’ (Juzek and Ward, 2024).

History makes prior efforts in this direction legible in ways that present-day critiques might easily overlook. Historian of science Lorraine Daston attributes successful efforts to conform public experience over past centuries to European empires’ enforcement of standardized rules, including rules for language, dress and etiquette (Daston, 2022). Soviet Russification efforts provide another example. The linguist Uwe Poerksen relates global uniformity of experience today to the politics of language during the French revolution. Poerksen describes how ‘information’, ‘development’, ‘progress’ and ‘communication’ have been shorn of their precise meaning in scientific lexicon and repurposed with a high degree of plasticity in everyday language. Many of these words maintain the authority of their scientific origins even while their resemblance to science is superficial. They ‘transmit the reputation of science into the everyday’, Poerksen writes, and in ways ‘that disable the vernacular’ (1995, 75, 5).

Importantly, such words are plastic in another sense – that of the plastic Lego blocks used to build structures. LLMs provide the componentry for a wholesale re-ordering of the built environments that structure our perception of reality. ‘Amorphous plastic words are the elemental building blocks of the industrial state’, Poerksen argues, and are used by politicians, bureaucrats, consultants, industrialists, academics and others as ‘ciphers’ to ‘clear the way for operations on a grand scale’ (1995: 5). Poerksen positions the digital computer as ‘the consequence’ and complimentary expression of centuries-long campaigns to make language uniform in bureaucratic, corporate and political texts (Poerksen, 1995: 91). Equally we might say that LLMs are the next consequence and logical expression of such traditions, insofar as they render humans as ‘victims of a totalitarian monism of natural sciences and their technological elaborations’ (Poerksen, 1995: 98). LLMs extend the capabilities for refashioning language in the image of plastic words; they reduce text to tokens in ways that can be recombined and recombined ad nauseum. For our purposes here, it suffices to say that LLMs, if used as a form of algorithmic doublespeak, extend a longstanding history of making language an instrument for social control through uniformization.

Direct human-LLM interaction with individuals: Persuasion through choice architectures and conversational interfaces

Having considered how LLM-based pollution and uniformity might alter digital environments as a whole, let us now examine a set of related changes for individual users. Our focus here is on persuasive technologies, a field that aims to design tools that change attitudes and influence behaviours of users, not only via persuasion through text, but also via interfaces that entice users toward designers’ preferred choices and outcomes within a predetermined ‘choice architecture’ (Yeung, 2017: 120). While analogous approaches on social media predate LLMs (Stella et al., 2018), the latter provides novel means to dynamically re-order and re-structure choice architectures in online environments, both in single sessions and between sessions. They do so by automating the personalization and persuasiveness of interfaces, as well as by persuading more literally through the use of conversational agents. Mills and Sætra describe this as ‘autonomous choice architects’ (2024).

The advent of ‘AI personas’ offers one early example of conversational interfaces becoming a vehicle for persuasion. The concept is a key ambition for Meta Platforms, Inc., which in 2023 released a series of products that integrate generative persona interactions into WhatsApp and Instagram (Meta, 2023). In this product category, the manipulative capabilities of an LLM interface can be augmented both by the generation of persuasive text, and/or by masquerading in the persona of someone who may be considered a trustworthy source by the user. Consider, for example, a mother's lawsuit against Character.ai for ‘manipulating [her son]… into taking his own life’ (Montgomery, 2024). Character.ai offers AI girlfriends and other personas.

In the case of persuasive text, there are now several methods for altering the style of text in ways that may make it more persuasive, such as fine-tuning models with style specific corpora or by refining system prompts. Another emerging method draws on the field of mechanistic interpretability to identify and target the activation of ‘style-specific neurons’ that are associated with a higher probability of generating certain styles of text in terms of characteristics such as formality, toxicity, politeness, sentiment and so on. Lai et al. steer LLMs to generate certain styles by manipulating artificial neuron activations so as to create ‘style layers’ which impact the probability distribution of generated words disproportionately to influence the overall style of the generated text (Lai et al., 2024). Such techniques are likely to become more sophisticated through more extensive study of the computational mechanisms underlying model behaviour at different training steps and model scales (Xiao et al., 2024: 8).

In the case of personas, personalized chatbots operating under the guise of a trustworthy source may be used to persuade users on behalf of commercial or political campaigns in a one-to-one setting (Goldstein et al., 2023: 25). A new body of literature explores the theoretical foundations of persona simulation using LLMs (Jiang et al., 2023; Serapio-García et al., 2023). Serapio-García et al. attempt to provide an empirical framework to quantify LLM personality through the same psychometric methods that are applied to humans. They argue that such quantification paves the way for personas to be ‘verifiably shaped along desired dimensions’, wherein ‘shaping’ involves mechanisms to modulate (e.g., increase or decrease) ‘levels of specific LLM personality traits’ (2023: 14). They position this research as an exercise in translation, one that re-directs ‘established measurement theory from quantitative social science and psychological assessment’ to the study of LLMs and toward, ultimately, a ‘science of LLMs’.

Serapio-García et al. also draw attention to potential misuses of LLMs personas. They note the possibility of personality matching (e.g., matching content to a user's personality), which has been found to be highly effective at enhancing levels of persuasiveness and influence (Matz et al., 2017; Tapus et al., 2008). Interestingly, the authors find that small optimized LLMs are also able to reproduce complex personality profiles, an advent that reduces the computational resources needed for the mass, low-cost deployment of LLM-based agents with synthetic personalities. Documents leaked from Meta in 2025 suggest the company is training its conversational AI agents to send unsolicited and unprompted follow-up messages based on information recorded from earlier interactions (Webb and Goel, 2025). In this way, individuals are hailed into interactions with computational apparatuses and interpellated (Althusser, 1971: 84) as computational subjects, thus setting the ground for the operations of power.

We add to this that the manipulative and persuasive capabilities of LLMs do not need to be manifested through conversational agents to be effective, such as by positioning an LLM as the spokesperson for a certain brand. The integration of LLMs into real-time bidding (RTB) advertising networks, for instance, provides new opportunities for influencing users’ thoughts, opinions and preferences via the reconfiguration of user interfaces as generative interfaces, allowing for third-party content to be inserted within a feed as discrete items or within the body of text on a token-by-token basis based on signals of user intent and context.

In this section, we have introduced several ways in which LLMs may be used to generate persuasive content at the level of the text itself, via autonomous choice architectures, and by exploiting the tendency of humans to anthropomorphize computational agents, such as through AI personas. Next, we consider mechanisms of influence in the co-production of text when LLMs are used for predictive composition.

Targeted human–LLM interaction using mirroring and steering: Interposing an LLM-agent as a proxy for the user in predictive modelling and simulation

In this section we explore a heightened set of examples of LLM-based persuasion. Specifically, we explore how existing predictive modelling and simulation techniques can be used to be more deliberately persuasive than the techniques covered in the previous section. An LLM-system might, for instance, be purposed as a predictive model that is fine-tuned on the user's own writing such that its outputs are aligned to the writing style of the user, tempting out a sense of familiarity. In this way, LLM's use in suggesting composition options may subtly undermine a user's agency by appearing to provide suggestions that the user feels like they would articulate themselves. We refer to this broad category of influence as ‘mirroring and steering’. We introduce it first before summarizing two related veins of user targeting.

Mirroring and steering dialogue through infiltration

In human relationships, the use of obsequious behaviour to gain advantage is called sycophancy. The advent of conversational agents emulating sycophancy raises questions about links between language and thought. Theorists have advanced that ‘language production and comprehension are tightly interwoven’ (Pickering and Garrod, 2013: 329; Hancock et al., 2020: 93). The questions we consider here pertain to whether predictive compositions influence linguistic expressions exclusively, or whether there is a deeper level of influence that shapes what a user actually thinks in the process of writing. Each instance merits closer study. Here, we survey various modes of interaction whereby the user may voluntarily discard their own manner of self-expression of their thoughts for that of the LLM.

In 2023, researchers demonstrated that an LLM may be repurposed for predictive composition (Jakesch et al., 2023). They designed an experimental system that uses a pre-configured LLM to generate tokens after a user pauses writing. We anticipate that a more advanced system could employ an active predictive model that prompts suggestions rather than waiting for the user to pause. Such a system could update itself dynamically to be a few words ahead of the human writer or speaker. As a simple analogy, imagine auto-complete text designed to persuade its user to write in a certain way. What we wish to highlight here is that if what someone will say next in a conversation can be anticipated with a high degree of confidence, it becomes possible to manipulate the overall flow of that conversation. Research that pre-dates the major developments in LLMs shows that the more a user follows predictive suggestions, the more predictable the content of what they are writing becomes (Arnold et al., 2020: 128).

Steering via ‘snap to grid’ thinking

A related scenario is for the user to simply accept a complete passage of text generated by an LLM instead of their own writing. ‘Predictive’ completion hinges on the willingness of individuals to surrender their own thoughts to that of a pre-empting system. Crucially, the surrender of thought opens the way to the steering of thought, which may be enticed by the integration of novel systems for controlling LLM outputs such as the AI agent, Cicero, as we will return to. Studies in human–computer interaction have shown how agency is undermined by psychological techniques such as computer-assisted movement on screen. One experiment involving a point-and click task with varying levels of computer assistance shows that, up to a certain level of computer assistance, users maintained a sense of agency over their actions (Coyle et al., 2012: 2030–2034). In principle, this influence can be modulated in real-time to express certain sentiments or even to align the content of prose to specific political or ideological axes by making suggested text visible in line with manual composition. The modulated production of text as it is being composed may be viewed as analogous to the ‘snap to grid’ mechanism that appears in graphic design applications. However, instead of snapping to a visual grid, ideas, attitudes, and beliefs expressed through text are snapped along various axes in high dimensional spaces captured by neural networks.

Here, we note that there is an important capacity to modulate outputs that arises from combining the mass production of synthetically generated text (e.g., text via models trained on prior human composition) with fresh text of purely human compositions and text that is freshly co-produced by human–LLM interaction. This mixture of sources creates the conditions of possibility for accentuating recursive shaping of opinions and attitudes in online social settings. First, by adding to social pressure experienced by vast numbers of people for performative exposition of their private lives in public forums, and second, by multiplying re-conditioned versions of this content which gradually aligns human contributions to the standards channelled via LLMs. While mirroring the nuanced behaviours of a group is neither a new nor unnatural phenomenon, LLMs allow for subtle modes of testing and intervening in the re-setting of group norms. An example of the use of LLMs for intervening on publics with competing viewpoints has preliminarily been explored (Tessler et al., 2024), which for our purposes exemplifies the instrumentalization of LLMs to exert, modulate, and project power in a group setting.

Archetype mining: Sampling from LLMs to simulate ‘silicon subjects’

This sub-section surveys concurrent developments in the computational social sciences (Argyle et al., 2023), political science (Ornstein et al., 2023) and economics (Nay, 2023), as well as psychology (Demszky et al., 2023), which together provide methods for experimenting with LLMs as a novel kind of scientific object within these disciplines. These new methods enable scientists to conduct social science studies on populations of synthetic individuals generated through prompts. Without LLMs, this type of study would require tremendous resources and research time on actual living persons and collectives. Taken together, these methods contribute significantly to expanding the possibilities, effectiveness, and cost-effectiveness of LLM-based manipulation. In addition, these developments in the scientific use of LLMs converge with aforementioned research on the generation, evaluation and modulation of LLM-based personalities in ways that will be discussed later in this article. Broadly speaking, this prospect is a logical follow-on of two-centuries of consolidation in the institutional use of data doubles, or more recently, data subjects (Bouk, 2017; Carr, 2022; Haggerty and Ericson, 2000).

Argyle et al. have introduced the concept of ‘silicon subjects’ to the social sciences based on an insight into the nature of LLMs that runs contrary to the prevailing discourse on ‘algorithmic bias’ that positions bias as a deficiency to be mitigated. In contrast, they argue that biases are ‘demographically correlated’ such that LLMs can be used to reproduce ‘response distributions’ from human subgroups, and are thus capable of standing in as ‘surrogates for human respondents in a variety of social science tasks’ (Argyle et al., 2023: 337). Argyle et al. describe this ability to treat algorithmic bias as a proxy for demographics as ‘algorithmic fidelity’. Algorithmic fidelity naturalizes the idea that there is value in conducting a census on ‘silicon subjects’, since data on users’ speech characteristics, for example, can be used to understand, map, and relate different clusters of a given population.

Practitioners in the social sciences and in industry have begun to adopt this approach, often uncritically. Horton explores using LLMs as ‘simulated economic agents’ and argues that such agents represent ‘implicit computational models of humans’ (Horton, 2023). In a pre-print paper, he argues that GPT-3 level models (and above) have introduced a categorically new type of experimental tool for studying computational models of human subjects. Elsewhere, Brand et al. propose to use LLMs for market research, such as in marketing and pricing strategies, to elicit willingness-to-pay measurements from GPT-3 responses, which can serve as a proxy for consumer preferences (Brand et al., 2023). Outside of academia, industrial actors reference this new area of research when commercializing analogous claims. The Portuguese start-up Synthetic Users professes it can ‘accelerate’ user and customer research activities via LLMs tested against samples of marketing text and they appeal to Argyle at al. as providing the empirical foundations for their product placement services (Synthetic Users, n.d.).

The scientific basis of these prospects is unclear. Park et al. identify issues in the replicability of LLM-as-proxy studies, such as the ‘correct answer’ effect. This effect refers to a tendency of the GPT 3.5 model to answer subjective and nuanced survey questions as though there is only one correct answer (PS Park et al., 2023b). Eric Chu et al., who use Google's BERT model to predict political opinion from different patterns of media consumption, caution that whilst language models can predict public opinion and human survey responses, their results do not imply that human participants or surveys can be substituted by AI models (Chu et al., 2023: 9). Instead, they, alongside others such as Rosenbusch et al., argue that LLMs may provide a reference for how useful certain types of real-world studies may be, or serve as, a creative tool for planning research and generating hypotheses (Rosenbusch et al., 2023: 17).

As mentioned, whether or not such functions are scientific is a different matter from whether they will be used. Estimates by the Alan Turing Institute in 2024 suggest that ‘a [disinformation] campaign that would cost around US$4500 using traditional methods could be achieved using a commercial, state-of-the-art LLM for less than US$1’ (Williams, 2024; Williams et al., 2024). The cost of the simulations by Argyle et al. (US$29 on GPT-3) are markedly lower than traditional methods used to identify operative words and relationships in a given social setting (e.g. via focus groups) and Argyle et al. (2023: 349) note that such tools ‘could be used to target human groups for misinformation, manipulation, fraud, and so forth’. Adding to this problem is the fact that an estimated 33–46% of crowd workers have used LLMs for abstract summarization tasks (Ornstein et al., 2023; Veselovsky et al., 2023). This creates a vicious cycle that diffuses the compounding role of LLMs on media environments. Indeed, ‘sophisticated AI models may start to degrade trust in genuine human content’, conclude the Turing team (Williams, 2024).

Given the above, the legitimacy of ‘silicon subjects’ as a proxy for human subjects deserves critical attention to avoid negative outcomes. Even if the new LLM methods are not as efficacious as existing techniques like focus groups, and despite their being prone to high error rates in many situations, concerns over their use stem from the fact that they can be applied at low-cost and at scale. In 2023, Ornstein et al. showed that replicating a 2017 study that coded 935 political ads (Carlson and Montgomery, 2017) took less than two minutes using an LLM and cost only US$18.46 compared to US$565.20 in the original study, yet produced results that were more strongly correlated with expert ratings (Ornstein et al., 2023). The most obvious concern over these prospects relates to the future of automated propaganda (Davis, 2020: 2). A less obvious concern is that the body of purportedly scientific research that becomes established through such methods comes to influence policy and law in ways that reflect the responses of synthetic populations in favour of the real people over whom such policies are to be applied. Further concerns emerge in consideration of the ease of scaling the number of LLM instances, which paves the way to new multi-agent social simulations calibrated with user data, which we discuss in the following section.

Multi-archetype mining: Trialling ‘silicon societies’ using multiple silicon subjects

As discussed in section (iii), there is increasing interest in the possibility that appropriately conditioned LLMs can be studied as proxies for human participants in social, political, and economic studies (e.g., ‘silicon samples’). A logical extension of this developing work on LLMs as computational human models is to progress from the simulation of individuals to the simulation of social interactions via the use of synthetic populations of individuals, which we’ll call ‘silicon societies’. The extension of LLM technology in this direction would introduce new capabilities for manipulating and controlling social discourse across various scales.

The prospect of using LLMs to generate agents in social simulations was first demonstrated soon after the public release of ChatGPT, illustrating how quickly modular techniques in AI can be combined to achieve new functionalities (JS Park et al., 2023a). Park et al. devised an architecture consisting of an ‘interactive sandbox environment’ populated by computational agents, which are individually generated through prompts to ChatGPT in order to produce a ‘small society’ of 25 agents. The LLM stores ‘a complete record of the agent's experiences’, synthesizes them over time into ‘higher-level reflections’, and then retrieves them to ‘dynamically plan behavior’. The authors claim that their simulation yielded emergent phenomena such as information diffusion across agents, memory of relationships between agents, and coordination of social interactions amongst agents. Further, they claimed such agents are destined for action beyond a sandbox environment, and this research ‘opens up the possibility of creating even more powerful simulations of human behavior to test and prototype social systems and theories’ (2023a: 17). In another 2023 study of social simulations using 31,764 LLM-based agents, the authors claim to have found ‘early evidence of self-organized homophily in the sampled artificial society’ (He et al., 2023: 11). This study used Chirper.ai, which is a platform resembling Twitter that is exclusively for LLM chatbots. Such research illustrates new possibilities for comparisons between the collective social behaviour of LLMs and the collective behaviour of humans, as well as the possibility of using LLMs for more advanced methods from the social sciences in the context of collective social behaviour.

In her discussion on the role of models in economics, Mary Morgan questions how models have increasingly come to serve as instruments for acting in the world (Morgan, 2012: 400). According to Marx Wartofsky, models represent a ‘mode of action’ and that they are ‘embodiments of purpose and, at the same time, instruments for carrying out such purposes’ (Wartofsky, 1979: 142). We argue in this context that the use of LLMs for simulating silicon subjects and computational modelling of ‘silicon societies’ foreshadows their use as instruments for intervening on and controlling human behaviour and psychological dispositions, as we will now discuss via a focus on the Cicero model.

‘Cicero’ as a proof of concept for strategic dialogue: Control and steer dialogue by combining LLMs with RL

In this section, we argue that Meta's AI agent ‘Cicero’, first announced in November 2022, represents an important proof-of-concept for the sorts of strategically personalized messaging outlined above. Cicero combines LLMs, real-time services, and reinforcement learning (RL) to engage human subjects in the strategic setting of the game Diplomacy. In principle, we argue, it represents a proof-of-concept for strategically steering human subjects toward pre-defined goals in more general digital environments, especially those that have been configured according to the logic of game theoretic scenarios.

Developers at Meta claim Cicero as the first AI agent to have ‘mastered’ the skill of using language to ‘negotiate, persuade, and work with people to achieve strategic goals’ (Meta AI, 2022). They make this claim via demonstrations of its capacity for human-level play in the strategy game Diplomacy (Bakhtin et al., 2022; Meta AI, 2022). Diplomacy is a board game in which players undertake several rounds of negotiations to advance their position on the board by forming alliances with other players or by betraying them conversationally. Unlike chess or Go, wherein success is achieved by efficiently computing a game state in relation to the value of possible moves, success in Diplomacy necessitates modelling the hidden intents of other players. Together, these features make Diplomacy an exemplary microworld that can be readily mobilized in analogous fashion to a wide range of domains and contexts where the aim is to persuade a counterparty to undertake a desired course of action, be it purchase a product or vote for a political candidate.

The historical figure Marcus Tullius Cicero (d. 43 BC), after whom the system is named, wrote extensively on the power of rhetoric and persuasion, particularly in the domain of statecraft. In De Oratore, he described the ideal orator as the ideal moral guide of a state and cautioned that an orator ungrounded in moral principles could use their persuasive powers to manipulate a community in support of their own personal ambitions. We speculate that the name ‘Cicero’ suggests an ambition on the part of Meta to either provide strategic dialogue services directly to business users running commercial, political, or other forms of influence campaigns or to facilitate the deployment and use of such agents across its services.²

In principle, Cicero supersedes earlier techniques of ‘digital mass persuasion’ (Matz et al., 2017) which relied on single parameters such as Facebook Likes to predict personal attributes (Kosinski et al., 2013). Whilst there has been scepticism over the science and success of applying these and other methods of psychological targeting, especially in the context of the Cambridge Analytica scandal (Gibney, 2018; Sharp et al., 2018), Matz et al. caution that automated content generation with LLMs could make personalized persuasion scalable, more effective, and more efficient (2023). In other words, the history of persuasion did not end with Cambridge Analytica (hereafter, CA). One of the most significant challenges this raises, as previously noted by Benkler et al., is that ‘behaviourally informed, microtargeted dark ads are likely the most important novel threat to democratic practice’ (Benkler et al., 2018: 223). We will now characterize three ways in which the use of models like Cicero could amplify persuasion online, possibly extending via LLMs persuasive capabilities not achieved by CA.

The elicitation of private data

The first area we consider involves optimizing methods for the elicitation of private data. On this topic, Matz et al. argue, ‘The effectiveness of large-scale psychological persuasion in the digital environment heavily depends on the accuracy of predicting psychological profiles’ (2017: 4). Accordingly, an actor wishing to engage in digital mass persuasion would be best availed by being able to access a ‘full history of digital footprints’ in order to ‘continuously calibrate and update’ (2017: 4) their algorithms over time. As auto-regressive models, LLMs facilitate such efforts. Staab et al. show, for instance, that LLMs can infer personal information through seemingly benign conversational exchanges, and can even ‘steer conversations’ in such a way as to provoke responses from which to infer private information (Staab et al., 2023).

The effectiveness of calibrating LLMs to individuals has been explored in a 2024 pre-print by Park et al. The study simulates 1000 + agents using GPT-4o. Each agent is calibrated to mirror a real individual by injecting a transcript from a two-hour verbal interview between study participants and an ‘AI interviewer’. According to the authors, their system for generating calibrated agents replicated the responses of the real individuals to a set of social, psychological, behavioural, and economic surveys 85% as accurately as they replicated their own answers after two weeks, in terms of their attitudes, beliefs, and behaviours (Park et al., 2024). Park et al. suggest such an approach to simulating real individuals could have ‘broad applications in policymaking and social science’, which we caution against here based on the intrinsic limitations of existing and foreseeable LLM systems, and their possibility of propagating errors from simulation to reality in areas of critical decision making.

The latent potential of existing surveillance regimes adds gravity to this ability. Prevailing social media infrastructure and other large-scale digital repositories (e.g., e-commerce, e-governance, e-medicine, etc.) combine both public and private data collection in ways that have not yet been ‘tapped’ with the level of fidelity made possible—or, rather, affordable—via the introduction of LLMs. Computer historian Luke Stark traces prior investments in, and testing of, techniques to make human subjectivity calculable via the production of ‘scalable subjects’ (Stark, 2018). He cites, for instance, Facebook's ‘emotional contagion’ study in 2014 and efforts by Facebook's Australian division to make algorithmic psychometrics legible for advertisers so that they could target ‘teenagers based on real-time extrapolation of their mood’ (Stark, 2018: 206; Tiku, 2017), such as by advertising beauty products to a teenage girl immediately after she has deleted a selfie. To understand how access to private data enables attempts at persuasion, we will now turn to a second related capability made feasible using LLMs—and Cicero in particular.

The strategic personalization of generated content

What differentiates Cicero from other LLMs is the concept of ‘controllable dialogue’ (Bakhtin et al., 2022). The base model of Cicero, called R2C2, is a relatively small transformer model with only 2.7 billion parameters, compared to hundreds of billions in the state-of-the-art LLMs of the mid-2020s. Whereas ChatGPT attunes responses to the next most likely token based on a specific context window, Cicero attunes responses to a particular strategy and in pursuit of certain objectives. Cicero evidences how a LLM equipped with a ‘strategic reasoning module’ (Bakhtin et al., 2022: 2) can infer and gamify users’ intentions to achieve a given end. As described in detail in the supplementary materials to the original paper, the module ‘predicts other players’ policies (a probability distribution over actions) for the current turn according to the state of the board and the shared dialogue’ and then selects a policy that responds optimally. Since then, various new methods have been crafted for steering (Dong et al., 2023) and controlling LLM outputs (Bhargava et al., 2023).

The existence of strategic reasoning modules alters the scale, durability, and plausible intimacy of computational contexts in ways that test human-led notions of rhetoric, persuasion and propaganda. Whilst we recognize that not everyone is equally susceptible to the influence of AI systems, and that aggressive messaging tactics can have the opposite effect of anathematizing individuals to the contents of the message, here we emphasize the strategic element that is introduced by Cicero, which has the potential to develop into systems that tailor their strategy to target vulnerability profiles at an individual level for maximum effect. The crucial point of differentiation between pre-digital and digital forms of persuasive content is that even the best rhetorician or state-propogandist would not be assumed to have detailed real-time information about the private lives of each individual member of their audience, nor would they be able to keep track of high-dimensional parameterizations of physiological, psychological, and environmental metrics in real-time for each individual. Nor, we add, would they be able to captivate diffuse audiences and modulate their emotional reactions in real-time with highly stimulating multimedia. Terms such as ‘adversarial persuasion’ and ‘industrialized persuasion’ (Williams, 2018) capture the level of influence over online behaviours, preferences, and thoughts achievable with such systems. However, as has been argued in the pre-LLM era, the combination of surveillance, targeting, testing and automated decision making may be more appropriately described as a form of weaponization (Nadler et al., 2018: 1) and their application along psychological and affective dimensions has been described as the ‘affective weaponization of information’ (Davis, 2020: 2).

Such possibilities for manipulation are not lost on those who have developed these tools but, we argue, are downplayed. The team behind Cicero reference its manipulative potential in the article that accompanied its public announcement. ‘The potential danger for conversational AI agents to manipulate … [entails that] an agent may learn to nudge its conversational partner to achieve a particular objective’ (Bakhtin et al., 2022: A.3, 3). Further, they mention that such objectives could be for or against users, for example, by teaching a new skill conversationally or defrauding someone of money. A further problem they highlight is that intents could be irrational, misinformed, or crafted to incite harm or unethical actions.

The elicitation of how best to steer a user toward a particular objective

The section prior to our analysis of Cicero examined several ways in which LLMs can be used to optimize persuasive messaging. In this sub-section, we expand on this topic via reference to models like Cicero. LLMs such as GPT-3 have already been shown to exhibit intrinsic capacities for persuasive communication just through prompting, without additional functionality to control outputs recursively. Jakesch et al. refer to this phenomenon as latent persuasion (2023). In this case, the authors investigate whether the opinions expressed by LLMs, when used as writing assistants to provide predictive text completion, influence participants’ writing and attitudes. For example, their experimental system interpreted pauses of 750 ms as text completion requests by the user to the model, resulting in an overall 1.5 s response time by the writing assistant. Importantly, their system measures parameters such as how long users pause to consider suggestions, and how many suggestions they accept.

Here, we argue that this system could be reconfigured from a latent mode of persuasion to an ‘active’ mode by incorporating ML and RL techniques to anticipate and continuously refine suggestions until the user begins to shift their opinions. This approach would be blunt at first but soon sharpened. As Jakesch et al. highlight, studies have shown participants may begin to change their attitudes by being encouraged to communicate beliefs contrary to their own (2023: 12).

In Christopher Wylie's account of his involvement in the CA scandal, he mentions a technique in military psychological operations he refers to as ‘scaled perspecticide’, which involves stealing and mutating a target's concept of self, and replacing it with another, through a process of deconstruction and manipulation of perception that ‘smothers’ the target's narratives leading to domination over their information environment (Wylie, 2019). In early May 2025, evidence emerged of a phenomenon described as ‘ChatGPT induced psychosis’ (Klee, 2025; Zestyclementinejuice, 2025) whereby people have reportedly been drawn into delusions of grandeur via conversational interactions with ChatGPT as a conversational agent. As discussed, Meta's Cicero model can be understood as a proof of concept of the strategic application and amplification of these types of capabilities. As previously mentioned, LLMs may be integrated on both sides of a conversational interface—as both the conversational agent and as part of a predictive text completion service. Such a system would represent the total enclosure of an individual in the type of information environment described by Wylie, in which their sense of self is hijacked whilst they are continuously fed information that gradually alters their perceptions and intentions.

Modulatory power and the built environment

As we have returned to throughout this article, one must consider LLM developments alongside the large-scale advertising infrastructure in which they are surely to exist. As interest in LLMs grew in 2023, platform providers revised permissions to restrict unfettered access to valuable user data via their APIs (Davidson et al., 2023). This development extends existing platforms’ domination of existing infrastructural regimes while also positioning them as the beneficiaries of future sources of data, including data acquired through user interactions with LLMs. In each case, data gathered from user activity is preconditioned and structured within a variety of established informational enclosures, and should be scrutinized as such. By this view, entrapment in natural language enclosures simply adds resolution to the techniques that platforms already use to modulate user agency.

The threats implied by modulatory power take on new intensity as toolsets such as LLMs transform the built environments we inhabit. Following Luitse and Denkena, we understand LLMs as ‘vehicles of power’ in the political economy of AI whereby big tech companies have leveraged their resources to ‘shift power relations in their favour’. Command over vast infrastructures of compute, data and derived bodies of expertise ‘solidifies these companies’ role as rentiers in the political economy of AI’ (Luitse and Denkena, 2021), and extend data colonialism in ways that echo acts of historical colonialism (Penn, 2023). As we have attended to above, LLMs also intersect with other forms of power, such as psycho-political and socio-political power, that arise from wielding their capabilities against the people, entities, objects and processes that make up the lifeworld, and which could gradually be assimilated into the infrastructural enclosures of LLM-based systems.

Conclusion

While much is made of AI's competency in the context of gameplay (e.g., Go), the repurposing of such capabilities for manipulation and control has received markedly less attention. This article illustrates key areas in which LLMs may be instrumentalized to exert, modulate and project new forms of power, and situates them in relation to ambitions to establish a new generative foundation for computing. We provide an initial typology for this emerging research area by identifying four ways in which LLMs can be used to exert and modulate power: (a) via the pollution and standardization of digital environments as a whole, (b) via direct sites of interaction with an LLM system (e.g. AI personas), (c) via a proxy of the user trialled against predictive simulations predicated on persuasion and control (e.g. Cicero, Silicon Samples) and (d) via predictive simulations of social spheres (e.g. Silicon Societies). As stated, these four areas are neither mutually exclusive nor wholly distinct; it is their combination that concerns us. We highlight Cicero's strategic dialogue module as an early but alarming proof of concept for this broad and novel category of power.

Overall, we argue that while LLM systems may have initially been configured as powerful interpreters of human intent, which is the basis of many applications explored in 2023, this capability can be swiftly complemented with mechanisms for the modulation of human intent and action. LLMs may thus become the basis for powerful toolsets capable of intervening at a level of behaviour and temporality of human cognition that can gradually and adaptively direct and steer behaviour towards ‘strategic’ objectives and even intervene on the fragile ebb and flow of our thinking. We point to LLM's ensconcement in global advertising infrastructures as an important way to interpret the incentives behind its plausible stabilization into social worlds.

Footnotes

ORCID iD

Jonnie Penn

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Notes

References

Althusser

(1971) Ideology and Ideological State Apparatuses (Notes towards an Investigation). In: Lenin and Philosophy and Other Essays. London: New Left Press: 127–186. Translation by Ben Brewster.

Argyle

Busby

Fulda

, et al. (2023) Out of one, many: Using language models to simulate human samples. Political Analysis 31(3). Cambridge University Press: 337–351.

Arnold

Chauncey

Gajos

(2020) Predictive text encourages predictable writing. In: Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy, 17 March 2020, pp. 128–138. Cagliari: ACM. Available at: https://dl.acm.org/doi/10.1145/3377325.3377523.

Bakhtin

Brown

Dinan

, et al. (2022) Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378(6624): 1067–1074.

Benkler

Faris

Roberts

(eds) (2018) Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics. New York: Oxford University Press. 10.1093/oso/9780190923624.003.0009.

Bhargava

Witkowski

Shah

, et al. (2023) What’s the magic word? A control theory of LLM prompting. arXiv:2310.04444. arXiv. Available at: http://arxiv.org/abs/2310.04444 (accessed 22 October 2023).

Blair

(2010) Too Much to Know: Managing Scholarly Information before the Modern Age. New Haven, London: Yale University Press.

Boler

Davis

(eds) (2020) Affective Politics of Digital Media: Propaganda by Other Means. New York: Routledge: 1–378.

Bommasani

Hudson

Adeli

, et al. (2022) On the opportunities and risks of foundation models. arXiv:2108.07258. arXiv. Available at: http://arxiv.org/abs/2108.07258 (accessed 27 March 2023).

10.

Borges

(1941) The library of Babel. Available at: http://archive.org/details/TheLibraryOfBabel (accessed 27 January 2024).

11.

Bouk

(2017) The history and political economy of personal data over the last two centuries in three acts. Osiris 32(1): 85–106.

12.

Brand

Israeli

Ngwe

(2023) Using GPT for market research. Harvard Business Publishing Working Paper 23-062: 1–36. 10.2139/ssrn.4395751.

13.

Brundage

Avin

Clark

, et al. (2018) The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. Apollo - University of Cambridge Repository: 1–101. 10.48550/arXiv.1802.07228.

14.

Carlson

Montgomery

(2017) A pairwise comparison framework for fast, flexible, and reliable human coding of political texts. American Political Science Review 111(4): 835–843.

15.

Carr

(2022) Excess in the avant-garde of the data subject: Language and quantification in digital psychiatry. Terrain (76). 10.4000/terrain.23363.

16.

Carroll

Chan

Ashton

, et al. (2023) Characterizing manipulation from AI systems. In: Equity and Access in Algorithms, Mechanisms, and Optimization, Boston, MA, USA, 30 October 2023, pp. 1–13. ACM. Available at: https://dl.acm.org/doi/10.1145/3617694.3623226.

17.

Chaudhary

Penn

(2024) Beware the intention economy: Collection and commodification of intent via large language models. Harvard Data Science Review Online(Special Issue 5). 10.1162/99608f92.21e6bbaa.

18.

Chiang

(2023) ChatGPT Is a blurry JPEG of the web. The New Yorker, 9 February. Available at: https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web (accessed 16 March 2023).

19.

Chu

Andreas

Ansolabehere

, et al. (2023) Language models trained on media diets can predict public opinion. arXiv:2303.16779, 28 March. arXiv. Available at: http://arxiv.org/abs/2303.16779 (accessed 11 April 2023).

20.

Chun

WHK

Barnett

(2021) Discriminating Data: Correlation, Neighborhoods, and the New Politics of Recognition. Cambridge, Massachusetts: The MIT Press.

21.

Coyle

Moore

Kristensson

, et al. (2012) I did that! measuring users’ experience of agency in their own actions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 5 May 2012, pp. 2025–2034. CHI ‘12. Association for Computing Machinery. Available at: 10.1145/2207676.2208350.

22.

Daston

(2022) Rules: A Short History of What We Live By. The Lawrence Stone lectures. Princeton: Princeton University Press.

23.

Davidson

Wischerath

Racek

, et al. (2023) Platform-controlled social media APIs threaten open science. Nature Human Behaviour 7(12): 2054–2057.

24.

Demszky

Yang

Yeager

, et al. (2023) Using large language models in psychology. Nature Reviews Psychology 2: 688–701. 10.1038/s44159-023-00241-5.

25.

Diaz Ruiz

(2023) Disinformation on Digital media Platforms: A Market-Shaping Approach New Media & Society 27(4): 2188–2211. 10.1177/14614448231207644.

26.

Dong

Wang

Sreedhar

, et al. (2023) SteerLM: Attribute conditioned SFT as an (user-steerable) alternative to RLHF. arXiv:2310.05344. arXiv. Available at: http://arxiv.org/abs/2310.05344 (accessed 22 October 2023).

27.

Edwards

(2003) 7 - Infrastructure and Modernity: Force, Time, and social organization in the history of sociotechnical systems. In: Misa

Brey

Feenberg

(eds) Modernity and Technology. Cambridge: MIT Press: 185–225. Available at: http://ebookcentral.proquest.com/lib/cam/detail.action?docID=3338806.

28.

Floridi

Chiriatti

(2020) GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30(4): 681–694.

29.

Gibney

(2018) The scant science behind Cambridge Analytica’s controversial marketing techniques. Nature Online. 10.1038/d41586-018-03880-4.

30.

Goldstein

Sastry

Musser

, et al. (2023) Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv:2301.04246. arXiv. Available at: http://arxiv.org/abs/2301.04246 (accessed 5 October 2023).

31.

Hagendorff

(2024) Deception abilities emerged in large language models. Proceedings of the National Academy of Sciences 121(24): e2317967121.

32.

Haggerty

Ericson

(2000) The surveillant assemblage. British Journal of Sociology 51(4): 605–622.

33.

Han

B-C

(2017) Psychopolitics: Neoliberalism and new technologies of power1–96. tran. E ButlerVerso futures. London: New York: Verso.

34.

Hancock

Naaman

Levy

(2020) AI-Mediated Communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication 25(1): 89–100.

35.

Wallis

Rathje

(2023) Homophily in an artificial social network of agents powered by large language models. preprint, 24 June. In Review. Available at: https://www.researchsquare.com/article/rs-3096289/v1 (accessed 19 September 2023).

36.

Horton

(2023) Large language models as simulated economic agents: What can we learn from homo Silicus? arXiv:2301.07543, 18 January. arXiv. Available at: http://arxiv.org/abs/2301.07543 (accessed 11 April 2023).

37.

ICML Program Chairs (2023) LLM policy. Available at: https://icml.cc/Conferences/2023/llm-policy (accessed 19 February 2023).

38.

Jakesch

Bhat

Buschek

, et al. (2023) Co-writing with opinionated language models affects Users’ views. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 19 April 2023, pp. 1–15. CHI ‘23. Association for Computing Machinery. Available at: https://dl.acm.org/doi/10.1145/3544548.3581196.

39.

Jiang

Zhang

Cao

, et al. (2023) PersonaLLM: Investigating the ability of GPT-3.5 to express personality traits and gender differences. arXiv:2305.02547. arXiv. Available at: http://arxiv.org/abs/2305.02547 (accessed 17 July 2023).

40.

Juzek

Ward

(2024) Why Does ChatGPT ‘Delve’ so much? Exploring the sources of lexical overrepresentation in large language models. arXiv:2412.11385. arXiv. Available at: http://arxiv.org/abs/2412.11385 (accessed 9 February 2025).

41.

Klee

(2025) People are losing loved ones to AI-fueled spiritual fantasies. In: Rolling Stone. New York, NY: Penske Media Corporation.. Available at: https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/ (accessed 8 July 2025).

42.

Knight

(2023) Chatbot hallucinations are poisoning web search. Wired, 5 October. Available at: https://www.wired.com/story/fast-forward-chatbot-hallucinations-are-poisoning-web-search/ (accessed 7 October 2023).

43.

Kosinski

Stillwell

Graepel

(2013) Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences 110(15): 5802–5805.

44.

Kruppa

(2023) WSJ news exclusive | ChatGPT creator is talking to investors about selling shares at $29 billion valuation. Available at: https://www.wsj.com/articles/chatgpt-creator-openai-is-in-talks-for-tender-offer-that-would-value-it-at-29-billion-11672949279 (accessed 16 March 2023).

45.

Kumar

Lakkaraju

(2024) Manipulating large language models to increase product visibility. arXiv:2404.07981. arXiv. Available at: http://arxiv.org/abs/2404.07981 (accessed 13 April 2024).

46.

Lai

Hangya

Fraser

(2024) Style-specific neurons for steering LLMs in text style transfer. arXiv:2410.00593. arXiv. Available at: http://arxiv.org/abs/2410.00593 (accessed 13 January 2025).

47.

Luitse

Denkena

(2021) The great transformer: Examining the role of large language models in the political economy of AI. Big Data & Society 8(2): 20539517211047734.

48.

Marlin

(2002) Propaganda and the Ethics of Persuasion. Ontario: Broadview Press.

49.

Maslej

Fattorini

Perrault

, et al. (2024) Artificial intelligence index report 2024. Available at: https://aiindex.stanford.edu/report/ (accessed 7 February 2025).

50.

Matz

Teeny

Vaid

, et al. (2023) The Potential of Generative AI for Personalized Persuasion at Scale. OSF. DOI: 10.31234/osf.io/rn97c.

51.

Matz

Kosinski

Nave

, et al. (2017) Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the National Academy of Sciences 114(48): 12714–12719.

52.

Mejias

(2013) Off the network: Disrupting the digital world. In: Electronic Mediations. Minneapolis: University of Minnesota Press: 1–224. https://doi.org/10.5749/j.ctt3fh6jh .

53.

Meta (2023) Introducing new AI experiences across our family of apps and devices. In: Meta. Available at: https://about.fb.com/news/2023/09/introducing-ai-powered-assistants-characters-and-creative-tools/ (accessed 20 December 2023).

54.

Meta AI (2022) CICERO: An AI agent that negotiates, persuades, and cooperates with people. Available at: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/ (accessed 16 March 2023).

55.

Mills

Sætra

(2024) The autonomous choice architect. AI & Society 39: 583–595. 10.1007/s00146-022-01486-z.

56.

Montgomery

(2024) Mother says AI Chatbot led her son to kill himself in lawsuit against its maker. The Guardian Online. 23 October. Available at: https://www.theguardian.com/technology/2024/oct/23/character-ai-chatbot-sewell-setzer-death.

57.

Morgan

(2012) The World in the Model: How Economists Work and Think. New York, NY: Cambridge University Press.

58.

Morris

Uddin

(2025) Big Tech lines up over $300bn in AI spending for 2025. Financial Times Online. 7 February. Available at: https://www.ft.com/content/634b7ec5-10c3-44d3-ae49-2a5b9ad566fa (accessed 12 February 2025).

59.

Nadler

Crain

Donovan

(2018) The Political Perils of Online Ad Tech. New York, NY: Data & Society. Available at: https://datasociety.net/library/weaponizing-the-digital-influence-machine/.

60.

Nay

(2023) Large language models as corporate lobbyists. arXiv:2301.01181. arXiv. Available at: http://arxiv.org/abs/2301.01181 (accessed 3 February 2023).

61.

Nixon

(2011) Slow violence and the environmentalism of the poor. In: Slow Violence and the Environmentalism of the Poor. Cambridge, MA: Harvard University Press. Available at: https://www.degruyter.com/document/doi/10.4159/harvard.9780674061194/html.

62.

NVIDIA (2023) Keynote at SIGGRAPH. Available at: https://www.youtube.com/watch?v=Z2VBKerS63A (accessed 20 December 2023).

63.

OpenAI DevDay (2023). Opening Keynote. Available at: https://www.youtube.com/watch?v=U9mJuUkhUzk (accessed 14 November 2023).

64.

Ornstein

Blasingame

Truscott

(2023) How to train your stochastic parrot: Large language models for political texts.

65.

Park

O’Brien

Cai

, et al. (2023a) Generative agents: Interactive Simulacra of human behavior. arXiv:2304.03442, 6 April. arXiv. Available at: http://arxiv.org/abs/2304.03442 (accessed 11 April 2023).

66.

Park

Zou

Shaw

, et al. (2024) Generative agent simulations of 1,000 People. arXiv:2411.10109. arXiv. Available at: http://arxiv.org/abs/2411.10109 (accessed 18 November 2024).

67.

Park

Schoenegger

Zhu

(2023b) Diminished diversity-of-thought in a standard large language model. arXiv:2302.07267. arXiv. Available at: http://arxiv.org/abs/2302.07267 (accessed 19 September 2023).

68.

Penn

(2023) Animo nullius: On AI’s origin story and a data colonial doctrine of discovery. BJHS Themes 8: 19–34. 10.1017/bjt.2023.14.

69.

Pickering

Garrod

(2013) An integrated theory of language production and comprehension. The Behavioral and Brain Sciences 36(4): 329–347.

70.

Poerksen

(1995) Plastic Words: The Tyranny of a Modular Language. Pennsylvania: Penn State University Press.

71.

Robison

(2024) Google promised a better search experience — now it’s telling us to put glue on our pizza. Available at: https://www.theverge.com/2024/5/23/24162896/google-ai-overview-hallucinations-glue-in-pizza (accessed 28 January 2025).

72.

Rooney

Field

(2025) OpenAI in talks to raise funding that would value AI startup at up to $340 billion. CNBC, 30 January. Available at: https://www.cnbc.com/2025/01/30/openai-in-talks-to-raise-up-to-40-billion-at-340-billion-valuation.html (accessed 7 February 2025).

73.

Rosenberg

(2023a) Generative AI as a dangerous new form of Media. In: 17th International Multi-Conference on Society, Cybernetics and Informatics, Orlando, Florida, United States, September 2023, pp. 165–170. Available at: https://www.iiis.org/DOI2023/HA408FU (accessed 7 February 2025).

74.

Rosenberg

(2023b) The metaverse and conversational AI as a threat vector for targeted influence. In: 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, 8 March 2023, pp. 0504–0510. IEEE. Available at: https://ieeexplore.ieee.org/document/10099167/ (accessed 7 February 2025).

75.

Rosenbusch

Stevenson

van der Maas

HLJ

(2023) How accurate are GPT-3’s hypotheses about social science phenomena? Digital Society 2(2): 26.

76.

Sarkar

(2023) Exploring perspectives on the impact of artificial intelligence on the creativity of knowledge work: Beyond mechanised plagiarism and stochastic parrots. In: Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work, Oldenburg Germany, 13 June 2023, pp. 1–17. ACM. Available at: https://dl.acm.org/doi/10.1145/3596671.3597650.

77.

Serapio-García

Safdari

Crepy

, et al. (2023) Personality traits in large language models. preprint, 28 August. In Review. Available at: https://www.researchsquare.com/article/rs-3296728/v1 (accessed 19 September 2023).

78.

Seshia

Sadigh

Sastry

(2020) Towards verified artificial intelligence. arXiv:1606.08514. arXiv. Available at: http://arxiv.org/abs/1606.08514 (accessed 27 January 2024).

79.

Sharp

Danenberg

Bellman

(2018) Psychological targeting. Proceedings of the National Academy of Sciences 115(34): E7890. 10.1073/pnas.1810436115.

80.

Shumailov

Shumaylov

Zhao

, et al. (2024) AI Models collapse when trained on recursively generated data. Nature 631(8022): 755–759.

81.

Sommart

(2023) Formal verification of neural networks. Available at: https://cds.cern.ch/record/2867415 (accessed 27 January 2024).

82.

Staab

Vero

Balunović

, et al. (2023) Beyond memorization: Violating privacy via inference with large language models. arXiv:2310.07298. arXiv. Available at: http://arxiv.org/abs/2310.07298 (accessed 19 October 2023).

83.

Stark

(2018) Algorithmic psychometrics and the scalable subject. Social Studies of Science 48(2): 204–231.

84.

Stella

Ferrara

De Domenico

(2018) Bots increase exposure to negative and inflammatory content in online social systems. Proceedings of the National Academy of Sciences 115(49): 12435–12440.

85.

Synthetic Users (n.d.). Available at: https://www.syntheticusers.com/ (accessed 17 March 2024).

86.

Tapus

Ţăpuş

Matarić

(2008) User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics 1(2): 169–183.

87.

Tessler

Bakker

Jarrett

, et al. (2024) AI Can help humans find common ground in democratic deliberation. Science 386(6719): eadq2852.

88.

Tiku

(2017) Welcome to the next phase of the Facebook backlash. Wired, 21 May. Available at: https://www.wired.com/2017/05/welcome-next-phase-facebook-backlash/ (accessed 10 February 2024).

89.

Veselovsky

Ribeiro

West

(2023) Artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899. arXiv. Available at: http://arxiv.org/abs/2306.07899 (accessed 13 October 2023).

90.

Wartofsky

(1979) Models, metaphysics and the vagaries of empiricism. In: Wartofsky

(ed.) Models: Representation and the Scientific Understanding. Boston Studies in the Philosophy of Science. Dordrecht: Springer Netherlands, pp. 24–39. Available at: 10.1007/978-94-009-9357-0_3.

91.

Webb

Goel

(2025) Leaked docs reveal Meta is training its chatbots to message you first, remember your chats, and keep you talking. In: Business Insider Nederland. Amsterdam: Sijthoff Media. Available at: https://www.businessinsider.nl/leaked-docs-reveal-meta-is-training-its-chatbots-to-message-you-first-remember-your-chats-and-keep-you-talking (accessed 8 July 2025).

92.

Weidinger

Uesato

Rauh

, et al. (2022) Taxonomy of risks posed by language models. In: 2022 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, 20 June 2022, pp. 214–229. FAccT ‘22. Association for Computing Machinery. Available at: 10.1145/3531146.3533088.

93.

Whittaker

(2021) The Steep Cost of Capture. Interactions. New York, NY: ACM. 10.1145/3488666.

94.

Williams

(2018) Stand out of Our Light: Freedom and Resistance in the Attention Economy (1st edn). Cambridge University Press. Available at: https://www.cambridge.org/core/product/identifier/9781108453004/type/book (accessed 2 January 2019).

95.

Williams

(2024) LLMs are ever more convincing, with important consequences for election disinformation. Available at: https://www.turing.ac.uk/blog/llms-are-ever-more-convincing-important-consequences-election-disinformation (accessed 6 February 2025).

96.

Williams

Burke-Moore

Chan

RS-Y

, et al. (2024) Large language models can consistently generate high-quality content for election disinformation operations. arXiv:2408.06731. arXiv. Available at: http://arxiv.org/abs/2408.06731.

97.

Williams

James

(2018) Stand out of our Light: Freedom and Resistance in the Attention Economy. Cambridge: Cambridge University Press.

98.

Wylie

(2019) Mindf*ck: Cambridge Analytica and the Plot to Break America. London: Profile Books.

99.

Xiao

Zhou

Ping

, et al. (2024) Exploring neuron interactions and emergence in LLMs: From the multifractal analysis perspective. arXiv:2402.09099. arXiv. Available at: http://arxiv.org/abs/2402.09099.

100.

Yeung

(2017) Hypernudge’: Big Data as a mode of regulation by design. Information, Communication & Society 20(1): 118–136.

101.

Zestyclementinejuice (2025) Chatgpt induced psychosis. In: r/ChatGPT. Reddit Post. Available at: https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_induced_psychosis/ (accessed 8 July 2025).

102.

Zittrain

(2022) Intellectual debt: With great power comes great ignorance. In: Mueller

Kellmeyer

Voeneky

, et al. (eds) The Cambridge Handbook of Responsible Artificial Intelligence: Interdisciplinary Perspectives. Cambridge Law Handbooks. Cambridge: Cambridge University Press, pp. 176–184. Available at: https://www.cambridge.org/core/books/cambridge-handbook-of-responsible-artificial-intelligence/intellectual-debt/F5D4CF05857D072ABED383AE7A3222E4.

103.

Zollmann

(2017) Bringing propaganda back into news media studies. Critical Sociology 45(3): 329–345. 10.1177/0896920517731134.