Abstract
In a world increasingly shaped by generative AI systems like ChatGPT, the absence of benchmarks to examine the efficacy of oversight mechanisms is a problem for research and policy. What are the structural conditions for governing generative AI systems? To answer this question, it is crucial to situate generative AI systems as regulatory objects: material items that can be governed. On this conceptual basis, we introduce three high-level conditions to structure research and policy agendas on generative AI governance: industrial observability, public inspectability, and technical modifiability. Empirically, we explicate those conditions with a focus on the EU’s AI Act, grounding the analysis of oversight mechanisms for generative AI systems in their granular material properties as observable, inspectable, and modifiable objects. Those three conditions represent an action plan to help us perceive generative AI systems as negotiable objects, rather than seeing them as mysterious forces that pose existential risks for humanity.
Keywords
Introduction
Across the world, governments are now confronted with the complex challenge of setting up oversight mechanisms for generative AI systems, from democratic to authoritarian contexts. Consider Italy, which, in response to concerns about violations of user data privacy, imposed a temporary ban on ChatGPT in early 2023 (Satariano, 2023). Or take China, which has been noted for rolling out some of the world’s earliest generative AI regulations. China’s Interim Administrative Measures for Generative AI Services, which came into effect on 15 August 2023, combine several oversight instruments. Generative AI service providers are required to file their algorithms for a security assessment, which authorities list in an “algorithm registry” (Sheehan, 2023). Other tools are disclosure obligations regarding training data sets used to develop generative AI applications and the technical integration of mandatory watermarks for AI-generated content. Those types of measures show that the toolkit of oversight approaches propelled by the sophistication of generative AI systems is rapidly evolving.
Maneuvering through this volatile empirical landscape is challenging, not only due to inconsistent governance approaches at an international level but also because new variations of generative AI technologies emerge every week. Consequently, there is a need to develop a conceptual vocabulary for high-level oversight conditions—categories that are not limited to one particular regulatory context but can serve as a systematic basis for cross-country comparisons and inform research and policy agendas. In other words, what are the structural conditions for governing generative AI systems? In a world increasingly shaped by generative AI systems such as ChatGPT, the absence of benchmarks to measure the efficacy of oversight mechanisms is a problem for research and policy. Without a clear conceptual framework to interpret these fleeting, quick-paced developments as empirical expressions of structural conditions for the governance of generative AI, it is difficult to navigate the global AI policy landscape and compare measures across countries.
To address the governance question, we introduce a nested structure of three oversight conditions for generative AI systems: industrial observability, public inspectability, and technical modifiability. The first condition refers to the macro-level observation of the AI industry, including the relationship between consumer-facing applications and computational infrastructure. The second condition points to the bitwise inspection of the technical underpinnings of generative AI systems, especially their underlying “foundation models” (Bommasani et al., 2021). The third condition addresses the extent to which those foundation models can be modified, such as by enabling digital watermarks. Importantly, those conditions are interdependent. It is only when they come together that they create a coherent framework for research and policy upon which regulators can then act. Observation of industry structures alone is futile without the ability to inspect and modify generative AI systems. Conversely, both the inspection and modification of those systems never occur in a political-economic vacuum, but always within identifiable industry and government structures that need to be observed closely.
The crux of our framework lies in the fact that those three anchor points enable us to perceive generative AI systems as inherently negotiable objects, rather than seeing them as mysterious forces that pose existential risks for humanity. Their negotiability derives from a recognition that we can ground the study of oversight structures for generative AI systems in their granular material properties as observable, inspectable, and modifiable objects. Systems can be more or less observable, more or less inspectable, and more or less modifiable. As Rieder and Hofmann (2020: 3) argue, “unlike transparency, which nominally describes a state that may exist or not, observability emphasizes the conditions for the practice of observing in a given domain.” The same nuance applies to inspectability and modifiability. It is not a yes or no question; it is a more-or-less question—a negotiable field of action. Nonetheless, for this argument to carry empirical weight, it must be developed vis-à-vis a specific regulatory framework; it cannot remain an abstract claim. Therefore, we explicate the real-world implications of those three conditions with a focus on the EU’s proposed Artificial Intelligence Act, analyzing to what extent the regulation accounts for the granular properties of generative AI systems. 1
As these systems permeate economic, political, and cultural interactions, we must understand how they can be opened to regulatory scrutiny. In order to render generative AI systems governable, we first situate them as “regulatory objects” (Fisher, 2014) in science and technology studies (STS) scholarship. On this basis, we use the EU’s AI Act to think through the regulatory manifestations of our three oversight conditions. We conclude by stressing how both research and policy agendas can build on those conditions.
Generative AI systems as regulatory objects
Generative AI systems are built on top of foundation models that detect and store patterns of dependencies between elements (e.g. words or pixels) in training data sets. Foundation models can be fine-tuned to downstream tasks after training, but they can also generate outputs based on the learned patterns, starting from user-generated prompts (Bommasani et al., 2021). Such new outputs may be text, images, music, speech, and video. We specifically focus on large language models (LLMs), which are trained on large text data sets in order to generate text outputs. Examples of foundation models are OpenAI’s Generative Pre-Trained Transformer (GPT) model, which underlies ChatGPT, Google’s LaMDA model which powers Bard, and Meta’s LLaMA model.
In this section, we situate generative AI systems as regulatory objects. According to Fisher (2014: 163) a “regulatory object” can be thought of as a “thing that is understood by regulatory actors as the ‘thing’ to be regulated. For something to be a regulatory object it must be capable of being understood as a material item that can be governed.” To outline the implications of this fundamental assumption for generative AI systems, we approach them through three analytical themes: material referentiality, algorithmic opacity, and regulatory co-production. These three themes not only help in conceptualizing generative AI systems as multi-layered regulatory objects, but each theme also drives the formulation of a subsequent condition for generative AI governance. Each of those themes underscores the importance of developing oversight instruments to observe, inspect, and modify quickly evolving regulatory objects, from trading algorithms to genetically modified organisms.
Material referentiality: how to observe the regulatory object?
The first theme, material referentiality, poses the fundamental question: How to observe “the thing to be regulated” (Fisher, 2014: 163)? The field of science and technology studies (STS) has played a pivotal role in scrutinizing the dynamics between complex regulatory objects and governance frameworks. Offering a clear response on how to trace a regulatory object proves challenging due to the complexity and concealed nature of its material properties, as well as the degree to which these properties may be in a state of change. By observing, we mean tracing how regulatory objects are situated in wider industry structures.
A case in point is the governance of high-frequency trading algorithms that are used in financial markets to execute transactions. Seyfert (2021: 6) demonstrates in his analysis of the German High Frequency Trading Act that “the demarcation of a manipulative trading algorithm is only a derivative second step after objectifying the algorithm as a distinct object.” In this case, the trading algorithm needs to be meticulously distinguished from both the trading platform and the trading firm. Although these three governance entities are inherently interconnected, it is pivotal to differentiate them analytically. Without a clear specification of what constitutes the regulatory object (and how it can be defined), it is impossible for regulators to observe it over time. As Seyfert (2021: 12) explains, “trading firms were obliged to redraw the demarcations of algorithmic objects within their socio-technical systems, strictly separate algorithmic and human activities.” In this case, clear-cut analytical separations made the operations of quickly changing and secretive trading algorithms observable to regulators.
While generative AI systems are therefore not the first algorithmic regulatory objects that invoke thorny questions about material referentiality, they are also not the first regulatory objects that are difficult to trace empirically. Consider the regulation of genetically modified organisms (GMOs), such as transgenic agricultural seeds that are developed to improve crop productivity. For example, transgenic seeds are used to grow corn that is genetically modified to resist destructive pests, leading to increased yields. In his study of GMOs as governance objects, Lezaun (2006: 501) traces the “administrative practices and detection instruments able to track GMOs throughout the food production system, from the farm to the table.” Those “infrastructures of referentiality” (Lezaun, 2006: 505) include both bureaucratic nominalism (i.e. giving a clear name to the regulatory object to make it bureaucratically unambiguous) and the standardization of detection methods (i.e. identifying the regulatory object as such). Without an in-depth understanding of the transgenic seed industry, it would be impossible to maintain an overview of different seed variations and their later metamorphosis into food products.
Bureaucratic nominalism and the standardization of detection methods are also highly relevant in the context of generative AI systems. Granted, their material complexities differ from other regulatory objects studied in STS scholarship. Consider the example of aircraft. Aircraft comprise many components like engines, propellers, and other parts (Law, 2002). Before they can enter the market, regulatory bodies must grant approval for all components. Foundation models, by contrast, consist of fewer component types, essentially artificial neurons, but the multitude of connections between them allows for a mind-boggling variety of architectural configurations that affect the generation of outputs. The example of the aviation industry clearly illustrates that the issue of material referentiality is critical. If different authorities within one jurisdiction respond with different understandings of what the regulatory object is, this disparity would severely hamper their power to oversee the aviation industry. At the same time, if there were significant differences in material referentiality between jurisdictions, the potency of cross-border regulatory regimes would be dangerously limited—a highly relevant aspect in light of the globally interconnected nature of AI governance (Veale et al., 2023).
Inspired by this strand of research, we contend that a similar analytical demarcation of interconnected governance objects is urgently needed in the context of generative AI systems. Without specificity of what exactly the constitutive elements of the regulatory object are (and how they relate to each other), it cannot be made publicly inspectable or technically modifiable. Because generative AI systems are not isolated lines of code that exist in a societal vacuum, their roles as regulatory objects need to be studied in conjunction with the political-economic context within which they proliferate and reconfigure cultural practices. Comparable to trading algorithms, generative AI systems are also extremely changeable and volatile objects, as they are modified constantly (e.g. by training or fine-tuning). This structural mutability raises the question of how the workings of those systems can be inspected by regulators in precisely defined ways.
Algorithmic opacity: how to inspect regulatory object?
The fact that unambiguously demarcating and delimiting generative AI systems as regulatory objects is extremely challenging leads to the second analytic theme: algorithmic opacity. The key question of this theme is: how to inspect the regulatory object? It is no secret that high-profile commercial algorithms—irrespective of whether or not they are labeled as AI—are impervious to outside scrutiny. A significant body of research has dealt with the thorny issue of algorithmic opacity (Brevini and Pasquale, 2020; Burrell, 2016; Seaver, 2017). A common way to approach algorithmic opacity is to see it as an epistemological limitation: a barrier to knowledge production and, by extension, regulatory action. Burrell (2016: 2), for example, aptly points to the “mismatch between mathematical optimization in high-dimensionality, characteristic of machine learning and the demands of human-scale reasoning and styles of semantic interpretation.” However, there is a danger in interpreting mathematical complexity as an unregulatable realm. “Inspecting” the regulatory object does not mean that all of its intricacies can or should be revealed (as if there were no intellectual property rights or trade secrets). Instead, the ability to inspect means that there are regulatory mechanisms to inspect its workings in standardized ways—whether its behavior or its outcomes.
To stay with the case of high-frequency trading algorithms, regulators left “behind the idea of an algorithm as a black box to be opened and regulated” (Seyfert, 2021: 9). Instead, the algorithm’s behavior was meticulously categorized on the basis of specific parameters. Identifying what those properties are in the case of generative AI systems—rather than pointing to their structural algorithmic opacity—is a key matter. If no regulatory authority would curtail its agency by classifying chemicals or genetically modified food as mysterious or ungovernable objects, why should this be the case for AI systems? Ascribing algorithmic opacity as an inevitable feature of generative AI systems represents a discursive maneuver by their proprietors to avoid oversight measures such as transparency obligations. OpenAI, for example, has not disclosed any critical details about the training data set it used for training ChatGPT. We only know some basic information, such as that an early version of ChatGPT was trained on a vast data set of 45 terabytes, equivalent to around 300 billion words. While OpenAI has repeatedly called for new regulations, the firm has a vested interest in defining and influencing regulatory norms and principles, and prefers corporate oversight approaches (Perrigo, 2023). Corporate oversight means that the same companies producing AI systems should also set the conditions for overseeing their repercussions.
To push back against arguments that prefer corporate oversight approaches, we argue that it is possible to appreciate that generative AI systems are indeed mathematically complex objects that partly transcend human-scale reasoning (Burrell, 2016), while also stressing why their developers may deliberately hide information about those systems. In the case of large language models, for example, their opacity is a direct result of corporate strategies of not disclosing details about essential components, such as training data sets. At the same time, training data sets are not the only type of relevant information. Without precision about what types of information need to be accessed and why, regulatory responses fall short in capturing points of leverage to “render visible the relational interactions between algorithmic strategies” (Seyfert, 2021: 8), such as the AI-enabled generation of harmful content.
Rather than claiming algorithmic opacity, we argue it is more important to carve out granular levels of inspectable technical details that need to be made available to regulators. As the next section demonstrates, this knowledge about the inner workings of generative AI systems is also relevant to how they can be modified as regulatory objects.
Regulatory co-production: how to modify the regulatory object?
The third and final theme, regulatory co-production, looks beyond how regulatory objects and regulatory frameworks are positioned to each other; it asks how they are co-produced by each other. The principal question of this theme is: how to modify the regulatory object? How can its material properties be changed, and why? As Fisher (2014: 165) writes in the context of chemical objects, “the role of co-production may be recognized in relation to the question of the safety of a chemical but not much the identity of the chemical itself.” The crux is how the problematic properties of regulatory objects (and their potential implications) trigger regulatory responses that purport to change those material features. In the example of chemicals, safety concerns prevail. In the case of high-frequency trading algorithms, issues of financial manipulation necessitate regulations. In the case of generative AI systems, concerns about misinformation, manipulation, and the unauthorized use of sources (e.g. copyright infringement) may feed regulatory motives. Importantly, those systems can be used for a wide range of purposes (Helberger and Diakopoulos, 2023). Therefore, the broader context of where, how, and why a regulatory object can be made modifiable needs to be explicated.
For our purposes, the key consideration is how generative AI systems can be rendered modifiable as part of AI oversight structures. When examining technical modifiability in the context of generative AI systems, we understand their granular material properties as layered arrangements that are continuously co-produced by regulations. The idea of co-production goes back to Jasanoff’s (2004) groundwork, which assumes that “knowledge and its material embodiments are at once products of social work and constitutive of forms of social life.” For this article, it can be stated that generative AI systems are at once products of regulatory frameworks and constitutive of regulatory responses. For example, if generative AI systems were not opaque arrangements in the first place, there would not be a need for transparency requirements. Those transparency obligations might not directly change the material properties of AI systems, but they might inform downstream modifications to particular system features after inspection and recommendation. Investigating how providers of commercial foundation models, such as OpenAI’s GPT-4, could be forced to make their models modifiable so that they comply with domestic laws requires a sensitivity to the recursive nature of regulatory co-production. In other words, technical modifiability is not a one-way street, especially not in such a rapidly evolving domain as the governance of generative AI systems.
To summarize this section, the role of generative AI systems as regulatory objects cannot be explained without adequately accounting for how they are rendered observable (and to whom), what layers of information about them are made inspectable (and to whom), and how (and why and by whom) they can or should be modifiable—before they can be governed as material items. In the next section, we substantiate this argument by turning those themes into three operationalizable oversight conditions.
Observe, inspect, modify and the EU’s AI Act
In this section, we use three conditions for probing the efficacy of oversight mechanisms over generative AI systems: industrial observability, public inspectability, and technical modifiability. Empirically, we explicate those conditions with a focus on the EU’s AI Act, which is widely considered a blueprint for other countries and thus a particularly salient case study to underpin our argument.
At its core, the AI Act relies on a risk-based approach, whereby some AI technologies are labeled as posing “unacceptable risk” (e.g. facial recognition software) while others are deemed “high risk” and “limited risk.” The AI Act proposes a stringent certification regime for systems of high risk and less stringent transparency obligations for AI systems of limited risk. AI systems categorized as posing unacceptable risk would be banned. The initial proposal by the European Commission in April 2021 did not include specific provisions for generative AI systems. In part triggered by the release of ChatGPT, this has changed. Although negotiations are ongoing at the time of finalizing this article, the most recent draft shared by the European Parliament in June 2023 does not classify generative AI systems as high risk, and provides specific provisions for generative AI providers. Those requirements include but are not limited to “disclosing that content was generated by AI,” “preventing the model from generating illegal content,” and “publishing summaries of copyrighted data used for training” (European Parliament, 2023). As the preliminary text of the AI Act does not include technical or enforcement details, those requirements would benefit from a more granular explication of what needs to be regulated by whom for what specific purposes (Busuioc et al., 2023).
In the following sections, we first introduce the respective oversight condition on a general level and subsequently explain its manifestations in the context of the EU’s AI Act. We concentrate not only on “what” needs to be governed, but also on “how” and “why” to make generative AI systems observable, inspectable, and modifiable.
Industrial observability: how to observe generative AI systems?
To make generative AI systems governable for regulatory oversight as material entities, we must begin by elucidating their constitutive elements and their place within broader industry dynamics. Without an empirical analysis of how different layers of generative AI systems—such as foundation models—fit within platform ecosystems, and how they relate to other entities (e.g. platform companies as infrastructure providers), we cannot identify them as regulatory objects. As STS scholarship (Lezaun, 2006) shows, a clear delineation of what needs to be governed according to precisely defined technical parameters and detection methods is crucial. Only after pinning down the what, we can address how generative AI systems can be governed.
Crucially, a dynamic and processual perspective is required when dealing with ever-changing generative AI systems, rather than relying on static or rigid governance categories. In Rieder and Hofmann’s (2020: 14) words, to “account for the distributed and dynamic character of platform life, as it emerges from the interaction between policies, design choices, and use practices, continuous observation needs to reach beyond legal and technical specifications.” As such, it would be insufficient to focus on only one dimension, such as highly visible consumer-facing applications like ChatGPT. Rather, the starting point for observation is to acknowledge “generative AI” as a complex relation, in which computational infrastructure, foundation models, and consumer-facing applications are intricately intertwined (Figure 1).
Components of generative AI systems as observable objects.
We interpret Figure 1 through the “platformization tree” metaphor (Van Dijck, 2021). At the top of this tree are consumer-facing AI applications (e.g. chatbots), which depend on generative foundation models beneath them to run on a daily basis. Foundation models act as the trunk of the tree, representing remarkably resource-intensive computational processes, requiring high amounts of processing power. High-profile foundation models like GPT-4 are becoming quintessential gatekeeping services situated at the heart of the AI ecosystem, where their providers leverage the power of platforms to control digital markets (Gillespie, 2018; Van Dijck, 2021). At the root level of the tree are the providers of computational resources, such as cloud computing services, data center capacity, and energy resources, to run models. Cloud computing divisions of Big Tech companies such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform dominate the provision of computational infrastructure for foundation models. They attempt to foster the adoption of AI systems in the most widespread way, as their function underlies both foundation models and consumer-facing applications built on top of them. However, Big Tech companies are also dependent on chipmakers like Nvidia and semiconductor firms like Taiwan Semiconductor Manufacturing Company Limited (TSMC).
While consumer-facing applications receive the most attention from policymakers and the public, the granular material properties of foundation models often remain underreported. Arguably, the important role of models in forming the tree’s trunk can neither be separated from its branches (applications) nor its physical roots (infrastructure), and the constitutive layers of generative AI systems cannot be easily taken apart. For example, both consumer-facing applications and foundation models evolve quickly, as shown by the optimization of ChatGPT based on Reinforcement Learning with Human Feedback (RLHF 2 ). Nonetheless, in the remainder of this article, we single out foundation models as inspectable and modifiable layers of generative AI systems. The reason for this decision lies in the fact that the EU’s AI Act defines “providers of foundation models” (European Parliament, 2023) as the key actors to be regulated, subjecting them to designated transparency obligations and technical modifications. For example, the disclosure of training data sets that are covered by copyright laws.
The question of how to operationalize and enforce such transparency obligations is at the forefront of EU regulatory debates. To influence AI regulations according to their interests, industry-dominating foundation model providers can leverage consumer pressure. For example, Sam Altman, the CEO of OpenAI, which owns ChatGPT, has raised the prospect of withdrawing from the European Union’s Digital Single Market should the company find it impossible to adhere to the EU AI Act. As of January 2023, reports indicated that ChatGPT was being used by more than 100 million individuals daily (Milmo, 2023). The substantial user base affords OpenAI’s political pressure, as EU policymakers do not want to be seen as obstructing AI innovation or technological progress. In a similar way as the ride-hailing company Uber has set up consumer petitions aimed at regulators, municipalities, and federal governments in the pursuit of corporate lobbying, similar efforts of consumer mobilisation are likely in the case of OpenAI.
In order to capture foundation models as regulatory objects, three components need to be considered: neural network architectures, training data sets, and training procedures (Figure 1). Public debates predominantly focus on the role of training data sets, primarily due to evident concerns surrounding copyright issues and privacy violations. However, neural network architectures and training procedures are also essential components for the enforcement of oversight mechanisms. In terms of neural network architectures, most foundation models rely on the transformer architecture, 3 introduced by Google researchers in 2017 (Vaswani et al., 2017). Google’s seminal paper introducing this architecture is publicly accessible, and it later served as a technical basis for OpenAI to build its proprietary GPT model, which now underlies ChatGPT (Luitse and Denkena, 2021). It is much more challenging to render proprietary neural network architectures observable than their open-source counterparts. The same caveat applies to the training procedures for fine-tuning models for specific tasks (e.g. conversational use), which commonly remain inscrutable to regulators. Far from being solely shaped by mathematical laws, neural network architectures, training data sets, and training procedures are also intricately shaped by the economic imperatives and restraints of foundation model developers—and by their role in underpinning consumer-facing applications and their reliance on computational infrastructure (Figure 1). Fostering “ethical” self-regulation of foundation model providers would hamper oversight as there is no economic benefit in making foundation models inspectable to regulators.
Proceeding from the bird’s eye view of observing industry structures to the bitwise inspection of models, data, and code, the next section specifies what layers of information about foundation models need to be made inspectable to regulators. Observation alone is futile without regulatory access to key parameters of those complex models.
Public inspectability: how to inspect generative AI systems?
The condition of “public inspectability” raises the question of openness to the prying eyes of outsiders. “Inspectability” means that generative foundation models should become available for deep regulatory inspection at the most detailed level of saved network parameters and hyperparameters. “Public” means they should be treated as instruments of public knowledge and common goods. As such, public inspectability poses the dilemma of whether foundation models should be fully open (as some proponents of open-source argue) or fully closed (as proprietors claim). Solaiman (2023) distinguishes six levels of inspectability along a gradient between two poles: fully closed; gradual or staged access; hosted access; cloud-based or API access; downloadable access; and fully open. Fully closed means that generative AI systems remain sealed off by their developers and thus inaccessible to regulators and the public, while the fully open option means that they are fully accessible to the public. In between those two poles are several other granular access levels; openness is not a binary question.
However, we argue that the condition of public inspectability is more complex than a question of whether a model is closed or open, as there is a precarious balance between the public values of safety/security and openness/democratic control. Public inspectability also relates to concerns about the detection (and control) of misinformation, manipulation, and unauthorized use of sources. In advocating for seeing openness as a continuum, we attempt to counterbalance corporate attempts to monopolize the power in deciding which parts of those systems are accessible to public scrutiny (Liesenfeld et al., 2023; Widder et al., 2023). Model cards, for example, have evolved as bottom-up standardization devices for AI developers to state all essential components of a foundation model, including a specification of (domain-specific) training data sets, but also biases and ethical considerations (Mitchell et al., 2019). In the case of semi-closed or fully closed models, opacity sometimes comes as thinly veiled transparency: model cards previously released by OpenAI and Meta were criticized by scholars (Birhane et al., 2021) as well as policymakers (Blumenthal and Hawley, 2023) for being seriously—perhaps deliberately—underspecified.
Therefore, the condition of public inspectability poses a nuanced question: how can generative foundation models be made inspectable to whom and for what purposes? Which precise aspects/levels of these models should be made inspectable to experts (for regulatory purposes) and which should be open to the public at large? If we compare the regulation of foundation models to the levels of “public inspectability” in the aviation industry, we can witness how different levels of openness and inspectability apply to qualified and certified experts (e.g. regulators licensed to control and test technical safety valves) and the public at large (e.g. the aviation industry needs to publish engine specifications). In order to elaborate on the means by which generative foundation models may become subject to public inspectability and technical modifiability, it is essential to have a grasp of their underlying operations: what information is necessary for inspection? We propose to distinguish between five layers of information (Table 1).
Five layers of inspectable information about generative foundation models.
Training data sets for foundation models: Commercial models tend not to give access to the full data sets they have been trained on, and at best give non-exact pointers to the data sets (Ferrari et al., 2023). Open models tend to give pointers to the data sets, but often leave out technical details on selections and applied pre-processing methods. OpenAI’s GPT models were trained on openly available data and data acquired under third-party licenses. GPT-3.5, for example, was trained on 45 terabytes of text data, which adds up to approximately 300 billion words extracted from public sources like Wikipedia, Common Crawl, and GitHub, but also from undisclosed other sources.
Domain-specific training data sets for fine-tuning: Commercial systems typically shield these data sets, citing competitive reasons. A counter-example is Google’s Med-PaLM, where Google has been open about which databases were used for fine-tuning PaLM for medical purposes, including more than 200,000 question-answer sets from medical exams, and consumer questions and reference answers by the US National Institute of Health (Singhal et al., 2023). Open systems often refer to widely-used benchmark data and evaluation scores of the systems themselves on these data.
Neural network architectures: Open systems typically give these in full (e.g. on the Hugging Face platform); closed (commercial) systems do not. In the case of closed models, those architectures are underspecified in model cards. Open systems often make architectures fully available to the public; an example is the BLOOM model, made available through Hugging Face (Scao et al., 2023).
Trained models (with all trained parameters): Like the configuration of the neural network architectures, open systems typically make trained models available in full (i.e. as a weights file), while closed (commercial) systems do not.
Scripts for training and output generation: Closed systems may offer code to work with their system’s APIs (while shielding the model itself from direct access), even extensively so (especially with paid services). Open systems tend to come with all types of scripts and code, often contributed by multiple users.
Our contention is that access to all five layers of information is essential in order to verify the possibility of replicating specific elements of the procedure, evaluate the integrity of AI-generated outputs, facilitate fine-tuning, and enable direct research into debiasing on these models. To implement robust technical safeguarding measures, it is advisable to engage licensed external experts to conduct model auditing, ensuring the verification of training sources and the detection of biased or hateful content. In its current form, the EU’s AI Act neither specifies any requirements for such auditing procedures nor adequately addresses the five levels of information explicated in Table 1. Different layers of information have different formats (e.g. config file vs text labels) and require different practicalities of inspection (e.g. studying files vs combing through data sets). As the EU AI Act’s draft text states, “given the nature of foundation models, expertise in conformity assessment is lacking and third-party auditing methods are still under development” (European Parliament, 2023). More research on how those audit procedures should be designed and implemented to capture the five layers of information shown in Table 1 is necessary (Mökander et al., 2023).
The second part of public inspectability relates to the extraction and transformation of “public” knowledge into commercial products (Ferrari et al., 2023). If text data processed by foundation models—which are trained using openly available and often publicly-funded data sets—lead to the formation of shared knowledge, the foundation models trained on these collectively created data sets ought to be made accessible to the public as well. However, regulatory oversight should not stop at the stages of observation or inspection; it requires an analysis of how democratic values relate to concrete pathways for technical changes. Herein lies the focus of technical modifiability.
Technical modifiability: how to modify generative AI systems?
The term modifiability can be defined as making basic or fundamental changes to an object to give a new orientation or to serve a new end. Therefore, we do not see this modifiability as an end in itself but rather as a means to achieve a higher-level end: the compliance with oversight mechanisms. As such, the key question is: who should have the power to modify the properties of generative AI systems, and to what ends?
For example, if a normative aim is to curb the spread of “misinformation” churned out by generative AI systems, a clear interrogation of technical modifications, such as stronger safety filters, watermarking, or more potent content moderation systems, is paramount. The use of watermarks in AI governance is not a speculative future scenario, but a regulatory reality in some countries. For example, China’s Cyberspace Administration enforced a set of regulations that restrict the production of AI-generated media without clear labels, ruling that citizens may not resort to “technical means to delete, tamper with, or conceal relevant marks” (Edwards, 2022). In this case, mandatory watermarking acts as a means to exert censorship. This demarcation and subsequent modification of generative AI technology which is categorized as damaging “the legitimate rights and interests of the people,” and endangering “national security and social stability” provides autocratic oversight regimes with abundant room for interpretation and censorship enforcement. Defining generative AI systems that need to be technically modified in such wide-ranging terms means that regulators have more leverage in deciding what types of AI systems fall under the scope of the restrictive regulatory framework.
By contrast, in the context of European regulations, the modifiability of foundation models should be firmly grounded in democratic values, including openness, privacy, and autonomy. Although provisions to ensure the use of mandatory watermarks are relevant when it comes to democratic oversight within the EU, there should not be a monopoly of co-production by dominant firms such as OpenAI or Google in deciding how watermarking techniques are used. Regarding democratic oversight, we need to ensure a proper balance between corporate, government, and civil actors when it comes to a shared responsibility to control and modify generative AI systems. Corresponding with the above-mentioned five layers of information about foundation models, we contend that those layers entail different modes and rationales for modifiability (Table 2).
Five layers of modifiability of generative foundation models.
GDPR: General Data Protection Regulation; RLHF: Reinforcement Learning with Human Feedback.
Training data sets for foundation models: With pre-trained models, this stage has been completed, and very large LLMs are not easily retrained with modified data sets. End users cannot modify the training data (should they have access to it). Only home-grown LLMs pre-trained from scratch, with no earlier pre-training having taken place, allow the determination and modification of training data sets. Data sets might not be selected for training in the first place, or modified to address problems of bias or curb the generation of harmful outputs. This explicit type of debiasing, also called intrinsic debiasing, is itself a challenging topic of research (Gonen and Goldberg, 2019). Most intrinsic debiasing work has targeted gender debiasing, using methods that either mask or counterbalance gender-specific terms such as gendered pronouns, first names, and other gender-specific terms (Orgad et al., 2022). Beyond the reduction of bias, the enforcement of data protection regulations like the General Data Protection Regulation (GDPR) requires the modifiability of training data sets.
Domain-specific training data sets for fine-tuning: This may be open to the user for models that allow some sort of fine-tuning on top of the pre-trained foundation model. If so, this should be entirely modifiable by the user. However, if fine-tuning concerns proprietary data that is an integral part of the released model, such as the RLHF-based layer in ChatGPT that post-selects among multiple possible outputs of the foundation model GPT-3.5 or GPT-4, it may be vital to have access to this data set to be able to understand better which toxic, badly formed, and other unsuitable output is filtered away (and which is not) by this post-processing layer. The modifiability of this type of data is also crucial for the enforcement of data protection regulations (e.g. patient data covered by the GDPR to train domain-specific medical chatbots.)
Neural network architectures: If the pre-trained model is not hidden behind an API, this might be part of a downloadable package in which the config file is readable and, if it complies with some open-source standard like PyTorch or TensorFlow, is interpretable. Modifying it makes no sense with a pre-trained model, since this is fixed after training; the model will only work with this unaltered configuration. With an open trainable network architecture, the user can fully modify this when the aim is to pre-train the model from scratch, for example, with different data (see [1]), or to train a model faster and/or with lower energy consumption. The consumption of fewer energy resources, which is a requirement of the AI Act, can be attained both by software or hardware innovations (Li et al., 2023). In terms of hardware, foundation model providers rely on Big Tech’s processing power (Ferrari, 2023).
Trained models: If not hidden behind an API, with pre-trained models this might be part of a downloadable package in which the weights are readable and, if it complies with some open-source standard like PyTorch or TensorFlow, interpretable in combination with the architecture config file. Modifying it is of no use; it is fixed after training and will only work with the network architecture configuration used during training. Even if the model is fully open and trainable, the model weights are simply the end result of neural network training, and modifying them manually makes no sense (as it will likely harm performance).
Scripts for training and output generation: Only some output-related code might be shipped along with a pre-trained foundation model, e.g. for watermarking, fine-tuning, replication of reported results, or API access; modifying these is necessary for different downstream tasks and fine-tuning. When training from scratch, the user will typically want to modify the training scripts (along with the architecture configuration, see [3], and training data, see [1]).
The level of granular modifiability described above is relevant in the context of the EU’s AI Act, as it includes a requirement to “train, and where applicable, design and develop the foundation model in such a way as to ensure adequate safeguards against the generation of content in breach of Union law” (European Parliament, 2023). Nonetheless, resembling the vagueness of AI transparency obligations, there are currently no precise technical details about how such modifications can be rendered enforceable in policy practice. Table 2 provides a pathway to specify the technical modifiability of generative AI systems based on different layers. However, the intricacies of technically modifying models require more detailed comparative studies, and they should also be discussed as communal (open) standards for future compliance.
Conclusion
After grounding the study of oversight structures for generative AI systems in their material properties as inherently observable, inspectable, and modifiable regulatory objects, a fourth feature presents itself: they are now negotiable as material items. We explicate how specific components of these systems are rendered observable; what layers of information about them can be made inspectable; and how they can be modifiable. Those three conditions should be thought of as reference points that open up algorithmic opacity to licensed auditors and may eventually lead to a robust regulatory co-production of generative AI systems. This layered classification is not an endpoint; it could be refined when and where necessary and may lead to common standards after deliberation resulting in agreement. For example, regarding the EU AI Act’s disclosure obligations for training data sets and the integration of safeguards in model design, Table 1 and Table 2 open up the possibility of precise analysis and intervention. Therefore, those three conditions will be relevant to regulatory negotiations and international standards development, exemplified by the UNESCO’s (2023) generative AI guidance.
Beyond those negotiations, those three conditions also act as anchor points to inform comparative research on generative AI governance. Around the world, governments attempt to oversee the same technological phenomenon: the infiltration of social, economic, cultural, and political spheres by generative AI systems. But while the computational underpinnings of those technologies converge across contexts, exemplified by the widespread adoption of large language models, there is a stark disparity of oversight approaches. To systematically examine this disparity, it is worth resorting to high-level governance conditions. However, we do not claim that there are no other conditions outside of this framework. Legal compliance, for instance, is a highly relevant aspect as well, particularly in the context of Big Tech companies that are infamous for undercutting regulatory efforts to ensure democratic transparency (Gorwa and Ash, 2020). Our framework suggests that providers of generative AI systems need to be held accountable to agreed-upon standards. The question of how accountability can be enforced in practice requires further research. We singled out foundation models as the inspectable and modifiable layers of generative AI systems, but it is clear that those oversight conditions also need to be applied to consumer-facing applications and computational infrastructure. There is a pressing need for further research on governing generative AI applications (e.g. by specifying inspectable paramaters like prompting logs and modifiable safety filters) and AI infrastructure (e.g. by specifying inspectable parameters to measure ecological impacts, and modifiable features to mitigate those impacts).
AI systems may seem more complex than other technologies, such as genetically modified organisms, but there is no reason at all to believe that they cannot be governed. Regardless of how generative AI systems will present themselves to public scrutiny in the future, the study of oversight mechanisms needs to be grounded in their granular material properties—not in speculative ideas about human extinction. Normative beliefs of how generative AI systems mediate power entail sweeping ramifications for how they can be governed. If we perceive AI systems as neutral harborers of existential risks, their right to exist precludes democratic negotiation. There is an urgent need to dispel this misleading notion of AI systems as inescapable forces imposed upon society, instead recognizing them as observable, inspectable, and modifiable objects. Only in this way, democratic negotiations will become inescapable forces imposed upon generative AI systems.
Footnotes
Acknowledgements
The authors thank Bernhard Rieder and Joanne Kuai for their helpful feedback. An early version of this article was presented at the AI Infrastructures Conference in Amsterdam as part of the Junior Scholars Network on AI, Media and Democracy. The authors thank the organizers, Anna Schjøtt Hansen and Hannes Cools, and the participants for their feedback.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors were supported by the Spinoza program of the Dutch Research Council (NWO).
