Prediction as extraction of discretion

Abstract

I argue that prediction is not primarily a technological means for knowing future outcomes, but a social model for extracting and concentrating discretionary power. Prediction is a ‘relational grammar’ that governs this allocation of discretion: the everyday ability to define one's situation. This extractive dynamic extends a long historical pattern, in which new methods for producing knowledge entail a redistribution of decision-making power. I focus on two contemporary domains: (1) crime and policing are emblematic of how predictive systems are extractive by design, with pre-existing interests governing what is measured and what persistently goes unmeasured. (2) The prediction of productivity demonstrates the long tradition of extracting discretion as a means to extract labour power. Time after time, making human behaviour more predictable for the client of prediction (the manager, the police officer) often means making life and work more unpredictable for the target of prediction (the employee, the urban citizen).

Introduction

The promise of prediction has become central to the real-world implications of AI and data-driven decision-making. When applied to human behaviours and social outcomes, prediction goes far beyond narrow technical definitions like “a model's output when provided with an input example” (Google, n.d.), and increasingly involves asserting probable outcomes as bases for definitive judgments - despite mounting evidence that much of what we call AI is “fundamentally dubious” (Narayanan, 2019) for assessing social phenomena like job performance or recidivism (also see Raji et al., 2022). I argue that prediction is primarily not a technological instrument for knowing future outcomes, but a social model for extracting and concentrating discretionary power: that is, people's ordinary capacity to define their situation. This dynamic extends a familiar historical pattern, in which discretionary power is extracted from the target of prediction (the citizen, the suspect, the employee) and concentrated in the client of prediction (the employer, the police officer, the manager).

Viewed this way, prediction extends the long history of how new methods of producing knowledge generate a redistribution of epistemic power: that is, who declares what kind of truth about me, to count for what kinds of decisions? (e.g., Deringer, 2018; Poovey, 1998; Shapin and Schaeffer, 1985) Below, I describe prediction as a relational grammar which generates social expectations on how facts are made and what accuracy or objectivity looks like when it comes to human behaviour (similarly on the notion of fairness, see Hoffmann, 2021). It follows that such grammar prioritises certain kinds of factmaking over others. Focusing on crime and policing, I then examine how predictive systems are extractive by design. Choices in what kinds of predictive models are created to pursue what kinds of objectives generate systematic disparities between what tends to get measured and what does not. The result is that novel mechanisms of capture and analysis regularly reproduce pre-existing social asymmetries. The paper demonstrates this historical pattern by turning to predictions of productivity – a site where the technological extraction of discretion extends capital's long extraction of labour power. In sites like the Amazon warehouse, making human behaviour more predictable for the client of prediction frequently means making life and work unpredictable for the target of prediction. I show how the cutting edge of the predictive workplace inherits the strategies and priorities of over a century of workplace automation. Data-driven systems promise to “render all agonistic political difficulty as tractable and resolvable” (Amoore, 2020: 10). But in this equation, it is us – and our discretion – that is being defined as the risk and error.

This approach to prediction extends the growing critical conversation around concepts like fairness, transparency, and the very notion of AI ethics. The question of how to pursue normative values in AI has increasingly raised the problem of how to define those values, and who should get to do that defining to what ends (Bietti, 2020; Green, 2019; Hoffmann, 2019; Phan et al., 2021). Today, these framing questions are being settled not in the proverbial roundtable of independent thinkers engaged in good faith, but a quagmire of industry-funded lobby groups, corporate ethics teams that mistreat and fire their own ethics experts, and active co-option of critical vocabulary into forms of ethics washing (Greene et al., 2019). In terms of funding, computational resources and prestige, Big Tech “make[s] the water in which AI research swims” (Whittaker, 2021: 53), as visible not only in expensive signature projects like the Chan Zuckerberg Initiative's $500 million donation to Harvard, but also the rapid growth of corporate affiliated authorship in AI research (Birhane et al., 2022). One result is that ethical concepts are narrowed and instrumentalised: bias becomes a form of numerical error to be corrected with better datasets, and ethics becomes a bureaucratic checklist to be inserted into the production flowchart (Birhane et al., 2022; Scheuerman et al., 2021). Often, paying for predictive technology is less a way to establish a more objective foothold on future outcomes than it is a way to reallocate discretionary power in one's favour. To extract from one is to concentrate on another.

Prediction as relational grammar

The history of technology amply demonstrates that new technologies do not enter as blank slates, but that their meaning and usefulness are derived from preexisting social relations and conflicts (e.g., Marvin, 1988). The social impact of technologies also tends to exceed their actual capabilities or implementation. The history of 20th century automation, for instance, was dominated by institutional and economic responses to what people expected automation to be and to do – responses which often persisted even when the technology never quite arrived as projected (Hind, 2021; Hong, 2021). Such expectations shape our perceptions on what kind of technological arrangement is inevitable, or what kinds of reform, abolition, and alternatives are considered plausible. The frequent declarations that everything about human beings can be predicted with enough data and computational power, or that such predictive machines will ‘know you better than you know yourself’, reflect and perpetuate these kinds of shifting distribution of epistemic power.

Prediction is particularly important to these mythological dimensions because it is at the vanguard of both (allegedly) concrete, real world benefits and the enveloping fantasy of long-term progress. The claim to predict a tangible ‘problem’ like recidivism or productivity promises a powerful empirical basis for thinking about new data-driven technologies as desirable and inevitable. Tech firms promise that algorithms can, for example, automatically analyse U.S. prisoner phone calls to predict violent crimes and suicide (Sherfinski and Asher-Schapiro, 2021), identify suspect individuals based on their head vibrations (Wright, 2021), and use facial features to predict prospective employees’ personality (HireVue). This remains the case even when many such predictions remain unproven or, at times, specifically disproven. COMPAS, a widely used recidivism prediction tool that leverages over 100 indicators, performs about as poorly as laypersons’ guesswork (Dressel and Farid, 2018). AI-driven tools were celebrated as potential difference-makers during the COVID-19 pandemic, but were soon found to be largely useless, plagued by poor input data and other fundamental problems (Heaven, 2021). Yet even as individual applications of prediction might be criticised and lampooned, such promises of real-world payoffs continue to generate emotional and cognitive investment in the idea of prediction (also see Finn, 2017).

These reports reflect a broader pattern in which social life is so often driven by the disjuncture between the substance of what passes as true in a given context and a broader sense of what kinds of things are likely to look, feel, and smell true. Foucault famously distinguished between what statements are treated as true, and statements that are considered sayable in the first place. The latter are less likely to become visible as formalised rules; in the context of Kuhnian paradigms, Ian Hacking (1992) suggests that scientific communities operate through ‘styles of reasoning’, basic tendencies for relating things and making connections that undergird and often outlast any particular theory. From this perspective, descriptions of a facial recognition tool as ‘predicting’ criminality or a generative adversarial network as ‘learning’ are not simply (misleading) descriptions of technical method; they perpetuate a more basic sentiment about how things must relate to each other in an age of supposedly intelligent machines.

But what does it mean to relate? The anthropologist Marilyn Strathern (2020) writes that the English word ‘relation’, in its modern sense, allows us to think the world in terms of discrete phenomena which may be joined, separated, sorted, mixed, in highly modular and indiscriminate ways. Saying something relates conveys a strange neutrality about the kind of epistemic operation that I am undertaking: it relieves the pressure of identifying a more precise interaction, such as ‘extracts’ or ‘causes’, and all the ethical and theoretical commitments that those designations would carry with them. (All these considerations make ‘relation’ an indispensable word for academics.) In the context of big data and machine learning, prediction is funneled into a flexibly neutral space, where a scientist or an engineer is required only to let ‘the data speak for itself’ and to document those relationships. And if the discovery – for instance, the use of facial recognition to allegedly predict sexual orientation – should facilitate clearly harmful uses of technology, it becomes easier to dismiss such consequences as ‘side effects’ that simply could not be helped.

This very idea of modular and general relation is neither timeless nor universal. It is very much a modern way of seeing, and one which resonates with the long advent of quantification as a dominant method for understanding human and social phenomena (Deringer, 2018; Porter, 1995). In this sense, the language of data-driven prediction imposes its own relational grammar: not so much the substantive claim that specific visual data of the face accurately identifies, say, sexual orientation or criminality, but the backgrounding tendencies in how we reason and model that make it ‘plausible’ to try and predict one from the other in the first place. Dan McQuillan calls data science an ‘organising idea’ which imposes a very specific model of knowledge that he describes as neoplatonic: a “belief in a hidden mathematical order that is ontologically superior to the one available to our everyday senses” (2018: 254). Others have described similar tendencies as an ‘enchanted determinism’ (Campolo and Crawford, 2020). While such worldviews are often demurred in polite conversations, they tend to persist in partial and half-avowed ways. As Golumbia (2009) notes, only some people will claim that human beings are just like computers (i.e., ‘the myth of computationalism’), but many more will flirt with the idea in practice. Today, public polls continue to show widespread expectation that artificial general intelligence is imminent, or that automated prediction of phenomena like crime is inevitable (Smith and Anderson, 2017; Zhang and Dafoe, 2019). When Clearview AI's notoriously unethical facial recognition technology became controversial through a New York Times report, it was telling to hear one investor fallback exactly on this kind of broad article of faith: “you can’t ban technology. Sure, that might lead to a dystopian future or something, but you can’t ban it.” (Hill, 2020)

Crucially, any description of data-driven systems as objective and consistent represent an aspirational ideal rather than actual practice. In their history of scientific objectivity, Lorraine Daston and Peter Galison write that “all epistemology begins in fear—fear that the world is too labyrinthine to be threaded by reason; fear that the senses are too feeble and the intellect too frail”. (2007: 372) Prediction as relational grammar does not simply dictate a smooth, totalising blanket of mathematical reason. Equally significant is what gets concealed or devalued in the process of imposing that grammar. Critical media scholar Tara McPherson (2012) calls it a lenticular logic, referencing the lenses in old 3D postcards where one can flick between multiple images but never see them together. She argues that the history of computing and its attendant lessons are often sealed off from other domains. While McPherson takes up this point specifically around to race and computing, we can also consider how computing imposes its own ways of seeing on those by other domains. Drawing from James Scott's work on ‘seeing like a state’ (1998), Ali Alkhatib argues that the indiscriminate proliferation of a data-driven way of seeing adds up to an asphyxiating utopia: an expectation of calculability which drives out everything that does not fit. “The models AI researchers train and deploy construct worlds without space for the dimension of our lives that make us who we are” (Alkhatib, 2021: 3). How we attribute predictability – for instance, that a single statistical model can be ‘scaled’ to predict housing prices (Zillow) or ‘suspicious eye movements’ (Proctorio) across many different lives and locales – has direct repercussions for what kinds of actors are considered rational, and what kinds of behaviours are considered optimal. Crucially, the grammar of prediction tends to prefer particular kinds of data and relations over others: a preference driven not purely by epistemic reasons, but by economic and institutional conditions that make some datasets and problems more available than others. Often, datasets treated as general benchmarks, like ImageNet and GLUE, are built not through rigorous definition of what is truly representative of the target reality, but look “more like samples of convenience” (Raji et al., 2021: 5). Subsequently, any arbitrariness or partiality in such benchmarks tends to flow downstream into derivative applications in a ‘blood diamond effect’ (Prabhu and Birhane, 2020). These limitations are not purely technological in the sense that more diverse datasets and models are simply impossible; rather, they tend to replicate longstanding economic, political, and institutional asymmetries in what kinds of data and knowledge get produced, documented, reported, and archived, in the first place. As Scott himself would later note: “legibility doesn’t come cheap.” (2021: 513)

Extractive by design

Bad questions

Prediction, like any knowledge regime, has a self-fulfilling element: it sees what it knows to see, and it measures what it is used to measuring. These tendencies are shaped through longstanding economic and political asymmetries, whose influence is then set aside as ‘externalities’. But when prediction troubles get compartmentalised as ‘just a dataset problem’, they obfuscate how patterns of extraction shape the research questions and the choice of what to measure (and what to dismiss without measuring). When predictive successes are defined as ‘beating’ humans in tests of accuracy, they reify the model's own parameters as the world that matters, further burying these externalities beyond consideration.

The sociologist Luc Boltanski, with Laurent Thévenot (2006) and Eve Chiapello (2007), has shown how capitalism constructs regimes of justification: the rules of the game by which we assess its own success and failure. Such regimes are also exercises in managing expectations. We learn to feel that it is surely unrealistic to expect that an employer will not squeeze their workers as hard as they can to maximise profit, or that if a technology is more ‘accurate’, it must serve the common good. Some researchers have already adapted Boltanski's notions to contemporary technoculture, showing how technologies like hiring systems provide justificatory scaffolding for firms (Dencik and Stevens, 2021), and help maintain collective belief in aspirational values like convenience and speed (Huberman, 2021). Prediction is useful precisely because it offers both the technical mechanisms and the logic of justification through which pre-existing extraction of discretion can be replenished. This motivation is the starting point for the shape that predictive systems tend to take today, rather than just a side effect.

Today, prediction regularly appears as an epithet of approval, in which the technique du jour (such as convolutional neural networks) is rapidly scaled onto an improbably wide range of complex social situations. Such claims often derive their attractiveness precisely by claiming to clarify human behaviours and propensities that have long remained contested and ambiguous. Crime and policing are emblematic, partly because the very history of crime as a measurable object, and of modern policing as an institution, are defined by this project of extracting discretion. In one notorious case, a Harrisburg University (2020) study claimed that “with 80% accuracy and with no racial bias, [their] software can predict if someone is criminal based solely on a picture of their face.” Although this particular study was rejected by Springer, the publisher, following widespread international criticism (Hatmaker, 2020), efforts to graft machine learning tools onto physiognomic worldviews are increasingly common (Agüera y Arcas et al., 2017; Stark and Hutson, 2022). The Harrisburg study was part of a pattern, not an aberration; it followed a now familiar grammar which effectively reduces any human condition into discrete empirical states. In this logic, ‘criminality’ appears as a state that inheres in the person, echoing centuries of imputation of bad blood or evil affliction. A similar exercise is performed with the human face, usually by relying on a variation of the facial action coding system (FACS) and its systematic reduction of facial expressions into objective, decontextualised data points (Gates, 2011: 168). Once frozen into such ossified forms, it only remains to demonstrate some statistical relation between the two artificially stable objects (‘face’ and ‘criminality’) to complete the equation.

On the surface, predictive systems rarely admit any commitment to a specific theory of criminality or the face. The grammar of data-driven prediction allows, and even encourages, the researcher to avoid asking such questions in the first place - the argument being that it is unnecessary to understand what criminality is as long as we can produce actionable measurements bearing its name. Orit Halpern recounts computation's long rearticulation of objectivity away from the pursuit of ‘external truth or reality’, and towards a calculable grid of action, measurement, and replication (2014: 83). This refusal to theorise is often claimed as an apolitical and amoral path; in practice, the very choice to predict a social entering into politics of erasing ambiguity (Birhane, 2021: 10–11). Articulating criminality or productivity as an object of prediction invariably leverages working definitions and heuristics already employed by the investors, engineers, and entrepreneurs involved in the production process, but rarely those of ‘targeted’ populations.

Bad measures

These patterns extend into the measurements themselves. As critical researchers of race and technology have shown with particular clarity, pre-existing inequalities shape what kinds of predictions roam the world in the first place, endowed with what kind of legitimacy (e.g., Benjamin, 2019; Browne, 2015). To suggest that the neutrality of data or algorithm cleanses predictions of their own historical provenance effectively describes a variation of money laundering, in which decisions like what kind of data is and is not gathered, or where the resulting ‘insights’ are deployed, are written off as someone else's problem. The predictive output, like ‘clean’ money, can then be treated as a neutral object of use. However, contemporary predictive systems’ need for massive dataset makes them heavily reliant on existing records and practices to source the data, with all their incumbent theories and assumptions about crime and policing. Ben Green (2019) describes how one University of Southern California project began by seeking to predict ‘adversarial groups’ like ISIS, but soon pivoted to targeting ‘criminal street gangs’. They ended up drawing data from the notoriously problematic LAPD gang data for the model, effectively amplifying and legitimating its errors and prejudices. All this happens despite the fact that “there is no agreement as to what predictive [policing] systems should accomplish […] nor as to which benchmarks should be used”, and even if there were, “like all evaluations of police technology, confounding factors make it impossible to measure directly its effectiveness at reducing crime” (Shapiro, 2017: 459).

At the same time, those who develop and sell surveillance technologies are incentivised to constantly produce useful justifications on the dangerousness of crime and the necessity of advanced policing. Technologies like PredPol and ShotSpotter are directly sold to law enforcement clients, preying on those clients' anxieties about being left behind and their perpetual interest in expanding budgets (Amoore, 2020: 2–4). Consumer-facing surveillance similarly increases fears of urban crime. In what has been called ‘digital vigilantism’ (Trottier, 2017), platforms like Amazon Ring's Neighbours incite and collate user data – the video footage and the neighbourhood alerts – into a picture of the world in which crime is always just around the corner. Citizen, the most aggressive company in this market niche, repackages dramatic cases of crime thwarted into ‘magic moments’ for consumption, and even deploys ‘street teams’ to make content when none is organically forthcoming (Ashworth, 2020). With Amazon Ring signing up over 400 local police departments for two-way data sharing, such business models generate new, ‘efficient’ pathways for connecting existing prejudices and perverse economic incentives with the aura of technological innovation (Bridges, 2021).

The choice of what to measure and how to measure caters to these existing social scripts, often entrenching their distribution of discretion across different subjects. But this is not simply a zero-sum game, where greater automation or rationalisation leads to less human discretion. Rather, the deployment of predictive systems can serve to protect and relegitimise bureaucratic discretion by integrating those judgments into the technology. The move towards data-driven policing often involves new systems of documentation and data-collection, as well as existing data taking on new uses and importance, following a longer pattern of modern policing as police media (Reeves and Packer, 2013). The mundane work of documenting, photographing, and ‘writing up’ targets thus becomes a site where police workers’ discretion feeds into large data-driven systems. Consider LSI-R (Level of Services Inventory-Revised), a longstanding risk assessment tool for predicting individuals’ likelihood of recidivism. One version used by the Idaho Department of Correction (EPIC, 2019) shows how details like prior convictions, “dissatisfaction with marital or equivalent situation”, or “unfavourable attitude towards convention” contribute to the quantification of individual risk. Each measure requires interpretive work, some more than others: a score of 1 out of 3 in ‘peer interactions’, as the training documents explain, would describe an offender who “actively dislikes co-workers or has only limited contact with them [… and] often lets angry feelings toward others build up inside.” In practice, LSI-R has fared poorly in tests of interrater consistency, and evidence suggests that its correlation with recidivism is low (Harcourt, 2007: 82–3). When predictive models are baked into decision-making processes, they do not simply shift the entire apparatus towards inhuman objectivity, but rather empower new norms on who gets to impose their discretion upon whom. Often, the promise of data as a universal illuminator conceals the reality in which it is data for me, and not for thee.

The unmeasured

The common thread running through these different sites of data productionis that where the police remains the primary purchaser of these predictions, the kind of data and models we “get” as a society will invariably reaffirm their definitions of crime and safety. In the US context, conspicuously missing in this thriving industry is any major efforts to improve the disastrous undercollection and invisibility of data around police violence and misconduct. Kelly Gates writes that such asymmetry in the sheer availability of data entails a ‘symbolic annihilation’ (2015: 4), in which one's experience is not simply underappreciated, but directly overwritten by police-oriented data production. From Michael Brown and Eric Garner in 2014 to George Floyd and Breonna Taylor in 2020, high-profile police killings of Black Americans precipitate repeated public debate on just how prevalent such killings are. Yet each time, it is data on Black crime that tends to be readily available, while data on police misconduct and violence is underfunded, neglected, and sometimes actively suppressed. Even as protestors demanded widespread reform in the wake of George Floyd's death in 2020, Chicago's police union was working to destroy records of police misconduct, as they had been doing for years as part of their contract with the city (Hancock, 2020). Investment in predictive policing and other new technological systems deepens a vicious feedback loop of legitimation by “assum[ing] the credibility of the underlying crime data - and the policing methods that generate these data in the first place.” (Burrington, 2015)

Such asymmetries condemn certain kinds of suffering and lived experience as ‘merely anecdotal’, forced to an uphill struggle to count as ‘data’, while the police receive the numbers they need by default. Samuel Sinyangwe, co-founder of the Mapping Police Violence project, describes the recurrent barrier: ‘How often have you heard, “We don’t have the data. We don’t know if what you’re telling us is true.”’ (in Foss et al., 2017) Although more robust databases of police misconduct are becoming publicly available, they tend to come from small, independent alliances of researchers, journalists and activists working with limited resources. While the police are legally obligated to produce some kinds of misconduct data, they routinely engage in strategies of obfuscation and delay, turning theoretical availability into practical absence (Cox and Freivogel, 2021). As a result, whatever becomes transparent about those killed by police—their misdeeds, their personal life—tends to be far more exhaustive than the data that we can get about the police themselves (Gates, 2019). Predictive systems are often used to delegitimise the kinds of lives and experiences that are already too disadvantaged to generate rich ‘data’ in the first place.

Entrenching discretion

Defining the situation

Discretion typically describes the ability to assess how a rule might apply (or not) to the case at hand. Thus in legal scholarship, Dworkin (1963) memorably defined judicial discretion as the judge's ability to ‘choose a solution’, while in sociology, Lipsky's (2010) influential study of ‘street-level bureaucrats’ describes discretion as a means of managing the tension between document and person, government policy and individual needs, that dominates everyday bureaucratic labour. But discretion has repercussions far beyond the interpretation of formalised rules. To exercise discretion is also to judge which rules should apply to a given situation, and to define the situation as requiring a rule-based resolution in the first place (or not). In her history of algorithms as rules, Lorraine Daston (2022, Ch2) describes how the Benedictine Abbey, commonly imagined as the apotheosis of suffocating, mechanical rule-governed life, actually places those rules at the mercy of the abbot's discretio as the most crucial virtue. Here, the most important use of such discretion was to judge when the rules should not apply, or at least not apply as rigidly as they are normally. Discretion thus describes a broader and more fundamental power to define one's situation – one which shapes the practical reach of human agency and autonomy.

These qualities link prediction indelibly to discretion. After all, predictive technologies are tasked with rationalising judgment, in both common senses of the word: they establish rule-bound processes for decision-making, and further provide scripts for justifying that judgment to insiders and outsiders (also see Hall, 2017). These functions impinge directly on the exercise of discretion. This is not to say that prediction always enforces formalisation and Kafkaesque inevitability at the expense of discretion. Rather, prediction serves to reallocate discretionary power across different actors, and additionally to obfuscate the continuing role of discretionary power in decision-making. Police departments might commission predictive systems to relegitimise their existing discretionary practices upon the urban poor, while workplaces might see it as a way to squeeze out workers’ control over their work process. Discretion does not always provide relief from control any more than prediction always renders judgment objective. Indeed, it would be rather subjectless to say that discretion manages the rule and leave it at that. Issa Kohler-Hausmann's analysis of misdemeanour prosecutions in New York City demonstrates how defendants, once captured through a parking ticket or turnstile jumping, get sucked into not only the opprobrium of rule and all its painstaking requirements, but also the prosecution's ability to exercise discretion over that rule. Hence William Stuntz’ famous dictum that criminal law constitutes “items on a menu from which the prosecutor may order as she wishes” (in Kohler-Hausmann, 2018: 12). Street-level bureaucrats might employ discretion to relieve some of the stupidity of the rule, or to mitigate their own working circumstances, but they can also wrap their own discretionary judgment in the cloak of the rule and wield it against the citizen.

In so many places in social life, discretion is something one exercises upon another, imposing one's definition of the situation over others. At the macro level, there is a dyadic struggle; the history of labour automation and prediction, as I discuss below, is a series of managerial extraction of worker discretion. But if new technologies enable new strategies for such extraction, deploying them into action often entails new groups of intermediaries, the petit-sovereigns. Police officers are classic street-level bureaucrats in their interactions with citizens, but predictive policing technologies are newly encroaching into their everyday discretion by introducing greater surveillance and control over officers’ patrolling activities. Notably, in the US, officers have been pushing back by asserting their professional rights to discretionary power. Their existing level of protection through powerful police unions empowers them to resist in ways that would not be practical for many other more precarious workers (Shapiro, 2020). Such cases demonstrate how predictive technologies often redistribute discretionary power within bureaucracies as well, from individual truck drivers to clerks monitoring them through electronic logging devices (Levy, 2022) or from ground-level customs officers in airports to centralised data analytics teams (Côté-Boucher, 2016). Some have argued that algorithmic systems are enacting a broader shift from street-level bureaucracies to systems-level bureaucracies (Zouridis et al., 2020).

Discretion, then, describes the always unequal distribution of the power to define the situation – a distribution which data-driven prediction seeks to actively reconfigure. After all, the very justification for the deployment of predictive systems in the first place elevates data and interpretation above other kinds of evidence. This remains broadly true even where model outputs are subject to human review. A risk scoring system for child abuse and neglect may only issue non-binding recommendations, but they will tend to nudge and influence the judgment of overworked social workers, as well as how they justify their discretion to themselves and others (Eubanks, 2018: 141–142). However probabilistic the insight, the decisions that result from them are singular and binding for each individual. To extract from one is to concentrate in another.

Predictable workers, unpredictable work

I now turn to efforts to predict productivity and worker behaviour. As I noted earlier, prediction's transfer of discretion from the target of prediction to its client often dovetails with other historical patterns of extraction, and in particular, that of labour power. Consider the case of Amazon. As one of the largest private employers in the world, Amazon is both test-bed and ground reality for prediction as the extraction of worker discretion. Its warehouses (‘fulfilment centres’) and delivery networks exemplify the synergy between intense, data-driven automation of labour on the one hand, and the systematic deprivation of information, predictivity, and discretion for the workers on the other. Here, the master metric is “the rate”: individual performances like boxes stowed, picked or packed per minute and hour. The rate is regularly used within the company as a reliable base measure for decisions including the hiring and firing of individual workers; it provides an always cooperative reference point for justifying Amazon's abnormally high rate of firings (Lecher, 2019). Yet notably, the exact rate that a worker must hit at a given warehouse is never disclosed to the workers themselves. Sociologist Nantina Vgontzas (2021), who has worked at Amazon warehouses and joined in workers’ organising efforts, recounts being trained against the rate of 400 picks per hour, and later being told by a human trainer that the actual expectation was around 360. Meanwhile, the demand for a machine-readable and predictable labour rate is further enacted through a sensory opprobrium. Coloured graphs warn employees to work faster; ‘project loving energy’, counsels a workstation screen (Delfanti, 2021b: 56).

These quantified expectations governing the algorithmic workplace cater to managers and employers’ desire for a certain kind of inhuman clarity, in which the many variations and ambiguities inherent in any act of labour are not actually eliminated, but simply neglected. The consequence is that for the worker, their own work and life becomes both less predictable and less discretionary (also see Ganesh, 2020). The Amazon ‘picker’ is constantly adapting to the algorithmic redistribution of boxes and goods, unable to accumulate their own rhythms for effective and safe work. Journalist Alec MacGillis relates the story of Hector Torrez, a San Francisco tech industry professional turned an Amazon warehouse worker: “the challenge wasn’t so much the weight as that you couldn’t really tell, based on size, whether a box was going to be heavy or not when you went to pick it up. Your body and your mind never knew what to expect.” (MacGillis, 2021: 4) This deprivation of someone's ability to anticipate, plan, and adjust their own conditions of working and living is but one episode in which making labour more predictable for some requires making it less predictable for others.

This disparity in control results in all too familiar harms to working conditions. Amazon's warehouse workers have long experienced significantly higher rates of severe injury compared to industry competitors (Greene and Alcantara, 2021). Amazon's delivery drivers are subject to similar pressures: top-down dictations of punishing rates, sometimes of up to 400 deliveries per 10-hour shift, provide profits at the macro level, while workers pay in the form of extraordinarily high injury rates and the now infamous practice of drivers having to pee in bottles or trash bags to meet the rate (Gurley, 2021). Amazon's response to these problems are to intensify this dynamics of profit-oriented datafication, rather than to restore discretion on the floor level. In 2021, Jeff Bezos’ final letter to shareholders as the CEO of Amazon prescribed “new automated staffing schedules that use sophisticated algorithms to rotate employees among jobs that use different muscle-tendon groups” (Ongweso Jr., 2021) - a solution that would implement new forms of unpredictable, opaque decision-making on behalf of workers, while retaining the non-negotiability of ‘the rate’ and punitive productivity standards.

Amazon may be a pioneer in the algorithmic destruction of workers' bodies, but its underlying economic and technical incentives are not at all unique. Coupang, South Korea's answer to Amazon (though heavily dependent on SoftBank's Saudi-backed funds, as is the case for many Silicon Valley companies), has also seen rocketing stock value and a reputation for superfast delivery on the back of algorithmic surveillance, high rates of workplace injury, and several cases of death from overwork (Lee, 2021; Seol, 2021). In delivery platforms like DoorDash, the on-demand worker is shephered into surge areas in one moment, then made to kick their heels curbside in another, never knowing how much they’ll make or how to pace their work (Gregory and Sadowski, 2021: 7; Shapiro, 2020: 2). Chinese food couriers, working for platforms like Ele.me and Meituan Waimai, describe sudden periods of intense pressure in which a delay of seconds might cost several orders’ worth of pay, the result of a business and technological model which profits from its ability to “stratif[y] the value of people's time” in rapid, finely-grained ways, extracting every possible sliver of ‘downtime’ (Chen and Ping, 2020: 1563). For the worker, it is their lived time chunked up into a pulsating mess of alarms and nudges, distractions and panic; for the manager and employer, it is a vast predictive matrix which externalises everything that the model would prefer not to predict and funnels the cost to the worker. What the model refuses to count cannot hurt it.

One extraction for another

This disparity between the predictor and the predicted reprises over a century of labour struggle, throughout which the extraction of discretion has served as a crucial instrument for the extraction of labour power. ‘Surveillance capitalism’, currently one of the most common critical frames around prediction, describes not a seismic revolution, but a variation upon the theme that is capitalism itself (Morozov, 2019). Delfanti (2021a) argues that automation separates the worker from their skills and knowledge - a separation which then enables the factory to operate with deskilled workers, and to turn that information around to evaluate and control worker behaviour. In Amazon warehouses, human ‘stowers’ handle incoming items of staggering variety, grouping them into bins in giant ‘pick towers’, a task that involves a certain degree of discretionary judgment. When this process is laced with data-gathering systems, however, each worker action - such as the stower's scanning of the item, their rate of pieces per hour, and their bathroom breaks – provides data grist for improving predictions of the stowing process. Individual judgments and tacit knowledge are siphoned into a unique source of knowledge for managers, but not for the workers themselves. The key innovation is not merely to pack seven boxes instead of six, but ensuring that it is the manager who can set the ‘rate’ of seven, or eight, or two hundred, in ways that are precise for the manager, and opaque to the worker.

This dynamics runs right through the history of modern automation. Harry Braverman's landmark work, Labour and Monopoly Capital, argues that this separation was one of the key legacies of Charles Babbage's work, and especially the principle that any labour process should be subdivided into ultra-fine fragments, such that each fragment could be regulated for minimal cost and maximum labour (1998: 55). Not only does this presage contemporary microwork, Babbagification also requires ever more detailed apparatuses of measurement and justification through which those fragments can be priced and punished. David Noble (2011) has shown how General Electric's postwar investment into machine shop automation was an explicit response to the success of nationwide union strikes in 1946, and the fear that skilled workers might exercise discretion over the conditions of their workplace and to make demands of the employer. GE Vice President Lemuel Boulware, whose name would supply the label for the company's aggressive anti-union strategies of ‘Boulwarism’, argued that GE must “eradicate” the “fantasy” that “the employees … were in the driver's seat.” (Noble, 2011: 156) Such attitudes shaped what kinds of automation technologies would be accepted, and which functioning alternatives would be consigned to obscurity. GE's favoured ‘record-playback’ approach to automation eventually lost out to numeric control in part because the managers and employers making the acquisition decisions wanted the technology to transfer discretionary power away from the worker. In one instance, GE designed an automation system for a steel company, only for the client to complain that “operators were [still] controlling production, determining the output”, engaging in well-worn techniques of ‘stints’ and ‘pacing’ (worker-determined production quotas and rates) (Noble, 2011: 164).

These recurrent efforts to separate the worker from their discretion come hand-in-hand with longstanding cultural tropes about the nature of the worker and what is necessary to make them work. As with crime and policing, such presumptions enter into the social life of prediction before and beyond any question of statistical bias in a dataset or the appropriateness of particular object labels. In many such applications, the worker is presumed to be, by default, a potential thief (of wages via low productivity, or directly of company property). Amazon has consistently cited package theft by delivery drivers as a key motivation for its Ring door cameras (Amazon, 2018), pushing that fear onto consumers as a way to sell the devices (Bridges, 2021: 5). In recent years, Amazon has also aggressively implemented new computer vision technologies for driver surveillance. In March 2021, Amazon began pushing its drivers to sign away ‘biometric consent’ to keep their jobs, authorising a new suite of surveillance tools including claims of AI detection for ‘risky driving behaviours’, in addition to existing means for tracking and evaluating their rate of delivery (Vincent, 2021). Perhaps the most strikingly explicit example of this presumptive criminalisation comes from an anonymous Microsoft engineer, who describes travelling to Kazakhstan to advise an oil production partnership between the Kazakh state and Chevron. After giving a presentation on technologies for mapping the ground to identify oil sources, the managers in the room – mostly American expats – ask instead for AI-driven worker surveillance. Their presumption was clear: “We have a lot of workers in the oil fields. It would be nice to know where they are and what they are doing […] If they are doing anything at all”, said one. (Zero Cool, 2019) The worker is the suspect, and it is this a priori declaration that determines what role the data will play to begin with.

Predicting to stay the same

Prediction grammatises – renders flexibly replicable, habituates, provides a template for – the widespread extraction of discretionary power: the spaces of practical ambiguity, the gap between rule and case, the moments of situational judgment, that were always unequally distributed across different subjects in the first place. From predictive approaches in policing and incarceration to their analogues in workplace surveillance, we find a consistent pattern. It is the subjects of measurement who are preemptively defined as objects of suspicion and danger, whose exercise of discretionary power over their own circumstances is primarily seen as a source of unwelcome uncertainty. All this ‘seeing’ is done from the perspective of the managerial clients of prediction, for whom systems of datafication offer new ways to entrench their desired definitions of a productive worker or otherwise 'good' subject as neutral and objective facts.

When understood as the extraction and redistribution of discretionary power, data-driven prediction appears more clearly as a reproduction of longstanding strategies for social control, rather than any radically novel approach towards human bias and irrationality. David Noble relays an evergreen definition of workplace automation from none other than Peter Drucker, perhaps the closest available personification of corporate managerial logic: “what is today called ‘automation’ is conceptually a logical extension of Taylor's scientific management [in which] productivity required that ‘doing’ be divorced from ‘planning’.” (in Noble, 2011: 231) From the gig economy worker left to kick their heels waiting for the algorithm to bestow the next job, to Amazon's patents for smart devices guiding warehouse workers on which boxes to dig for the delivery item (Hong, 2020:101), this separation of doing from planning deprives the ‘targets’ of the very information and experience that can become the basis for theorising and contesting their own situation.

To adapt Joseph Weizenbaum's famous dictum on computing, not everything that can be predicted should be predicted (1976: x). The very act of declaring something predictable, of turning it into a prediction ‘problem’, already establishes the conditions for extraction of discretion. The drive to predict deploys well rehearsed templates for decision-making in which the clients of prediction are empowered to consistently speak over and speak for the targets of prediction: the police departments over inhabitants of overpoliced neighbourhoods, and the managers and executives over warehouse workers. As technologies like facial recognition increasingly impose themselves over diverse social situations, they deliver not a rationalised environment in which the data speaks for itself, but a thin refurbishing of familiar asymmetries in which the same few might continue to judge the many with impunity.