Algorithmic governmentality and the space of ethics: Examples from ‘People Analytics’

Abstract

Does human reflexivity disappear as datafication and automation expand and machines take over decision making? In trying to find answers to this question, we take our lead from recent debates about People Analytics and analyze how the use of algorithmically driven digital technologies like facial recognition and drones in work-organizations and societies at large shape the conditions of ethical conduct. Linking the concepts of algorithmic governmentality and space of ethics, we analyze how such technologies come to form part of governing practices in specific contexts. We conclude that datafication and automation have huge implications for human reflexivity and the capacity to enact responsibility in decision making. But that itself does not mean that the space for ethical conduct disappears, which is the impression left in some literatures, but rather that is modified and (re) constituted in the interplay of mechanisms of closure (like automating decision making, black boxing and circumventing reflexivity), and opening (such as disclosing contingent values and interests in processes of problematization, contestation and resistance). We suggest that future research investigates in more detail the dynamics of closure and opening in empirical studies of the use and effects of algorithmically driven digital technologies in organizations and societies.

Introduction

Recent scholarship suggests that the novel ‘data-driven sciences’ provide an improved knowledge base for managing individuals and populations (Mayer-Schönberger and Cukier, 2013: 18), making it possible to ‘view society in all its complexity through the millions of networks of person-to-person exchanges’ (Pentland, 2015: 12). Unlike the analogue forms of statistical knowledge historically developed in western societies (Barry, 2019), so-called ‘actionable knowledge’ extracted from the ‘ocean of data’ promises to reveal unexpected insights and fine-grained predictive analyses. ‘People Analytics’ is an emerging field of scientific development and managerial approaches that epitomizes the promises and perils of this development. Grounded in behaviorist approaches to human conduct (Pentland, 2015; Zuboff, 2019), it recommends the adoption of digital technologies in work-organizations to enhance efficiency and reduce the noise of ‘unstructured, subjective opinion’ purportedly shaping decision making (Bodie et al., 2017: 964; Leonardi and Contractor, 2018). Sophisticated algorithms are designed to collect, process and mine the vast data traces that people leave on sensors and digital devices, turning data into manageable outputs that magically come to appear as value neutral, capable of predicting, optimizing and managing human behavior. While People Analytics is embraced and promoted by some researchers and practitioners (Gelbard et al., 2018; Isson and Harriott, 2016; Waber, 2013), critics voice concerns about the marginalization of human reasoning and reflexivity as algorithms become self-learning, machines autonomous and managerial surveillance in work-organizations expand (Gal et al., 2020; Giermindl et al., 2021; Manokha, 2020). Beyond work-organizations, People Analytics shed light on important aspects of digitization and algorithmic regimes, such as the datafication of citizens’ activities, emotions and social relations, and risk profiling across domains as diverse as health and education, finance and transportation, policing and security (Amoore, 2013; Brayne, 2017). Writing critically about People Analytics in this extended sense, Zuboff (2019) sees a new form of authoritarianism emerging that undermines democracy, human dignity and autonomy. Pasquale (2015) draws attention to the unquestioned values and prerogatives ‘hidden in the black-box’ of algorithmic regimes. Still others are concerned with the ‘harms’ of algorithmic decisions (Tufekci, 2015), questioning that accountability and transparency can ever be built into complex automated systems (Ananny and Crawford, 2018; Diakopoulos, 2016). Eubanks (2017: 12) argues that algorithmic decision making ‘automates inequality’ and provides the needed ‘ethical distance’ to make ‘inhuman choices’ easier.

These observations beg an important question: how does the adoption of algorithmically driven digital technologies in work-organizations and societies at large shape ethical conduct? We argue that answers to the question can be found in an analysis of (1) how algorithmically driven digital technologies form part of managing and governing practices in specific contexts, and (2) how the space of ethics (Iedema and Rhodes, 2010) is modified and reconfigured as these technologies increasingly shape social practices. Our analysis rests on the following premises. While algorithmic procedures and technologies are often presented as purely technical and neutral, we concur with critical studies that emphasize their ethical-political character. Algorithmic procedures and technologies make specific things visible and others invisible. They produce classifications and categorizations, ‘sort’ people and objects, including some and excluding others (Alaimo and Kallinikos, 2021; Bowker and Star, 2000; Gillespie, 2014). They help generate profiles of persons and are constitutive of valued and devalued identities. While these technologies are political in terms of constructing and normalizing some versions of reality rather than others, they are also ethical and value-laden (Introna, 2005). In this way, they have a considerable impact on how we perceive the world and understand our place in it, as well as on how we make judgments and explain to ourselves and others why we act in certain ways (Gal et al., 2020: 5).

Here, our analytical focus is on how the use of algorithmically driven digital technologies in work-organizations and society at large shapes the conditions of ethical conduct. This requires some conceptual work and selection of illustrative examples. Section one sets out to theorize the intersections of technology, governing practices and ethical conduct. Specifically, we explain the concept of ‘algorithmic governmentality’ (Rouvroy and Berns, 2013) and link it to a practice-based understanding of ethics. This provides a non-deterministic view of how technologies shape the conditions of possibility of ethical relations to self and others. Section two builds on this framework to analyze the space of ethics in relation to the modes of objectivation and subjectivation afforded by these technologies as well as their implications for human reflexivity. We take facial recognition and drones as exemplary cases. Facial recognition technologies involve computers and cameras controlled by an operator. They help capture and analyze images of people’s facial expressions to extract information on gender, age and emotions, or the propensity to commit a crime (Andrejevic and Selwyn, 2020; Bueno, 2020; Gates, 2011). Drones are aircrafts controlled by computers and operators on the ground who receive images from the aircraft. Both technologies are used today in organizational work-place contexts, for instance to monitor workers’ compliance with health and safety regulations (Parsons, 2017), and in civilian, humanitarian and military affairs (Gates, 2011; Pugliese, 2011; Van Wynsberghe and Comes, 2020; Weber, 2016). We zoom in on these digital technologies in processes of corporate recruitment and drone warfare from the perspectives of relevant institutional actors and users, to trace out implications for the space and practices of ethics. Here, we draw on ‘practical texts’ (Foucault, 1992: 12); that is, ‘texts, which are themselves objects of a “practice” in that they were designed to be read, learned, reflected upon’. This includes material accessible on corporate and governmental websites promoting or instructing the use of these technologies; we examine personal testimonies by people involved in their practical use, based on publicly available interviews conducted by researchers and journalists, as well as on individual comments uploaded on websites such as Reddit. Theoretical texts on these technologies are only consulted here for clarification.

Based on our analysis, section three discusses how algorithmically driven digital technologies shape the space of ethics in a dynamic interplay of mechanisms of closure and opening. Usually operating anonymously in the background and without the consent of those targeted by them, these technologies help make decisions crucial for human life. At the same time, they potentially obscure the human frustration and suffering caused by or lurking behind their use. This, however, does not necessarily preclude awareness about their potentials and dangers nor the problematization of them. The concluding section reflects on our contributions to literatures on algorithmically driven digital technologies.

This is a conceptual article in which illustrations of the use of these technologies are considered rich empirical reference points for understanding the variety of ways in which people experience them, relate to them and make use of them in their relations to self and others. We do not pretend to present a full-blown empirical analysis, however, but instead wish to create a critical opening that future and more exhaustive empirical research can delve more systematically into.

Conceptual background

Exploring the governing and ethical implications of algorithmically driven digital technologies requires conceptual work that links the political dimension of such technologies with a concept of ethics that considers the possibilities of acting and deciding in different ways.

Algorithmic governmentality

Government is a form of power that refers to the ‘conduct of conduct’ as it structures ‘the possible field of action of others’ (Foucault, 1983: 221, 1991). Government relies on representing the world in terms of problems to be identified and in need of amelioration – be it terrorism, disease or unemployment – through the mobilization of technologies and intervention. Particular ‘governmental rationalities’ inform the diagnosis of problems and the exercise of government, which also includes the governing practices of non-state actors. ‘Technologies of government’ are the actual mechanisms through which authorities – public, private or in-betweens – seek to shape, normalize and instrumentalize the conduct and thoughts of others to achieve the desired objectives. Government is exercised at a distance and technologies help link calculations in one place with action in another, establishing loose assemblages of agents and agencies, humans and non-humans (Miller and Rose, 1990). While government relies on technologies and practices that render a domain knowable in specific ways, such constitution of objects of knowledge (objectivation) is linked to specific modes of forming subjects in power relations (subjectivation) that tie the subject to specific identities (Foucault, 1983: 212). With governmentality, subjectivation is not mainly about imposing predefined models of behavior on subjects and cannot be reduced to disciplinary subjugation or ‘subjection’. Rather, subjects are invited to reflect and work on themselves to become specific subjects or kinds of persons. Power operates as much through practices that ‘make up’ subjects as ‘free persons’ by encouraging specific modes of being and ways of seeing through practices of exclusion and denial (Raffnsøe et al., 2019).

Before the advent of the internet and big data, studies of governmentality focused on the governing practices of and by the offline individual in the classical liberal and neoliberal registers. Liberal governmentality took a collective view and focused on governing the population, while neoliberalism turned to the individual and sought to foster and govern the ‘enterprising self’ (Bröckling, 2016). Algorithmic governmentality addresses the particularities of governing practices in our contemporary digitized world (Barry, 2019; Introna, 2016; Rouvroy, 2013). It expands the focus on the individual while addressing the interactive modalities afforded by digital technologies and the specific rationality associated with them. Governing comes to rest on the automatized sorting of fragmented online data footprints left by individuals, turned ‘dividuals’ (Deleuze, 1995). These are aggregated and objectified as profiles that are used for various purposes (Hildebrandt, 2013; Newlands, 2021). The human is thus linked to material technologies in novel and subtle ways, forming ‘cognitive assemblages’ (Hayles, 2016a) in which machines and technical systems can operate as ‘cognizers’ (Hayles, 2016b: 789). All forms of government rely on knowledge of the individual and the population, but in contrast to the classical off-line assemblages, millions of datapoints derived from online activity, images and sensors, now form an immense and rapidly growing knowledge reservoir subject to extraction by machines.

Algorithmic governmentality is characterized by the enormous speed of the ‘data gaze’ (Beer, 2019). Speed shapes the conditions of human reflexivity as shown in studies of cognitive assemblages (Hayles, 2016a, 2016b). In High-Frequency Trading (HFT), for example, algorithms execute innumerable trades before human traders even notice and can respond (Beverungen and Lange, 2018). While we would usually have time to reflect, ‘reflexivity, is short-circuited by the speed of digital technology. This speed taps directly into impulses rather than reflexivity’ (Rouvroy, 2020: 3). Such short-circuiting of reflexivity typically comes with a strong instrumental focus on algorithmic procedures and the pursuit of efficiency. In the extreme case, the subject is ‘evacuated’ (Rouvroy, 2020: 3). When everything is reduced to procedures driven by machines developing organizing categories at high speed, there is a new form of subjection, which Lazzarato (2014), drawing on Deleuze and Guattari 1987 term ‘machinic enslavement’. It represents an ‘entirely different form of capture of subjectivity’ (Lazzarato, 2014: 25) where the individual is considered as ‘cog’ or ‘gear’ of a machinic assemblage. As a mode of governing, it implies managing parts of a system and ‘ensuring the cohesion and equilibrium of the functioning whole’. Here, human and non-human agents function as ‘component parts’ of technical machines. Since ‘enslavement works with decoded flows (abstract work flows, monetary flows, sign flows, etc.) which are not centered on the individual and human subjectivity but on enormous social machinisms’ (Lazzarato, 2014: 28), there is indeed no space left for decision making of autonomous (ethical) subjects. Yet, as Beverungen and Lange (2018: 92) have shown, even in the case of HFT, often taken as a prime example of an area where automation marginalizes humans, the subjectivity of traders is not simply removed, and traders are not ‘passive parts of a system which was wholly designed by others’. Rather, they develop different ‘modes of awareness’ and actively relate to the algorithms through a number of containment strategies. Finally, algorithmic governmentality emphasizes predictive analytics, which serves to make specific futures and potential future persons knowable and ‘actionable’ (Flyverbom and Garsten, 2021). In so doing, it creates normative expectations about how to conduct oneself and what to become, and by implication, it shapes the conditions of possibility of relating and responding to oneself and others.

The space and practice of ethics

Ethics is conventionally understood as philosophical reflection of morality, providing foundations of moral rules and regulations, and the a priori justification of moral principles and yardsticks that allow evaluation of the rightness or wrongness of an action or practice. For instance, in studies of algorithms, such a normative, rule-based concept of ethics has been applied to assess and judge the practices relating the development and uses of algorithms from an external (transcendent) point of view (Mittelstadt et al., 2016; Zarsky, 2016). Conversely, practice-based approaches within organization and algorithm studies take an internal point of view, focusing on how people experience, act and decide with (or without) algorithms within a specific field of practices (Amoore, 2020; Clegg et al., 2007; Iedema and Rhodes, 2010; Introna, 2007; 2016; Markham et al., 2018; Weiskopf and Willmott 2013). Building on this approach, we conceptualize ethics as a practical activity, a reflexive practice of freedom within the normative field of action subject to analysis (Foucault, 1992: 25–32, 1997: 284). Ethics in this sense presupposes that subjects are not blindly following rules, norms and procedures – including algorithmic procedures – but can relate to prescriptions and expectations in a variety of ways: depending on the ‘mode of subjection’ (Foucault, 1992: 27) they can affirm, problematize, resist them or make use of them in ways that support them in their own projects of self-formation. Ethics thus revolves around subjects’ practices of questioning and problematizing of the ‘prescriptive ensemble’ (Foucault, 1992: 25) in which they are embedded, and concerns the possibilities of relating and responding to oneself and others in particular ways. Here, it is not defined in advance what is ‘ethical’ or ‘unethical’. Rather, ethics is the emergent product of multiple practices in which subjects enact normative orientations in their relations to self and others. In the process of subjectivation, subjects are formed in ‘folding’ the forces back on them (Deleuze, 1988: 94–123).

Iedima and Rhodes’ (2010: 201) conceptualization of the ‘(undecided) space of ethics’ is most relevant in this context since it considers how people are both conditioned by practices and technologies and potentially ‘active and reflexive in deciding themselves to change their conduct’. Building on this insight, we argue that algorithmically driven digital technologies can reduce or close the space of ethics, but they can also open it up. Processes of closure transform the ‘undecided’ space into a decided normative order. Drawing on post-foundational thinking, closure can thus be understood as a process of ‘hegemonization’ in which interests and values of some are in/excluded and stabilized, whereas the contingency of this order is hidden and ‘depoliticized’ (e.g. Introna, 2005; Laclau and Mouffe, 2001; Mouffe, 2005; see also Hay, 2014; Willmott, 2005). In our context, closure works for instance by excluding certain interests from the process of designing technologies, by automating decision making, black-boxing and denying the ethical relevance of specific categories, decisions or acts, or by short-circuiting or circumventing reflexivity (Rouvroy, 2013: 144). In turn, opening is a process of dis-closing the normative and ethical basis of established practices and technologies. It reveals the contingency of established practices and technologies and (re)politicizes the taken for granted, for example by provoking new forms of problematization, contestation or resistance. The latter make it possible to consciously act in relation to algorithmic procedures and decisions and may in effect support subjects’ ‘ethical work’, that is, reflexive practices in which they transform parts of their selves and seek to become what or who they aspire to be (Foucault, 1992: 27; Magalhães, 2020).

To summarize: while all governmental technologies, including algorithmically driven digital technologies, shape the space of ethics in the sense of providing closure, they cannot fully determine human conduct and ethical practice. Algorithms are embedded in heterogeneous socio-material practices and never simply execute a set of instructions (Introna, 2016: 20). There will always be a space for human action and responding to the expectations, constraints and imperatives that are embedded in technologies and practices.

We now examine how facial recognition and drone technologies shape the space of ethics and influence the conditions of possibility for ethical practice. We analyze three features of algorithmic governmentality and its relationship to the space of ethics: the specific modes of objectivation implied in these technologies, the implications of removing human reflexivity from the process of categorization that always goes into the work of datafication and data analysis and, finally, the modes of subjectivation associated with these technologies. As we will demonstrate, the space of ethics is constituted in a dynamic interplay of closure and opening.

Facial recognition and drone technologies at work

Objectivation: Managing ‘datafied objects’

Objectivation is the process in which human subjects and their experiences are rendered visible and transformed into an object of knowledge. There are different modes of objectivation (Foucault, 1983: 208). The disciplinary mode is epitomized by the classical examination. This procedure ‘establishes over individuals a visibility through which one differentiates them and judges them . . . (and) it manifests the subjection of those who are perceived as objects and the objectivation of those who are subjected’ (Foucault, 1976: 184–185, translation modified). As a paradigmatic way to extract and constitute knowledge, the examination is central for the development of diagnosis, measurement of performance (Townley, 1998), systems of accounting (Roberts, 2009) and metrics like rankings (e.g. Espeland and Sauder, 2007; Hansen, 2015).

Algorithmic governmentality reconfigures the processes of extracting knowledge and modifies the process of objectivation. The use of facial recognition technology exemplifies this process. Increasing image quality of cameras in digital devices and the expansion of cheap apps have accustomed people to facial recognition technologies. Apple has developed a software (‘Face ID’) used to unlock devices, making payments or accessing specific data. Here, 30,000 infrared dots are projected onto the face and produce a forgery-proof objectivation of the face by reading the patterns (Apple, 2021). Restaurants and retail providers have developed pay-by-face technology (Guszkowski, 2020). Airports screen travelers by matching face scans to online images, watch lists and criminal databases (CNN, 2019). Schools use facial recognition technologies (Alba, 2020) while law enforcement agencies deploy them to identify criminals and search for missing persons (Rector, 2021). These forms of objectivation would be impossible without algorithmically driven digital technologies transforming the human face into a data object, which necessarily reduces the complexity of the actual face. The face is first scanned and subjected to a graphical analysis, which involves identifying points, such as mouth, tip of the nose and eyes. These points are connected to form a grid model of biometric measurement. An algorithm encodes the face and creates a ‘faceprint’, a unique numerical code. Once the system has stored the faceprint, it can compare it to the millions of faceprints in the database, divide populations and sort (in)dividuals (Introna, 2005).

The US company HireVue has produced face recognition software, which is increasingly used by organizations to improve recruitment and the quality of management decisions in the evaluation of candidates. It offers video interviews and AI-driven assessments, launched in 2014 as an add-on to its video interview software. According to the company:

The traditional candidate screening process is inconsistent. CV screens are just that, quick scans that are seconds or minutes long, focusing on characteristics that are known to be less predictive of performance like grades, former company, and years of experience. Phone screens are also highly unstructured, leading to even greater variability in which talent makes it through your hiring funnel. Structuring the interviewing process right from the start with assessments and video interviews brings consistent and fair evaluation of candidates, yielding better, more inclusive hiring outcomes. (HireVue, 2021a)

The candidates do not sit face to face with the HR manager, but in front of their laptop or cellphone camera and present themselves. Candidates are asked a set of questions. The technology not only evaluates responses and compares it to ‘ideal responses’, it also scans and analyzes facial movements, the choice of words and voice to assess the honesty of responses. The datafied face is transformed into a measurable object, which is compared with ideal types that categorize the applicant accordingly and result in an ‘employability score’. While this score is contingent on assumptions and preconceptions of programmers and designers, the selection and weighting of factors, the quality of data, as well as their interlinking, a quasi-scientific ‘objectivity’ and superiority over traditional methods is claimed:

Unconscious bias compounds at every step of the traditional hiring process. HireVue’s AI-driven approach mitigates bias by eliminating unreliable and inconsistent variables like selection based on CV and phone screens, letting you focus on more job-relevant criteria. AI empowers you to assess each candidate in a large pool quickly and fairly, so you can make high-quality inclusive hiring decisions. (HireVue, 2021a)

Another firm, Faception, presents itself as a ‘facial personality analytics technology company’. It promises to ‘reveal personality from facial images at scale to revolutionize how companies, organizations and even robots understand people and dramatically improve public safety, communications, decision-making, and experiences’ (Faception, 2021). The idea of ‘revealing personality’, or inferring emotional states, or even sexual orientations (Kosinski and Wang, 2018) from measurable characteristics of the human face has often been criticized as ‘pseudo science’ in the tradition of phrenology and physiognomy (Metz, 2019). Yet, Ekman and Friesen’s (1978) ‘Facial action coding system’ (FACS), which objectifies the movements of the facial muscles, dividing them into 27 ‘action units’, and links them to ‘basic emotions’ (like anger, fear, sadness, joy, disgust and surprise), is still used. Such classifications are used in automated systems that differentiate facial expressions and link them to specific emotional states. In this vein, commercial applications like EmotionNet, promise to ‘empower both humans & AI with deeper understanding of emotions to make more intelligent, compassionate and socially-sustainable decisions’ (EmotionNet, 2021). EmotionNet is deployed by organizations for purposes of marketing and political communication (Benitez-Quiroz et al., 2016).

It is worth recalling that in social interactions we usually interpret the variety of forms of expression of the face in context, and we engage in meaning negotiations (Goffman, 1967). But facial recognition and drone technologies all embody a technological vision that assigns codes according to a binary logic. The human face, as a dynamic expression linked to the history of the subject, is transformed into a measurable and decontextualized object that travels easily in digital infrastructures and across domains (Bueno, 2020). Furthermore, the algorithms that select, filter, frame and thereby constitute the face as an object are saturated with politics. Since they not only reflect the choices and prejudices of programmers, but also the possible biases that are embedded in the set of training-data, the generated objects are all but neutral representations (Raji and Fried, 2021). For example, white skin color is typically reified as ‘normal’ while people of other colors are subject to ‘misidentification’ (Harwell, 2019b). A group of reporters from the German Bavarian Broadcasting (Bayerischer Rundfunk) recently undertook experiments with facial recognition software promoted by companies promising the production of faster, fairer and objective behavioral personality profiles in recruitment processes (Bayerischer Rundfunk, 2021). The software divided personality into five dimensions: ‘openness’, ‘conscientiousness’, ‘extraversion’, ‘agreeableness’ and ‘neuroticism’. Test persons were awarded scores between 0 and 100 for each of the personality traits. A test person portrayed by the system as being particularly conscientious when not wearing glasses or covering her head with a scarf, saw her scores dwindling once when she wore glasses or a scarf. The study also showed how the chosen background image influences the process: a bookshelf behind the face-image alters the results and creates a different person.

Consider also the example of drones, which make it possible to observe and intervene physically in the world from a great distance and at high speed. A much-discussed area of application is anti-terrorism, especially drones used for the ‘targeted killing’ of people classified as (potential) terrorists. The drone is often presented as a ‘precision instrument’ and morally superior to conventional killing, purportedly because they work ‘cleanly’ and ‘rationally’ and, at least on the side of the drone owners, avoid personal injury. Drones rely on objectivation in the form of a ‘more precise visualization of the patterns of human life’ (Pentland, 2015: 12–13). A drone attack requires that the target be defined and fixed on a ‘kill list’ (Weber, 2016), with suspicious persons identified as ‘dangerous’ or ‘terrorists’.

The distinction between ‘personality’ and ‘signature’ strikes is important (Chamayou, 2015: 46–47). A personality strike assumes that a terrorist is known by name. Suspicion arises from past actions, from indications of terrorist intentions, statements or membership in groups. In signature strikes, the object is not a known person but a bundle of data generated by algorithms. Algorithms derive abstractions based on the identification of patterns in data, which correspond to a ‘signature’ of predefined behavior. The ‘pattern of life’ analysis mentioned above was developed for mapping relationships between people, places and things (Franz, 2017). In extreme cases, the target of an attack or killing is not a human being but a data signature. The following quote demonstrates the logic:

If Bill and Ted live at the same address, rent a truck, visit a sensitive location and buy some ammonium nitrate fertilizer (known to be used for growing potatoes but also for the production of home-made bombs), they would exhibit a behavioural pattern that corresponds to that of a terrorist signature, and the algorithm designed to process this data would then sound the alarm. (Chamayou, 2015: 3–4, in Cheney-Lippold, 2019: 43)

In this way, life can be captured as an object and placed into a risk-matrix and liquidated ‘with the swivel of a joystick’. The drone’s scanning technologies are connected to the ‘pattern of life’ analysis, its instrumentalist mode of objectivation and associated production of ‘exterminatory violence’ (Pugliese, 2011, quoted in Schwarz, 2016: 64). In all, the above examples demonstrate how facial recognition and drone technologies turn subjects into ‘datafied objects of simple strategic convenience’ (Cheney-Lippold, 2019: 42), aligned to the programs, projects or commercial activities of the authorities in charge, be it military forces hunting ‘terrorists’ or companies recruiting ‘successful candidates’. Such objects can be managed at great distance and speed, and purportedly more efficiently than the types of engagement with subjects known in the classical offline world. Still, such data objects are connected to processes of categorization and classification in intricate ways as discussed in the following, and with implications for the space of ethics.

Removing human reflexivity from the process of categorizing

Governing relies on categories for describing and ordering the world as well as for their application to specific situations. The process of arriving at categories is crucial in defining the space of ethics. In pre-internet governmentality, categories and objectifying measures were typically defined by experts, professionals or scientists, for example in profiling employees or criminals (Bernard, 2019; Townley, 1998). While the process of arriving at relevant categories is always complex, it would still be possible, at least in principle, to identify the designers of systems and hold them accountable for the use of specific categories and outcomes like false positives or false negatives. But in the context of algorithmic governmentality, categories are increasingly generated by machines (Alaimo and Kallinikos, 2021; Hildebrandt, 2013; Introna, 2005). This makes it difficult to give an account of how and why specific categories are arrived at and used (Burrell, 2016: 1; Mittelstadt et al., 2016: 6). Algorithms typically are not single authored but produced by multiple teams, and they evolve over time. Not even programmers and designers can give an account of how specific categories were developed and why they are used. As Coeckelbergh (2020: 117) explains:

This is a problem for responsibility, since the humans who create or use the AI cannot explain a particular decision and hence fail to know what the AI is doing and cannot answer for its actors. In one sense, the humans know what the AI is doing (for example, they know the code of the AI and know how it works in general), but in another sense they don’t know (they cannot explain a particular decision), with the result that people affected by the AI cannot be given precise information about what made the machine arrive at its prediction.

All this suggests that responsibility, understood as a process of giving an account of one’s actions and decisions and of responding to the demands of various others that are affected (Butler, 2005) is removed from the processes of categorization and classification. This is especially so in machine learning, where categories are outcomes of machine operations and often impenetrable even for the experts who run the machines (Burrell, 2016: 10; De Laat, 2018). The space of ethics, which is where responsibility is enacted, is fundamentally modified. On the one hand we observe a ‘de-responsibilisation of human actors’ (Mittelstadt et al., 2016: 12), allowing machines to make ‘autonomous’ decisions. On the other hand, the space of ethics is dispersed. Ethical conduct and responsibility are not simply removed but distributed across the assemblage of human and non-human actors (Hayles, 2016a: 45–54).

Consider again the example of facial recognition technologies. Here, algorithms following instructions designed by programmers to achieve certain goals are increasingly replaced by ‘machine learning algorithms’ reflecting the growing size of training data-sets and complexity of learning models (Raji and Fried, 2021). Drawing on such technologies, HireVue develops AI-driven assessments based on video interviewing. After a job candidate has completed the test, the system produces a report on the candidate’s competencies and behaviors, which is used to rank candidates. HireVue feeds the system with data from interviews with already ‘top-performing’ employees so that machines can continuously learn how to filter and sort. But little is known about how the system generates the individual assessments in practice: ‘HireVue offers only the most limited peek into its interview algorithms, both to protect its trade secrets and because the company doesn’t always know how the system decides on who gets labeled a “future top performer”’ (Harwell, 2019a). Similarly, the computer scientist Katharina Zweig commented on the facial recognition systems investigated by the Bavarian Broadcasting team mentioned above: ‘The problem with face recognition machine is, that we never know exactly which pattern in an image these machines are responding to’ (quoted in Bayerischer Rundfunk, 2021).

Subject to controversy, law enforcement authorities now use similar facial recognition technologies for criminal justice purposes. The categorizations that underpin the process produce ‘actionable output’ largely beyond public debate and resulting in an ‘accountability gap’ (Bennett and Chan, 2018: 806). This gap is exacerbated by the proliferation of sophisticated cheap software produced by start-up businesses operating. The firm Clearview AI, for example, has developed a groundbreaking facial recognition app that US law enforcement officers say they use to help solve shoplifting, identity theft, credit card fraud, murder and child sexual exploitation cases. But little is known about Clearview and who is behind it, and the app is not just a piece of innocent toy. It can help ‘identify activists at a protest or an attractive stranger on the subway, revealing not just their names but where they lived, what they did and whom they knew’ (Hill, 2020).

Take also the example of drones. Typically, drones collect visual data, infrared data and so-called ‘signal intelligence’ via sensors, providing the aircrew with ‘“more-than-human” vision abilities’ (Williams, 2011: 386). Optical sensors can zoom in from large distances. They rely on software that codifies and sorts the world for the decision-maker. For example, is the object a military vehicle or a school bus with children? Is it a target or non-target? At first glance, the technologies may be said to increase the possibility of responsible decision making, if they are based on reliable knowledge about how the data are compiled, aggregated and analyzed, and the values incorporated into the decision process. For instance, facial recognition technologies are said to provide ‘evidence’ that can lead to more responsible decision making in law enforcement (Goriunova, 2019). Likewise, drones are said to provide greater awareness of what is happening on the ground: ‘From his control station at the Pentagon, Dan (a drone pilot) is not only watching the target in real time; he has immediate access to every source of information about it, including a chat line with solders on the ground’ (Bowden, 2013). From this lens, these technologies expand the space of ethics. Improved ‘evidence’, it is often argued, allows for more precise and ethically superior forms of interventions.

On closer inspection, however, these technologies are typically driven by opaque algorithms, which map the world for the decision-maker, but may fail to capture real life on the ground. Excessive faith in precision instruments may displace or eliminate doubt as an inherent element of responsible decision making (Amoore, 2020). A drone operator explains: ‘We can develop those patterns of life, determine who the bad guys are, and then get the clearance and go through the whole find, fix, track, target, attack cycle’ (Chamayou, 2015: 46). Consequently, the space of ethics, which is where responsibility is supposed to be enacted in relationship to self and others, is modified. The ‘find, fix, track, target, attack cycle’ illustrates how responsibility shifts to procedures and machines. At the same time, moral agency is distributed across the whole network, making it difficult or impossible to attribute responsibility to some entity – an individual, an organization, an author or designer of the code (Hayles, 2016a).

Subjectivation: Circumventing reflexivity and self-formation

The constitution of objects of knowledge – objectivation – is linked to modes of subjectivation; that is, the specific ways in which subjects are formed and form themselves in relation to normative expectations of behavior (Foucault, 1983: 212). With governmentality, individuals turn themselves into objects of knowledge or take themselves as objects of reflexive action in transforming their relations to themselves and others. They can relate to prescriptions and fold them back on themselves in a variety of ways. They may internalize models of behavior inscribed in specific technologies and in this way self-regulate to comply with what they think and anticipate algorithms or the designers of algorithms are expecting (De Laat, 2019; Introna, 2016). Yet, as reflexive subjects and ‘users’ they may also engage in problematizing, questioning and resisting the prescriptions and effects of the technologies in question, or even use them actively in their own projects of ethical self-formation (Magalhães, 2020: 6).

To illustrate, facial recognition technology has subjectifying effects that range from the internalization of expectations inscribed in the technologies as such to the problematization and questioning of their effects. Facial recognition is often presented as a form of ‘personalization’, which provides access to the ‘real person’, yet it remains an abstraction generated by algorithms. Indeed, such language of personalization and authenticity may seem to have more in common with a commercial offer or relationship than a trustworthy consideration of individual needs and desires (Rouvroy and Berns, 2013). But while these technologies help establish a kind of ‘truth’ about individual persons, imposed on them as profiles or models of behavior, they also seek to align subjects to these models by inviting them to present themselves, reflect on what they are or what they could become. More than anything, subjectivation in this disciplinary sense is shaped by prescriptions for conduct and criteria of normalcy. Consider again the example of HireVue. The company addresses potential job candidates on its website and invites them to reflect and work on themselves in preparing for the ‘successful interview’:

Do the practice questions: Taking the practice questions will help you get comfortable with the style and timing of an On Demand interview. These are not your real interview questions, and they will not be seen by the company. Distraction free environment: Keep in mind that everything your webcam sees, and your microphone hears, in the background is recorded. A quiet, distraction free area helps you give the best impression possible. Be yourself: Interviewing can be nerve-wracking. Doing an On Demand interview lets you take the interview in your own space at your own time. Get comfortable and be confident in yourself. (HireVue, 2021b)

Importantly, such prescriptions for conduct and ensuing criteria of normalcy are not restricted to the given situational encounter between HireVue software and the interviewee but are stretched out to include societal institutions providing ‘resources’ and ‘tips’ for mastering online interviews. Some US universities now provide guidelines to help students prepare HireVue interviews (University of Dallas, 2021). After establishing the centrality of being ‘authentic’, the guide developed by University of Dallas lists the themes brought up in interviews, such as ‘decision-related’, ‘integrity’ and ‘dealing with adversity’. It concludes with the following advice: ‘If there is a feature where you can get rid of the picture of yourself on the screen, it makes it easier to look directly into the camera.’

While such guidelines seek to govern candidates’ self-regulation and align them with models of ‘successful’ behavior, we also find job candidates reflecting critically about their experience with HireVue on the website Reddit, suggesting an effort at reconstituting relations to self and others in terms that problematize the prescriptions for conduct and their associated criteria of normalcy:

So I applied . . . and around a few hours later I got an email for a Hirevue interview. As soon as I opened the site and tried doing some practice questions, I immediately hated it . . . I bring this up because my (then) current manager told me she hired me based on my personality alone. How hard is it for them to tell me to come by the store for a 30 minute interview for an entry level position? I felt alienated and completely awkward, and it wasn’t even 30 seconds before I closed the site and ‘nope’. (Reddit, 2021)

Another candidate problematizes and questions the technology in a much wider sense, reflecting on its perceived large-scale consequences for self and others:

I was mortified, what if you don’t have a phone that supports the app, don’t have a pc etc.? Are you just screwed?? Also what if you are blind, deaf, deaf and no speech therapy, or have a speech impediment? The time it gives you to answer the questions isn’t enough for a bad stutter, or sign language. What if you just . . . Mess up? I feel this also breeds discrimination, the employer can look at the video and go ‘this quality is potato. . . NEXT’ or “That one is wearing a hijab. . . NEXT’ . . . Edit: I just found out they use an AI to analyze everything you do from your pupils dilating to your tone of voice. (Reddit, 2021)

We see in these examples how the use of algorithmically driven digital technology like HireVue governs subjectivation. It not only shapes the speaking self but also the speaking self’s perceptions of what the software or technology does to others. Yet, subjectivation here cannot be reduced to subjection to procedure and predesigned norms, but also includes elements of problematizing these very norms.

As to the technology of drones, we have already shown how it objectifies the targets and with the ‘signature strikes’ presents targets as pattern-based abstractions. Still, much like the facial recognition example above, the drone technology also has subjectifying effects, ranging from the internalization of expectations inscribed in technologies to the problematization and questioning of the effects of these technologies. Drone operators conform to specific rules when they execute technical operations. They are typically located thousands of kilometers away from the specific intervention. They undergo lengthy training and work on scheduled shifts in airconditioned rooms watching the distant action unfolding on high-definition screens. They interpret electronic signals, and – from a safe distance – occasionally eliminate selected targets.

Operators are sometimes colloquially called ‘cubicle warriors’, who develop a ‘PlayStation mentality’ in this working environment. This mentality epitomizes a certain attitude towards self and others that is also expressed in language use. The term ‘bugsplat’, for example, is used to denote a successful attack. It has become official terminology used by US authorities to refer to ‘the individuals killed by a drone, as the dead bodies resemble squashed bugs when rendered as pixels on a screen’ (Keene, 2015: 22). Successfully pulling the trigger and causing the ‘bugsplat’, the drone operator may regard the drone as superior to ordinary manned military aircrafts, supporting in this way its bureaucratic rationale. Here, drone operator Justin’s remarks on the difference between the two reflects an internalization of normative expectations inscribed in drone technology for military use:

On balance . . . the (drone) . . . is hugely advantageous in terms of being able to act more ethically I believe . . . So in answer to the simplistic question: ‘Is it different?’ Yes, it is, but different in a good way. (‘Justin Thompson’, interviewed by and quoted in Cole, 2017)

These examples reveal how the drone operator as a bureaucratic subject produces ‘bugsplat’, and as such is framed by the technology and the truth claims inscribed in the algorithms and the surrounding and largely supportive institutional discourse. None of these arrangements address ‘subjects’ as reflexive moral agents but rather attune subjects to the informational and physical environment and to the procedures and calculations enacted by the machines.

These machines are complex assemblages where technical and human components are involved. While algorithmic procedures and infrastructure preform choices and the space of decision making, human interpretations and evaluative judgments are still involved (Hayles, 2016a: 50). This is illustrated in an interview with the drone operator Dan, conducted by Bowden (2013). Dan reports how he often watches the people-targets for a long time before he pulls the trigger. He also experiences the stress of his soldier colleagues on the ground whom he is tasked to support. Once the trigger is pulled, he sees the violence close up and in real time, even to the point of watching the blood and severed body parts of victims, as well as the anguish of relatives and friends:

There is a very visceral connection to operations on the ground . . . When you see combat, when you hear the guy you are supporting who is under fire, you hear the stress in his voice, you hear the emotions being passed over the radio, you see the tracers and rounds being fired, and when you are called upon to either fire a missile or drop a bomb, you witness the effects of that firepower. (Dan, quoted in Bowden, 2013)

Slim, another drone operator, compares his work and experiences to pilots who physically fly into an area and release their weapons:

While fighter pilots have to worry about being shot down, they rarely see the results of their attack . . . After an engagement, we have to conduct surveillance for quite a long time. Yes, we may only be seeing it, but sometimes, we’re seeing it for hours on end, and that is part of the traumatic impact of the mission. It’s a definite form of stress on the operator in and of itself. (Slim, interviewed and quoted in Chow, 2013)

These examples demonstrate how the drone operator’s sense of attachment to the people on the ground and to related subjects, is made possible by the drone and its algorithmically produced visualizations, which shape the operator’s relationship to colleagues and those who are framed as ‘targets’. While drones allow for the avoidance of face-to-face encounters and in this way reduce the space of ethical reflection, they can at the same time ‘bring the faces back’, provoking reflection if not a (re)problematization of the relations to self and others.

Shaping the space of ethics: Closure and opening in algorithmic governmentality

We have analyzed three central features of algorithmic governmentality and its relationship to the space of ethics, as exemplified by facial recognition and drone technologies: the mode of objectivation that reduces humans and human life to data-packages; the removal of human reflexivity and accountability from the categorization and extraction of knowledge from data; and the modes of subjectivation that seek to circumvent reflexivity and nudge behavior in specific directions. Our analysis conjures with Rouvroy’s (2013) claim that algorithmic governmentality tends to close down the space of ethics as it avoids addressing people as moral subjects. But our analysis also illuminates practices that potentially open the space of ethics, bringing new forms of questioning and problematizing into play, often as an unintended side-effect. Thus, the space of ethics is not simply replaced by the calculative rationality and logics of algorithms (Lindebaum et al., 2020; Zuboff, 2019) but rather (re)constituted in the dynamic interplay of closing and opening.

Facial recognition technologies involve the automation of decision making and thereby raise the risk of excluding ethical considerations and reflection. The technology illustrates how sedimented categories are taken for granted and tend to ‘depoliticize’ the sorting involved in the process. Yet, despite the appearance of objectivity that tends to close down the space of ethics, the potential of reflexivity, contestation and resistance against these algorithmically driven digital technologies, also may effectuate their politicization, as is manifest in the aforementioned HireVue. In addition to job candidates’ reflections, frustrations and anxieties expressed on public websites like Reddit, non-profit organizations and institutional actors have begun to problematize and take steps to regulate the interview platform (Harwell, 2019b). The company recently conducted a third-party audit, which has spurred it to drop facial monitoring in interviewing (Kahn, 2021). Projects like Gender Shades (Buolamwini and Gebru, 2018) and the mentioned German Broadcasting project, have disclosed the discriminatory effects of these technologies, opening a space for ethical problematization and political contestation. Facial recognition technologies are contested, not only because they add yet another layer of surveillance and intrusion into the private sphere (Hill, 2020), but also because of biases that result from training data on which these systems depend for recognizing an object or misattributions based on incorrect data (Amoore, 2020: 70).

Other recent studies have shown how users can actively engage in algorithms and practice what Velkova and Kaun (2021: 535) call ‘repair politics’. By this they mean ‘a corrective work that works through improvisations, patches and ingenuity, together with and within algorithmic systems, to make them generate unintended, alternative outputs to respond to the “brokenness” or biased representational politics of algorithms.’ Kellogg et al. (2020) have mapped multiple forms of ‘algoactivism’, which comprises heterogeneous practices of individual and collective strategies and tactics of escaping algorithmic control, contesting algorithmic systems and opening a space for public debate around excessive surveillance, algorithmic fairness, accountability and transparency. Disclosing the violation of privacy norms and rules and values of human dignity (i.e. non-discrimination), as practiced by organizations like Algorithm Watch or the Algorithmic Justice League, opens as space for debate and potential transformation on organizational and institutional levels.

The closure and opening dynamics can also be seen in the drone-example. ‘Smart’ drones have a capacity to close down the space of ethics by selecting targets and taking over decision making. By creating a moral distance to objectified targets, they tend to ‘set responsibility afloat’ (Bauman and Lyon, 2013: 91, 94; Schwarz, 2016). At the same time, the very technology that enables acting at a distance, also creates a new form of proximity, allowing the operator to zoom in on the ‘target as if it were close’ (Chamayou, 2015: 114, emphasis in original). Drone technology ‘fuses extreme visibility with extreme distance’ (Pinchevski, 2016: 65). While there is no doubt that it helps establishing and maintaining a huge power imbalance between the drone (and its operators) and the targets, operators are not simply acting in a machine-like way, dispassionately executing imperatives that are built into the machine. Post-traumatic symptoms and stress of drone pilots point to moral conflicts and struggles prompted by this form of closeness and demonstrate how drone operators experience ‘moral injury’ when personal standards of good conduct are violated (Enemark, 2019). While proximity is still mediated by technologies, it can nevertheless instigate moral reflection and open spaces of questioning, problematizing and politicizing the technology and one’s relation to it. Coeckelbergh (2013: 89) calls this ‘emphatic bridging’, which ‘may be regarded as an unintended mutation or indeed “ethical hacking” of the otherwise distancing technologies’. Finally, ethical-political activities may enact a ‘disclosive ethics’ (Introna, 2007), which seeks to make the ‘morality of technology’ visible and discussible. Dis-closing moral values and decisions that have been ‘folded’ into specific algorithmic procedures and technical systems, scrutinizing choices of scientists and designers (of algorithms) that went into the formation of systems and bringing them the surface potentially opens a space for debate and transformation, a re-politicization of what at first seemed normalized, depoliticized.

Practices of designing systems do not only affect social and moral values that are built into systems technologies and algorithmic procedures. They can, as Shilton (2012) has shown, also encourage ethical reflections and action, for example when the technical work of engineers or programmers raises ethical challenges around data use and surveillance. Ethical reflection can also be made an explicit part of the design practice. In our practice-based understanding, this includes practices that consciously build specific moral values and ethical considerations into machines and algorithms (Verbeek, 2008). While we would insist that ethics cannot be reduced to the execution of a program (even if the program is consciously designed so as to comply to specific moral values), certain approaches to reflexive design (Sengers et al., 2005) or critical design practices (Agre, 1997) could contribute to opening the space of ethics, insofar as they make it possible to discuss, contest or (re)negotiate otherwise hidden values built into technologies.

In summary, our examples illustrate that the closure of complex and heterogenous assemblages of algorithmic technologies can never be final and fixed. Algorithmic assemblages are dynamic rather than static, open to modification and change. This is clear from the ways in which various forms of politicization of what appear to be natural and depoliticized orders, are taking place with a potential to re-form relations to self and others.

Conclusions

The aim of this article has been to explore how the use of algorithmically driven digital technologies in work-organizations and society at large shapes the conditions of possibility of ethical conduct. To this end, we developed a conceptual link between algorithmic governmentality and the space of ethics. Taking facial recognition and drone technologies as examples from the broader field of People Analytics, we have demonstrated the capacity of algorithmic governmentality to shape the conditions of how we relate to self and others. The use of these technologies reconfigures our sense of distance and proximity in ways that are different from technologies of the pre-internet age, with significant implications for human reflexivity and the capacity to enact responsibility in decision making. This influencing does not work in a linear way but displays considerable variation. Even if the logic is ‘avoiding confrontation with subjects’ (Rouvroy, 2013: 161), perhaps making ‘inhuman choices’ easier, as Eubanks (2017: 12) puts it, it is important to consider the fissures within these regimes, their various modes of subjectivation and the potential for transformation they inhabit. The space of ethics is not disappearing. Indeed, it is constituted in the interplay of mechanisms of closure, such as automating and black-boxing, the denial of ethical relevance or the circumvention of reflexivity, and mechanisms of opening. Mechanisms of opening can emerge as an effect of a broad range of ethical-political practices, ranging from ‘ethical hacking’ to problematization or contestation of algorithmically driven digital technologies in the public sphere and its more specialized sub-spheres targeting specific issues. This is exemplified by the rise of algoactivism and conscious design interventions.

In reclaiming the space of ethics within algorithmic regimes, we contribute to the critique of algorithmic governmentality yet avoid deterministic interpretations and dystopian visions. While it is indeed important to point to the ‘daily insults to human autonomy’ (Zuboff, 2019: 309) and the dangers of an ‘evacuation of the subject’ (Rouvroy, 2020), all of which make the ‘death of politics’ and the disappearing of the space of ethics imminent, our approach suggests also to ‘follow the distribution of gaps and breaches, and watch for the openings this disappearance uncovers’ (Foucault, 2003: 380). Following this line of thinking, future research in organizations or wider societal domains could begin to address the dynamics of opening more systematically. Future empirical studies might scrutinize, for example, how exactly and with what effects the politicization of otherwise taken-for-granted digital technologies is playing out in specific situations and contexts, and occasionally across different social arenas. For instance, the day-to-day workplace employee contestation and resistance of automation, the character of academic and expert activism on the ground and in media fora exposing the problematic workings of some digital technologies; the proposals of new technological designs and the proliferation of social movement organizations addressing wider issues of inequality, privacy and data ethics, and much more. Finally, we should not forget here to consider in more detail the politicization that takes place through the channels of formal political systems at various levels, including the lobbyism by civil society groups and business, which, as also illustrated in this article, occasionally translates individual, organization and society level problematizations into regulatory interventions.