Algorithms as complementary abstractions

Abstract

The text diagnoses two opposing tendencies in the research on algorithms: the first abstracts and unites heterogeneous developments under the term “algorithm”; the second emphasizes specifics such as data sets, material conditions, software libraries, interfaces, and so on, thus dissolving that which apparently algorithms do into more fine-grained analyses. The text proposes a research perspective that resolves this tension by conceiving of algorithms as a relation between the abstract and the concrete that allows to capture both in their interdependence. This approach is informed by two motives: first, the necessity to connect detailed analyses of specific information technologies with general political concerns; and second, the application of recent feminist critiques of epistemology to the analysis of algorithms. The ensuing relational perspective on algorithms is connected to the genealogy of algorithmic technology before being demonstrated regarding the mutually complementing relationships: algorithms-materiality, algorithms-data, algorithms-code, and algorithms-interfaces.

Introduction

In recent years, many discourses, debates, or concerns regarding information technology have crystalized around the concept “algorithm.” In that sense, the concept unites and supersedes roles that have been played by “software,” “artificial intelligence” (AI), “automation,” and also just “computers” or “information and communication technologies.” We read about “the relevance of algorithms” (Gillespie, 2014), “algorithmic cultures” (Seyfert and Roberge, 2016), the “algorithmic turn” (Uricchio, 2011), “algorithmic governmentality” (Rouvroy, 2013), and also “governing algorithms” (Introna, 2016).

This currency of the term has drawn criticism from various sides. Many see a problematic loss of conceptual clarity in this wide usage of the term algorithm (Beer, 2017; Dourish, 2016). For other approaches that take a view toward the empirical and particular, for example, in science and technology studies (Bergermann et al., 2018), ethnography (Seaver, 2017), or software studies (Fuller, 2008), algorithms are at best one of many elements in the complex interplay that constitutes what we commonly know as information technology or digital media. Some positions see algorithms not even as contributing to this complex interplay, but as a mere effect of “intersections where software and hardware, labor and nature come together to produce key algorithmic operations” (Munn, 2018: 25). Many recent studies, particularly regarding bias, emphasize data as the key component linking social and technical issues (Buolamwini and Gebru, 2018; O’Neil, 2017).

Thus, the flourishing field of research on algorithms in the recent years reveals a tension between abstraction and dissolution: on one hand, using algorithms as the central abstraction to theorize widespread and complex developments in concepts like “algorithmic culture” or “algorithmic governmentality,” which infuses the term with diverse and growing aspects of meaning; on the other hand, the dissolution of the concept into the very specifics of software, machine code, data sets, chips, storage media, cables, rare earths, labor, generation of energy, and much more.

Importantly, both approaches are not only concerned with empirical or conceptual clarity. Most often, they have a social, ethical, or political angle. Speaking of “the relevance of algorithms” (Gillespie, 2014) or “algorithmic governmentality” (Rouvroy, 2013) argues that algorithms have momentous influence, for example, on the structure of public discourse or the functioning of state power. Similarly, the emphasis on the specific interplay of many components that produces what apparently algorithms “do” is often meant to highlight that the social or political issues might be located, for example, in a biased data set (Campolo et al., 2017) or in the recruitment of an overworked and underpaid workforce to control posts on social media (Gillespie, 2018). This social or political angle will be maintained in the following.

I will develop a view on algorithms that takes a middle ground in the tension between dissolution and abstraction. Algorithms are conceived in a relational manner: as an abstraction that is inherently related to a necessary complement of that abstraction in something concrete and particular.

Algorithms as complementary relations

Accounts of algorithms often begin with the etymological roots in the name of the Persian mathematician Muḥammad ibn Mūsā al-Khwārizmī, and thus its relation to calculation in general. However, contemporary notions of algorithms depend on the developments of mathematics in the early 20th century, when Gödel (1931), Turing (1937), Church (1936), and others inquired into elementary features of formal systems. From that moment on, certain forms of abstraction (Gödel numbers, Turing machines, and Lambda calculi) were complemented with an idea of computability which Turing (1937) based on the executability of a (very formalized) “machine”. Here, an algorithm abstracts a task but in order to make it executable on a Turing machine. This is what I intend by complementary abstraction: algorithms are abstract, but they have their complement (here the Turing machine) already built in.

This inherent relationality of algorithms is generalized in this article. There is not just one complement of algorithms—a mode of execution—but many. Another one would be data: To say that, for example, a biased data set is causing problems, we have to know what less problems would mean, for example, what the algorithm that uses this data set is meant to do—say controlling passports at a border. Then, understanding which role automated border checks play in a general turn toward “algorithmic governmentality” is important to gauge the scope of the consequences of a particular element, such as biased training data. In consequence, asking what an algorithm is meant to do affords a suitable level of abstraction to ask what this task—here border control—when turned into an algorithmic task means in a wider perspective. Only from that level of abstraction can the concrete impact of a particular element like a data set be ascertained. Vice versa, we would not understand the consequences of automated border control without taking into account that dependence on training data. Both tendencies in research on algorithms—abstraction and dissolution—presuppose each other. In the following, I want to argue for the necessity to maintain this relational middle ground, before illustrating it regarding some central relations in the following sections.

From the abstract toward the particular

Abstract views on algorithms are prone to conceive of them merely as the implementation of a task or intention, resulting in a focus on the programmers and the users determining the tasks and to forget all the other influences on the outcome of an algorithm (Chun, 2008). In particular, algorithms are often thematized as the essential element hidden behind glossy interfaces or input–output relations. The notorious metaphor of a “black box” (Kitchin, 2017; Pasquale, 2015) illustrates this very well. In contrast, it has been shown that algorithms need to be seen as becoming what they “are” only in relations. For example, Chun (2008) emphasizes the complex socio-technical mediations that are necessary before a programmer’s intention can be executed on a machine. Bucher (2017) shows that interfaces do not just hide algorithms, they also change how users imagine algorithms, which changes the users’ behavior, which in turn changes the efficacy of algorithms. Thus, what we gain when opening the “black box,” for example, by access to source code or data sets, is not the essential or even just a better perspective, but a different perspective (Brey, 2005; Introna, 2016; Matzner, 2019). This difference amounts to a switch in complementary relations: for example, from algorithms-interfaces to algorithms-source code. Depending on which relation we take into view, we deal with a different view of algorithms.

A related set of rather abstract views on algorithms does not necessarily consider them as expression of intention or defining element, but uses the term “algorithm” or “algorithmic” to denote the entirety of very complex systems. Often, this goes along with considering the algorithmic as inhibiting an essential (usual social or political) quality. For example, Rouvroy (2013), in her analysis of algorithmic governmentality, finds that algorithms reduce any form of human interaction to a form of “data behaviorism.” She contrasts this form of interaction with a richer human form of sociality that allows for a mutual constitution of subjectivity rather than the immersion into an environment of “stimuli producing reflex responses” (Rouvroy, 2013: 155). Thus, in contrast to my term of “complementary abstractions,” such views could be called “oppositional abstractions,” for the argumentative load is carried by an opposition of algorithms to something else. Usually, this opposition entails social or political concepts. Rieder (2020: 54) critically engages with such views in demanding “to approach some of the most complex techniques in computing from a humanistic perspective with regard to their technicity and not just their social effects and entanglements.” With this focus on a “properly technical substance that sits at the center of technical practice” against the backdrop of its “political relevance” (Rieder, 2020: 54), he already takes a much more relational view. However, he still essentializes technology and thus algorithms—not in an oppositional operation like Rouvroy—but in following Simondon in arguing that technicity itself is “an essential mode of existence for man” (Rieder, 2020: 58). He does so partly not to lose important distinctions in a flattening relational ontology. This problem is discussed below, and I will show that it can be curtailed without such a strong ontological commitment.

This discussion helps to qualify the idea of complementary abstractions. The issue is not to show that algorithms are inherently—or essentially—relational. Rather, their relationality demands us not to dwell on essential qualities on an abstract level, but to tie that abstraction and their social entanglements to concrete specifics. In other words, this is not to only say that every algorithm needs to be complemented with material infrastructure, data, or interfaces, but also to ask which specific material infrastructures, data, or interfaces are involved.

This situatedness is a central tenet of Amoore’s (2020: 158) “Cloud ethics.” She argues “that one cannot stand outside the algorithm to judge its morality”. Consequently, in

place of an ethics that seeks to make an illegible algorithm legible to the world, a cloud ethics recognizes the nonclosure in all forms of writing and pushes the fabulation of the algorithm beyond what can currently be read of its logic. (Amoore, 2020: 160)

This situated approach avoids both taking the algorithm as the decisive element that needs to be made transparent (and consequently may be less biased or more explainable) and the tendency to essentialize the algorithmic. Rather, Amoore trains a general ethical sensibility of avoiding closure on algorithms. She urges to care for the “incomputable” or “unattributable” (Amoore, 2020: 170). However, at times, Amoore (2020: 161) appears to read algorithms as inherent agents of such closures and thus the “unattributable” as an outside constitutive of ethics or politics rather than a situated perspective: “If one understands politics as expressing the impossibility of resolving all problems with a solution, then algorithms appear to circumvent the political, because they adjust the parameters of the problem to experiment abductively with the ‘good enough’ solution”. These depoliticizing effects of algorithmic operation in many areas are certainly a huge ethical problem. However, it should not be turned into an inherent algorithmic problem because it touches upon the problem of politicization per se. For example, Judith Butler has shown that every process of creating meaning—or any “form of writing” as Amoore herself states—produces something unattributable, some form of exclusion. Furthermore, Butler (2005) argues that the best we can achieve politically is to include specific others whose exclusion we have recognized—but never achieve a state where exclusion itself is eradicated. Karen Barad, building on Butler’s work, has shown that this political situatedness is not just a problem of meaning. Rather, it extends to what she calls “material-discursive practices” (Barad, 2007: 148). Yet, again, each such practice enables specific observations while excluding others. Introna (2007, 2016) has transferred Barad’s epistemology to the question of “governing algorithms.” Here, each theoretical position corresponds to a “cut” between observer and observed, which brings about particular governing or governed subjects. I extend this perspective to other algorithmic relations.

From this perspective, algorithms are not something that exists and can consequentially become the object of researchers’ attention. Rather, they are themselves a perspective of analysis. This perspective abstracts from many of the details of information and communication technologies, which allows to connect them to wider social, cultural, or political developments. At the same time, this broader view is inherently oriented by particular, concrete aspects. As a material-discursive practice, such an analysis needs to be trained to the ethical sensibility that whatever is of current interest–algorithms and data, algorithms and matter, algorithms and subjects, and so on—precludes other perspectives. Furthermore, using this framework, Amoore’s concern of algorithms as agents of closure can itself be situated. Within each perspective, we have to ask which specific ethico-political potentials algorithms foreclose—but may be also open up.

From the particular toward the abstract

Almost all views that focus on the particularities of information technologies, and thus tend to dissolve algorithms, are relational. Rather than emphasizing a central element or an essential quality, they highlight the complex interplay of many factors. However, in light of the discussion of situatedness in the preceding section, some epistemic issues arise here as well.

For example, the “Anatomy of an AI System ¹ ” provides an impressive view on the many elements that are necessary for Amazon’s “Echo.” However, it also presents itself as final and closed. The anatomical map shows all there is. In contrast to the fractal used on the same website for mainly aesthetic purposes, the map has no depth. Furthermore, all elements on the map become equalized; the assembly line worker is a part of the anatomy just as the chemical element lithium or a neural network. Finally, each element exists independently, there are no constitutional relations. It is literally an anatomy where the entire object is at the disposal of the observer, ready to be dissected into individual components for a view from above. Such tendencies at closure that do not aim at essence but at completeness and flatness have been politicized and criticized more strongly in other fields where circumspect theories aim at global closure—for example, in the discourse on the environment or the Anthropocene (Yusoff, 2018). Ironically, as Amoore shows, such tendencies of closure are also an ethical problem algorithmic applications themselves tend to produce. Thus, if we inquire algorithms for ethical or political reasons, the politics of the own perspective needs to be reflected. Indeed, Barad’s (2007: 389) own writing on ethics and responsibility tends to focus on such an element of completeness, when she assembles many relations into yet another flat map where everything becomes equally (un)important. She argues that objectivity—as a prerequisite for ethical responsibility—“in an agential realist sense requires a full accounting of the larger material arrangement (i.e. the full set of practices) that is a part of the phenomenon” (Barad, 2007: 390). Inspired by Barad’s earlier work on exclusion and the emphasis on situatedness from the last section, a more political perspective is possible. If politics, on a very general level, means that the same is different to a plurality of perspectives (Arendt, 1998), then the issue is not fullness or completeness but rather to elucidate how algorithms relate to these particular differences and which roles the many complements of algorithms play.

A helpful approach in this regard is Dourish’s (2016) text on “Algorithms and their others.” Similar to the aim of this article, he does not want

to dissolve the algorithm in a sea of relations, but rather to understand how algorithm—as a technical object, as a form of discourse, as an object of professional practice, and as a topic of public or academic concern—comes to play the particular role that it does. (Dourish, 2016: 2)

However, most of the article aims at demarcating algorithms and “their others,” often relying on “the limits and particularities of that term’s use as a members’ term, its emic character” (Dourish, 2016: 2). In many parts of Dourish’s paper, relationality is a matter of addition. For example, in his analysis, algorithms plus a determinist practice of translation create code. Such views are indeed influential in computer science; consider, for example, Wirth’s (1976) famous dictum: algorithms + data structures = programs. However, Seaver (2017) cautions that algorithms are much harder to stabilize as an emic term than many practitioners and their observers would believe. Furthermore, it is important not to carry this additional logic over to analysis of algorithms—otherwise this brings us again to demands of completeness or they entail a departmentalization of issues and responsibilities. When analyzing complex software architectures, Dourish (2016) himself uses Barad’s epistemology,

recognizing that the “algorithm” is itself an “agential cut,” a means of constituting some semi-stable object within a dynamic and unfolding socio-technical assembly. This does not diminish the power of “algorithm” as a way of accounting for the operation of a digital assemblage, by any means, but it does imply that “algorithm” may dissolve into nothing when we drill down into the specific elements of a system [. . .]. (p. 5)

This, however, would leave behind the algorithm as a necessary orientation not only for the operation of the assemblage but also for the relevance of its specific elements.

In consequence, both the algorithm and its complements in their mutually constitutive relatedness are necessary. Tying both movements together—from the particular to the abstract and vice versa—we arrive at algorithms as a situated, relational perspective of analysis, where the abstract and the concrete complement each other. In the following, this perspective is illustrated regarding some key relations. Following the argument just made, this list cannot be complete. For example, a lot could be said about algorithms and commodities, algorithms and platforms, or algorithms and human subjects (Matzner, 2019).

Algorithms and material conditions

Material conditions are one important way in which algorithms are dissolved. In contrast, I aim at an analytical perspective that can integrate the manifold studies on material conditions of computing with a view on algorithms. The term algorithm here conflates issues that—from a different perspective—can be analyzed as the relation of software and algorithms, which is the matter of the “Algorithms and code (software I)” and “Algorithms and data (software II)” sections. This underlines that algorithms are conceived as research perspective that abstracts from certain things to clarify the relation to others.

Many media theorists have conceived of software or algorithms as mere representation of a lower layer: program code can be translated to machine code which can be translated to opcodes which can be translated to circuits in the CPU, and so on. This is why Galloway (2012: 60) calls the relation between the layers purely “technical.” Such views follow common stack or layer models for CPUs, networking, or storage (Bratton, 2015). In the meantime, many more material aspects have been added to the picture: mines for rare earths, electric energy generation (Starosielski and Walker, 2016), cables and networks including their geopolitical locations (Starosielski, 2015), data centers and their surrounding infrastructure (Holt and Vonderau, 2015), and more.

These studies extend the importance of material conditions. At the same time, this distributed and networked form of material conditions challenges the possibility of reducing algorithms to hardware. When a “descent” through the layers is attempted on a platform for distributed and networked computing, one is confronted with a dynamic mediation, which can only vaguely be predicted. Only when executing a program, decisions take place that locate execution in a particular data center somewhere on the globe, on a particular machine, a specific chip. At the next execution, the chip could be a different type, in a data center on another continent (Holt and Vonderau, 2015). Thus, the material is dynamically made into that which runs the algorithms by additional algorithms, on additional hardware, and so on.

This allocation where hardware and algorithms mutually constitute each other shows how such an analysis opens up a political angle: it depends on economic structures (like Amazon’s determining market position in cloud computing), geographic conditions (including increasingly the climate catastrophe) (Krapp, 2016; MacKenzie, 2018), political histories (particularly colonialism) (Starosielski, 2015), and much more. All of these factors influence how the material conditions complement the algorithm. For example, Edward Snowden has revealed that Internet routing is not just a matter of decentralized efficiency and military resilience (Dourish, 2017) but also the gathering of intelligence in a geopolitical residuum of the British Empire (Greenwald, 2014). Recently, the global demand for material complements that current algorithms necessitate has led to new ways of politicizing them, for example, for their environmental impact (Schwartz et al., 2020).

Similarly, new algorithmic applications drive material allocation and developments. In this sense, materiality is complemented by new algorithmic possibilities. For example, MacKenzie (2018) shows the material efforts by some enterprises to reduce signal runtimes for high-frequency trading. These improvements are based on physical differences between electromagnetic waves in air or fiber in the order of milliseconds. However, these material differences could only become a factor in financial business with the advent of algorithmic trading systems that could process orders fast enough so that such short timespans could make a difference in the first place. Thus, material differences matter, but in important regards via their algorithmic relation to markets, where physical differences in time become differences in profit—and milliseconds can mean millions of dollars (MacKenzie, 2018).

Paul Dourish importantly reminds us that materiality is not merely an issue of infrastructure or background conditions but also part of the daily encounter with computers. Some aspects of this will be covered in the section on interfaces, but the “materialities of information” (Dourish, 2017: 22) encompass more. For example, relational databases introduce the distinction of structure and content, where the first is rigid and the second is malleable. This difference cannot be found materially on the disk where “the database is composed entirely of bits” (Dourish, 2017: 226), but neither is it a pure matter of the presentation on the interface. Rather, it is tied to the algorithmic functionality relational databases offer (in comparison to other types) (Dourish, 2017: 222). As such, it is much more concerned with the institutional and economic concerns of the database users and manufacturers (Dourish, 2017: 232) than the processes in chips and on disks. More recent developments, such as the MapReduce framework, tie such concerns also to the distributed nature of material infrastructure (Dourish, 2017: 250).

Algorithms and code (software I)

Code is a further common way in which algorithms are dissolved: What we commonly discuss as features of algorithms are really the features of code. For example, open source and hacker communities distinguish themselves among other things by the form of code they produce (Coleman, 2012). Implicit presumptions and prejudices of the programmers (Campolo et al., 2017) or the Western/Anglophone bias in programming influence its functionality. Such studies set out to show that code is subjected to institutional, cultural, aesthetic, and economic logics that impact the algorithms it purportedly just “implements.”

Rieder (2020) approaches this issue in a more relational manner. He emphasizes that the practices of coding are important, but they do not just deal with algorithms as abstract functionality, but as a concrete functionality for particular human beings; a functionality that itself is framed by conventions, communities of practice, archives, trainings, and so on. He calls this “algorithmic techniques” and they are translated into code. This shows how a relational approach changes both relata. Algorithmic techniques live partly in practices of coding themselves, and also in research papers, tools, programming libraries, as well as organizational and economic concerns that far exceed the realm of computing—as Rieder shows using the genealogy of the Bayes classifier going back to the history of statistics beginning in the 17th century. Thus, coding itself, but already the algorithms it purportedly just “implements” (or spoils via human prejudice or other forms of influence), is an inroad of political issues.

This also concerns authors who emphasize code as commands for machines. Galloway (2006), as many others, claims, “Code essentially has no other reason of being than instructing some machine in how to act”. This verdict underestimates the contemporary quality of code, which has a deeper relation to the algorithmic. Only initially were programming languages intended as more human-readable representations of the codes for basic instructions for machines. Already in the 1970s, Backus (1978) argued that programmers still wrote with a particular machine in mind, although in the meantime they were using more comfortable “higher” programming languages. They no longer programmed for the computer but for an abstract model of it. In consequence, he argued that this model need not be based on the material computer. In a sense, he demanded that code be not an expression of what the machine should do but what the program should do. In my words, the complementary relation algorithm-code should supersede the relation algorithm-hardware.

Backus was worried about efficiency and mathematical properties. These concerns were amended by others: organization of programmers’ work, commodification of code and many more. Recent developments of “software as a service” radicalize these organizational and economic aspects. Programming increasingly is not writing code, but combining predefined services that can be best described as “composition,” “choreography,” or “orchestration” (Neubert, 2015). In today’s economy where the platform is a central form of information technology, such orchestration hinges substantially on the demand that software needs to function within the predominant “platforms-as-infrastructure” (Helmond et al., 2019). Enterprises like Google or Facebook, but also environments like those that Apple or Android devices provide, demand the use of particular application programming interfaces (APIs), compatibility with app-stores and payment methods, and much more. Coding thus is strongly guided by the socio-economics of platformization via their algorithmic operationalizations as services or libraries to be composed.

This complementary relation of code and algorithms also exists even when algorithms are treated as an object of study in themselves, because even then they rely on some form of code. Sandvig (2014) shows that there are all kinds of codes that are intended to denote algorithms but that are not directly programming languages: diagrams, the so-called “pseudo-code,” and so on. Furthermore, he argues, algorithms now have their own public relations (PR). They are illustrated, named, and branded. This happens, for example, for educational purposes but also for PR activities in the strict sense of the word, for example, when search engines announce changes to their famed algorithms (like Google’s PR). This is of course quite removed from many of the aspects that pertain to the relation of algorithms and source code. Yet, these other “codes” extend one important element from programmers to users: While many of these aspects will be discussed below regarding data and interfaces, it is important to notice that many algorithms rely on users to provide a certain form of input. Algorithmic PR, for example, in press releases about a filtering algorithm for feeds, can be used to directly address the algorithm to change users’ behavior. This will certainly be different from a mathematical presentation of the algorithm, but should not be considered as mere rhetoric or other form of (bad) representation. Important features of the algorithm, for example, if it is accepted on the market, depend on that “rhetoric” rather than its mathematical properties.

Again, it is not the case that algorithms are a pure entity and then the implementation in a programming language or “algorithmic PR” adds all the economic, social, and cultural baggage to it. All algorithms depend on some textual, graphical, diagrammatic, or other elements that exceed the implied purity of platonic takes on algorithms—albeit in a different form in each context.

Algorithms and data (software II)

The idea of data as objective representation of the world has long been criticized (Daston and Galison, 2010; L. Gitelman, 2013). However, algorithmic data analysis has been seen as a return to a new form of “positivism” (Kitchin, 2014) or the “transcendental signified” (Burkhardt, 2015). Yet, there seems to be something particular about algorithmic data analysis that is read as positivism by its critics but expressed by its advocates in terms of induction, discovery, or generation rather than positivist representation. As Cramer (2018) argues, “the paradox of Big Data is that it both affirms and denies [the] ‘interpretative nature of knowledge’” (with reference to Drucker, 2011). In his view, recent data analytics do acknowledge that data always need to be interpreted, but assume that this interpretation can now be done better by algorithms. Cramer (2018: 52) shows that such verdicts often imply a difference to a humanist subject, but can also mean to view the human as enmeshed in a systems or cybernetics based view of society, where the distinction between a human and an algorithmic interpretation of data becomes blurry. Even if opposing, both of these views presuppose that interpretations by humans and algorithms are comparable as interpretations of the same data that are equally accessible to all kinds of human and algorithmic processes. Such views both over- and understate the roles of algorithms. They overstate it because algorithmic interpretation is usually only a part of a practice that renders data meaningful or actionable, and it understates it because it does not see how much the algorithm is already in the data. Again, my analysis aims for a mutual complement of both these views.

Differences in human and algorithmic interpretations of data are evoked regularly, for example, by the diagnosis that machine learning (ML) is susceptible to “clever Hans behavior.” Like the infamous calculating horse, ML processes follow cues that are totally unrelated to the intended training task, for example, labels in images rather than their content (Lapuschkin et al., 2019). This is even more problematic if these cues are not unrelated but related via discriminatory assumptions. For example, an infamous study mistook what is probably a detector for certain hairstyles and grooming as a visual predictor of homosexuality (Hirschman, 2017). Such illustrations of the knee-jerk “stupidity” (Cheng and Steyerl, 2017) of AI have led to an increased attention of the importance of training data curation and processing—done by humans (Campolo et al., 2017; Casilli, 2017; Mühlhoff, 2019). These factors certainly influence the performance of an algorithm, and in this sense human interpretations of data complement the algorithm.

At the same time, this research illustrates that data become data for specific algorithms through intricate and complex practices. Lee et al. (2019) provide some pertinent illustrations. Such practices differentiate data for algorithms from data for other forms of “interpretation.” For example, a bias in a data set is not just there—rather it is there in relation to a particular algorithm—and sometimes, it is there even because of the processes that are necessary to make the data for the algorithm. (The aforementioned “predictor” of homosexuality relied on social media images by self-proclaimed homosexual persons to get “labelled” training data.) Consequently, the common talk of biased data runs the risk of implicitly reducing algorithms to a neutral channel that transports whatever socio-political structure is fed into them.

Vice versa, the quality of such training data can only be ascertained by evaluating the performance of the ML system that has been trained using those data. In this sense, the algorithm complements the training data because the latter only gets its meaning as training data—and as, for example, biased—through its algorithmic processing. This dependence can become problematic for critical analysis. For example, a study that intended to show a racial bias in gender classification algorithms had to follow the logic of the algorithm in determining a fixed gender binary based on appearance in the training data to determine a racial bias relative to that “given” categorization (Buolamwini and Gebru, 2018). Thus, the focus on bias in data fails to do justice to the intersectional perspective the article promises in its title because the notion of bias depends on this algorithmic classification of the data.

This problem illustrates that the complementarity of data and algorithms has to be extended beyond the small scale of the training feedback loop. What the many studies on data point to is the social, economic, and cultural contexts that exist on both “ends” of this loop. From this larger perspective, algorithmic data processing is not just a co-constitutive process of making data and making the meaning of that data. It is also a process that creates new relations between contexts through data. From that point of view, important elements of the meaning of data do not derive from the particular analytic capability of an algorithm, but result from the connection to certain social norms that it creates (Matzner, 2016). Regarding the example of gender classification, this would mean to leave the representational frame of bias toward a relational view, which, as has been discussed in the “Algorithms as complementary relations” section, connects to particular social differences, for example, in showing how producing (rather than classifying) a gender binary always already implies a racial positioning (Crenshaw, 1995; hooks, 2010).

It is this feature of algorithmic data processing that links social practices or contexts that stand in the center of many contemporary discussions of the social impact of information technology. Maybe most prominently, algorithmic data processing can invest almost any activity with economic logics of platforms (Srnicek, 2016; Zuboff, 2019). The specific combination, where the data that are created by a search algorithm can be accessed by an advertising algorithm, relates—through an interplay of data and algorithms—a purely quotidian and often private activity like typing a term into Google’s interface with an extensive form of capitalism or commodification.

In sum, algorithms are not just carriers or media of contextuality like biases in data; they themselves do something to the data and their meaning. Vice versa, the data are “biased” toward a particular algorithm through complex processes of data creation. On a larger scale, the meaning of data also depends on the social contexts in which they emerge and how they are algorithmically related to the socio-economic context of application such as platformization.

Algorithms and interfaces

Interfaces can mean several things (Fuller, 2008: 149): connections between hardware and users (e.g. a mouse), between hardware and hardware (e.g. a network switch), between software and hardware (e.g. device drivers), or between different pieces of software (e.g. APIs). Thus, many aspects of the relation algorithm-interface really fall apart into two relations: algorithm-software and algorithm-hardware. However, particularly user interfaces are an important object of research and of designing, using, and selling information technology where social and material factors intersect in a complex fashion. Thus, it makes more sense to spell out the consequences of an algorithmic research perspective for user interfaces directly.

Influential theories of interfaces operate on a double logic of suspicion: either the interface hides the true functionality (e.g. it subtracts or dissimulates) or it performatively produces something in addition to the algorithmic functionality, like dark patterns or affective design (e.g. the interface adds or imposes). This suspicion can be found in the concept of “depresentation” (van den Boomen, 2014) or inquiries into interfaces using questions of representation and indexicality (Distelmeyer, 2018). Galloway, although he describes the interface as an “effect,” still ties this to ideology and thus also remains within that frame of suspicion. In contrast, Chun (2004) argues that for all the necessary focus on calculation and code, visual knowledge persists. Distelmeyer’s (2018) use of the concept of “operational images” similarly conceives of the interface as an inherent part of the functionality of information technology.

Giving the relation algorithms–interfaces consideration of its own extends this perspective. In fact, it is often history of use that suggests that different versions of the interface can be seen as addition or subtraction to the algorithmic or machinic “reality.” For example, command lines have been superseded by graphical user interfaces that can appear as much more removed from the actual processes. However, a list of files in a black command-line window and a series of icons in a graphic user interface window are both presentations, and neither is per se closer to the structure of the file on a disk, just different. Neither, for example, shows the distribution of a single file over several blocks, that is, the form in which it is stored on a hard disk (Kirschenbaum, 2008).

However, it would be quite misleading to say that a file “really” is this sequence of blocks—or even a pattern of differently magnetized areas on a disk. To the contrary, for users a computer file is in many regards an algorithmically created presentation on an interface: a text document, an mp3 file, and so on. In more detail, we have a structure where two groups of algorithms intertwine: one assembles the file from the bits on the disk, complemented by storage hardware, and one presents its contents, complemented by graphics hardware. Both these processes, however, cannot be seen separately and in turn complement each other, because the file is never totally assembled anywhere but immediately falls apart into memory cells, processor registers, graphics processors, and so on. In this sense, the file only “exists” through a sustained execution of that entire process. Thus, the “truth” is neither “behind” nor “on” the interface but in a series of algorithmic processes that link what happens “on” the interface to other elements.

This complementary character of the interface-algorithm relation recontextualizes important differences. Let us consider again the example of the list of files in a command window as compared to a series of icons. There are differences within the level of the interface itself: They belong to different logics of organizing or demanding user input (keyboard and mouse), they address different practices and cultural spaces of meaning (e.g. representing a folder by a little image of a paper folder or by a special character), they work in different registers of accessibility, and so on. Thus, the functionality we often ascribe to an algorithm, for example, sorting a list of files by name, is in important regards complemented by the interface because only on the interface does a list of files with filenames as alphanumeric characters for which sorting makes sense in the first place exist. On the other hand, it is no pure function of the interface, for example, because one of several possible sorting logics is involved that is not present on the interface.

This aspect gains particular significance in the context of ML and data-driven applications. The many elements for feedback that define social media, CAPTCHAs, so-called “dark patterns,” and so on, all are parts of the user interface that activate users to generate the specific forms of data necessary for ML algorithms (Casilli, 2017; Mühlhoff, 2019; Zuboff, 2019). In important parts, the algorithms that create timelines, feeds, or targeted ads based on ML run “on the interface” (Schulz and Matzner, 2020). Thus, interfaces as an integral part of the affordances of platforms (Bucher and Helmond, 2017) are as essential in creating the commodification through data tracking and “prediction products” (Zuboff, 2019) as the analytic capabilities of ML.

However, the interface cannot be just included into the algorithm in a neo-cybernetic fashion where the user is just an input–output mechanism as well, coupled to the platform via the interface. Although such views are usually expressed with a political angle in mind (Rouvroy, 2013), they conflate the complex politics of human–machine interaction. In fact, the structure of mutual complementarity is reproduced on the level of users’ practices. Taina Bucher (2017) has shown that an idea of an algorithm that structures a practice is important not only for programmers but also for users. Even if users do not know the precise algorithm the way programmers do, many know that their activity is algorithmically processed and adapt to it. In light of the discussion so far, this “algorithmic imaginary” (Bucher, 2017) must not be understood as a more or less correct grasp of what “really” goes on. This imaginary influences users’ actions and thus has real effects—effects that are creating and modulating the inputs which in turn influence the data-driven algorithms. While the platforms exert enormous power on the users, technically, economically, and socially, they still do not determine their behavior. There is a space for appropriating their affordances differently, may be even subversively. Thus, it is no wonder that platforms no longer just program algorithms, but also try to address this imaginary, in part on the interface, in part via “algorithmic PR” that has been discussed in the “Algorithms and code (software I)” section. Social media platforms and their users are just one example of this level of complementarity; the complex interplay of search engine optimization and search providers is another case in point.

Conclusion

What is presented in this article provides an alternative to a common motive in engaging critically with algorithms. To that aim, often an “opposite” of algorithms is constructed. This can be done using algorithms as abstractions—frequently in a way that turns algorithms into a qualifier: algorithmic judgment, to pick just one example, is opposed to other forms of judgment. The flipside of that theoretical move is finding essential qualities of the algorithmic. An alternative of an opposite of algorithms is dissolving them into their conditions or constituents. What has been proposed here is to engage with algorithms specifically in their relations—or more precisely as structures of complementary abstraction. This perspective neither looks for an essential difference to algorithms, nor does it give up the perspective on algorithms for the sake of the concrete, detailed, particular.

In consequence, a critical account of algorithms need not find the definitive definition of algorithms nor the definitive place where the features of information technology emerge. To remain within in the vocabulary here developed, it is not about deciding for the right side of a relation that is better understood as complementarity. A critical account of algorithms, in consequence, should not play off the perspectives from engineering or programming against perspectives from social sciences or cultural studies. It should not play off what happens “in the black box” against users’ perspectives. Rather, it means to take all these perspectives in their relatedness seriously as bringing to light a particular aspect that cannot be grasped by one perspective alone.

By employing this relational perspective, it also continues the critical thrust of discussions on algorithmic judgment, algorithmic governmentality, and so on while maintaining the importance of the constitutive relata of algorithms: materiality, code, data, interfaces, and more. At the same time, a perspective on algorithms as complementary abstractions orients and connects the many detailed studies on materiality of information technology, on data, on code, on interfaces to wider socio-political developments. A critical perspective on algorithms, then, is situated within different configurations of these complementary relations—not on either side of them.