Abstract
Artificial Intelligence (AI) in the form of different machine learning models is applied to Big Data as a way to turn data into valuable knowledge. The rhetoric is that ensuing predictions work well—with a high degree of autonomy and automation. We argue that we need to analyze the process of applying machine learning in depth and highlight at what point human knowledge production takes place in seemingly autonomous work. This article reintroduces classification theory as an important framework for understanding such seemingly invisible knowledge production in the machine learning development and design processes. We suggest a framework for studying such classification closely tied to different steps in the work process and exemplify the framework on two experiments with machine learning applied to Facebook data from one of our labs. By doing so we demonstrate ways in which classification and potential discrimination take place in even seemingly unsupervised and autonomous models. Moving away from concepts of non-supervision and autonomy enable us to understand the underlying classificatory dispositifs in the work process and that this form of analysis constitutes a first step towards governance of artificial intelligence.
This article is a part of special theme on Knowledge Production. To see a full list of all articles in this special theme, please click here: http://journals.sagepub.com/page/bds/collections/knowledge-production.
There is a long tradition of equating knowledge with classification—in the physical sciences, the classification of subatomic particles is a core endeavor; in chemistry, Mendeleev's table was a fundamental breakthrough which gave us classes of elements; in botany and biology, the Linnean classification system is still at the root of scientific work and the central pursuit of cladistics is classification. The argument that Big Data can do without large scale classificatory work has been made at opposite ends of the spectrum by editor of Wired Chris Anderson, and social philosopher Bruno Latour (Anderson, 2008; Bowker, 2014 for critique; Latour, 2002; Shirky, 2005).
In their magisterial study about the use of “Big Data”, Lehr and Ohm among others (Barocas and Selbst, 2016; Diakopoulos and Koliska, 2017; Lehr and Ohm, 2017) point to the profusion of layers at which social and political factors can enter into the deployment of Big Data to “automatically” and “dynamically” assign (in our reading) classes on the fly: for instance data collection; data cleaning; data partitioning; model selection; model training (including tuning and assessment); and model deployment. In this article, we shall consider the most relevant layers for understanding classifications as they arise in artificial intelligence (AI) and machine learning with the aim of making visible knowledge production.
Data collection is clearly one area in which data classification can occur. We may find the number of tweets per day (some 500 million in 2018—https://blog.hootsuite.com/twitter-statistics/, accessed 27 November 2018) or the number of Facebook users (2.3 billion active users in 2018—https://zephoria.com/top-15-valuable-facebook-statistics/, accessed 27 November 2018) staggering; however we always need to remember that these numbers do not of themselves provide representativity of the total population outside social media (Bechmann and Vahlstrup, 2015; Lomborg and Bechmann, 2014). What we get reflected back from large scale studies of these platforms is not society as it is, but a society that is classified immediately into users (of interest, accessible) and non-users (not of interest, inaccessible).
Another step at which classification work gets done is data cleaning. Walford (2014) has written beautifully about the work of data cleaning in the canopy of the Brazilian rain forest. Certain results from streaming sensors of the environment, which we may think of as objective representations of reality, get routinely rejected from the databases being built. If a temperature reading or a window reading is outside of the permitted range, it will be excluded in the scrubbing process. Thus anomalies are weeded out before they can be spotted—the world has a classified set of behaviors that can only exist within certain parameters.
Model training is a third step of classificatory work. Jaton (2017) has shown in the case of a new machine learning algorithm for detecting complex photographs (with more than one object in focus) that the innovators had to try to establish their training set in order to get their results accepted. The problem was that their model did not work as well as the finely tuned models when looking at a single focus image. Ultimately, their work was rejected because of this flaw. There is an argument, then that the scientific and organizational authority to create a training set was a core part of the process. And again, only recognition algorithms that worked optimally over a certain class of objects were considered valid.
We recently have witnessed how the largest communication platform in the world Facebook has weaponized political propaganda. This became especially clear in the case of Cambridge Analytica. The company collected data on millions of users through Facebook third party apps to understand the correlation between psychological profiles and platform behavior such as “like” patterns (Kosinski et al., 2013). This data inferred knowledge allowed the company to target more precisely voters with the specific message appealing to particular geo-located, profiles and potentially win over votes (Cadwalladr, 2017). According to John Dewey (1927), democracy can only be performed if different groups interact flexibly and fully in connection with other groups through “free” and “open” communication. Political micro-targeting brings into question whether such “open” communication is taking place. The increasing entrenchment of privately owned media ownership into an international oligopoly again questions “free” and “open” communication—particularly since we have limited access to the logics and structures on which social media are built.
One such logic is the use of AI in the algorithms of social media and how the reasoning of such machines on top of these structural problems can potentially create problems of visibility, redlining and other discrimination such as targeting, favoring and normalizing some people over others (Caliskan et al., 2017; Citron and Pasquale, 2014; Eubanks, 2017; Howard, 2005; Levin, 2016; Sweeney, 2013). AI and machine learning are concepts often used as synonyms to describe widely used yet controversial computational models employed to cluster and make sense of data to inform and predict actions in the Big Data era (see also Russel and Norvig, 2010).
Despite the as yet imperfect state of these models for interpreting and predicting action from data, they have an increasingly significant influence on decisions made in an increasingly data-driven society. In line with critical algorithmic scholars (Ananny and Crawford, 2018; Boyd and Crawford, 2012; Cheney-Lippold, 2017; Citron and Pasquale, 2014; Elish and Boyd, 2018; O'Neil, 2016; Sandvig et al., 2016), we argue theoretically and show through empirical case studies that such models and associated classification dispositifs are central objects of study in order to provide a critical yet informed discussion on the knowledge production of AI. This article aims to provide a framework for analyzing social, cultural, and political classification dispositifs of supervised and unsupervised machine learning models as models that are seemingly autonomous.
We will theoretically discuss classification as a central knowledge producing concept and how it has been applied to AI in general. This will be followed by examples on the use of two different models with different degrees of classification—topic modelling with text2vec (unsupervised) and deep neural network picture pattern recognition with inception v.3 (supervised) applied to Facebook data in one of our labs. The purpose is to detail at what point in the work process of applying AI classification, as defined theoretically in the previous sections, takes place. The article contributes to the existing literature on knowledge production in AI and potential problems with accountability (Ananny and Crawford, 2018; Barocas and Selbst, 2016; Burrell, 2016; Dwork, 2006; Hardt, 2011; Kroll et al., 2017; Lehr and Ohm, 2017) by focusing even more on the human nexus in the work process and at what point classification is carried out that can result in counter-productive outputs for democratic societies and their shared human values.
AI, machine learning, and the role of classification
Theories of AI have historically distinguished between strong/general and weak/narrow AI (Searle, 1980; Slezak, 1989). Searle (1980) describes weak AI as “a powerful tool” that, for instance, allows for us to examine larger amounts of data in a more rigorous and precise way than what we as human brains would be able to. An example of such processing is using Baysian methods to create data inferred classifiers (Rieder, 2017). With the current fascination of Big Data, such uses of AI are widespread in all areas of the digital layer of everyday life through predictions that lead to actions, be it within robotics, communication optimization and manipulation, or behavior adjustments.
However, weak AI—as the most widespread use of AI—is best understood in the historical context in which strong AI as an “imitation game” (Turing, 1950) has played a major role as driver for AI research and developments, over the past 70 years and still present today within the development of humanoids (Kanda et al., 2004). In comparison to weak AI, strong AI has the goal of imitating human behavior and communication. Strong AI is defined by Searle as a computer mind with intentionality acting as a human mind that is able to “understand and have other cognitive states” (p. 417). Similar visions can be traced back in time (Buchanan, 2005), but newer vision is often connected to the Turing test (1950), where Turing asks “can a machine think?”. For a program to pass the test, the human must not be able to tell in a dialogue that the person is speaking to a computer (pretending to be a woman). In this vision, humans are used as a benchmark to measure computational success in a simulation optic. Many examples demonstrate that it is difficult to establish a structural classification scheme or an artificial understanding of the implicit rules and tacit knowledge (Polanyi, 1966) socially inferred from a specific society. With a growing global media arena that complexity only increases. Inspired by the Turing test, many developers have tried building software bots that chat with human/bot peers within communities such as Twitter but have failed to encode classifiers of such tacit knowledge. Using AI on social media will, on the surface, seem like strong AI, because the machine would have to decode logics of human communication and behavior and adapt to changes in this.
An example of such an attempt was the release of the Twitter chatbot Tay powered by Microsoft in 2016 that turned into a female hating, Nazi sympathizer and had to be shut down only 16 hours after release. Why did the experiment fail? Alba (2016) suggests that the feedback loop is problematic because the AI acted on top of the input it was provided with. So, if the input data intentionally or unintentionally display unacceptable classes of behavior or social values, the AI will mirror these unless there is an intervention in the programming phase, e.g. hardcoded adjusted thresholds or black listing outcome variables. As digital social scientists, we would add that it could also play a role that the logics changed without the social chat bot noticing the social cues. From trying to mirror a traditional Twitter conversation, the chat turned into a game where the teenagers tried to make the social chat bot deliberately display “unacceptable” social values, triggering classifiers that were not acceptable in the wider community. The debate about simulating human thought, and of originality and intentionality (strong AI) versus human enhancing and computing power to process enormous amounts of data (weak AI) is important to the discussion on classification. We will argue that it is precisely the quality of the classifiers that result in the perceived success or failure for a given social context. As the discussion on weak and strong AI suggests, classifiers are present in both cases. In weak AI applications, classifications are generated from a lot of training data and learning iterations. In the strong AI vision they are meant to imitate humans' way of approaching the world, training and learning being a significant part of this, but also being able to recognize social cues in shifting contexts.
Classification theory, social categories, and claimed lack thereof
Classification is a natural part of human reasoning and is important in order to understand the world around us (Foucault, 1971); as the boxes that we structure the world around and in. However, despite such boxes being dynamic, these widely accepted boxes only allow us to see the world from certain cultural and historical standards, often exclude those at the margins and the “residual categories” that do not fit into our system or negations of the standardized classifiers: “deciding what will be visible and invisible in the systems” (Bowker and Star, 1999: 44).
Legally protected classes of people may well be covered in classification systems, however computer systems are often blind to other kinds of potential discrimination: for instance, towards left-handed people, people with allergies (Star, 1990), people who are predicted to be depressed or pregnant in the near future, or receptive to political campaign messages (Tufekci, 2014). In the contemporary media landscape where platforms create increasingly international media fora transgressing different cultures and cultural classification systems, an American standard would also be blind towards Indian, Egyptian or Korean classes inferred from the specific history and culture in these societies. Classes are also ubiquitous and can create “cumulative mess” (DiPrete and Eirich, 2006; Strauss et al., 1985). When we approach classes ecologically, we find that individuals, for instance, can be classified as many things at the same time, and some of the classes can (from the standards we work with) seem opposite or working against each other, leading to confusion for the algorithmic outcome. Such contradictory classes may be due to classes deriving from a certain context and not taking into consideration that the person can interact differently in different contexts as we saw with the chatbot Tay; that the classes are layered, textured, and tangled (Strauss et al., 1985), and that “one size does NOT fit all!” (Gasser, 1986). We need to work with parallel or multiple representational forms (Bowker and Star, 1999).
Statistics “is immanently a science of classification” (Farr, 1985: 252) and classification is another word for generalization. AI builds on statistics and other mathematical principles, and the main interest for platforms is often to create personalization and amplification to maintain and monetize user attention, presenting users with the norm trained on this specific person's own data traces paired with, for instance, relational data to find the right person to target with the right posts or ads. “Ground Truth” (Jaton, 2017) forms the apodictic basis that is used to train the learning algorithm to improve the prediction that is otherwise based on the historical behavior of the user (posts, likes, shares, comments, group memberships), similar users and the user's network as a prerequisite for what the person wants to do or to see in the future news feed. Still, such ground truths create problems if we turn to the classification theory of ubiquitous classification and cumulative mess; the basic assumption behind such ground truth being that; (1) there is a ground truth (disregarding confirmation bias); (2) scores can indicate whether ground truth has been reached; (3) we will repeat our past actions and preferences in the (as it has come to be) broad context of social media, ranging from different contextual uses of social media as in itself a ubiquitous service transgressing locations and incentives for use; (4) future predictions will match the ground truth registered for past interactions without model and system developers influencing the users and user behavior/incentives in question.
As pointed out in the late 1990s, if we work with too few categories, the information is not useful, and if we work with too many categories, the result will be increased bias, or randomness, on the part of those filling out the form: the Goldilocks zone is well described by Ashby in his “law of requisite variety” (1956). However, in the late 2010s, the information is not filled out in the system manually. This does not reduce the problems of too many categories; in fact, we argue that we see increased biases and randomness in actions built on top of multi-categorical processing. Categories are not a priori constructed, but highly context sensitive, following the cultural context of the person categorizing (Bowker and Star, 1999: 107) as well as being subjective, case- and site-specific as already suggested by Roth in the 1960s (1966). We assume that such qualitative findings in the 1960s also apply to the labelling industries in the 2010s (e.g. Mechanical Turk) that are widely used to identify features and other labels in training data; but what are the political consequences of such subjective processes and can system designers and developers apply AI without classification?
One site for the alleged disappearance of classification is the programing procedure of assigning “dynamic proxy classes”. In object-oriented terms, this means that if you have a given classification built in (say humans as an example of species) then you can on the fly extend the category (to include, say, Neanderthals, as some would argue) by assigning Neanderthals automatically to the proxy class for humans—so that the object “Neanderthal” would act like the object “human”. Through this technique, you can generate substantial drift from an initial category set. Significantly, however, we are still talking about classification work all the way down—the only issue is how visible and how a priori that work is. Instead of discussing lack of classification we argue that we need to account for how classes and social categorization arise in the design process as deliberate and unintentional consequences of decisions made.
Two concepts within AI deserve to be outlined and discussed in continuation of the weak and strong AI discussion: the concepts of supervised and unsupervised learning (Alpaydin, 2016). The concepts describe algorithms that work with classifiers or labels to generate predefined outputs (supervised) or algorithms that do not have predefined outputs (unsupervised). In unsupervised learning data is placed in clusters or other pattern recognition outputs according to the structure in the data, here conceptualized as inductive classifiers or data inferred classification. “Inductive inference” was debated critically by, for instance, Slezak in the late 80s: "these programs constitute ‘pure’ or socially uncontaminated instances of inductive inference, and are capable of autonomously deriving classical scientific laws from the raw observational data” (1989: 563). Slezak, among others, questions how this computational method can distinguish “mere contingent co-occurrence from causal connection” (p. 565). Like Searle, Slezak was also skeptical about the ability to develop strong programs. Through a critique of Bloor's strong program that highlights social contexts as important to inferences and causal explanations (the sociological turn), he argues that such explanations would lead to insufficient explanations of the context following the critique of radical relativism, and instead argues for a turn towards social interests that are best outlined through cognitive science. The focus of cognitive science, however, has been on human brain processing and rational decisions (Hayles, 2017). What is rational to computers is not necessarily rational to humans because depiction of the situation and context differs, and the definition of the task may be more narrowly interpreted by algorithms, not taking into consideration an ecological approach to the consequences on other tasks such as societal inclusion and social cohesion—is inclusion a rational choice (e.g. in the case of Cambridge Analytica)? In this way, algorithmic ecology plays a role in governing AI as will be illustrated in the case studies of the next section.
Hidden layers of knowledge production in AI
As we have pointed out, there are several problems with data inferred classification. Despite the ability to modernize existing classes, they can be highly spurious and self-fulfilling. Furthermore, they can be difficult to backtrack logically, and the consequences of acting on top of such classes can be highly problematic if classes prove to follow a different social code than human society allows (Caliskan et al., 2017; Elish and Boyd, 2018). The lack of algorithmic ecology makes machine learning task oriented and not necessarily aware of the larger consequences of a closed decision process and the influence on the larger socio-economic and political climate.
Steps outlined in the process of applying AI.
In the following, we will outline the five phases and the associated classification questions and potential manipulative/discriminatory processes in two case studies of famous AI models on Facebook data (Bechmann, 2019). On social media classification happens on people or groups of people with the purpose of profiling and subsequently targeting them with advertisement and tailored content (as was the case with Cambridge Analytica). We test two famous algorithms in the setting of such profiling. First, we are interested in identifying the topic of the content people are exposed to in the Facebook News Feed, and secondly, we are interested in predicting uploader gender from a random picture users will upload (not portrait). Both case studies in this article serve as illustrative cases in the setting of classification and not as empirical findings on its own right (Bechmann, 2018; Bechmann and Nielbo, 2018).
Classification in LDA model application processes
The Latent Dirichlet Allocation (LDA) model Text2vec (text2vec.org) is a (seemingly) unsupervised model and a standard model for semantic analysis in textual data. As such we are interested in accounting for how the model can be applied to understand what content people are exposed to in the Facebook news feed to subsequently tailor messages that go viral in a certain population (Figure 1).
A case study of human choices and potential discrimination made through classification when working with LDA models.
As the case shows, a seemingly unsupervised model becomes extremely supervised due to classification work such as setting number of topics, cleaning data in a particular way with an a priori understanding of “meaningful” clusters and interpreting clusters with parent classes manually. Due to the high level of human control in natural language processing models (NLP) already marginalized groups are potentially underrepresented (see also Duarte et al., 2018).
Classification in CNN models application processes
Convolutional Neural Networks (CNNs) are supervised models designed to predict identified outcome classes (here female and male uploader), the seemingly opposite of LDAs and thus interesting to include here. CNNs are standard models for image recognition/processing in Big Data (Bechmann, 2017; Burrell, 2016; Shelhamer et al., 2017). As a social media profiling tool we are interested in training the model to predict uploader gender, feeding the model with only random Facebook pictures labeled with uploader gender. What would the accuracy be with such sparse data and what do the falsely categorized pictures tell us about the classification work of such models? (Figure 2).
A case study of human choices and potential discrimination made through classification when working with convolutional neural networks.
Classification in the applied AI and machine learning process.
Discussion and conclusion: Classification and AI governance
Our analytical framework has shown how seemingly mundane classification processes carry potential discriminatory consequences in a type of hyper-discrimination that combine “zero sense discrimination” (between quantities that might appear similar, being able to spell out a meaningful difference between apples, oranges and pears) and “discriminating against” by utilizing persistent patterns of social injustice (masking, redlining, data inferred biases). If returning to the introductory case of Cambridge Analytica such hyper-discrimination is also important to take into consideration here. When we are discriminated into smaller and smaller groups, experimented on through A/B testing, and then acted upon down to a very granular level such hyper-discrimination becomes highly political. Either because such targeting might become a proxy (Barocas and Selbst, 2016; Kroll et al., 2017) for discrimination against race and education level, or for people highly susceptible to manipulation that fall out of our democratic defined protected classes, yet undermines the integrity of the democratic process by providing an unequal approach to elections.
But how do we govern these fundamental issues? The speed by which processing takes place, often in combination with A/B testing, makes it difficult to govern these algorithms and services favor effective and fast models and processes over standards, balancing tests and documentation (see also Kroll et al., 2017). Furthermore, algorithms are protected by proprietary rights. In this article, we have suggested that we need to focus less solely on access to the models and algorithms as technical constructs by themselves and more on documenting the human choices made in the work process surrounding AI in order to act sustainable in relation to shared democratic values, avoiding masking, redlining, discriminating biases, and voter discrimination specifically. Such deliberate discrimination is already regulated against but needs to be governed effectively in the application of AI models. This supports the work of Dwork (2006) and Hardt (2011) that distinguishes discrimination as “blatant explicit discrimination”, “discrimination based on a redundant coding”, and “redlining”.
Many critical algorithmic scholars (Bostrom, 2017; Gillespie, 2014; Rogers, 2009; Sandvig et al., 2016) have discussed the issue of algorithms being opaque, arguing that we need to govern the algorithm through a larger degree of transparency so that we know how it sorts information. Kroll et al. disagree, suggesting that it is not a solution to make the rules or source code open and transparent (supported by Ananny and Crawford, 2018). Testing them through audits (Sandvig et al., 2014) and simple random tests would only create ex post analysis, but algorithms, environments, and populations change too quickly between tests in order to use such tests as a benchmark for measuring discrimination. They criticize black-box evaluation of systems as the least powerful of available methods to understand algorithmic behavior, but instead suggest that the algorithm ex ante is designed for governance and accountability in a type of what could be labeled value-accountability-by-design.
We believe that the analytical framework suggested in this article with a focus on the human nexus in knowledge production could also be a good starting point for governing against counter-productive democratic values. In order to make an ex ante value-accountability-by-design policy, we need for instance to re-inscribe anti-discrimination classes (see e.g. European Union, 2000; U.S. Equal Employment Opportunity Commission, 1964) beforehand so that it is possible to adjust for them and then bootstrap new categories to enable the machine to test for discrimination against these potential new protected categories. When the machine is “race-blind”, “gender-blind” or “income-blind”, and the categories deliberately omitted from the processing such as in the Cambridge Analytica case of unsupervised learning, any discrimination against such categories or proxies for such classes cannot be adjusted for in the process (for tests using categories see Kim et al., 2018; Kusner et al., 2017). This is especially important in A/B testing as the fundamental social test of our time in the data-driven society, and such tests are practically ungoverned at the moment (Leese, 2014).
In general, the role of amplification in the algorithms needs to have more attention in the policy work as these principles are nearly entirely unregulated at the moment. Encouraging algorithmic workers (in a broad understanding) through both education and regulation to test for discrimination on, for instance, anti-discriminatory classes moves us away from a populistic programmed consensus truth where discriminatory progressiveness is given towards questions of programmed anti-discrimination as a standard for inclusion. In the race to pursue the “right” and most effective solutions, we need a fair game in which protected classes are in fact protected against correlations of the best predict variable(s). We also need to use AI as a way to protect against potential new discrimination and new, as yet unknown, rising suppressive classes.
Footnotes
Acknowledgments
The authors would like to thank Userneeds, participants who shared their data for the purpose of research, DATALAB developers and assistants Peter Vahlstrup, Ross Kristensen-McLachlan, Henrik Pedersen and Anne Henriksen, Judith Gregory for comments on earlier version of the paper, special issue editors, and anonymous reviewers for their insightful suggestions.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Aarhus University Research Foundation (grant number AUFF-E-2015-FLS-8-55) and part of the research was performed while A Bechmann visited Evoke Lab at University of California Irvine.