Article Commentary: The two kinds of artificial intelligence,or how not to confuse objects and subjects

The following essay is a stylistic experiment for Interdisciplinary Science Reviews, reflecting on a personal research agenda and trajectory, in relation to the disciplines that the author has engaged with. Interdisciplinary enquiry often arises from the idiosyncratic experiences and decisions of an individual serendipitously following curiosity, alongside the practical contingencies that shape anybody's career. Such a reflection, if spanning multiple disciplines through the perspective of one person, cannot possibly be comprehensive, and will certainly expose the gaps in knowledge and loss of rigour that could have been corrected within a single discipline. The intention in presenting such a personal agenda is not to be definitive, but rather to open up discussion by pulling on the loose threads at the edges of discipline. The primary goal of the experiment is to unsettle established disciplinary perspectives, even where the same questions could have been addressed more authoritatively in another field. 1

Introduction

Artificial Intelligence (AI) is a research pursuit that appears ideally suited to Interdisciplinary Science Reviews, given the ambition of AI to advance both the scientific study of humanity, and of computational machines as a ‘science of the artificial’ (Simon 1969). The first of these aspires to self-knowledge (that is, the study of ourselves as subjects), while the second has become associated with mechanical and quantified objectivity (Blackwell 2022). These alternative perspectives are not easily reconciled, and in this essay, I will argue that AI research has not paid sufficient attention to the implications of these distinct disciplinary perspectives. In particular, I will argue that many popular commentators on AI, including journalists, investors and philosophers, have spent insufficient time understanding the mechanical ambitions of what AI engineers actually create. This argument is complementary to that advanced by Philip Agre, in his own Notes on Trying to Reform AI (1997), where he observed that AI researchers were engineers engaged in covert philosophy. Agre proposed that those philosophical ambitions should become more overt, advocating a critical technical practice. The consequence of insufficient understanding across disciplines, as criticized by both Agre and myself, has been to muddle together subjective and objective questions.

My own interdisciplinary perspective on AI has arisen idiosyncratically, through accidents of personal history. My first degree was in electrical engineering, specializing in control theory, the mathematical analysis of closed-loop feedback in servomechanisms, that is also described as ‘cybernetics’. I completed a second undergraduate major in comparative religion, then a master's by research in computer science, in a sub-field of AI fairly closely related to that of Agre (c.f. Agre 1988; Blackwell 1988). Eventually, after some years of employment as an industrial control systems engineer and then AI engineer, I returned to full-time study for a PhD in psychology. But by the time I studied psychology, I had already internalized a view of cybernetic servomechanisms, and of AI, as purely engineering techniques, wholly distinct from the cultural, even religious, question of self-knowledge – what it means to be human. The pragmatic experiences I had with each of these questions meant that I had very little patience for cognitive psychologists who used engineering-style ‘black-box’ models from AI or cybernetics as if they could be adequate accounts of the human condition.

The remainder of this article, while presumably shaped by the peculiar route that I have taken, seems in my own mind to be engineering common sense, rather than arcane interdisciplinary inquiry. However, the conclusions that I draw appear to be seriously at odds with the most influential public AI commentators. I hope that other AI engineers would find nothing surprising in what I say below, although it is odd that so little public commentary has included similar observations.

To be brief, I am going to argue that the essence of ‘AI’ as an imaginary future technology, or even semi-magical marketing solution to present-day problems, represents a misunderstanding of the technical operation of these systems. In particular, I am going to argue that the misunderstanding comes precisely from confusing ideas about cybernetics, ideas about machine learning, ideas about psychology, and ideas about what it means to be human. Having been educated (wholly by accident of personal history, rather than any admirable foresight) in the disciplines most relevant to each of these questions, I hope that I am in a useful position to unpick the confusions.

If it were not for these confusions, I believe that the field of AI would not have its current magical and imaginary character. In reality, the technical operations of many systems marketed with reference to science fiction robots and oracles are quite mundane, in ways that can be explained by reference to the appropriate discipline. In order to make this argument accessible to non-technical readers, I include some brief explanation of the fundamental operation of two kinds of AI: servomechanisms, which I will characterize as ‘objective’ AI, and the use of machine learning algorithms to capture human behaviour, which I will characterize as ‘subjective’.

Servomechanisms

The principles that I describe as the objective kind of artificial intelligence have been understood by engineers for centuries – a servomechanism is any kind of device that ‘observes’ the world in some way, ‘acts’ on the world, and can ‘decide’ to ‘behave’ in different ways as determined by what it observes. Familiar examples include clocks and steam engines that are automatically regulated to move at a particular speed, thermostats that regulate the temperature in one's house by turning heaters on and off, and automobile cruise control that can accelerate to move at a certain speed, or stay at a certain distance from the car in front or the side of the road. In these familiar cases, there is no mystery about the nature of the mechanical action, produced by wheels, engines, or boilers. The ‘observation’ capacity of servomechanisms is only slightly more intriguing, since it relies on mechanical or electronic devices that respond to speed, distance, temperature and so on.

Rather than mechanical actions or observations, the apparent intelligence of servomechanisms is in the way these devices change their behaviour. The mathematical foundations of control theory, involving technical terms such as state, stability, gain and damping, are not taught in high schools, and are thus not available in everyday conversation about the operation of these everyday devices. Instead, most people resort to anthropomorphic metaphor, explaining the behaviour of a heating or cruise control system in terms of how it ‘thinks’ it should act, what it has ‘remembered’, what we have ‘asked’ it to do, and so on. In the case of the simplest devices, the internal mechanism is so trivial that a curious child could, for example, open up a mechanical thermostat to see that its ‘memory’ is simply a metal strip bending backward and forward as it heats and cools. Unfortunately, when servomechanisms become more complex, and the workings are minuscule electric currents inside a silicon chip, anthropomorphic analogy becomes the only practical vocabulary for talking about these everyday devices, since the general population is unfamiliar with the mathematical and engineering principles.

Because these devices are so simple and familiar, we do not seriously consider the vocabulary of ‘thinking’, ‘seeing’ and ‘deciding’ as reflecting a belief that our car or heater is an intelligent agent. However, as closed loop control theory was applied to larger and more complex systems, those developments fed mid twentieth-century enthusiasm for ‘cybernetics’, grandly named by Norbert Wiener (1948) from the Greek κυβϵρνήτης for steersman or governor, suggesting that almost any problem can be solved with a feedback system that observes, decides and acts on the basis of observation. Allende's project Cybersyn in Chile notoriously attempted, and failed, to run the whole country on the basis of cybernetic theory. Control theory works reasonably well at what it was originally created for – mechanical regulation of a physical process – but is dubious as an overarching theory of human behaviour or society. After finishing my own training in control theory, I learned that the mathematics of mechanical control systems are often too complex to be easily predictable, and there is much challenging engineering work involved in making good thermostats, cruise controls, autopilots and so on. But these are challenges for engineers, not philosophers. These electro-mechanical systems have no ‘intelligence’, in the human sense, despite what marketing literature might say. A ‘self-driving’ car does not have an artificial driver, it is simply a slightly more complex cruise-control.

I hope that this has communicated the nature of what I call ‘objective’ AI. It measures the world, acts according to some mathematical principles, and may indeed be very complex, even unpredictable, but there is no point at which it need to be considered a subjective intelligent agent. We might talk about such devices and systems as if they had feelings, desires, or memories, but these are no more than slightly poetic interpretations of rather more mundane mathematical functions and algorithms, which could be explained to anyone with the patience and interest to ask. Pursuing philosophical enquiries on the basis of poetic analogies, when accurate mathematical descriptions are available, would be misguided at best.

Imitations of subjectivity

The other kind of AI is constructed on very different principles. It uses principles of ‘machine learning,’ in which very large amounts of data are collected to ‘train’ a stored model. A variety of training algorithms can be used, but they include approaches such as the ‘neural network’ that counts how relatively often some pieces of data are seen to be associated with other pieces. In recent years, the amount of data involved has become too large to easily imagine, while at the same time, the individual pieces of data might be very small indeed (a single pixel of a photograph, a barely-audible click, or one word among the thousand million edited contributions to Wikipedia). The possible numbers of connections between these tiny elements are so many trillions that no human could ever understand what they all represent.

Although describing these algorithms as ‘neural networks’ is even more likely to encourage casual analogies to the human brain than previously occurred with cybernetic servomechanisms, the engineering implementation of these systems has little practical resemblance to human reasoning. Instead, the computer does what computers do best, which is to store and retrieve extremely large amounts of data. This is important because the data most often stored by these subjective AI systems are real subjective demonstrations of human intelligence. For example, if a researcher wants to create a ‘neural network’ to subjectively recognize or express emotions, the network must be trained with examples of human subjects recognizing or expressing emotion, in order that it can later imitate human responses by replaying appropriate reactions. If the goal is to generate English language text, the system must be shown large amounts of text written by humans, so that it can play back its own pastiche of the texts it has already seen. This kind of text-generation algorithm has been described as a ‘stochastic parrot,’ (Bender et al. 2021) for the way that it wanders over the training data encoded in its network, picking out words and phrases that typically might follow each other. The results are generally perfectly grammatical, since grammar-checking is quite straightforwardly algorithmic, but often include wildly inappropriate leaps or faults in ‘reasoning’, given that there is no actual knowledge encoded beyond sequences and co-occurrences of words (Alexander et al. 2050/2021; Shane 2019).

Unfortunately, there is an ethical problem associated with this subjective kind of AI. The problem is quite different from the ethical problems associated with servomechanisms, which might indeed be used in unethical ways, but where the target, goal, or ‘objective function’ of the device has been clearly stated, as was indeed the case with the innovative guidance systems of Wernher von Braun's V2 rockets, whose principles of operation were so influential in the early development of cybernetics.

In contrast, the ethical problems of subjective AI are concealed, embedded within the infrastructure itself. They build on the tradition of hoaxes and fairground entertainments such as von Kempelen's chess-playing automaton, ‘the Turk’, created in the eighteenth century to conceal a human chess player hidden inside, so as to make a humanoid puppet appear artificially intelligent (Schaffer 1999). This fairground strategy has been directly copied by modern AI companies, most blatantly by Amazon in their product ‘Amazon Mechanical Turk’ (AMT). The purpose of AMT is to commission anonymous humans to carry out ‘Human Intelligence Tasks’ hidden inside an AI system, with the interface to these humans presented to engineers as if the hidden person was a software subroutine. Commissioning hidden intelligence via the piece-work of AMT ‘Turkers’ is an unregulated international labour market, managed in auction style that leaves some workers paid no more than a penny for their intellectual labour (Irani and Six Silberman 2013). The modern exhibition of AI, demonstrated by wealthy AI companies and research institutes, but underpinned by the most exploitative kinds of anonymized and alienated labour, fully realizes the problems of automation anticipated by Karl Marx, of ‘this automaton consisting of numerous mechanical and intellectual organs, so that the workers themselves are cast merely as its conscious linkages’ (Marx 1973).

AMT is particularly shameless, in naming its product directly after von Kempelen's hoax, and famous training data sets for machine learning systems have indeed been commissioned as piece-work from Turkers, but other companies have more subtle ways to hide human subjective intelligence inside their technical systems. Many of us have been required, in order to use some ‘free’ service or other, to spend time classifying images of traffic lights, street signs, etc – decisions that will be stored and used to train supposedly ‘self-driving’ cars with our own replayed intelligence. And in many other cases, intelligent human work is not even commissioned, but simply stolen. Every photograph uploaded to a Facebook page or Instagram feed, every blog entry or online conversation, might potentially be swept up by AI researchers, to be hidden among terabytes of other training data as unpaid intelligent work that is simply available for the taking.

In these cases, the very claims of subjective AI as a philosophical enterprise have ethical consequences. If it were really possible to create a genuinely artificial intelligence, having its own subjective agency, then this machine could be recognized as the original author of its own creative work. A company that owned such a machine would have acquired a magnificent intellectual slave, able at the press of a button to generate hundreds or thousands of new books, poems, journalism and so on, all owned by the owner of a machine that itself has no human rights. As I write this, substantial publicity is arising from a recent case in which a Google engineer did claim that one of these stochastic parrot systems has become sentient, and should therefore be recognized as having rights (Tiku 2022). The implications of such a decision could be devastating for a company reliant on digital slaves, and Google instantly responded by suspending the engineer. However, it is essential, for cognitive capitalism on such a massive scale, that the enslaved machine itself be recognized as the sole author of its outputs, and certainly not the hidden pieceworkers of AMT or authors of ‘free’ online work harvested from Wikipedia or elsewhere on the internet. Philosophical claims of original creation by machines, in these circumstances, are no more than justification for institutionalized plagiarism on a massive scale, a subjectivity factory that converts human intelligence input into artificial intelligence output. To quote Marx again, ‘Rather it is the machine which possesses skill and strength in place of the worker, is the virtuoso with a soul of its own in the mechanical laws acting through it.’ (Marx 1973).

What discipline have we constructed?

Drawing from one side on my training and professional experience in control theory, computer science and AI research, and from the other side on my education in cognitive psychology and comparative religion, my personal view is that there is a clear difference between the two kinds of AI. One is a purely objective physical servomechanism to control machinery acting in the world. As with many other kinds of automation, this kind of AI does displace, or rather frees, human labour. Before we had such automatic mechanisms, many humans were employed in mindless servo-mechanical tasks such as adjusting the wicks on lamps or regulating the speed of a steam pump. In my own experience as an automation engineer, factory workers whose meaningless jobs become replaced by automated devices are pleased that their own skills can then be applied by their employer in more meaningful ways. The second kind of AI is far more worrying – a subjectivity factory, supposedly motivated by philosophical enquiry into what it means to be human, but actually providing a smokescreen for institutionalized plagiarism, taking the genuine creative work of real humans, and disguising it such that the original authors need never be recognized or compensated.

Although it is clear to me that these two kinds of AI are quite different, the difference is not often remarked on by either public commentators on AI, or by technical AI researchers. Perhaps this is because it is relatively rare for one person to have my own combination of disciplinary perspectives, although there are certainly research teams that combine people with different kinds of training. Perhaps these products are simply so complex, often combining some components that are objective and others that are subjective (including IBM's ‘Watson’, Microsoft's ‘Cognitive Services’, Amazon's ‘Web Services’ and others), that the people building them have little time to ask critical questions, while those inclined to public commentary might struggle to appreciate such diverse technical principles of operation.

A particular challenge of interpretation is how to evaluate the significance and implications of technical demonstrations, such as the highly publicized successes of AI algorithms at playing board games like chess and Go, or simple video games. Such demonstrations are promoted by AI boosters as an important step toward ‘Artificial General Intelligence’. However, the argument of technical continuity from simple board games to the full complexity of human affairs seems surprising to any engineer, who understands that the challenge of solving a problem rests in the particularity of that problem. In fact, chess, Go, Breakout and the others are themselves algorithms – simply sets of rules, defining what turns may be taken, together with a criterion to decide who wins the game. In cases where the game is an algorithm, then the demonstration of ‘AI’ is simply one algorithm ‘playing’ with the rules of another algorithm.

Algorithms playing with other algorithms might be (and often is) mathematically interesting, but hardly relevant to ordinary human affairs, where diligently following every rule without exception is the least likely possibility. Such demonstrations also have limited relevance to control engineering, because the physical world is so resistant to formalization. Noise, dirt and stickiness in the real world make it frustratingly difficult for even simple robots to work reliably. At the time of my own contributions to the field, most robot planning researchers, including myself, Philip Agre and many others, tested their algorithms in ‘toy worlds’ – digital simulations without noise, dirt or stickiness – that were themselves little more than a videogame. The essential starting point for meaningful discourse across disciplines is being able to recognize what kind of problem can usefully be solved by repeatedly testing against a mathematically specified goal (for example, using ‘reinforcement learning’), and which are ‘wicked problems’, where neither mathematical specification nor repeated testing are possible (Rittel and Webber 1973), and where academic discipline itself is challenged (Blackwell 2008).

Which disciplines do we need?

What are the practical implications of all this? If we accept this argument for two fundamentally different kinds of AI, what should AI researchers and AI engineers do differently? In the sociotechnical research challenge that was problematized by AI researcher-turned-social-scientist Paul Dourish (2006), what are the ‘implications for design’? From my own engineering perspective, it seems possible to draw some reasonably clear prescriptions for effective AI research.

Firstly, what is the ‘ground truth’ of accuracy and evaluation for intelligent action in any given situation or problem? Is it a mechanical measurement, or is this a social situation reliant on human judgment and consensus? If the ‘ground truth’ is derived from the kinds of judgment that are only made by humans (for example when providing the labels for a training data set), this is a sign that both the ‘ground’ and the ‘truth’ might be open to human debate, as analysed in great detail by philosopher of software Brian Cantwell Smith (1996, 2019).

Second, if there is a clear mechanical measurement involved, then it is important to ask what kind of question can be answered by this objective measurement. Accurate measurement of a building's height is a good answer to the question ‘how high is this building?’. It is not a good answer to the question ‘how high should this building be?’. Unfortunately, there is a long history in science of measuring things only because we know how to measure them, not because those things are the most important or relevant to a particular question. Old proverbs and jokes tell us that ‘to a man with a hammer, everything looks like a nail’, and ‘the drunk looks for his keys under the lamppost because that is where the light is’. If the measures available do not reflect the real goal, then use of sophisticated AI to answer the wrong question seems just as foolish as these old jokes.

Third, if the purpose of an AI system really is to economically replicate or automate human actions, we need to ask when and why that is an appropriate thing to do. My own contribution to this field, in a book entitled Moral Codes (Blackwell forthcoming) sets out at length the engineering design opportunities that can be realized through automation of repetitious judgment tasks – precisely the thing that we have always been grateful for, when computer systems are designed well.

Fourth, we may be able to advance the study of human cognition through comparisons between what humans and machines can do. This is the domain of psychophysics and neuroscience, where the current level of advance is still uncertain about the precise function of the individual neuron, cells whose internal processes might perhaps be described using the mathematics of cybernetic control (MacKay and McCulloch 1952). There may also be some opportunity, in relation to the sociology of knowledge, to consider more carefully the patterns by which humans come to know each other, for example in the analyses made by Harry Collins in his books investigating What Humans and Computers Can Do (Collins and Kusch 1998), and Artifictional Intelligence (Collins 2018).

Finally, there have always been AI researchers who are motivated by a desire to gain insight into the human condition through constructing representations of humans. This is a goal shared with literature, painting, sculpture, and many other fine arts. Computer systems are dynamic, generative and interactive works of art that can indeed illuminate the human condition, and it is surely this promise that motivates so much continued interest from philosophers, beyond the relatively mundane question of how digital products and services should ethically be governed. This kind of AI research is most like a branch of literature, in its employment of language and narrative, and the manner of encoding via (source code) text. Like literature, this genre of AI research would benefit not only from more rigorous application of literary analysis, but also from provision of creative writing courses to better select and edit what the public audience must endure.

All these kinds of interdisciplinary science present interesting and exciting new challenges, that would benefit greatly from more appropriate disciplinary frames of reference, terms of analysis, and methods of investigation. However, AI is not a unitary phenomenon, and there seems little prospect that any single discipline can make the most necessary breakthroughs. The challenges and opportunities that I have summarized above are all worthwhile, but their investigation will be most profitable when the nature of each specific problem being addressed is properly aligned with the potential contributions of the discipline from which it is being investigated.