Shareable and un-sharable knowledge

Abstract

This article focuses on what it means to generate actionable but non-sharable information, and how this might relate to our understanding of what counts as knowledge, which typically entails some form of explanation. As automated systems sort and classify us for the purposes of dating, education, employment, health care, security, and more, we are going to want to know how and why these decisions are being made. Or, failing that, we will at least want to know, with as much clarity as possible, under what circumstances and to what uses, automated systems are being put to use. In either case, the role of narrative is inseparable from the call for transparency.

This article is a part of special theme on The Black Box Society. To see a full list of all articles in this special theme, please click here: https://journals.sagepub.com/page/bds/collections/revisitingtheblackboxsociety

Revisiting Frank Pasquale’s (2015) foundational work on the “Black Box Society,” it becomes increasingly clear that the relationship between knowledge and action lies at the heart of his critique. This relationship forms the essence of the notion of the “black box,” which generates an action or outcome while simultaneously barring access to the underlying process. The notion, of course, exists in metaphorical tension with the image of “light” or transparency at the heart of the Enlightenment conception of knowledge: that it is amenable to rational explanation and thus shareable and explainable. In his discussion of the need for improved transparency for automated processes, Pasquale (2015) invokes a famous Thomas Jefferson quote to highlight the benefits of transparency, “he who receives an idea from me, receives instruction himself without lessening mine” (p. 57). Jefferson is appealing to what economists describe as the “non-rival” character of information, which, unlike material goods, can be shared without diminishing the overall store. The familiar formulation goes as follows: If I give you a piece of cake, I can’t eat it myself; however, I can share what I know without losing that knowledge. Whether or not I lose some of the potential value that may obtain from keeping that information to myself is, of course, a different question—and one that underwrites the development of various forms of protection for intellectual property, trade secrets, and so on.

However, for the purposes of this reflection I want to focus on what it might mean to generate actionable but non-sharable information, and how this might relate to our understanding of what counts as knowledge, which typically entails some form of explanation. As automated systems sort and classify us for the purposes of dating, education, employment, health care, security, and more, we are going to want to know how and why these decisions are being made. Or, failing that, we will at least want to know, with as much clarity as possible, under what circumstances and to what uses, automated systems are being put to use. In either case, the role of narrative is inseparable from the call for transparency. As Pasquale (2015) puts it, “If we’re not going to be able to stop the flow of data, therefore, we need to become more knowledgeable about the entities behind it and learn to control their use of it” (p. 57).

One of the challenges posed by “black box systems” is that even when an explanation is demanded and supplied, the result is likely to by cursory: because the machine said so. In many cases, this is unlikely to be a satisfactory answer, given the persistence of Enlightenment conceptions of knowledge that lead us to expect a narrative explanation of some kind. The notion that knowledge is something that is sharable in a meaningful sense entails the possibility of providing both evidence and a reasoned account: one that can, in principle, allows others to reconstruct it for themselves. When a data-mining consultant tells an employer that job applicants who use Chrome or Firefox are more likely to be good hires than those who used Safari or Explorer (Pinsker, 2015), we naturally want to know why, and are tempted to provide our own explanations. Perhaps someone who is satisfied with a pre-installed browser is not likely to seize the initiative, maybe they are willing to settle for a lower quality outcome, who knows? The speculation could unfold in a variety of directions. For the consultant and the prospective employer, however, the “why?” question is largely beside the point. The machine has rendered its verdict. As Chris Anderson notoriously put it in a manifesto for post-explanatory knowledge: “Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves” (Anderson, 2008).

There is an oracular quality to such a formulation: the data has spoken (“I count the grains of sand on the beach and measure the sea; I understand the speech of the dumb and hear the voiceless” ¹ ). To the extent that such a pronouncement counts as knowledge, it is actionable, but not comprehensible: it does not expand or elaborate upon our understanding and is not shareable in the way an underlying explanation might be. This type of knowledge is practical—in the sense that is actionable—but not sharable in the sense that the entity that generated it can share its logic with the mind receiving it.

Such oracular “knowledge” is not simply an artifact from a superstition-laden past eclipsed by the Age of Reason. There are still plenty of contexts in which we rely on knowledge that is, in the explanatory sense, non-comprehended. One ready example is the fact that, “Some of the most recognizable drugs—acetaminophen for pain relief, penicillin for infections, and lithium for bipolar disorder, continue to be scientific mysteries” (Johnson, 2015). We know from past experience that they have a high likelihood of working, but we have yet to understand the underlying mechanism.

However to say that the underlying explanations are unknown is not the same as saying that they are, in principle, unknowable. In the medical realm as elsewhere, there are ongoing attempts to uncover underlying explanations, with the hope that understanding the mechanism might be useful in further developing treatments and cures. There are also instances in which we act on the basis of principles that are explicable but not necessarily consistent or reliable in their outcomes. We may, for example, adhere to ethical principles that do not reliably result in beneficial outcomes or just rewards (just as unethical behavior does not always precipitate just desserts), which is to say that the relationship between understanding, action, and outcome is not always a fixed or straightforward one. Nevertheless, there is a recurring tendency to provide an explanation of some kind—not least because of the role these play in accountability and legitimation. This tendency expresses an orientation to the world that exists in tension with the rise of “black box” systems.

One way to approach the issues raised by actionable but non-shareable “knowledge” is through the lens of narrative. It is worth emphasizing that the term non-sharable does not apply to the outcome—it is easy to tell someone that the algorithm has rendered a particular decision—but to the underlying explanation. The “knowledge” in such instances is, in this deeper sense, unknown and perhaps unknowable.

Nevertheless, for us, as of yet, the horizons of narrative and representation remain unsurpassable. We may not know why a particular outcome has been generated (for example: a recommendation to hire or not), but we need to know why it should be honored. This means generating narratives about why a particular process is reliable, how it avoids bias and increases efficiency and/or quality of outcomes. But even on a conceptual level, it is impossible for black boxes to reach escape velocity from the pull of narrative. When decisions are opaque and the processes that generated them black-boxed, there are plenty of stories to be told about how the data is collected and “cleaned,” how problems are framed, what questions are asked, and how outcomes impact individuals and groups.

The work of the narrative is in part the work of sharing. As Ricoeur (1986) puts it, the text, “is a mediation between man and the world, between man and man, between man and himself” (p. 25). The black box, by contrast replaces sharing with operationalism: the goal is not to tell or to explain, but to form a link in a process of decision or classification. The subtraction of the element of narrative explanation deprives the black box of what Ricoeur describes as “reflexivity” by short-circuiting the process of interpretation. For Ricoeur (1986), the interpretive process, what he describes as the hermeneutical problem, “attempts to discover new features of referentiality which are not descriptive, features of communicability which are not utilitarian, and features of reflexivity which are not narcissistic” (p. 27). We bring to the AI or the neural net our processes of interpretation, but these are, in a sense, external impositions upon its operation. The automated system has been designed to isolate a pattern, not to interpret or make sense of it, and this feature continues to make inroads into social processes.

There is, of course, a practical value, in many cases, to this logic, as the example of medicines with unknown mechanisms suggests. But there is also a potential threat to the benefits provided by shareable knowledge. Consider, for example, the case of so-called predictive policing. The promise of surveillance-based systems to be able to identify where and when crime is likely to occur threatens to launder historical bias through the algorithm. It also underwrites a resource shift away from explanation-based policies and toward pre-emptive ones. Why devote resources to researching the causes of crime when they can, supposedly, be thwarted at their moment of emergence?

The answer, of course, is that addressing underlying causes might obviate the need for ongoing pre-emption. But there are both institutional and political sources of opposition to causal analysis. At the institutional level, organizations may not be highly motivated to significantly reduce or eliminate the problems whose existence is their reason for being. At the political level, narratives of causality are subject to the increasingly destabilized conditions for public debate and deliberation. The state of the political debate over climate change in the United States is one of the more ready-to-hand examples. The political right has been particularly effective, in recent years, in mobilizing strategies that no longer seek to propound a dominant narrative so much as they work to undermine the possibility of crafting any coherent challenge to established forms of political and economic power.

Social media have proven to be particularly effective platforms for promulgating a din of misinformation, disinformation, and uncertainty that undermines the efficacy of narrative as a decision-making tool in politically charged contexts. The allure of pre-emptive approaches facilitated by automated, data-driven systems is that they ostensibly sidestep the vicissitudes of narrative. Politically charged arguments about social or economic policy can be circumvented by a focus on pre-emption. In the United States for example, the political paralysis over gun control creates an opening for data-driven systems with names like ZeroEyes and Athena that deploy smart cameras and machine learning systems in the schools to identify weapons and threatening behavior. These systems offer automated forms of response, such as locking down a school and notifying authorities in the face of a perceived threat and perhaps eventually disarming or otherwise incapacitating potential threats.

As Pasquale argues, however, automated systems have built-in biases—not least because, like narratives, they are always, in some sense, partial: they are shaped by the available data, by the priorities of those who deploy them, by the assumptions built into the decision-making process, or those that shape the classifications and correlations they generate. However, unlike narratives, they are not reflexive about their own processes. Google does not seek to divine our hopes and dreams when its automated systems sort through our email messages. Rather, it looks for fruitful correlations: does a particular pattern of words correlate with a greater or lesser likelihood of someone clicking on an ad or following through on a purchase? “In this model,” as Jeremy Packer (2013) puts it, “the only thing that matters are directly measurable results … did someone initiate financial data flows, spend time, consume, click, or conform?” (p. 298).

As the interpreters of the actions that machines take, we will always attempt to reintroduce some type of narrative explanation—and we may have the evidence to back this up. But one danger we face is the temptation to adopt the post-narrative position toward which the automated system gestures, despite its incoherence. If the online information flow has taught us one thing it is that we face an inexhaustible flow of narratives, that there is always more to be said, the story is never finished, closure is never complete. The quest for understanding looks like an increasingly vexed one: why bother to dive down the rabbit hole of interpretation if the numbers can speak for themselves? As we attempt to keep pace with the speed of information, interpretation just slows us down. Who has time any more? We don’t even have time to walk these days—as the burgeoning mobility-as-a-service landscape, littered with electric scooters and bikes reminds us.

It will be very tempting, in a range of contexts, to offload time and labor onto automated black box systems without holding them accountable. They are blissfully free from the vagaries of representation: there is no space between sign and referent for them: they simply operate on the data they are given. It is up to us to make the leap of representation: that is, to ask whether the output accurately corresponds to reality. If an automated monitoring system tells a college admissions office that an applicant has received a “low interest” score because they are not responding quickly to email, or not clicking through to emailed links, it is up to us to inquire whether this score accurately captures what we attribute to it. But it can be costly, time consuming, and efficiency-busting to continually scrutinize these outcomes.

If a system yields useful outcomes at some point in time, it will be tempting to imagine that it will continue to do so. If it simply makes a task easier, there will be a strong bias toward adopting it, even if we are not sure that it is doing the task better. As in the case of Australia’s “Robodebt” system, which automatically evaluates whether the recipients of welfare benefits have been overpaid, and thus owe the government money (because of changes in their income stream, for example), the burden will fall disproportionately upon those who are harmed by the process to demonstrate its flaws—a process that takes time and resources.

The danger is not that black box systems will fully exempt themselves from narrative explanations. It will always be possible, at least in principle, to assess their overall impact, even if we are unable to reverse-engineer the processes that led to it. With enough data and the power to analyze it, we can determine whether, for example, a system’s outcomes exhibit a particular pattern (of discrimination, for example), even if we cannot open the box to explain why. In many cases, however, the data will be hard to collect or proprietary, and the cost of collecting it and analyzing it prohibitive. This means, as Pasquale (2015) argues, that the growth of the Black Box Society and its ability to shape the life chances of those who are subject to its processes mean, “We need to hold business and government to the same standard of openness that they impose upon us—and complement their scrutiny with new forms of accountability” (p. 57).