On oracles and collective intelligence: An exchange of letters

Achieving collective intelligence requires information to guide action. In the ideal case, entities consult an oracle that provides accurate information immediately and at no cost. Such perfect oracles rarely exist; neither in the social world nor in nature. In this exchange of letters, Daniel Levinthal, an organizational scholar, and Deborah Gordon, a biologist, share thoughts on how, in the contexts that they study, collectives overcome the lack of perfect oracles. The exchange highlights how oracles differ in relation to rugged landscape problems, for example, fixed, high dimensional problems with interaction effects, as opposed to systems composed of interdependent actors who alter resources and payoffs through their actions. In the former, significant resources may be spent to approximate perfect oracles. In the latter, actors evolve or adapt protocols and messages to produce emergent, decentralized oracles. The letters raise several important questions: how do built and emergent oracles differ? How accurate, how immediate, and how low cost must an oracle be to enable collective intelligence to emerge? And, connecting to this journal’s first exchange of letters, contrary to our initial assumption, might the ideal oracle be noisy?

—Editor: Scott E Page

Oracleness and organizations

Dan Levinthal

Dear Deborah,  I am sending along some thoughts about how organizations find solutions to problems and coordinate activities with the concept of “oracleness” as a theme. I have described some of those ideas within the context of the rugged landscape model which originated in biology. As you are an expert in coordination and problem-solving by ants, I’d love to get your reactions to how I’ve applied some of these ideas to human organizations. After all, we know about the power of recombinations!

Oracles

An oracle in the classical tradition is someone who foretells the future. Cassandra’s unheeded prophecy that the abduction of Helen will bring on the fall of Troy is the iconic exemplar. An oracle provides a prediction about the world—will a new technology prove commercially viable, or as in the context of Cassandra, will abducting a certain King’s wife trigger major geo-political consequences? In the business world, we don’t have prophets, but we do have cadres of analysts extrapolating from existing patterns and incorporating their own judgments about possible futures. Such efforts at anticipatory wisdom and judgment establish a foundation for intelligent organization action.

While not negating these explicitly forward-looking efforts, it is also important to recognize that much of our insight comes from inferences from our prior experiences—whether explicit experiments such as an A/B trial or less intentional instances in which the learning is a by-product of action and the observations of the associated outcomes. Another form of “backward-looking” intelligence is of the Darwinian sort. The “intelligent design” of a Darwinian process hinges not on skillful prediction of future states but rather stems from the survival over past states.

In addition to this question of the extent to which the wisdom of our beliefs about future states of the world and appropriate actions to take in those states stems from a forward-looking calculation or backward-looking associations, which is a temporal issue, there is a “spatial” issue: the challenge that the wisdom of any given action is typically not independent of the broader pattern of behaviors of which it is a part.

John Donne famously noted that no man is an island. Donne made this observation to suggest our shared humanity; however, we can also adopt that sensibility when we consider the possible wisdom of any given action. Even when we consider a trivial, unambiguously individual decision—what do I want to have for lunch—that isolated individual decision is not independent of other actions and experiences: When is my next meeting? Am I going out to dinner tonight? How interesting does the new food truck that opened-up near my office look? More challenging is when the relevant context includes other actors—actors who are simultaneously engaged in their own process of choice, discovery, and search.

Two facets of collective intelligence

In this regard, it is important to distinguish two distinct facets of “collective intelligence.” One variant associated with the “wisdom of the crowd” points to the power of aggregating the diverse insights of a collection of individuals. Unquestionably, this is an important line of inquiry. However, we should not narrowly conceive of organizations as prediction machines.

Organizations core function is to facilitate coordinated interactions among a set of actors. In this light, while the notion of “routines” is often viewed pejoratively as reflecting inertia, Nelson and Winter (1982) in their development of evolutionary economics point to routines as central to organizational capabilities. Routines address the coordination problem of joint action. Alternatively, if actors engage in a simultaneous process of adaptation, the resulting collective performance may be very unstable.

But organizations do not face a pure coordination problem—such as the societal problem of whether cars should travel on the left or right side of the road. There are vastly different payoffs to alternative patterns of collective behavior. Does the organization “coordinate” on a particular set of applications of nano-technology to its products? Does the organization coordinate around a particular set of markets as focal for its business? Thus, organizations face a dual challenge of search, for better or worse strategies, and coordination—since there are important returns to coordinated collective action. A given routine may solve the “coordination” problem, but there is no reason to believe a given routine represents the most desirable pattern of behavior—the problem of search.

This joint problem of search and coordination is not unique to human organizations. Indeed, from the perspective of evolutionary biology, the evolution of the human genome confronts this challenge. Our genetic structure links in a complex, interdependent manner to influence our expressed phenotype or form. And of course, there is the broader dynamic of mutualism and competition with other organisms.

An important early analytical apparatus for addressing these issues within evolutionary biology is the idea of “fitness landscapes.” Sewall Wright (1933) introduced the basic framework highlighting the interdependence of genes in contributing to an organism’s reproductive fitness. This interdependence results in local peaks in performance landscape as a switch in an individual gene might lead to a decline in fitness but a change in some broader set of genes might increase reproductive success. Levin and Kauffman (1987) developed a formal representation of these ideas in the form of NK landscapes with “N” denoting the number of distinct elements in a form and “K” their interdependence, with each of the N elements interacting with K other elements. This structure was built upon by Kauffman (1993) and developed in the context of the processes of organizational learning and adaptation by Levinthal (1997).

What might constitute a more or less effective social structure in addressing this joint problem of search and coordination depends upon the nature of the task environment. In the limit, when faced with a problem of pure coordination, the example of whether traffic should flow on the left or right side of the road, hierarchy and authority structures are an effective form. However, unless we assume that actors at the apex of an organizational hierarchy are endowed with unique, God-like wisdom, when faced with a problem of search more decentralized structures that tap into distinct insights and perspectives of the broader collective are preferred (Page, 2007). The joint problem of coordination and search may suggest elements of asymmetry in influence processes, though not strongly hierarchical forms. Indeed, a wide array of organizational attributes—degree of centralization, strong or “loose” culture, level of individual autonomy—can trade off the degree to which coordination is readily achieved versus the variety of possible approaches that are engendered.

This issue of coordination poses another layer onto our consideration of the “exploration-exploitation” tradeoff that highlights the tension between leveraging current insights about practices and strategies that are effective (i.e., exploiting) and searching to identify and experiment with alternative bases of action with which one is less familiar but may have merit (Holland, 1975; March, 1991). One can consider the exploration-exploitation tradeoff from an individual’s perspective, and standard treatments of the exploration-exploitation as a multi-armed bandit problem do that (Posen and Levinthal, 2012). However, when we consider the challenge of balancing the dual imperatives of exploration (finding potentially superior bases of action) and exploitation (leveraging what is known to be effective for profits and survival “today”) from the perspective of collective intelligence, at least two distinct issues emerge.

First, an organization, at least an organization of greater scale and scope than a “lean start-up,” does not need to make a binary choice between exploration and exploitation. Some subunits, such as the R&D group, are devoted to more exploratory efforts while other subunits are pushing product out the door and engaging with current customers in ways that keep the lights on and current profits flowing. Second is the issue I posed of coordination. An organization faces a problem of mutual adaptation—it is not simply finding a good “arm” in a problem of search, but getting a diverse set of individuals working in ways that are complementary.

Specialization and diversity

Another important property of collective intelligence is the possibility of some division of labor between the identification of possible actions, new strategies, new products or markets etc., and the evaluation of the merit of these possible actions. A familiar “division of labor” is that of the aspiring corporate entrepreneur who identifies some new initiative that they feel has merit but may find themselves thwarted in the evaluation of that merit by their corporate superiors. Such situations are often interpreted as reflecting the pathology of the separation of the power of position (allocating resources) and the power of knowledge (expertise and awareness of the particulars of the proposed initiative). While such pathologies are certainly present, another issue is that of opportunity cost (Levinthal, 2017). The higher-level actor may allocate scarce resources, not just financial but also engineering talent and so on, across many competing claims with individual claimants likely to be strong advocates of the merit of their proposals.

One important contrast with biological processes, which are arguably myopic (Levinthal and Posen, 2007), is that this is a contestation over projections as to possible future outcomes. In that sense, organizations are composed of an ecology of oracles. Power, knowledge, and influence processes jointly determine the outcome of this “ecology.” Further, organizations can to some degree tradeoff their commitment to a particular anticipated future by experimenting, both sequentially in the form of staged investments and options and in the form of parallel “bets.” Diversity is critical to the identification of a rich set of latent initiatives. Furthermore, the effective culling, validation, and amplification of those initiatives within the organization is a function of both the diversity of perspectives as to how initiatives should be evaluated and how this contestation over evaluation and merit is adjudicated (Levinthal, 2021).

Yours,

Dan

Distributed oracles from the bottom-up

Dear Dan,

While ant colonies and human organizations both solve problems collectively, they do so in very different ways that I’ll explain.

Ants: guided by the present

To begin, ant colonies do not need single oracles of the form that you describe. They use distributed processes that accomplish colony tasks even though no ant can assess what needs to be done or decide how to do it. Their collective behavior is guided by changes in the present, not by ideas about how to prepare for the future. Of course, current adjustments change outcomes that make a difference in the future, but not because the ants intend it that way. Thus, in a colony, the ants’ perceptions of the current state, and their interactions with each other, collectively produce distributed outcomes that could be seen to function as a real time oracle, but that is not like the predictive oracles used by organizations.

Foraging

In colonies of the desert harvester ant (Pogonomyrmex barbatus), foragers search for seeds scattered by wind and flooding. Unlike some ant species that you may have encountered on your kitchen counter, or at picnics, harvester ants do not recruit others using pheromone trails that lead to a patch of food like the pizza crust on a plate on your counter, or the cookies at your picnic. Since the seeds are scattered, when a harvester ant forager finds a seed, this does not justify bringing more ants, as there are not likely to be more seeds nearby.

The foraging activity of the colony is regulated collectively through the decisions of individual foragers whether to leave the nest. There is no CEO deciding whether or not to coordinate activities on a new market opportunity. In a colony, the collective decision emerges from interactions among ants.

Colonies regulate foraging activity to adjust to a shifting tradeoff involving water loss. Ants lose water to evaporation when foraging in the sun, and colonies get water by metabolizing the fats in the seeds that they collect. A colony must expend water while searching, to obtain more water and food. A forager makes many trips each morning, engaging in a form of what an organizational scholar would call A/B testing.

The colony manages this tradeoff so as to regulate foraging activity, without any individual ant making an assessment of whether it is worthwhile to collect food on a given day with a particular humidity. They lack the capacity to make such calculations as individual actors. Yet, the colony makes a calculation using the rate of encounter that each forager experiences.

An outgoing forager does not leave the nest until it experiences a high enough rate of encounters with returning foragers that possess food. The encounters are based on smell; they consist of brief antennal contacts in which one ant smells the odors on the body of the other and the odor of the food the returning forager is carrying. These encounters regulate when the ants go out to search and when they do not.

The harvester ant system uses excitable dynamics, as do neurons, which use the rate at which they experience stimulation from other neurons to decide whether to fire. For a neuron, the stimulation provided when a linked neuron fires has a decay, as the electrical charge leaks down the neuron. We found that the timing of harvester ant decisions corresponds to a similar process (Ouellette and Gordon 2021); each encounter provides a stimulus that decays, and the effects accumulate (Davidson et al., 2016). This is how the ants can respond to rate without counting anything; if an ant experiences enough encounters before the last ones decayed, then it may reach a threshold past which it is likely to go out and forage.

The rate of forager return provides positive feedback associated with current food availability. Each forager searches until it finds a seed. The more food is available, the more quickly foragers find seeds, the more quickly they return, and the more they contact outgoing foragers who then go out to forage. This carries no spatial information—a forager returning from one place helps to inspire an ant inside the nest to leave to travel to another. While the rate of encounter is variable, and each ant’s response is not fully deterministic, in the aggregate this system tunes foraging activity closely to food availability (Prabhakar et al., 2012).

Foragers also respond to the humidity on their most recent trip, and this appears to affect how foragers decide whether to leave the nest (Pagliara et al., 2018). Colonies differ in how they regulate foraging. Some colonies are more likely than others to reduce foraging activity in dry conditions, thus conserving water but sacrificing food intake. We found that in drought conditions, the colonies that reduce foraging when dry are more likely to produce offspring colonies than those that maintain high foraging regardless of humidity conditions (Gordon, 2013); we are now investigating how this is changing as the drought in the southwest US has deepened (Sundaram et al., 2022). Recently, we learned that these differences among colonies are associated with dopamine—it seems that more dopamine makes a forager more willing to go out on the next trip, even if humidity is low (Friedman et al., 2018; Kocak et al., 2023).

Collective, distributed oracles

Thus, a colony is behaving as if it could predict how much water it can afford to lose, and how much food it needs to get, so as to be able to survive and reproduce. But no ant is making any assessment about food availability, what the colony needs or what is likely to happen next.

In some sense, the colony is making that assessment, but it is an outcome of their collective behavior rather than the result of a consultation with an outside information source, that is, an external oracle. Yet, as noted, the rate of encounter as experienced by the individuals performs the function of an oracle by giving a real time read of food availability. This method of assessment works in a world where the food supply changes slowly, where resources are distributed randomly so searching in one direction is just as good as searching in another, and where competition is low so the colony can afford to adjust slowly. (Gordon 2014). This slow adjustment through the rate of encounters at the nest would not work well for another species; for example, the arboreal turtle ants I study in the tropical forest, where everything changes quickly and competition is intense, use a much more local and flexible method of assessment to regulate foraging (Chandrasekhar et al., 2021).

Processes like this, in which collective behavior uses local interactions that reflect changing conditions, are widespread in nature. This is not aggregating the insights of many individuals. It is using the behavior of many individuals, each affected by conditions, to produce a collective outcome that allows the group to adjust.

This does not necessarily require any division of labor, and the allocation of function in natural systems is not based on knowledge or power. Instead, function can be assigned by a distributed process in which any individual that happens to encounter a particular situation, or engage in certain interactions, would respond in the same way.

The collective intelligence of natural systems works through interaction networks without the overlay of individual identity, ongoing accounts and reasons for what we are doing, and formulation of goals for the future that make human collective behavior unique. The question is how much of those processes are at work in human interactions as well.

Ants are very diverse, with more than 15,000 species in every habitat on Earth. They offer opportunities to learn about the many ways that evolution has shaped self-generated “oracles” to function in response to changing environments. By understanding how these distributed oracles function, we might also gain insights into how to enable self-generated oracles within organizations.

Yours,

Deborah