Making Affordances Real: Socio-Material Prefiguration,Performed Agency,and Coordinated Activities in Human–Robot Communication

Abstract

Usually, the alluring notion of “affordances” comes with the idea that technology makes some activities possible while constraining others. Our article departs from this dichotomic view and seeks to appreciate the multiplicity of socio-material prefiguration. Discussing three empirical examples from human–robot communication, we show that the affordances of “smart” technologies are not acted out in a smooth, planned process or through rational action alone. Rather, affordances are collective achievements that emerge within the interplay of humans and machines. This challenges the separation into active use and passive usability. It also demands us that we think through what types of agency are associated with these kinds of agents and what we take to define agency at all. Agency rests, we argue, on the capability to engage in intelligible encounters; it builds on purposive activities even though they might only realize a limited repertoire of tasks.

To capture the social and material settings that enable digital, networked communication, current scholarship in social research as well as in computer science frequently draws on the dubious, though alluring, notion of affordances. Usually, the decision to use this word comes with the assumption that technologies make some activities possible while constraining others. As such, the term is invoked to sidestep technological determinism on one hand and social constructivism on the other hand.

Without question, the dichotomy of constraint versus possibility has advantages in that it makes affordances measurable and comparable, for instance, across social media platforms (Majchrzak, Faraj, Kane, & Azad, 2013) or office software solutions (Rice et al., 2017; Treem & Leonardi, 2012). Yet, the binary options are achieved by curtailing other readings of the concept that would help us to embrace the variance and mutability of socio-material prefiguration that qualifies possible paths of action (Orlikowski, 2007; Schatzki, 2002). Otherwise, the notion would lack pizzazz: to state that a device affords a certain kind of activity might be obvious but also quite uninformative. Therefore, this loose usage generates, Rappert (2003) argues, “non-controvertible claims that border on the banal or unhelpful; it closes down debates in often arbitrary ways when they could be usefully opened up” (p. 573). In the same vein, Jarzabkowski and Pinch (2013) criticize that

often an affordance is simply equated with the ‘function’ of an object . . . The function of the object not only delimits the agency of the object but also ascribes a fixed intention or motivation to the person using the object. (p. 582)

Consequently, instead of treating affordances as bottom-line explanations for patterns of use, they can be turned into a topic for analysis in their own right (Bloomfield, Latham, & Vurdubakis, 2010; Davis & Chouinard, 2017). Besides asking “What is an affordance?,” it then seems plausible to also inquire, for example, “When is an affordance made present?,” “In what situation and how is an affordance rendered (un)available?,” or “How is an affordance configured?”

In our contribution, we refer to three empirical examples taken from the construction, prototyping, and deployment of robots in human-machine communication (Alač, 2016; Bischof, 2017). This concentration on material, locomotive, and (semi) autonomously functioning machines offers a lens through which the larger debate about interaction and sensemaking in-between human users and smart technologies that intervene in today’s social media comes into sight. From the Turing test and ELIZA onwards, a device or service’s capacity for intelligent behavior has commonly been demonstrated by its use of natural and convenient means of communication (Collins, 2018). The presumed intelligence of technologies such as chat bots or conversational companions entails the ability to participate in the creation of meaning shared by humans and machines. In turn, obstacles to integrating technologies in a real-world setting are typically framed as communication problems (Guzman, 2018; Licklider, 1960; Nass & Brave, 2005). Thus, because communication is key to agency and intelligence, Nagy and Neff (2015) rightly postulate that “communication research on emerging ‘non-human’ communication of algorithms, wireless devices in the Internet of Things, and data flows also needs expanded ways of theorizing socio-technical affordances for emerging technologies” (p. 2). To us, human–robot communication acts as a test bed and information-rich field where these emerging forms of joint socio-technical action and understanding can usefully be studied.

To this end, we proceed from scholarship in communication research, human-computer interaction, and social theory. What we take from this incoherent array of literature is an orientation toward situated actions whose enactment presumes the entanglement of material and social circumstances (Suchman, 2007). Our focus on this lived socio-material practice highlights the mutual associations between people and things that neither favor agency over structure nor vice versa structure over agency.

Engaging with these ideas, we take issue with three shortcomings in the present debate around affordances. First, the sense-making, interactional dynamics, and modes of sociability unfolding in human–robot communication challenge the gross simplification that the material and immaterial usability of hardware and software can be understood in terms of either constraint or possibility. Instead, they encourage us to appreciate the ways in which the mesh of practices, people, and things points to multitudinous paths of action. “Technologies have dramatically altered the nature, range and qualities of the contemporary ‘population’ of practices,” as Morley (2017) maintains, “by reducing and reconfiguring the contributions and qualities of human participation, how such practices are reproduced and whether and how they persist, evolve or dissolve” (p. 82). Such view departs from the concept of “technologies-in-practice” (Orlikowski, 2000, p. 407) that privileges the ongoing accomplishment of doings and sayings, and downplays the role of more durable cultural and material features relevant to a situation, too.

This perspective, second, requires us to think through what types of agency are associated with human agents and technical agents and what we take to define agency and empowerment at all (Barad, 2003). Questioning how we separate or blend objects and subjects is necessary for at least two reasons. For one, technologies increasingly have the capacity to produce an effect by analyzing evolving situations. In crafting and commodifying these communicative encounters, we witness differences between the strategies that designers and providers have adopted to stipulate choices and the tactics of users to construct a path out of them (Neff, Jordan, McVeigh-Schultz, & Gillespie, 2012). The variety and variability of the ways to come into action makes it difficult to smooth out the differences between what human users and technologies can do. Instead of a symmetry of agents, as Latour (2005) and other proponents of Action-Network-Theory have suggested, human–robot communication invites us to inquire how a multiplicity of affordances comes into being.

Finally, human–robot communication requires us to explain what we mean when we say that affordances are collective achievements and emerge from a conjunction of matter and sociality (MacVeigh-Schultz & Baym, 2015). Moving away from the idea that affordances are mainly visually perceived and cognitively interpreted, we explain how they arise from action cascades, where operations and procedures of different kinds of agents coalesce (Collins & Kusch, 1998).

Beyond the scenes discussed in this article, the day-to-day interplay of humans and smart machines is realized on a number of occasions, sometimes with less overt materiality than those of robot companions. Therefore, for instance, voice assistants and chatbots not only perform a considerable portion of communicative labor as they increasingly thread through and actually help to establish our social media activities (Guzman, 2018; Reeves, 2016). There may be situations in which these interfaces even enhance human capabilities. For instance, Bogost (2018) reports about the case of a blind man who utilized Alexa to maneuver his complex media ecosystem and connect with people. In this respect, any attempt to reduce the activities of this voice-activated technology and other social media to more menial tasks would be overly narrow.

Conceptualizing the Multiplicity of Affordances

If we want to take human–robot communication seriously, we cannot expect to find effective abilities. Instead, we face a culturally and historically contingent and socially contested accomplishment of enablements. It necessarily extends what might be called the received notion of affordances, which Hutchby (2001) has defined as “possibilities for agentic action in relation to an object” (p. 444) that artifacts should possess by virtue of their materiality.

The narrow distinction between constraining versus enabling attributes has been taken up in numerous approaches that aim to map and take stock of the actionable options inherent to technologies (Faraj & Azad, 2012). For all its conceptual incongruities, the notion of affordances allows them to surpass both under-socialized and over-socialized models for the appropriation of technology (Evans, Pearce, Vitak, & Treem, 2017). Instead, affordances are thought to establish an account that neither privileges the presumed objective obduracy of matter nor the subjective stubbornness of users; it likewise does not argue that physical objects are completely malleable (Bijker, 1995).

Yet, technologies that increasingly rely on artificial intelligence (AI), on analyzing aggregate data, and on learning algorithms that emulate social practices and enter into communication processes demand that we formulate a more nuanced concept of affordances (Nagy & Neff, 2015). This concept would need to consider the creative interplay of human capabilities and the capacities of more or less smart machines. And it should also pay attention to the kinds of agency employed in making affordances real and the effort that goes into enacting, stabilizing, or transforming them through situated actions (Suchman, 2007). In an attempt to thus broaden and detail the richness of prefiguration, Schatzki (2002) suggests directing our attention to the manifold ways in which technology makes courses of action

easier, harder, or simpler more complicated, shorter, longer, ill-advised, promising of ruin, promising of gain, disruptive, facilitating, obligatory or proscribed, acceptable or unacceptable, more or less relevant, riskier or safer, more or less feasible, more or less likely to induce ridicule or approbation—as well as physically possible or impossible and feasible or unfeasible. (p. 225f.)

Arguably, such a view seeks to dismantle the affordances concept as it originated in environmental psychology and design theory (Gibson, 1979; Norman, 1988). There, the problem with affordances starts with nominalization, as Bloomfield et al. (2010) declare. In this traditional account, affordances are treated as a property. For example, blogs possess qualities such as openness to reader input, fixity of permanent records, and juxtaposition of different viewpoints (Graves, 2007). However, such attempts to classify and count affordances run the risk of eliding the work necessary to make things affording particular activities (Siles, 2012; Treem & Leonardi, 2012). “The ‘affordances’ of technological objects,” Bloomfield and colleagues (2010) insist, “cannot be easily separated from the arrangements through which and amid which they are realized in practice” (p. 428). These multifarious arrangements encompass shared understandings, discourses and conventions, constellations of people, places and times, and social institutions and organizations.

To summarize these considerations, Evans and colleagues (2017) name three threshold criteria for conceptualizing affordances: first, they are found in the interaction of people and technologies, not in objects or features of objects. Second, affordances are not clear-cut outcome characteristics but are instead invitations to action. Third, these actions are not binary but variable.

Foregrounding process and agency leads us to recognize that artifacts can have a multiplicity of meanings. Multiplicity is, we admit, in itself a fashionable term of art that is used to underline the mutability and variability of things thought stable and given. However, besides this loose usage the idea can be employed in a more precise way since multiplicity does not mean arbitrariness. It rather makes us aware of the fact that affordances are, “different things in different practices,” to use Law and Singleton’s (2014) phrasing, because they are “being done or performed differently in those different practices” (p. 384). That is, affordances are, as Strathern (1991) put it, at the same time “more than one and less than many” (p. 35). Artifacts have multiple but not infinite meanings, some of them might be preferred and indicated through an object’s configuration and instruction, while others might be more ancillary or unplanned (Grint & Woolgar, 1997). Therefore, instead of being random, alternatives emerge within constellations of practical usage and design where agents with different skills, abilities, and interests operate. How certain affordances are privileged or sidelined can have considerable consequences for the implementation, diffusion, acceptance, and regulation of a technology. In effect, our task is to trace how a technology comes to channel a multiplicity of forthcoming actions that are simultaneously rather than individually present and interfering.

For example, during the Arab Spring, the proliferation, surveillance, or lock-down of platforms like Twitter, Facebook, and Telegram also depended on the ways in which their messaging services came to be associated with different repertoires of use and on how they were deployed for purposes by groups such as protesters, journalists, the police, or other state agencies (Gerbaudo, 2012). In that regard, their affordances multiplied in situations that differed in their use of one practice as opposed to another. There, affordances were not passive attributes or qualities of objects but were made to function within practices that existed alongside other practices in a social field.

Multiplicity furthermore suggests an analytical decision, one that opens up the possibility to study how affordances are achieved and how their varieties hang together. On such terms, “attending to the multiplicity of reality is also an act,” Mol (2002) holds. “It is something that may be done—or left undone. It is an intervention” (p. 6). This, then, is an empirical matter that is complicated by the assumption that affordances are not waiting within an object to be perceived and acted upon. Hence, the analytical task is not to identify innate cues or read off certain characteristics from a technology. Instead, we have to impute viable affordances in light of the configurations of users, devices, and services; the remediation of messages and contents; and the formation of environments in which they become conventionalized or contested (Lievrouw, 2014).

Affordances of Human–Robot Communication

We studied the construction and testing of robots within and outside scientific and corporate labs in the United States, Denmark, the Netherlands, and Germany. The ethnography combined the participation in six different sites with 36 in-depth interviews, more than 100 talks with researchers and representative users, living lab experiences and observations from scholarly and industry meetings, as well as the re-reading of existing academic studies. Hence, the study combines material generated in the field with a meta-analysis of scholarly work, mostly in the emerging area of human–robot communication. Methodologically, revisiting this multi-sited inquiry allows us to approach the question of how affordances are grounded in socio-material action from different angles.

At present, the field of cybernetics is especially interested in the expected similarity between human agency and computerized agency (Kurzweil, 2013; Reese, 2018). In contrast, the emerging field of human–robot communication is exploring how people and machines come to assume and realize capabilities for the joint construction of meaning rather than asking if or when androids will be able to act like humans. Therefore, the design of conversational robots for everyday situations as opposed to specialized environments like assembly lines represents a paradigmatic change in the engineering and conceptualization of what AI and agency can be (Brooks, 1999). Consistent with that, our ethnographic fieldwork, completed in between the years 2012 and 2015, was especially interested in the conceptual design, technical engineering, and real-world application of robots and how these revolved around situated actions. To this end, we examined the modeling and accomplishment of collective orientations toward an environment shared by humans and machines (Bischof, 2017).

One such instance, reported to us several times during lab talks and interviews, shows how a robot came to enter the role of a communicator within an interactional routine, in which its activities were seen as intuitive, intelligible, and accountable (Garfinkel, 1967). In 2002, a renowned AI conference announced a demo challenge that consisted of programming robots to behave like human participants at the following year’s conference. The tasks included entering the conference hotel independently and finding the room where a particular session was to be conducted. During the subsequent meeting, the decisive moment was when one of the robots approached the registration queue with its sensors not working properly. As other attendees were engrossed in conversation and had left gaps in the line, the robot had problems finding the line’s end. The detection algorithm registered one of the spaces left open by the people waiting in the queue and slipped in. The jury reacted with spontaneous applause and laughter. In their view, the robot was apparently not only able to follow social conventions but also to deliberately break them to gain an advantage, which would require a reflexive distance to the situative exigencies.

In a way, the robot’s defective circuits led it to misinterpret spatial intervals; yet, they also afforded queuing involving humans and the robot, whereby the technical agent appeared to act in a remarkably canny manner. This happened against the backdrop of the habitual expectations associated with the social practice of waiting in line. Without its interactional routine, the robot’s queue-jumping would have been interpreted as a failure. Yet, by confounding expectations in a way that allowed for an attribution of cleverness, the robot performed competently and sociably, allowing it to win the competition. As other inquiries have also noted, activities entailing a degree of obstinacy actually prompt people to understand their technological counterparts as intentional agents (e.g., Short, Hart, Vu, & Scassellati, 2010).

There are certainly limits to how we can interpret or handle a technology, but the proper challenge stems from those affordances that are not immediately perceptible or whose influence cannot be reduced to promoting or impeding actions. Therefore, MacVeigh-Schultz and Baym (2015) urge us to attend to the ways in which people themselves identify and make sense of the relationship between artifacts and the lived practices in communication and interaction. They thus introduce the notion of “vernacular affordances” (p. 3) to stress that people usually do not experience affordances in isolation or only through visual recognition. Instead, affordances emerge—or remain latent—within a setting where tools and messages, people, sense-making processes, and practices intertwine.

A striking example of an unaccomplished affordance comes from Alač, Movellan, and Tanaka’s (2011) study of the coordination between robots and humans. They followed a team of roboticists and charted their efforts to test robotic behavior with preschool children. They asked how a sophisticated object can achieve agency and examined the kinds of situated actions and accounts of such actions that unfold when humans and robots meet. In the episodes, which were meant to probe the allure of the material technology, a researcher was seated on a rocking chair next to a robot and simulated reading a book. The children showed immediate interest in the scene. As they entered the lab room, they approached the researcher, pointed to the robot, and asked for an explanation for what they saw. However, the researcher feigned indifference and did not react to their gestures. Gradually, the toddlers started to adopt this attitude of willful negligence, which Alač and colleagues (2011) called the “ignoring game” (p. 911): they minimized the attention paid to the robot while engaging more and more in other activities. Even when the robot physically moved, the children noticeably ignored it and finally left the room.

The incident illustrates that the assumed agency of the robot was highly dependent on the routine established in the situation. Although a number of potential uses had been technically inscribed and formed part of the toy robot’s sensorimotor equipment, they remained latent and were not enacted. In this case, voluntary disregard prompted users to ignore the material object. In the process of ignoring, the children actually demonstrated their interaction competence and common orientation. “Therefore,” Alač et al. (2011) concluded, “what is at stake is not simply a lack of shared attention, but a shared avoidance of visibly attending to the robot” (p. 914). Faced with unexpected uses or nonuses of technology and purposeful resistance, it is therefore difficult to determine the crucial characteristics of the device that would enable or obstruct a certain sort of action from the outset (Siles & Boczkowski, 2012). Rather than codifying affordances that enable functions, the relationship is turned on its head and the focus is on activities through which technologies come to afford certain types of handling.

A third case again makes clear that affordances of communicating with robotic counterparts do not emerge from a kind of natural situational setting but are instead explicitly provoked in communicative set-ups. The project we studied was an example of creating human-like androids. It was located in a lab where a researcher experimented with a robot that looked almost indistinguishable from the human investigator. The similarity went beyond replicating the researcher’s facial features in the robot’s silicone mask. At public events, they also wore matching suits, sported the same hairstyle and facial expression; they even set up a fully equipped office for the robot within the university premises. Although the robot itself had only very limited abilities to move and gesture, the head and parts of the face moved automatically, its synthetic eyes could follow its interlocutor’s eyes, and its appearance was well orchestrated and captivated visitors. This feeling was intensified when people spontaneously encountered the robot without introduction (Bischof, 2017). During our participant observation in the lab, maintenance staff referred to an informal initiation ritual they had. It involved sending new colleagues to the robot’s office while the others stood by waiting for the newcomers’ screams.

The uncanny humanness of the robot was not primarily a design feature. It was rather a discursive and practical production of human-likeness through the dressing, the setting, and the researcher’s own efforts to adapt to the robot’s looks. Therefore, the machine’s capability to represent a cogent counterpart was made possible by clearly designing the surroundings. In human–robot communication, this peculiar mix of the familiar and, at the same time, the disturbing presence and activity of an android, is referred to as the uncanny valley effect (Mori, MacDorman & Kageki, 2012). It assumes a linear relationship between the ever more human appearance of artificial devices and the human acceptance of these features up to the point where the technology behaves and looks as an almost indistinguishable replica. Then, the encounter becomes eerie.

It can be argued that the adjustment of human and machine features in this setting afforded a growing range of communicative interactions and allowed human and machine partners to generate meaning and coordinate activities. At the same time, the quest for similarity also stifled or at least subverted these possibilities for action. Not only did the doppelganger contribute to this dynamic, the humans entering and crafting the scene did so too. Therefore, the uncanniness was a feature of the situation, not a quality of the almost lifelike robot’s appearance per se. The uncanny valley effect, like the oft-described ELIZA effect, is therefore more likely to be something implied in the communicative constellation than a trait of the device (Hofstadter, 1996; Rosenthal-von der Pütten & Krämer, 2014).

This third example teaches us that researchers, constructors, and marketers of human-like robots purposefully use the tacit knowledge, expectations, and practices of everyday interaction to craft the affordances for human–robot communication. In line with Alač et al. (2011) observation of a failed human-robot interaction, the case, therefore, underlines that the presumed agency of artificial counterparts is shaped by the routines established in a situation. Routines and expectations associated with everyday situations are used to create a setting where the robot, despite its comparatively simple and certainly not intelligent technical functionality, is more likely to be perceived and addressed as an interlocutor. The same mechanisms, for example, worked in the case of a robot named Sophia, which delivered a speech to the United Nations and was granted Saudi Arabian citizenship, although technically it was a puppet that merely gave a chatbot a face.

Affordances as Collective, Socio-Material Achievements

Affordances cannot be determined in advance, but are collectively achieved in interactions between human and technological agents. Engineering and calibrating a robot that works in “unstructured” settings, that is, in places which are not factories or laboratories, is a fallible process. Thus, in the three empirical episodes, affordances were established, elided, and manipulated in a spatial and temporal sequence of interactional routines that were marked by improvisation and ad hoc interventions.

The situations in question here—which, in one case, afforded queuing by both humans and a robot but closed off affordances for communication in the other case and helped to erode possibilities for action in the third—were relational in character. They bridged technical and human realms. In their recursive relations, the socio-material prefigurations came primarily from lived practice and not through a planned process or rational calculation (Fayard & Weeks, 2014; Schatzki, 2002). Especially the second episode involving unaccomplished contingencies shows that affordances are achievements. Hence, they require effort to be seized, interfered with, and modulated; affordances are not discovered as functional aspects of things but emerge in the interplay of people and objects (Bloomfield et al., 2010).

On a related note, Orlikowski (2000) has introduced the concept of “technologies-in-practice” (p. 407). In line with our argument, it scrutinizes the idea that technologies embody inherent structures and instead assumes that the translation between material things and actions is not a one-way process in which human designers invent technologies whose construction goes hand in hand with the shaping and stabilizing of cultural knowledge, use scenarios, and prototypical adopters. Instead of assuming built-in arrays of fixed and embedded structures that somehow become available to users, Orlikowski (2000) discerns their structuring potential as something that needs to be instantiated to become effective and so only exists in conjunction with practices. Drawing on this idea, Costa (2018) even speaks of affordances-in-practice to stress that “affordances are not fixed and stable properties but are implicated in different ongoing processes of constitution, which may radically vary across social and cultural contexts” (p. 3651). Yet, our conception of affordances as collective, socio-material achievements differs from these ideas on the question of the relationship between structural features of technologies and the agency of users. In contrast, seeing affordances as collective, socio-material achievements seeks to balance given features and social usage. Rather than favoring a constant state of becoming or fixed structures, it treats affordances as emerging in associations between humans and things. As such, they are neither objective not in a constant state of becoming, but grounded in action (Fairhurst & Putnam, 2004).

Furthermore, as both cases of realized or muted affordances indicate, affordances are not only relational phenomena but constitute jointly accomplished prefigurations. In contrast to the implicit solipsism found in much of the research on affordances, which presupposes a dyadic tie between a user and a tool, the configuration of these enablements usually necessitates a collective of human actors and technology (Kaptelinin & Nardi, 2006). Whereas a relational perspective can still maintain two spheres consisting of an active social world and a passive material world that are connected and, at the same time, separated by human-made and machine-made connections, in a collectivist perspective it becomes difficult to draw a clear boundary between acting agents and nonacting things. Affordances are made to function in an entanglement that is inextricably social–material in character (Hodder, 2012). The conventions and expectations evolving around the commodification of technologies are typically formed within such mutual catenation of objects and people (Faraj & Azad, 2012; Fayard & Weeks, 2014).

It is important to note that even though affordances are collective achievements this does not imply equality, neither in terms of agency nor empowerment. Next to disparities between humans and machines, there are also distinct differences among people and their capabilities to manipulate available and manageable companions, both human and technological. For instance, designers and users differ in their capabilities to define the functional range of a device and also in their capabilities to imagine alternative configurations (Neff et al., 2012). Without a doubt, given the growing intersection of audiences and production, which Bruns (2008) called “produsage” (p. 33), such distinctions are far from straightforward and cross the corporate and the domestic as well as expert and lay circles. Such endeavors, however, must not obscure the at times significant power divides in access, ownership, financial resources, and political influence, especially when it comes to an industry pushing smart technologies where users might feel locked inside copious infrastructures. Despite their plasticity, affordances are there invoked as “default options” or “constraints,” and this rhetoric influences how users account for the material settings and their own capabilities to make decisions and express themselves. In return, imputing affordances can also help us to recognize such strategies of de-emphasizing potential enablements and to value instead oppositional tactics that try to challenge dominant directions for use (Shaw, 2015).

As regards agency, the idea of collectively achieved affordances once again does not need to level potential differences in the ability to create an effect. In general, practices rely on material utensils whereas materiality is, in turn, constructed, made functional, repurposed, or repaired in social activity (Jarzabkowski & Pinch, 2013). Explaining that connection, Gherardi (2012) posits that “within a practice, meaning and matter, the social and the technological, are inseparable and they do not have inherently determinate boundaries and properties; rather, they are constituted as relational effects performed in a texture of situated practices” (p. 40). This allows us to reconsider the distinction between social agency and machinations because it moves us away from detecting deficits in the cognitive abilities and interactional agency of technology to assess how humans and machines come to cope with the contingency and ambivalence of activity. Therefore, it takes our inquiry beyond the problem of identifying action possibilities toward a more holistic understanding of the socio-material disposition to act at all.

Agency in Action

In the three human-robot encounters we have described, the attribution of intentions, which is often taken as the key element for defining communication, did not depend on autonomous systems processing symbols. Rather, affordances were enacted or not based on social interactions, not on mental plans and semantic representations. Apparently, the human participants were given to ascribing intentions to machines when they displayed a sort of autonomous action (Agre & Chapman, 1990; Meister, 2014). Nevertheless, it seems that an inquiry into affordances cannot escape the question of what kinds of agency are associated with what kinds of actors. Neff and colleagues (2012) even argue that “instead of looking for affordances then we would be analyzing the ways in which reality is generated through various types of agencies and forms of action” (p. 310). One way to address this matter is to search for abstract criteria to specify agency and to map such axioms onto the properties of a biological being or technical system. In this respect, Emirbayer and Mische (1998) have usefully proposed three dimensions that characterize agency as a special sort of social engagement. It combines an iterative element of routine habits, a projective element of imagining future trajectories, and an evaluative element of making normative judgments among alternatives in evolving situations. Such a blueprint of full agency can be used to dispute or affirm the status of technologies as agents that are more or less equal to humans (Hayles, 1999; Pickering, 1995). Challenging this separation, Neff et al. (2012) speak of a technical agency, “technical in two senses, as in the agency that is possible by systems of technology, and as a limiting description of that agency or latent agency, a not-quite, but ‘technically’, agency” (p. 301). Technical agency then would always only be a surrogate set against the full agency of humans.

Yet, essentialist definitions of human agency versus technical agency fail to account for the association of people and robots in the collective accomplishment of affordances. In these instances, the seemingly intelligently behaving devices assume the role of agents and partake, together with human agents, in shared situated actions. Presumably, smart technologies are designed around this orchestrated activity and present themselves as personalized communicators like Siri, Alexa, or Jibo, who directly address their interlocutors (Gehl & Bakardjieva, 2017; Guzman, 2018). In contrast to the well-known experiments with conversational assistants passing off as human beings, these interactions are not framed around communicative deceit or mimicry but make up a growing portion of our everyday communicative commitments. Therefore, human and technological communicators take part in what Suchman (2007) has referred to as “a real-world activity in which we make use of language to delineate the collective relevance of our shared environment” (p. 178).

In such encounters, it does not seem helpful to mobilize anthropocentric distinctions to capture how sociality is jointly performed. Instead of asking about the qualities that characterize agents or making authoritative statements about the nature of agency, we can discover capabilities to act while studying how affordances are configured in a socio-material setting. After all, “agency refers not to the intentions people have in doing things but to their capability of doing those things in the first place,” as Giddens (1984, p. 9) remarked.

Being agnostic to a priori categorization does not mean to level out the differences between human agents and technical agents. In contrast to assuming a basic symmetry between the agency of humans and objects where agency mostly becomes a matter of attribution, we should attend more to the continuum of actionability established in human–robot communication. This extends our view on the skills possessed by machines and it, too, blurs the lines around humanness when defined in terms of intelligible, reasoned action. For example, we can become aware of bot-like behavior among users who execute monotonous duties such as deleting off-limit comments on Facebook or reverting acts of vandalism in Wikipedia (Bucher, 2014; Pentzold, 2018). Indeed, many activities in social media such as liking, linking, or friending can equally be done by technical agents, without the need to imagine them as ontologically equivalent to humans, as proposed by some strands of Actor-Network-Theory (Bollmer & Rodley, 2017).

Shapes of Coordinated Activities

By attending to the performance and expression of agency rather than treating agency as an innate property, we admittedly avoid ontological questions about the nature of a sentient and conscious machine and instead observe what robots can and cannot do. In so doing, we do not trace back their proficiency to a technical efficiency resting on high-capacity processors and self-learning algorithms but to socio-material communication in whose course of interaction the abilities to respond to indexical situations and to act intelligibly are enacted, ascribed, perceived, and evaluated.

Human and machine actions can take on different shapes. To this end, we distinguish polimorphic actions from mimeomorphic actions. Justifying this distinction, Collins and Kusch (1998) expend considerable effort in explaining the difference between people and machines. While people, they argue, have an understanding of society, machines do not. Yet, even without following them further into their inquiry of intentional states of mind and socialization processes, we can still employ their distinction because it stresses the execution of sensible performance, that is, what observers perceive agents as doing. Thus, “mimeomorphic actions are actions that we either seek to or are content to carry out in pretty much the same way, in terms of behavior, on different occasions” (Collins & Kusch, 1998, p. 31). Polimorphic actions, in turn, “are characterized by essential variability in the behaviors with which they are executed,” Collins and Kusch (1998, p. 33) state.

To be sure, these two broad categories are not contingent on the eye of the beholder. Rather, many-shaped and same-shaped activities are constituted in the interplay of different agents and in their respective abilities to make interventions into ongoing social and technical processes. This can give rise to mimeomorphic behaviors, for instance, when agents are asked to select from a limited set of menu options or are given strict instructions that try to limit their actionable repertoire, as we find in some sports, for example. There, as in much routine procedures, people might not only act based on rational considerations but in accordance with habitual dispositions and affects.

Polimorphic engagement, in turn, may be facilitated when a palette of activities is acceptable and can be realized. The moment we set aside the question of whether such variability must conceptually be linked to intentions and whether shown behaviors indicate mental states, we are able to determine the multifarious trajectories of interaction that evolve in human–robot communication. Therefore, they can bring about changes in the interpretation of artifacts, people, and their relations; they can adapt the meaning and use of things; and they also can help to reinvent and transform the entanglement of materiality and sociality (Siles & Boczkowski, 2012).

Affordances are constituted in the space between these intertwined activities, where some seem to repeat previous doings and others take on new or unexpected forms. It is important to note that these associations are not solely based on visual perception, even though such a scopic understanding of affordances has been prevalent in most of the literature on the concept ever since it emerged (Gibson, 1979; Norman, 1988). While it emphasizes human rationality, it overlooks alternative sense-making processes that involve multiple agents, encompass attitudes and anticipations, and leave room for misperceptions (MacVeigh-Schultz & Baym, 2015). Consequently, Nagy and Neff (2015) claim that affordances transcend distinctions between right or wrong interpretations and uses. They thus introduce the idea of imagined affordances that “emerge between users’ perceptions, attitudes, and expectations; between the intentions and perceptions of designers” (p. 5).

One way to determine how technological and human agents become entangled is offered by Leonardi (2012). He uses the metaphor of “imbrication” (p. 36) to grasp the interconnection of diverse agents and their actions; “they have distinct contours and that through their imbrication they come to form an integrated organizational structure” (p. 37). In a more detailed approach, Collins and Kusch (1998) propose thinking of these catenations as “action cascades” (p. 56). There, polimorphic actions and mimeomorphic actions are vertically coordinated in temporal sequences: an activity can provide the condition for other actions and rejoinders, which again might instigate subsequent reactions and follow-up activities. In Collins and Kusch’s argument, such hierarchical cascades build on unequal competences to organize the order of actions and subactions embedded in them. It, therefore, also contains proxy actions, which are done by agents on behalf of others.

Undoubtedly, these relations of delegation are found in human–robot communication, but we also came across moments that required a more horizontal coordination of activities, a form that Collins and Kusch (1998) call “action conjugations” (p. 64). To execute their practices, human and machine agents had to adjust and modulate their activities based on reciprocal perceptiveness. This was not a smooth process; it usually involved disruption, misbehavior, and breakdown. Yet, especially these instances underscore the challenging task of accomplishing an interplay between humans and robots to generate meaning.

Our cases of human–robot communication evolved along these chains of interlocking actions. They required interactional coordination so to sustain human attention to the robot and to cultivate the idea that it plays an active part in the social situation (Alač et al., 2011). The routines were maintained or disturbed by the people’s and the robot’s capabilities to act and react. Thus, the practices and material arrangements provided a context for each other. Just as humans sometimes vary their positions, involvements, and commitments, the smart companions, too, could take on different roles: in the ignorance game, the robot formed part of what Shove (2017) has described as an infrastructural relation where things “are necessary for the conduct of a practice, but are not engaged with directly” (p. 156). In the queuing scene and the doppelganger experiment, the robot was instead treated as a device that was directly mobilized and actively manipulated and which again intervened in how the situation proceeded.

Conclusion

In the extensive literature on the interplay of humans and technology, agency and affordances are treated as distinct aspects of social-material activities. Usually, affordances are placed on the technological side while agency is reserved for the people imagining, building, and commodifying tools and instruments. Indeed, the notion of affordances has often been employed to open up these spheres and stress the relationality of human abilities to act and technological resources.

The communicative human-robot encounters we observed further challenge a straightforward separation into active use and passive usability. Interrogating human–robot communication makes us aware of the “multiple forms of accountability” (Suchman, 2007, p. 203) that accrue in a bidirectional socio-material relationship. Accountability, in this respect, is not primarily about liability or responsibility. From an ethnomethodological perspective, such accountability more fundamentally refers to the expressed intelligibility of actions and the work that goes into connecting and interlacing activities (Garfinkel, 1967).

Affordances are collectively achieved in these concerted efforts. They are contingent dimensions of a situation, not inherent tokens of physical objects. Postponing the unresolved problem on whether robots should be seen as social actors in their own right and whether there are several sorts of material agency and social agency, the lived practice of human–robot communication we have discussed in this article actually happens in settings where agency is enacted by multiple agents (Kaptelinin, Kiselev, Loutfi, & Hellström, 2017; Šabanović, 2010). This agency rests on the capability to engage in intelligible encounters; it builds on purposive activities even though they might only perform a limited repertoire of tasks. Therefore, there are pronounced differences in the agency that are accomplished through the uneven ability and license to take up monotonous as well as creative activities.

Our exercise of treating affordances as collective achievements represents a move away from the question of what technologies allow or prohibit people to do. In the face of “socio-technical systems that act with and sometimes without or despite us,” as Nagy and Neff (2015, p. 2) write, we rather place the emphasis on the socio-material co-creation of multiple prefigurations. The cases suggest that the affordances of digital technology are not enabled in smoothly planned processes or through rational action alone. Instead, the sites involved a considerable amount of negotiation and problematization of what people and technology were able to sense, do, and decide in a given situation and with respect to certain tasks.

Finally, whatever we can make of claims about the reality and future of AI, starting from how agency is performed and expressed should make us cautious in the face of an attempt to state and ultimately overstep a threshold of true AI. Beyond the ethical disputes about the simulation of sociality and sense of personhood and the ambitious quest to pave the way for autonomous, self-improving machines, this skepticism toward any watershed moment for machine intelligence is rooted in our observations of mundane human–robot communication. There, it proved impractical and unhelpful to start from intellectual properties. Instead, we attended to the ways in which activities executed by robots and people as well as those left undone by them were bundled together. The human and the machine participants in the situations could share in more or less mechanistic or creative activities. In these intersections, activities were rendered intelligible so as to engage in the “continuous flow of conduct” (Giddens, 1984, p. 3) and to generate and circulate meaning. This sets aside the dichotomous conception of options versus constraints. It even goes against the idea that affordances either exist or are absent because they do not materialize as technological functions but emerge at the practical intersection of people and technologies.