Drawn into the future: The epistemic work of visual scenarios in the configuration of human–robot encounters

Abstract

This article looks at artistic impressions of future robotics and considers how they inspire research into human–machine interaction. Our analysis of visual scientific practices and the epistemic ramifications of these speculative drawings emerges from a long-term participant observation study in a multi-disciplinary project on smart and autonomous technologies in public spaces. We discuss the design, appropriation and modulation of visual scenarios, and scrutinize how these diegetic futurescapes are imaginatively engaging and suggestive of scientific progress and experimentation. The article argues that the future-oriented scenes defy common notions of post hoc scientific representations. Instead, they are ex ante presentations of the ambition to imagine human–machine relations in the future and to draw the large-scale research venture together. The register of evaluation thereby shifts from aesthetic criteria to scientific parameters. More than just visual tokens, the scenarios became a catalyst for collaboration.

Engaging with the future usually implicates the imagination. Even more so, foreseeing things to come or envisioning future prospects entails an element of imagery. Different ways of visualizing the future have already received considerable attention, including a diverse array of instantiations like diagrammatic displays of weather maps, futuristic architectural designs or modelled trajectories which aim at precise predictions (Pentzold & Fechner, 2020). These materialized ‘images of the future’, as Polak (1961) once named such prescient visions that carry collective orientations, differ in terms of concreteness and prognostic ambition, and range from reverie to prudent forecast. In spite of the cultural significance of such images, studies that focus on the impact of imagination on technological development and scientific projects tend to overlook its genuinely visual element (Franklin, 2008; Sobchack, 1987).

However, more than merely depicting possible worlds, images can help us grapple with contingent cultural issues, environmental effects and political consequences that are brought about by seminal innovations around astronautics, autonomous driving, nano particles or social robotics. In these areas, science inquiry and engineering conflate. In this ‘technoscience’, as Forman (2007: 5) puts it, the primacy of science as a source of technological development is swapped so that technology comes to denote science, a shift which inevitably brings up considerations of practical purpose and functionality.

Parallel to this inversion, researchers have broadened their view from a sociology of scientific knowledge to the social study of technology. It is guided by the assumption that science and technology are both entangled and socially co-produced. Thus, Jasanoff (2004: 2–3) postulates:

Scientific knowledge . . . embeds and is embedded in social practices, identities, norms, conventions, discourses, instruments, and institutions – in short, in all the building blocks of what we term the social. The same can be said even more forcefully of technology.

According to Woolgar (1991: 42), this underscores the ‘representation and constitution of abilities and capacities of entities, some of which become endowed with the status of “technology”’. The associated epistemic work, he adds, can happen in texts, academic discourse, or processes of interpretation.

Against this background, the guiding research question of our study is how forward-looking creative presentations come to feature in technoscientific visual practice. We ask: What is the epistemic significance and scientific purchase of images in configuring research around emerging technologies? To address this question, we look at speculative material visions of protean robotic technologies and analyse how they come to inspire scientific discourse around human–machine interaction. This requires dealing with the interplay of forms of visualization and fictionalization, and how they are implicated in science inquiry and technological development.

We proceed as follows: firstly, we conceptualize visual scenarios as diegetic futurescapes that are imaginatively engaging and epistemically relevant. We draw on interrogations of image practices in science and technology which we link to efforts that promote design fictions as a conjunction of imagination, prevision and creation. The scenarios discussed here pertain to prototypical technologies that form meaningful, functional and useful components of the environments in which they are placed. As such, these visuals come to resemble design fictions that produce what Dourish and Bell (2014: 770) describe as an ‘imaginative and speculative figuring of a world’ that brings a potential future into being before it is implemented. Arguably, besides films and written fiction, images offer a semiotic repertoire which can help visualize ‘diegetic prototypes’ that craft a coherent setting around an imagined technology’s plausible use and viability (Kirby, 2010: 41). Diegetic prototypes thus construe ‘“pre-product placements” for technologies that do not yet exist’ (p. 46).

From this conceptual background, we secondly look at how visual scenarios as diegetic futurescapes have the ability to anticipate human–robot encounters. The visuals we consider show highly artificial scenes featuring interactions between humans and embodied digital technologies (EDTs) like self-driving cars, robot companions or wearable exoskeletons. As EDTs, the technologies are autonomously interacting with humans within shared spaces, for instance, in warehouse logistics, urban road traffic, industrial production or health care; some are even connected to human bodies, such as digitally augmented protheses (Rahwan et al., 2019). The smooth coordination between artificial and living bodies requires joint action, the monitoring of intentions and prediction of movements – topics that form the core of an ongoing large-scale research exercise from which our investigation emerges.

The long-term participant observation was made possible by our involvement in a German Collaborative Research Centre (CRC). The empirical material that we have collected since its launch in early 2019 consists of field notes that record formal meetings and informal conversations, and is complemented with messages, screen shots and photographs, as well as documents. They were analysed following the coding steps offered by Grounded Theory. Zooming in on the work of scientists and engineers at the centre, we particularly scrutinize the visuals that were introduced as catchy illustrations for science communication materials such as public leaflets and websites but that became an instrument both for imagining human–machine relations in the future and for drawing the large-scale research venture together. So we argue that, in the emerging scientific inquiry entangled with practical engineering, visual scenarios were equally inspirational and instrumental. On the one hand, they came in handy as a way of envisioning the prospective use value and achievement of the project. On the other, they were employed to steer the collective research enterprise. As such, they were treated as design fictions like they are used in other areas of technological imagination too (Blythe and Encinas, 2018).

The study deepens our understanding of the productive conjunction between the articulation of science visions and the organization of science collaboration (Calcagno-Tristant, 2003; Gross and Harmon, 2013; Hüppauf and Weingart, 2012). It shows how drawn images come to assume an anticipatory significance by pulling the future towards the present. Visual scenarios are suggestive of scientific progress and experimentation. They invite us to appreciate the performativity of visuals as they guide concrete research actions and strategies. The examples we consider were not theory-laden representations which are often the focus of research but creative construals that nevertheless prompted the scientists’ and engineers’ curiosity.

Technoscience Visions: Conceptualizing Visual Scenarios

Images are crucial tools for making observations, communicating findings or supporting a statement. Beyond illustrating what is already known, visuals form part of the apparatus through which expertise and technological innovations are produced and contested (Pauwels, 2005). They are not illustrations which come after the fact and hence do not serve a ‘mechanical objectivity’, in Daston and Galison’s (2010: 115) terminology, an objectivity that is separated from more subjective or artistic imaginations. Looking at the area of nanotechnology, Daston and Galison find images which do not copy existing micro-scale constructions but form a chief element of their realization: ‘nanographers want images to engineer things’, they conclude (p. 385). Images are therefore destined to entice scientifically, technologically, and perhaps even entrepreneurially by means of ‘deliberate aestheticization’ (p. 402), for example through proportions and colour effects. They serve epistemic and practical purposes as they animate the production of knowledge and the construction of technology.

Nonetheless, the imaginative capacity of images is often side-lined (Wise, 2006). Instead of being a repetitive copy of insights or facts, they also have an evidence-producing function. This complicates the notion of representation because the images used in science do not necessarily require a referent in order to become a part of epistemic practices. They are, Daston and Galison (2010: 382) posit, presentations rather than re-presentations, ‘images in which the making is the seeing’. This transition from images-as-evidence to images-as-tool also involves seeing them ‘not as competing with art or even employing art but positioned as art itself’ (p. 385). However, only few analyses of artful scientific presentations consider how their vital entanglement presupposes the powers of the imagination, and these analyses are unaware of visuals that move from reproduction to artistic impression as they are also employed in the area of design fiction (for exceptions, see Frigg and Hunter, 2010; Jones and Galison, 1998; Kemp, 2000). What the analyses lack is a more detailed examination of the actual usage of imaginative images in science and of the ways in which visual presentations and science research interweave.

Science images and future imaginaries

Science is an inherently forward-looking affair driven by expectations and claims about the future. Its outlook is not only promoted by futurologists and forecasting experts but is the perspective taken by most of the academic community. In looking into possibilities and developments, science has become a formidable site for producing anticipatory knowledge that manifests itself in technological innovations. This is not a straightforward process in which scientists foresee upcoming developments and issues, and find solutions to mitigate them. The anticipations are neither future-proof nor infallible. In fact, technoscientific endeavours very much rely on trial and error in which failures and misguided previsions direct inquiry and development as well (Firestein, 2015).

The kinds of prevision that undergird much of the current engagement with the future in science and technology rely on visualizations of all sorts. These images of the future are, as Konrad et al. (2016: 475) explain, ‘shared assumptions through which the future is made concrete in discourse and presentations, enabling actors in the present to anticipate, act, and prepare’. Likewise, Kinsley (2010: 2771) notes that ‘practices to “make futures present” often yield discursive and material products, in the form of reports, stories, and, of particular interest here, images.’ These images not only serve illustrative functions but are viable substitutes for physical designs. Moreover, visual practices in scientific inquiry entail aspects of fantasy and speculation, for instance when looking ahead towards potential futures made possible by scientific progress, contextualizing a finding and drawing out its corollaries, or envisaging the hopes and fears that surround an emerging innovation. But the small number of publications that consider the use of fictional artworks for the forward-looking orientation of science commonly assume their influence to be indirect and dispersed. These artworks are hence believed to promote a climate of expectation or an atmosphere amenable to the construction and diffusion of a technology, yet their direct, palpable effects on a research venture are a more minor topic (Menadue and Cheer, 2017; Olson, 2015).

Images are not only a visual demonstrator for concrete technologies. They are also the visual substrate of more widespread social imaginaries in which collective practices and common understandings come together. Social imaginaries prefigure how people view their sociocultural belonging and place in the world. They are not confined to a particular sphere, but are concerned with the presuppositions, conditions and sensitivities prevalent in any given social sector. Consequently, Marcus (1995) argues that the socially and culturally embedded imaginaries shared by science investigators are reflective of their positionings and practices as well as being prescient of technoscientific possibilities. Following Marcus’s argument, the future figures here as a ‘cautiously imagined emergent future, filled with volatility, and uncertainty, but in which faith in practices of technoscience become even more complex and interestingly constructed’ (p. 4). In this vein, Fujimura (2003: 192) stresses that future imaginaries are not ‘mere fantasies’ but visions that do more than redirect scholarly outlooks. Instead, they also reverberate in other social arenas. Since the socio-technical imaginaries join people and funds with political projects and public interests, they are contentious. They compete with alternative schemes over the opportunity to steer technological imperatives, direct government rationalities and secure the material commitments needed to turn a vision into reality (Jasanoff, 2015). This means that fictional images furnish a useful visual repertoire which allows people to imagine themselves in the diegetic reality of an envisioned future. Further, it also enables them to conjure up rival projections.

Diegetic prototypes and design fictions

Visuals are key to articulating design fictions which craft a non-existent environment for a specific innovation. In their peculiar mode of fictionalization, they devise a speculative story world that envisions a prospective future where people, technology and practices intertwine. They operate in the liminal space between the state-of-the-art and frontiers of science and engineering. The technologies constructed from within these fictional worlds are props that Kirby (2010: 41) calls ‘diegetic prototypes’. In his analysis, such prototypes may inhabit the scripted worlds of sci-fi films and television series, where they are fully functional. So, in Lee’s (2019) terms, naturalizing diegetic prototypes in a story means crafting a fictional world in which the limitations of reality that might impede an innovation are replaced by an environment where such technology becomes possible. A setting with its own built-in logics could then inspire a kind of ‘fictional sublimination’, she adds, from which new scientific theories spring up and can be elaborated within a story’s universe (p. 705).

The capacity of narratives to establish such a diegetic reality has particularly been embraced in design fictions. As Lindley and Coulton (2015: 210) state, ‘a design fiction is (1) something that creates a story world, (2) has something being prototyped within that story world, (3) does so in order to create a discursive space.’ It stipulates the existence of an innovation, crafts a possible context and invites people to relate to a future via its proxy presentation. Whereas diegetic prototypes are placed in a fictional world, design fictions turn this relationship around and ask people to envisage a fictional world out of a speculative technology (Kirby, 2019).

Since design fictions try to figure settings which are believable and relatable, they have to reconcile the fanciful with the credible. Design fictions, Lindley et al. (2014) remind us, are not fictional in the sense of being ‘unreal’ or ‘made up’. This runs against Sterling’s (2012) definition of design fiction as the ‘deliberate use of diegetic prototypes to suspend disbelief about change’ and stresses that design fictions are about the creation of belief. Hence, Blythe and Encinas (2018: 87) comment: ‘It does not require us to become temporarily credulous, rather we are invited to accept a set of axioms or rules and enter a game space.’

That design fictions are imaginative and populated by non-existing props does not mean that they must necessarily be anticipatory fictions in the sense of being future-oriented. Diegetic prototypes are not required to have concrete predictive value; they may also put forward worlds distinctively different from ours that provoke critical questions about the social and political context of a technology and its ethical ramifications. At best, design fictions that incorporate diegetic prototypes animate discussions around the probable, preferred or potential properties and contingencies of a technology (Auger, 2013; Bleecker, 2010).

Visual scenarios

Design fictions hover between ideas and things; they are materialized thought experiments that can take a variety of formats such as design workbooks, speculative stories, comics, cultural probes, fictional constructs and prototypes. This includes visuals from sci-fi art and futuristic fantasies because ‘even static images then might convey narratives of one kind or another’, as Blythe and Encinas (2018: 65) note. Likewise, Kirby (2010) underlines that pictures are a pivotal device for showing how an innovation would function and why it is needed. Although static images lack the cinematic means of threading a narrative around speculative conceptualizations, they too create visions of technological possibilities and open up people’s imaginations.

We call these depictions visual scenarios which hover between the expectable and the wildcard. In principle, scenarios are ‘stories about people and their activities’, Carroll (2000: 46) states. They position prototypical technologies in an evocative socio-material surrounding, thus visualizing a world in which they have already been realized. Being a design method, such scenarios allow us to project futures which go beyond the technological potentials and social needs available at the time of their creation in order to explore more futuristic visions. As such, they resemble a kind of fictional prototyping that employs narratives to imagine paths for technology development (Johnson, 2011). In the context of human–robot interaction, this is for instance used to extrapolate the design and use of an empathetic care robot (Stahl et al., 2014), or envisage the long-term consequences of robots for civil protection (Carlsen et al., 2014). Like visual scenarios, this sort of fictional prototyping uses a story world to explore the implications of innovations. Different from design fictions, fictional prototyping is a more confined foresight technique, a ‘mixture of logic and fiction’ (Grimshaw and Burgess, 2014: 5). As a combination of both, visual scenarios may have little connection to the laws of physics or current technological possibilities but can also be based on science facts and sets of rules that guide a projection.

The forward-looking visual scenarios we refer to are deployed in settings where they chart a future so that it can be discussed as if it is the present. One of these settings is nanotechnology. Nano images not only magnify invisible small particles but, in addition to what is existent but optically unseeable, they also address what may lie ahead. In Ruivenkamp and Rip’s (2014: 177) account, such hybrid nano images ‘combine elements of traditional scientific representation . . . along with elements that anticipate what an invisible world might look like’. Hybrid nano images, or ‘pictions’ (p. 180), are stand-ins for both the promises of nanotechnology and concerns about its impact. Other such areas of engagement with visual scenarios can, for example, be found in aviation (Van Riper, 2004), factories (Marchand, 1992) or domestic life (Henthorn, 2006).

The plurality of visions often implies competition, with one vision being promoted at the expense of others. Visual scenarios cannot be neutral illustrations of an innovation, and they are not agnostic about the envisioned sphere of usage and engagement. Indeed, their epistemic potential hinges less on concision or elaborateness and more on the assessment and evaluative framing which their depictions seek to transport. Relatedly, Bleecker (2010) highlights how effective design fictions can create a compelling possible world which is better able to communicate the idea around a technology than an engineered prototype. In the same vein, current practices of material speculative design are eschewing solutionist pretensions and try not to maintain prevailing social and technological realities (Dunne and Raby, 2013). This decouples speculative design from its narrow focus on functionality and the limitations of seeing it as a ‘panacea for whatever ails’, as Yelavich (2014: 43) decried: ‘Politically neutral, never demanding, the popular perception of design threatens to override criticality.’ At the same time, even speculative design cannot escape being propositional and prospective, which means that all acts of design are themselves ‘small acts of future-making’ (Blauvelt, 2019: 90). Even when it agrees that not every issue can be framed as a problem to be solved by design and seeks to foster material speculations that are not an extrapolation from the present, this practice privileges some futures at the expense of others. It can ‘defuture’, to use Fry’s (1999) term, and thus design away some potential avenues of technological development and the forms of social life this would entail. Moreover, speculative design has been criticized for its inherent elitism; an ‘academic, gallery-based endeavor’ (Ryan, 2019: 15) that does not take responsibility for the political and social consequences of its designs and has little connection to the people affected by design choices.

Note that within future studies, there are many methods of scenario-building which do not necessarily involve the use of visuals. In this context, the notion of visualization takes on a different meaning that commonly refers to a particular procedure of meditation and intuition which is intended to provoke genuinely new ideas about the future. What this approach has in common with the visual scenarios discussed here is the creative impulse to come up with outlooks ‘that are analytically coherent and imaginatively engaging’ (Bishop et al., 2007: 5). It too links to table-top scenarios usually set up to simulate emergency situations and anticipate crises. While we concentrate on prospective scenarios, the same methods can also be used for retrospection. For example, because we know very little about dinosaurs, their skin colour or herd behaviour, we use imaginative dioramas to bring them to life (Noble, 2016).

Visual scenarios and the meaningful discourses they engender ultimately point us towards the inextricable conflation of technology with cultural reflections. Usually, these proceed chronologically: design problems first, commodification second. Part of the epistemic work inspired by visual scenarios is grounded in the inversion of this rationale. Therefore, questions of benevolence, utility or hazards are, as Dourish and Bell (2014: 777) postulate, ‘ones that arise not in the deployment of technologies but in the imagining of them – an imagining that arises before design’. In a sense, visual scenarios draw on questions which are otherwise usually delayed until after a marketable product becomes available. Visual scenarios are thus performative, meaning future-shaping, in that they pull tomorrow’s problems into today’s science and design.

Setting and Approach: Visual Scenarios of Human–Robot Encounters

Machine imitations of living beings and the automation of cognitive processes and physical actions are prime objects of today’s sociocultural imagination. Robotics are as much an engineering project as a cultural prospect. Therefore, the ‘robotic imaginary’, as Rhee (2018: 6) explicates, ‘does not abide by disciplinary boundaries, but instead announces . . . the inextricable entanglement of “technology” and “culture”’. Imagining the future of human–robot encounters has political, ethical and technological stakes. They surface in diverse places such as academic writing, journalism, policy, promotional texts, consumer ads, literature, films and art (Cave et al., 2020; Social Robot Futures, 2022). This imaginative thinking, happening both in fictional as well as non-fictional contexts, around smart machine companions stretches from antiquity to the latest visions of autonomous weaponry or robotic lovers (Campbell, 2021; Del Monte, 2018). There, images are a prominent device lending visual plausibility to potential robotic futures.

The study into the epistemic significance and scientific purchase of images in configuring research around emerging technologies offers reflections on this nexus of concerns and moves across different fields of research which contribute to advancing the interaction between humans and robotic machines. The analysis emerges from our participant observation in a four-year Collaborative Research Centre (CRC) which is funded by the German Research Foundation (DFG) and was launched in 2019. The multi-disciplinary endeavour, based mostly in psychology and engineering, brings together more than 100 researchers at all career stages working across 20 sub-projects to implement EDTs in public spaces. We are both active members of the CRC and thus colleagues of the researchers and the graphic designer working on and with the visual scenarios, where we manage science communications and act as participant observers who keep track of the epistemic work being conducted at the CRC’s communicative interfaces.

The scenarios we look at originated from the collaboration between researchers and a graphic designer, including the CRC’s public relations project. The visuals formed part of the centre’s outreach initiatives and featured on leaflets and the website, as well as in other science communication formats. The different scenes depicted in the drawings could be shown individually, which made the illustrations even more versatile. While they are out of touch with the current state of the art in engineering and implementation, the images give visual form to expectations and aspirations around protean EDTs. The images were not originally intended to be design fictions but became interpreted as such.

To call the images unrealistic misses the mark – they do not aim to be plausible depictions but are reminiscent of sci-fi comics and animated sitcoms like The Jetsons or Futurama. Their composition references picture puzzles or ‘wimmelbooks’ (Rémi, 2018: 158). These types of books and puzzles are typically made for children and invite readers to zoom in on one of the many individual scenes that are assembled in one of the panoramas, whether it is the road junction (Figure 1), the city park (Figure 2) or the market (Figure 3). All three drawings take an isometric view of the entire tableau. Their palette of mostly grey, blueish and turquoise shades complies with the CRC’s overall colour scheme.

Figure 1.

The visual scenario ‘Road junction’. Credits: Jacob Müller/TU Chemnitz.

Figure 2.

The visual scenario ‘City park’. Credits: Jacob Müller/TU Chemnitz.

Figure 3.

The visual scenario ‘Market’. Credits: Jacob Müller/TU Chemnitz.

Our approach is inspired by Suchman’s (2011) call for a design anthropology that interrogates how imaginaries, material prototyping and inquiry are interlinked in a creative, future-oriented process. It is also based on the ethnographic explorations of science practices and their association with presentational forms originating from science and technology studies (Coopmans et al., 2014). What we take from these contributions is an interest in the situations where visual scenarios are becoming an element of concern. The images evoked responses with which the scientists and engineers positioned themselves and their work towards the imagined scenes, and in which they came to assess the images’ utility and plausibility. Stimulated by the images, the CRC members made considerations about the social requirements for EDTs and their technological possibilities.

We scrutinized the images at different situations during the CRC’s ongoing operations. This included appointments of members of the executive board and science communication staff with the graphic designer tasked with developing the CRC’s signature style. The three provisional visual scenarios we analyse in this article were the outcome of these meetings. In fact, their refinement turned into an open-ended process: each time the images featured in CRC meetings and on CRC media, staff and faculty commented on their ingenuity, appearance or plausibility. Many of them not only suggested changes to the visual displays but, in a reflexive twist, also to the prospective research endeavours of the CRC. We also used this work in progress in the CRC’s graduate workshops, which focused on developing and assessing robotic futures.

Our fieldnotes include minutes, transcripts and protocols from more than three dozen project meetings, workshops, plenaries and conversations, a corpus of messages, screen shots of online meetings, photographs and written documents circulated among collaborators. The meetings were held in German and English between January 2019 and June 2020. We use verbatim quotes from two meetings with the permission of our colleagues but base our conceptualization on the entire material collected or generated in the field. Where we quote from German-language statements, the English translations are ours. In conceptualizing from these records, we are particularly interested in the material visual practices through which the CRC team members articulated their views towards prospective EDT implementations where humans and robots share a common public space, sense each other and move in concert. The assortment of notes and documents was imported into the qualitative data analysis software MAXQDA and conceptualized along the procedures offered by Grounded Theory (Glaser and Strauss, 1967). In a team of two, all materials were labelled with codes and the proposed codes were constantly compared and merged into more general concepts. Divergent codes and ambiguities were discussed in regular meetings. As a result, the reflections we offer emulate empirically grounded concepts around their dual purpose as an inspiration for designing human–robot encounters and as an instrument for steering a science collaboration.

Reflections: Companions in the Making

The three visuals we focus on were deliberately aestheticized. They envision highly artificial scenes full of encounters between many different kinds of EDTs and humans from all walks of life. Since the CRC has a research focus on joint movements and smooth coordination, the visual scenarios represent what one might call ‘vehicular utopias’ (Evans, 1999: 99) which show people in motion. At times, people in the pictures are carried by locomotive robots, delegate menial tasks to them or treat them as partners in joint action. These encounters happen without noticeable obstacles or accidents; people and machines seem to get along well. Therefore, they resemble other visions like Weiser’s (1991) ‘Sal’ scenario of a life surrounded by ubiquitous smart technology that omits conflict. This in fact sets them apart from most science fiction since a narrative is usually predicated on some kind of struggle (Blythe and Encinas, 2018).

In the situations we were able to observe, the visual scenarios were treated, on the one hand, as a creative impulse that draws out visions of the protean interaction with robotic companions. On the other, they were recognized as consensual images which helped to draw the different projects and the people involved in them closer together. They thereby aligned these human companions in a scheme for the second four-year funding term offered by the DFG. In that sense, the images had both epistemic and administrative implications: they stimulated conversation around the plausibility and functionality of the scenarios, and they encouraged speculation on their fit with experience, plausible assumptions and current research. Further, they were used to draw out connections between the CRC’s sub-projects, to formulate ideas about how they could move forward in concert and how to succeed in the competition for further funding.

Drawing out a future of human–robot encounters

The forward-looking visualizations of human–robot encounters defy common notions of post-hoc representation. Apparently, they do not depict existing arrangements. Instead, they were taken to be ex ante presentations of the ambitions around crafting human–machine relations. The presentative relationship did not stem from a mimetic correspondence between the drawings and what could be seen in labs, testing sites or hybrid public spaces. There were artificial prostheses, self-driving cars or locomotive robots, yet they were constructed and tested in highly controlled settings, some of them requiring physical co-presence, some happening virtually. They only had a faint resemblance to the smooth interactions envisioned in the images. In the processes of engineering and experimentation, the EDTs rarely completed sequences of interactions autonomously as they were supervised and their operations were parsed into small units of movement and processing.

While the images do not mirror reality, CRC researchers welcomed them as an inspiration and associated the visuals with what they saw as a worthwhile future to which they hoped to contribute with their work. This means that the diegetic scenarios were treated as prototypically presentative, both of the possible interactions between people and intelligent machines as well as of potential paths for inquiry and engineering. As Daston and Galison (2010: 392) highlight, this shifts a representative understanding of depicting reality to an interventionist approach which seeks to affect it instead. In other words, the aim is to ‘intervene in the world as a way of establishing what we actually understand and therefore what really is out there’. The visual scenarios were useful not because of their explanatory value, which was absent, but because they were efficacious and allowed viewers to engage in the development of human–robot interaction.

Due to the inverse relationship between the presenting images and the presented scientific context, photographs were deemed insufficient and inadequate for transporting the CRC’s essence. Of course, shots of built environments, the labs or the investigators were available. However, researchers feared that the mission and general objective would become unrecognizable if they were represented photographically. The reason for this is that, although the formation of the CRC was motivated by the profound and far-reaching transformations brought about by driverless cars, autonomous drones and artificial prostheses, it mostly promotes basic research. The CRC is driven by its funding requirements, and its advances, in fields including sensomotorics, spatial coordination and intentionality, are subject to evaluation. Researchers at the CRC believe that this fundamental research is necessary for understanding and crafting the coexistence of humans and machines in public spaces. However, this conviction did not translate to cogent images which would communicate the CRC’s contribution to bringing about such a hybrid society. This was not so much a problem of the research appearing visually unremarkable. Instead, for its members, including the steering committee, the trouble of presenting what the centre should aspire to was that its projects – psychological experiments, computer modelling and mechanical calibration, for instance – were science in the making, and did not yet have any presentable results. For CRC members, photographs or images derived from actual research would thus have visually exacerbated the distance between their current work and the larger picture. Popular commercial examples like Aldebaran’s humanoid robot Nao or Sony’s robot dog Aibo, which could have served as a blueprint for applied imagery, were rejected for being at odds with the CRC’s vision of future human–robot encounters. This lack of demonstration and exemplification proved particularly problematic for a large-scale research scheme which was firmly oriented towards future technologies. It did not examine a functioning setting but claimed to be instrumental for realizing a future in which the use of embodied digital technologies in public spaces would be aligned with human needs and skills. Its rationale was predicated on the sociotechnological possibilities made available by scientific innovation in the near future.

Visual scenarios came in handy in a situation like this. They offered a playful look into the sort of future the CRC is geared towards. Moreover, they resonated with the researchers’ demand for neat and easy-to-communicate images which perform a particular task: namely, not to represent their day-to-day scientific activity or the imminent problem they sought to tackle but to create the larger vision to which they felt committed. Put differently, the visual value of the diegetic prototypes lies in their capacity to tinker with imaginative and highly fictional scenarios which picture the presence of what has not happened yet and might not happen at all.

The visual scenarios were not prescriptive in any way and did lack specificity. They did not present any kind of roadmap for technology and were not committed to an actionable set of tasks. Quite the opposite: they figuratively envisioned panoramas which encouraged the investigators to ponder and discuss what forms of scientific progress were necessary to realize a particular future. Like other visual or audiovisual scenarios, they were an ‘imaginative fabulation’, as Kinsley (2010: 2776) puts it in his analysis of commercial video visions, which produced anticipation without an obligation to deliver what was imagined. ‘They are both “fun” imaginative artefacts of entertainment’, adds Kinsey, ‘and serious investments in the attempt to make particular futures present.’ This underscores their ambivalent status – even though they were obviously fictional, they still attracted critical interest and were read as showcasing prospects of human–robot life which are not yet possible.

The visual scenarios perform futurity in a recursive process of inspiration and adaptation. While they were made to look nice and to be intuitive, CRC members nevertheless challenged their visual performance. Despite their unmistakably stylized and fictional character, the images were judged to be too realistic and accurate. These assessments were based both on the current state of research and the more or less expert opinions about existing and forthcoming robotic solutions. Consider the example of a PhD student commenting on an image of a robotic device about to clean up a dropped ice-cream cone in the market scenario. Assessing the realism of this scene in terms of her own experience, the PhD student noted that ‘it’s so realistic, I mean, something like this I think already exists’ (transcript II, line 1906). In another meeting, one of the principal investigators (PI) affirmed the plausibility of the smart trash can in the city park scenario by pointing to trends in his own research: ‘just think of maintenance tasks among other things, and policing’ (transcript I, line 615). The visuals were also challenged in terms of their functionality, the plausibility of the diegetic construal, and the negative implications if they were to be realized. In that respect, a PhD student made the following remark about the road junction scenario:

I have another question or remark regarding the sharing of the road because you have the bus driving on the road, you have the bicycle driver, and then you have this robot on the same lane. And I would guess that their speed is actually very different, so I’m not sure if this is the optimal solution unless you want the bus to go very slowly. (transcript II, lines 822–834)

The depictions of machines, humans and their interaction were adjusted in response to the engagement with the visual scenarios. For instance, the images were tweaked by exchanging an avatar, adding more detail or drawing figures more closely together. These changes can be grouped into adjustments driven by propositions regarding the proper functionality of a device, on the one hand, and adjustments driven by research considerations, on the other. Questioning the usability of the floating bus stop sign in the road junction scenario, a PhD student suggested: ‘maybe a good alternative for the stop sign for the bus would be that we could switch the drone to a projection on the street, like a light or something, that says, okay, here is the bus stop’ (transcript II, lines 1041–1050). Another pertinent point of criticism raised multiple times was that the people in the scenarios had no eyes, whereas both the robotic dog and its living counterpart did. This design choice became problematic since a number of subprojects explored the role of the gaze in coordinating humans and machines.

Drawing collaborative research together

Besides drawing human–robot encounters into the future, the scenarios also became an instrument for drawing together the large-scale venture of the CRC. They proved inspirational as well as integrative because they enabled the formulation of shared visions across the CRC. In addition to executing their planned projects, the PIs were also concerned with the centre’s future development and sustainability. Almost immediately after launching the CRC, they began outlining potential goals for the next round of funding. Within these preparations, the visual scenarios were employed to gauge the viability of future research topics. In the debates, researchers relied on the visual persuasiveness of the scenarios to advocate for a specific route that the joint venture should take in the future. The available three scenarios were scrutinized for their fit with existing projects and, vice versa, the CRC members assessed the cogency of individual projects in reference to their place in the images. This two-sided comparison and alignment required tweaks being applied to the visuals and could question a project’s relevance for the CRC. In effect, the formulation of research goals for a potential next round of funding went hand-in-hand with intense efforts to imagine a scenario charged to do many things: it should transport the key scientific and technological challenges the CRC would address next, the viability of the problems posed, and the benefits of tackling them.

The forward-looking images therefore had tangible consequences for the centre’s organizational structure and running operations. This involved clustering research projects and channelling seed funds in order to chart upcoming areas of concern, offer short-term grants and prepare the ground for repositioning the CRC in line with promising future research areas. As this large-scale endeavour was undergoing vertical integration, only a limited number of images were deemed to accurately encapsulate the CRC and its future. More than simply being ancillary visual tokens, the scenarios thus became a catalyst that helped the CRC define its profile. In discussions about the amplification and adjustment of the visuals, speculation functioned as a kind of boundary work which defined the future character of the joint enterprise and its margins (Gieryn, 1999). For example, tasked with connecting a particular sub-project to a scene found in the three scenarios, a member of the Executive Board exclaimed: ‘No, forget it, simply forget about it’ (transcript I, lines 1178–1180). He also rejected the possibility of simply adjusting one of the presentations and instead advised removing it from the picture entirely.

I say it again, we don’t care about [the] project . . . this doesn’t fit here, it will always be nonsense. We don’t build anything into the scenarios to make the project . . . fit into them. That has nothing to do with this. (transcript I, lines 1202–1208)

Similar dynamics of administrative closure have been observed in nano science, where images of nano-sized structures function as a tool for creative foresight which one might use to formulate material demands. ‘In depicting possible futures’, as Ruivenkamp and Rip (2014: 186) note, ‘there is no sharp boundary between images that articulate directions of further research or designs of molecular machines and images that present open-ended visions of new functionalities that might be achieved.’

For the CRC, this future orientation was closely aligned with its current activities. It sought to connect the dots from the envisioned future back to the present in order to properly set itself on track for its long-term trajectory. In this process, the diegetic settings which the scenarios encapsulated could eventually converge into a ‘projectory’, that is, in Messeri and Vertesi’s (2015: 56) account, a ‘material instantiation of the “imaginary,” made concrete and traceable through circulated documents that codify a community’s orientation toward future technological states’. The projectory in a research endeavour has timelines, milestones and deliverables relating to target results and innovations. In no way do these plans protect against failure. They are simply a framework for setting immediate project tasks and specifying long-term commitments. The visual scenarios were useful devices because they provided both the conceptions of the future and the ‘plot patterns’ (Edelman, 1995) into which the joint endeavours could be translated. The conceptions included some constellations of humans and robots, and left out others. They opened up varied yet finite scenes which featured a limited set of technologies and use cases. They thus restricted expectations of what human–robot encounters should look like.

However, even the repeatedly reviewed and revised scenarios dispensed with the criticality associated with proper design fictions. For example, the visual scenarios were not concerned with the fact that the future will be an ‘accretive space’ (Foster, 2013) which is full of both new and old technologies. Moreover, a postdoc criticized the conspicuous absence of conflict and malfunction. They unrealistically lacked friction which was at odds, she argued, with the CRC’s research interest in situations where human–robot interaction breaks down.

What I noticed in all three scenarios is the absence of any explicit conflict. It is as though if there had been any conflict at any point, it would already have been dissolved . . . Of course, I’m not suggesting that our visual representations contain too much negativity so that we only focus on potential problems. But maybe there is a way to at least hint that we might need a sort of additional level, sort of an additional meta level that solves these types of conflicts should they occur at some point. (transcript II, lines 383–403).

Overall, the visual scenarios enticed CRC scholars to position themselves in relation to visualizations of the future. Their plausibility was not so much a matter of visual design but of reception and usage. Credibility or believability were attributed or refuted by scholars engaging with the scenarios. They interpreted and tweaked these images so that they would match what they understood to be the focus of inquiry. The evaluation criteria for the scenarios’ visual performance thus shifted from aesthetic to scientific parameters. For example, in relation to the floating traffic lights and signs mentioned before, one PhD student proposed that the ‘transport thing would be interesting, yes, because we are also dealing with transport systems that are driving, but [it] ; would also be possible that they are flying . . .’ (transcript II, lines 792–735). Feasibility was spelled out in terms of future research agendas in order to make tangible what had already been anticipated. This calibration process meant aligning futuristic visuals with future research agendas, depending on what could or should be deemed achievable within the scope of the CRC.

Conclusion

In our study, we looked at how visual scenarios open up a discursive space which invited researchers working in a large-scale scientific endeavour to imagine and discuss future human–robot interaction. Analysing their discussions and suggestions prompted us to consider the reflexive relationship between visual science practice and the visual diegetic prototypes in which they serve as both an inspiration for technoscientific inquiry and development as well as an instrument for orienting the collaborative efforts. The visual scenarios operate in the fictive mode of imagining alternative worlds that present highly artificial and positively connoted settings.

The utility of the visuals was not based on their limited ability to generate visceral experiences. In fact, there was an ‘experiential gulf’ (Candy and Dunagan, 2016: 26) between the fictive images and the CRC members. In contrast to the widespread efforts to create more immersive simulations which allow people to imagine themselves within a scenario, the images stimulated sensemaking that was both speculative and concrete. For the researchers responding to the visuals, the lack of specificity and detail did not necessarily pose an obstacle to their ability to speculate about the potential of embodied digital technologies. In fact, what the conversations teach us is that ‘reasoned speculation’ (Elsden et al., 2017: 5393) was possible not despite of but because of the fact they were conspicuously unrealistic scenes, as if in a comic-book. Ignoring the question of how a sketched innovation might operate technically can free the imagination and produce accounts unconcerned with matters of feasibility and implementation. Yet the researchers we worked with were not inspired by such suspended disbelief. On the contrary, the visual scenarios were consequential precisely because they challenged the researchers to define the aims and scope of their inquiry in relation to drawings, which left much to the imagination. Rather than an invitation to suspend disbelief, the scenarios were about the creation of belief (Blythe and Encinas, 2018).

Arguably, the reception of the three scenarios considered in this article was not specific to the research initiative we studied but is a vital feature of all open-ended scientific exercises. According to Kirby (2010), fictional diegetic prototypes and the environments they inhabit not only disseminate scientific knowledge to the public but shape scientific epistemology. Indeed, the popularization of science through films, books or images can free scholars from disciplinary norms and academic doxa, and may allow them to ponder more speculative ideas (Hilgartner, 1990). This implies that popular representations of scientific issues can be a source of inspiration for specialists. These types of images may offer a release from their professional restrictions, which limit creativity. Bucchi (1996: 386) suggests that films, imagery or journalism ‘can in this sense provide an open space where stimuli, ideas and information may be merged and exchanged’. Because fictional media do not share the same conventions as science, they are particularly suitable venues for speculations which break with scientific consensus. That way, the visual scenarios functioned as a kind of modelling space in the CRC’s quest for smooth human–robot encounters. Since they were easy to change, the images sometimes provoked the CRC’s members to ask for more radical visions to be drawn, for instance with entities leaving their humanoid shells behind and instead embodying more experimental sensory and locomotive apparatuses. On these occasions, the scenarios envisioned more far-reaching ventures which could help us rethink the hybrid societies of humans and robots of the future.