Climate futures: Machine learning from cli-fi

Abstract

This paper introduces and contextualises Climate Futures, an experiment in which AI was repurposed as a ‘co-author’ of climate stories and a co-designer of climate-related images that facilitate reflections on present and future(s) of living with climate change. It converses with histories of writing and computation, including surrealistic ‘algorithmic writing’, recombinatory poems and ‘electronic literature’. At the core lies a reflection about how machine learning’s associative, predictive and regenerative capacities can be employed in playful, critical and contemplative goals. Our goal is not automating writing (as in product-oriented applications of AI). Instead, as poet Charles Hartman argues, ‘the question isn’t exactly whether a poet or a computer writes the poem, but what kinds of collaboration might be interesting’ (1996, p. 5). STS scholars critique labs as future-making sites and machine learning modelling practices and, for example, describe them also as fictions. Building on these critiques and in line with ‘critical technical practice’ (Agre, 1997), we embed our critique of ‘making the future’ in how we employ machine learning to design a tool for looking ahead and telling stories on life with climate change. This has involved engaging with climate narratives and machine learning from the critical and practical perspectives of artistic research. We trained machine learning algorithms (i.e. GPT-2 and AttnGAN) using climate fiction novels (as a dataset of cultural imaginaries of the future). We prompted them to produce new climate fiction stories and images, which we edited to create a tarot-like deck and a story-book, thus also playfully engaging with machine learning’s predictive associations. The tarot deck is designed to facilitate conversations about climate change. How to imagine the future beyond scenarios of resilience and the dystopian? How to aid our transition into different ways of caring for the planet and each other?

Climate futures: Machine learning from cli-fi

The climate crisis calls for novel ways of imagining the future. As writer Amitav Ghosh (2016) states, ‘let us make no mistake. The climate crisis is also a crisis of culture, and thus of the imagination’ (9). Imagining can be ‘understood as a way of seeing, sensing, thinking, and dreaming the formation of knowledge, which creates the conditions for material interventions in, and political sensibilities of the world’ (Yussof and Gabrys, 2011: 1). Attention to the imagination enters the climate debate in relation to a ‘cultural turn’ in climate research. In addition to the scientific understanding of climate change, there is a need to foreground climate change’s aesthetic, affective and cultural dimensions. Artistic interventions and artistic research are well-suited for addressing these more experiential aspects of the climatic impact of the Anthropocene. Experts, for example, are now seeking to expand modes of climate science production to reconsider the social spaces of climate interaction at the science-policy-public interface. They promote new forms of the coproduction of knowledge between different communities of practice such as science–art or art–public collaborations’ (Yussof and Gabrys, 2011: 2).

There are multiple sites for imagining climate futures, with recent calls to explore them also in relation to materiality, which emphasise that ‘mental conceptions of the world are not independent from our technologies’ (Davoudi and Machen, 2021: 4). For example, scientists collect massive datasets about the climate and model future scenarios that align with societal climate goals (Kaack et al., 2020). Artificial intelligence is envisioned to support better forecasting of electricity needs, help optimise buildings for efficiency, model energy and water consumption and predict future weather patterns. Science and technology studies have critiqued such scientific future-making sites (Latour, 1987; De la Puig de la Bellacasa, 2017). For example, situating the models historically, as Bailey (2020) from the Manifest Data Lab has done, denaturalises their effects. Bailey also points out, for instance, that small islands might fall off the grid when modelling. Modelling practices, for example, appear ‘like petri dishes for growing fictional Earths’ (Konior, 2020: 66) that invite drawing ‘parallels between simulated social scenarios in cli-fi and simulated scientific scenarios in climate models’ (Konior, 2020: 55). Members of the Manifest Data Lab also promote exploring embodied and material understandings of climate modelling as a way to foment the public understanding of science. Davoudi and Machen (2021) ‘compare key features of prolific yet contrasting mediums-climate poetry and scenario models- and analyse their mediation effects and the ways in which they enable or constrain imaginaries in complex and non-linear ways’ (8).

Environmental projects also use machine learning (a form of AI) to raise awareness. Through visualisations they aim to engage viewers ‘rationally and viscerally’ (Schmidt et al., 2019: 1) with the effects of climate change. For example, Schmidt et al. (2019) developed a tool that shows ‘in a personalised way the probable effect that climate change will have on a specific location familiar to the viewer’ (2). In other words, acting as a sort of time machine, the tool generates ‘images of future climate change impact’ (Schmidt et al., 2019: 2). Similarly, the project WorldUnderWater.org (CarbonStory, 2014) employs ‘Web GL and Google Street View to give people the chance to search, explore and share any location submerged in water’ (Van Gurp, 2014). Zylinska (2020), who studies AI art, criticises these types of corporate-sponsored projects. They often follow a techno-solutionist mindset. Artistic intervention, she argues, can instead help expand ‘the range of survival scenarios beyond the technofixes that include the de-extinction of the vanished species, space colonisation and an intelligence race against robots. AI-driven art that responds responsibly to the Anthropocene Imperative could thus help us expand the current imaginary, allowing for the emergence of new visions and vistas for the world to come’ (Zylinska, 2020: 45). For example, Zylinska draws attention to feminist views on machine learning and how they may serve climate change projects.

Climate fiction books are sites for imagining and modelling climate futures too. Climate fiction (or cli-fi) is speculative writing that sets its stories in a world impacted by human-induced climate change (Sperling, 2019; Johns-Putra, 2016). Popular examples of the genre include Paolo Bacigalupi’s The Windup Girl (2011), Margaret Atwood’s Oryx and Crake (2004) and The Year of the Flood (2009). Climate fiction does not focus on ‘imaginary technologies or faraway planets. Instead, the pivotal themes are all about Earth, examining the impact of pollution, rising sea levels, and global warming on human civilisation’ (Ulrich, 2015). For example, The Windup Girl is set in a dystopian 23rd century, where sea levels have risen to devastating levels, carbon fuels are depleted, and people generate energy manually.

An article for The Guardian (2017) asserts that cli-fi can ‘help us understand what future climate may be like, or what current climate effects are' and 'make experiences far more real than endless graphs or plots of temperature variations’ (Abraham, 2017). Literary scholar Trexler also argues that fiction has the unique capacity to interrogate ‘the emotional, aesthetic, and living experience of the Anthropocene’ (Trexler, 2015: 6) and make ‘the so-called invisibility of climate change to be seen, felt, and imagined in the present and the future’ (Corbett and Clark, 2017). In all, climate fiction responds to the idea that engaging with ‘global warming requires much more than assenting to scientific data’ (Trexler, 2015: 3). Studies on persuasive narrative have also shown that reading climate fiction can make people, at least temporarily, more open to climate issues (Schneider-Mayerson et al., 2020).

That these literary climate imaginaries 'lack diversity’ is a critique currently gaining momentum. For example, in Anthropocene Back Loop, Stephanie Wakefield (2020) explains that she is ‘exhausted by the apocalyptic and hateful images being forced on us by political culture in this and all regards, and by the rigid modes of discourse that now portray life in the Anthropocene as survival amid entangled ruins of a broken world. Annihilation! Annihilation! Annihilation!’ (Wakefield, 2020: 10). Ruins ‘are not novel imaginaries that help us generate other possible ways of living—as I believe many in this field hope’ (Wakefield, 2020: 13). It is necessary to remain attentive to the urgent changes ahead and sobering prognostics while also asking: how to imagine the future and our place in it, beyond scenarios of resilience and the dystopian? How to aid our transition into different ways of inhabiting and caring for the planet and each other?

In this chapter, we introduce and contextualise Climate Futures, an artistic research project that repurposes AI as a ‘co-author’ of climate stories and a ‘co-designer’ of climate-related images to facilitate reflections on present and future(s) of living with climate change. We trained machine learning algorithms (i.e. GPT-2 and AttnGAN) using climate fiction novels as a dataset of cultural imaginaries of the future. We prompted these algorithms to produce new climate fiction stories and accompanying images, and then edited and combined them to create a tarot-like deck and a k (published as a podcast series), thus, playfully engaging with machine learning’s predictive associations. The tarot deck is designed with practical applications: facilitate conversations about the climate and work towards escaping the crisis of imagination that surrounds the current climate debate (Ghosh, 2016; Wakefield, 2020).

In line with the notion of critical technical practice, Climate Futures has involved engaging with machine learning both technically and from the critical and practical perspectives of artistic research to explore new modes of engagement with the issue of climate change. Critical technical practice is a research practice t ‘uses material engagement with technology to open up and extend critical social reflection’ (Ratto and Hertz, 2019: 18). It involves ‘working with technical systems from a humanities-based perspective’ (Dieter, 2014: 217). Indeed, our goal has not been automating cli-fi writing or mimicking humans (as is done in many product-oriented applications of AI). Instead, Climate Futures converses with longer histories of experimental writing and computation where imagination has also been a matter of concern. These include surrealistic ‘algorithmic writing’ experiments, the first recombinatory poems of the 1950s, and what is known today as ‘electronic literature’ (Rettberg, 2018; Hayles, 2008) and ‘big data poetics’ (Jhave Johnston, 2017). Ultimately, Climate Futures is an experiment in writing with devices and the work of imagining climate futures, both through storytelling and ‘cultural probes’ (Gaver et al., 1999) such as ‘climate tarot cards’ that may help create and evoke much-needed new climate imaginaries. In the following section, we position Climate Futures in relation to electronic literature as a way to reflect on creative uses of AI and the project’s potential contribution to conversations about climate imaginaries. Afterwards, we share a selection of outputs from our experiment and how we have worked with public engagement.

Climate futures as electronic literature

In Climate Futures, we employ GPT-2, an open source language model developed by OpenAI. In this context, language modelling uses statistical and probabilistic techniques to ‘predict the next word, given all of the previous words within some text’ (Radford et al., 2019). GPT-2 is transformer-based, meaning it uses attention mechanisms to predict which section of the input text is the most relevant, and it is pre-trained on a dataset of 8 million web pages. This dataset tries to emphasise diversity and quality of content by using ‘only pages which have been curated/filtered by humans—specifically, outbound links from Reddit which received at least 3 karma’ (Radford et al., 2019).

GPT-2 is envisioned for general use, including use-cases such as writing assistance (e.g. grammar assistance, autocompletion in prose), creating ‘games, chat bots, and amusing generations’ (Clark, 2019), producing credible patent claims (Lee and Hsiang, 2020) and ‘exploring the generation of creative, fictional texts; aiding creation of poetry and other literary art’, such as writing poems indistinguishable from those of Maya Angelou (Köbis and Mossink, 2021). Ultimately, GPT-2 can be used to create new text from a single prompt while fine tuningg its writing style by training it with text from a specific genre. In Climate Futures, we used 20 cli-fi novels to train GPT-2 further. ¹ To generate images, with which we created collages that were formatted into a tarot deck, we employed another AI called AttnGAN, an Attentional Generative Adversarial Network. AttnGANs allow ‘attention-driven, multi-stage refinement for fine-grained text-to-image generation [...it] can synthesise fine-grained details at different subregions of the image by paying attentionsn to the relevant words in the natural language description’ (Xu et al., 2018).

The qualities of the text and images generated by models such as GTP-2 or AttnGAN can be evaluated, for example, by comparing their resemblance to a text written by a person. From a critical data studies perspective, one may question the assumptions embedded in these models in terms of the type of data used for training, biases and blind spots—for example, ‘because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true’ (Clark, 2019). Critical technical practice could engage with text modelling through ‘technography’ (Bucher, 2018) or explicit critical making practices. For example, from the points of view of speculative design, one could explore ‘what if’ scenarios or from data feminism (D’Ignazio and Klein, 2020) bring feminist principles such as ‘embrace embodiment’ to computer science and tool-making. In Climate Futures, we use machine learning for writing and image generation playfully, namely, by engaging with AI’s capacity to predict, detect and simulate patterns and defamiliarise its publics (more on this later), thus approaching machine learning as a tool and a collaborator for aesthetic engagement and exploration. In doing this, Climate Futures engages with practices already developed in electronic literature, a field of creative technical and aesthetic experimentation and research on writing that explores the capabilities of the computer and networked media and also offers examples of critical technical practice.

In 'combinatory poetics,' a genre of electronic literature (Rettberg, 2019), writing is done by applying combinatorial and predictive computational processes to textual data. An early example is the programme Love Letter Generator (1952), designed by British computer scientist Christopher Strachey. The programme used a thesaurus and coded grammatical instructions to produce variations of love letters. Jim Carpenter’s The Electronic Text Composition Project’s (ETC) Poetry Engine is a contemporary example of combinatory poetics. It is a ‘suite of software components that allow a user to generate aesthetic texts. Drawing word associations from its language database, the Engine’s grammar uses a probability-based approach to constructing syntactic constituents, which it aggregates into utterances, which it in turn aggregates into compositions’ (Carpenter, 2004: 1). Using as input poems from authors such as Emily Dickinson and Sylvia Plath, Carpenter ‘mixed their words and styles together algorithmically to produce new poems’ (Rettberg, 2019: 43). He published them under the virtual persona, Erica T. Carter. Similarly, artist David Jhave Johnston published Human + A.I. poetry, a collection of 12 poetry books written with neural net augmentation. Climate Futures, our project, is in some regards similar to these existing examples of algorithmically produced literature.

Curating the dataset that will serve to train and fine-tune a model requires reflection, in and outside the practice of electronic literature. For example, Strachey programmed the algorithms that ran The Love Letter Generator to use lists made of different types of words. He described the process in the following terms:

Apart from the beginning and the ending of the letters, there are only two basic types of sentences. The first is ‘My — (adj.) — (noun) — (adv.) — (verb) your — (adj.) — (noun)’. There are lists of appropriate adjectives, nouns, adverbs, and verbs from which the blanks are filled in at random. There is also a further random choice as to whether or not the adjectives and adverbs are included at all. The second type is simply ‘You are my — (adj.) — (noun),’ and in this case the adjective is always present (Strachey, 1954 in Wardrip-Fruin, 2005).

The resulting ‘love letters’ are surprising in how the algorithm leads to ‘a relatively small number of initial materials to be arranged, following certain rules, into a vast number of possible configurations’ (Wardrip-Fruin, 2005). Letter-writing and romantic conventions both appeared strangely formulaic. However, the word ‘love’ is not in any of the lists that Strachey made available to the programme. People who have studied Strachey’s work interpret this as a meaningful silence and a commentary on his sexual orientation: as a gay man, love was a word that could not be spoken.

In Climate Futures, the corpus we use to train the algorithms consists of 20 mostly western best-selling climate fiction novels. We used Amazon.com’s search engine to collect these best-selling cli-fi titles. One may see these books also as best-selling climate futures. They include Paolo Bacigalupi’s The Windup Girl (2010) Margaret Atwood’s Oryx and Crake (2003) and The Year of the Flood (2009). The Windup Girl takes place in a dystopian 23rd century, where sea levels have risen to devastating levels, carbon fuel is depleted and people generate energy manually. Atwood’s work speaks of deregulated neoliberalism. While follow-up projects also include climate-themed visual s, climate storytelling from across the world (including non-anthropocentric narratives) and climate disaster films to serve as input, we decided to start close to the mainstream. The goal is thus not to create new imaginaries that could end up in the next climate fiction bestseller per se. Rather, we approach the combinatorial, associative and predictive aspects of the algorithm as means to view these common imaginaries through new eyes and break them open to be retold, from different perspectives. Climate poetics, in our case, generated with AI, can help ‘dislocate routinized patterns of categorisation or thought’ (ibid) in ways that can enable radical or alternative “imaginaries” (Davoudi and Machen, 2021: 6).

Using AI in this creative way challenges more utilitarian and productivity-oriented approaches to software that, as we see it, align with the work already being done in the field of electronic literature. Electronic literature is not about using software to improve writing, in a technical sense, for example, in terms of clarity or readability, or automating or outsourcing storytelling for the sake of productivity. Rather, in line with critical technical practice and artistic research, the involvement of computation sets into relief assumptions about the process of writing and facilitates aesthetic exploration of the materials. Charles O. Hartman, poet and author of The Virtual Muse (1996), reflects on the process of composing music and writing poetry with computers. The purpose writing a chorale programme, he argues, was not to produce chorales or automate the work of harmonising them. In his own words, ‘the computer was a way of exploring the nature of what I knew about music. Later, when I began to use the computer in poetry, it would participate in a similar process of thought. Yet it would also turn out to change the poetry I would write, with or without a computer’ (Hartman, 1996: 14). The question ‘isn’t exactly whether a poet or a computer writes the poem, but what kinds of collaboration might be interesting’ (Hartman, 1996: 5). Johnston, author of Human + A.I. poetry also describes his relationship with AI in terms of a collaboration, in which choices and agency in creation are intentionally shared, in part, with a non-human entity. For him, people’s inspirations are ephemeral and their energy, finite but a ‘neural nets empowers and nourishes limited human creativity’ (Johnston, 2021). To work with AI is not to fetishize the computational or admire its creative outputs as such, but rather, explore different avenues of imagination through the process of making with AI.

In his historical review f electronic literature, Scott Rettberg (2019) connects combinatory poetics and the involvement of an external (non-human) agency in the process of creating with experimental writing already found in da Daism, Fluxus and surrealism. For example, Dada artists embraced proceduralism and chance in their writing. Poetic instructions could involve writing a poem by finding a newspaper article, cutting all the words, placing them in a bag, and then reorganising them in the order in which they came out of the bag (Rettberg, 2019). Also, surrealist artists employed techniques like collage, ‘exquisite cadaver’ and automatism. These techniques also served as practice-based critiques to notions of individual genius and romantic ideas about inspiration. Setting the imagination free was, in part, the goal.

In electronic literature, elements like databases, programmability, algorithms and automation introduce ‘elements of chance and surprise, constraint, and variability, and explore the poetic and narrative potentialities of text machines’ (Rettberg, 2019: 38). In combinatory poetics with AI, the process, however, is not left to ‘chance’ but rather one counts on a form of intelligent prediction, emphasising, even more, the metaphor of collaboration with non-human entities. Furthermore, when using text-generating algorithms, the outputs are generally not ‘completely human’: ‘we have observed various failure modes, such as repetitive text, world modelling failures (e.g. the model sometimes writes about fires happening under water), and unnatural topic switching’ (Radford et al., 2019). Depending on the context of use, results like the ones above constitute errors to be improved or productive impossibilities to be explored; a poetic use of language. While discussing AI art, Zylinska (2020) describes the situation as ‘creating machines that subsequently create works which those artists could not accomplish by themselves. Naturally, artists do not construct these machines just to get “help” but rather to probe the limits of the human idea of creativity and of human-machinic assemblages. These works are thus described as collaborations between the artists and the robotic systems those artists have designed’ (60).

Poetic uses of machine learning also call for distinctive reading practices. Machine-generated literary texts are often interesting in terms of the process behind them and how it unsettles the author-text relationship. However, literary machine-generated texts often fail to engage audiences as literature per se, be that because of their quality (a perceived lack of je ne sais quoi) or lack of intentionality, as the writing is disconnected from actual experience. Indeed, there are those who argue that the construction of texts ‘does not actually occur within the software—these constructions, absent authorial intent and divorced from any underlying message, assume their status as poems only as they are read’ (Slought, 2014). Also, studies show that perception of poems change when people learn they are machine-generated, a fact that resonates in debates about algorithmic transparency. Alternatively, the literary machine-generated text calls back for human involvement. In Johnston’s case, the computer creates and the human edits (Johnston, 2021). A new role for the ‘human poet: as editor for poetry entirely generated by machine intelligence based on a process that remains both invisible and largely incomprehensible for the human poet’ (Rettberg, 2019: 53). Editing becomes a form of deciphering and finding meaning in the ‘slightly off’; a form of meditative carving: ‘A block of A.I.-generated text, massive and incomprehensible, can exude the presence of solid stone. Here, the cursor exists like a chisel; I called this human-editing part of the process, carving’ (Johnston, 2021).

The eeriness of other machine-generated outputs and how they defamiliarise common objects and scenes also invite ‘meditative carving’. Popular present examples are the psycholodelic images generated by Google’s DeepDream model (Miller, 2019). The programme allows for looking into different stages of a machine making sense of images. The distorted versions of animals and humans that it produced were not seen as errors to be corrected but as strange aesthetic spaces to be explored; here one may think also of Glitch art (Menkman, 2011). Similarly, when speaking about the use of adversarial networks in his work ‘Memories of Passersby I’, where several GANs work in a feedback loop to continuously create portraits, artist Mario Klingemann, says, ‘what one model creates, the other one deforms. And then, I really like playing with feedback loops. So, at some point, I feed the output of the sixth model back into the input of the first and then you get to these systems, which start behaving, hopefully, in an interesting way’ (Follett, 2020). The question is present too in science fiction, when we are confronted with artificial eyes: how do the machines see the world and us? About these approaches to art and technology, artist and writer Zylinska asks ‘is there an ontological difference between early computer generated art, net art and the more recent forms of AI -driven art? Or is it just a difference of degree, i.e. of the mode and intensity of technological entanglement?’ (Zylinska, 2020:13). The questions that art made with AI raises about ‘collaboration’ and exploration with machines are indeed ongoing and researched in practices of making.

Like David Jhave Johnston’s AI poetry collections, Climate Futures is a collection of climate fiction stories written with machines and edited by humans. GPT-2, the model we trained with cli-fi, can learn from data (in this case, existing cultural imaginaries of climate change from cli-fi novels) in an unsupervised manner. We set out to find themes and images in the outputs (our own process of editing and ‘carving’ using Johnson’s term) and make familiar ideas into (un)familiar scenes. Indeed, in Climate Futures, instead of aiming to understand how a text or an image by GPT-2 is created, we lean on their patterns and ‘world modelling errors’ as spaces to be explored, not corrected or understood per se. Climate Futures, as we explained earlier, aims to respond to a crisis of imagination in the climate debate and thus situates itself outside a delimited literary realm. After creating the texts with the help of AI, the next step was to develop formats for sharing the texts and images in ways that would invite engagement with these imaginaries. For this, we collaborated with a range of artists and designers, including a sound artist, a novelist and a choreographer, to create evocative stories and objects, two of which we will present in the sections below.

The pilot: Turning to Birds

Our pilot experiment builds on science fiction’s capacity to transport readers into possible/probable futures. When talking about climate fiction, for example, author Kim Stanley Robison (2021) says: ‘Imagine that you’re in the future. You look around at a changed world […] Then you look back at your own time, seeing it as if it were already history’. A fictional visit to the future ‘can turn utopian. Having seen a good future, you can decide to make it happen’ (Stanley Robison, 2021). We aimed to explore whether and how cli-fi literature and other cultural collections could help create a kind of time machine to escape our Western, anthropocentric worldviews and the archetypal images of climate change to which we are exposed to time and time again. How could we receive messages from the future, postcards from the post-Anthropocene?

From both a technical and creative perspective the initial challenge was learning how to prompt the model. A prompt involves ‘a piece of text inserted in the input examples, so that the original task can be formulated as a (masked) language modelling problem. For example, say we want to classify the sentiment of the movie review “No reason to watch”, we can append a prompt “It was” to the sentence, getting No reason to watch. It was ____’. It is natural to expect a higher probability from the language model to generate "terrible" than "great"’ (Gao, 2021). Prompting can be used as a way of fine-tuning a model and its outputs. From the c, prompting became a key aspect of the process of ‘collaborating’ with the machine, which calls for reflection. In other words, prompting became a question-and-answer game.

The pilot started with the aforementioned set of twenty cli-fi novels, which we compiled in a single document of one million characters used to train GPT-2 for 40,000 iterations. After the training, with the help of Google Colab notebooks ² found online, GPT-2 needed a prompt to generate new text. How could we start writing without a narrative arc, let alone character motivation? We tried using news headlines, landscape descriptions and mundane sentences, like ‘I was walking down the road....’ as prompts. The results were, however, wildly different. We returned to the original texts and the genre of cli-fi more broadly to learn from its aesthetics and develop a more appropriate prompt. Much speculative fiction, science fiction and climate fiction work with the common literary format of the diary. We prompted the machine with a date: Monday, the 2nd of October. It was this last attempt which worked. We kept feeding dates to the machine, and it would tell us what had happened that day in the first-person. We repeated this process many times until we had built up an extensive range of days. We chose the most coherent and compelling ones from those diary entries, and we lightly edited them for readability and consistency (for example, in the consistent uses of pronouns of each story’s main character).

The editing was done with writer and editor Janine Armin, who was immediately struck by how good the AI was at writing fiction ³ . Being a climate fiction novelist herself, she valued how the writings ‘had no restrictions’ and demonstrated an innovative approach to language (Sanchez-Querubin & Niederer, 2022). For instance, one story describes how a baby doll’s face is being watched by thousands of tiny birds. In another, floating dartboards are the abandoned props once used by humans for their entertainment. Or, in the story below, water is described to look ‘PURPLEY different’. According to Armin, ‘When you are writing sci-fi, you are loading a narrative by using words that forebode a particular idea or situation. And so, reading the story with that structure in mind, […] everything becomes detritus. The utopian idea seems to be that we have learned to become sustainable and at the same time, everything has become fused, and thus meaningless. When you read it, it makes sense as something that could happen’ (Sanchez-Querubin & Niederer, 2022).

Episode 3. Out around the river

Monday, September 3rd.

Shame, shame. We got so filthy rich that now it’s nice to see something back on the Wall, even if we’re only on it for the week. The Captain brought us a boat to the end of the watchtower. It’s an oareb (OC boat). When I got to the water I stripped off my clothes and jumped into the boat. The boat circles around the river like a pool on a pool table. The Captain didn’t remind me of anything, but I still felt kind of weird. My first fifteen minutes on the water were total silence. We just sat in the boat and stared at the sky.

The second thing I noticed was the colour change. The water looked PURPLEY different. There is no such thing as a dull shade of blue. I was finally seeing some of the shades pop out around the river, like someone using a colouring program to colour pictures on a monitor. It’s like someone going on a trip to the colour revolution. The one that makes people sick is called Van Gogh’s sun bomb.

When we made these stories public through a podcast published on Spotify, Janine Armin edited and narrated them. The podcast series holds 11 short climate fiction stories co-authored with our trained cli-fi machine. Together, they narrate 11 days in a future that is seemingly no longer centred around people, describing daily rituals in altered landscapes that present glimpses of the inner turmoils of the narrators. We made the stories available to the public in the context of different cultural events. ⁴ We were interested in how listening to them could foster reflection on their immediate environments. One participant, for example, listened to the stories ‘while seated next to a large Monstera plant in his studio and looking out the window at the plants and trees in the courtyard’ (Sanchez-Querubin and Niederer, 2022). Another participant talked about the stories in terms of enabling noticing ‘the sound walk prompted her to think deeper about the situatedness of the forest she was exploring and discover what was specific about that very place. She described how she felt enabled to see the forest both through the lens of the sound walk and in contrast with the world presented in the audio work’. The sound landscapes include references to houses engulfed by nature and offer strange metaphors that emerge from having the world of climate fiction reinterpreted by machine learning. New strange sites overlap with the familiar scenes of people’s neighbourhoods and parks. One is then neither here nor there. These reflections are not yet scalable, however, they represent a step in the broader process of thinking through stories.

The next step was to create illustrations to accompany these stories (as album art for the podcast episodes) which we did with a different algorithm: the Attentional Generative Adversarial Network (Xu et al., 2018), a e model that can synthesise fine-grained details of an image by paying attention to the relevant words in a description. Using machine vision to ‘represent’ climate change also has precedents in art, specially, in works that engage with climate data. As with the case of electronic literature, here one inquiry into how machine vision may be employed with artistic goals. If the objective is not accuracy or prediction (as with a traditional data visualisation), what does machine vision add?

Generally speaking, climate data provides insights into a range of invisible phenomena describing vast geological timescales, complex interdependencies, atmospheres, biotics and other planetary inscriptions. Art has allowed exploring and visualising climate change data differently. For example, Lost Skies is a project by artist Maja Petricin in collaboration with computer scientist Mihai Jalobeanu. Petricin searches online for images of landscapes affected by climate change. Then, using software she ‘processes all the found images, analyses their salient properties and summarizes the data into one image — or an image within thousands of evolving images’ (Petricin, 2017). In the context of their project, AI becomes an ‘artistic tool that combines many images (Big Data) into individual archetypal images’ (Petricin, 2017). When using the more-than-human perspective of the machine for visual analysis — what has also been termed ‘cultural analytics’ by Lev Manovich – ‘the purpose of the images, she explained, is intended to express the essence of nature affected by climate change’ (Petricin, 2017).

In the context of climate futures, we selected an evocative sentence from each of the diary entries and used it to prompt the model, and it generates an image conditioned on the input text. The synthetic images generated this way were then lightly edited and processed for Risograph printing ⁵ . We designed one printed image per story. One of the critiques that machine vision projects and artistic renderings of climate change are faced with is being too focused on ‘representing climate change’. Moreover, seeing climate change as ‘art’ may, in itself, create an inherent distance from the viewer and from real life, where ‘disintegrating ice becomes as beautiful as suffering in Renaissance paintings of martyrdom’ (Miles, 2010: 32). The intention of the Climate Futures is not to represent in a traditional sense, but rather to create images to ‘think with’ and that may set in motion processes (using AI for artistic research) reflection. The aforementioned need informs the second iteration of the project, namely, moving from texts and images into cultural probes.

Second iteration: All Gone

After the pilot project, we set out to put the stories and the images to use. We aimed to further develop them into tools that could enhance people’s ability to (re)imagine our future with climate. We found inspiration in the diverse visual traditions of tarot, which are illustrated card decks that combine archetypal imagery with ‘symbolic situations’ (Jung, 2014). Tarot decks are sometimes used for divination and prediction of the future. Still, we were particularly inspired by their use as a tool for reflection, in which a focused and active dialogue deepens the users’ understanding of one’s personal stance towards a current situation. We wanted to create a tarot deck that would offer its users a tool to engage with one’s personal perspectives on climate futures. Decks of cards are also gaining popularity as tools for communicating research outputs and g reflection, as is done in design as well as in media studies, a recent example being the Meme Tarot by Luther Blissett, found in the Critical Meme Reader (2021). Tarot-like cards, in addition to specific modes of visual communication, also lead to knowledge creation through the method of ‘reading a spread’. In a spread, the images in the cards have meaning in themselves but also are interpreted in terms of combinations of images. Rather, than a one-way close conversation, cards prompt conversations around questions.

We explored the visual languages of tarot, from the iconic Tarot de Marseille, one of the oldest and most renowned tarot card decks, to Madre Paz (MotherPeace), a deck of round tarot cards inspired by the Goddess movement and second-wave feminism, created in the 1970s in California. Similar popular esoteric metaphors of tatot are also present in s of machine learning and AI (Singler, 2020). For example, ‘as our technology grows, it allows us to “see” deeper and deeper into the structure of the natural world. Is it possible that just as technology that imitated the eye has allowed us to see what the eye could not see, so technology that imitates the mind will allow us to perceive what the mind cannot perceive?’ (O’Hara, 2019). While remaining critical to such interpretation, instead of distancing ourselves from this language of machine learning in favour of exclusively technical terms – this also in line with critical technical practice – we embraced metaphors, including that of ‘deeper seeing’ and knowing that often accompanies big dataism. It became a way to extend our collaboration with AI in co-creating the images for the tarot cards (also referred to as ‘arcana’).

In the concrete process of making the cards, we worked with another text-to-image model: X-LXMERT (Cho et al., 2020). This offers more elaborate image generation capabilities than previous generative models, including AttnGAN. Starting from the opening passage of ‘All gone’, we manually selected evocative words (an example can be seen in the fragment below), which we used as prompts to generate a set of synthetic images. In all, we generated around five different images for each story (Figure 1), with which we manually designed digital collages.

“It’s cold outside. I curled up on the bed again, no wind, no stars, less than an hour from the first heavy rain of the year. I’m so close to the lake and the creek, maybe twenty yards, and I’m thinking about paddling back out there on the trail, my heart beating like a big drum, and I’d be lying if I said it’s not the worst thing I’ve ever seen. It’d be an adventure, everything – the mountains, the creek, the trees, the animals – all gone.”

Figure 1.

Images made with textual prompts from the ‘All Gone’ story, with the X-LXMERT demo model, available at https://vision-explorer.allenai.org/text_to_image_generation. The images were used to design the digital collage of the arcana ‘All Gone’.

Each collage, designed with the generated images, could then function as an arcana of a cli-fi informed, machine co-authored tarot deck that invites us to charge these images and narratives with our personal vision of possible futures with climate change. In creating the arcana, we tried to find new archetypes, strong images that could be somehow general yet detailed enough in their nature. Figure 2 is an example of the results of this process Figure 3.

Figure 2.

All Gone, the first arcana designed for the cli-fi tarot deck by Carlo De Gaetano., is a digital collage of the machine-generated images presented in Figure 1.

Figure 3.

The house engulfed with trees, one of the main arcanas designed for the cli-fi tarot deck by Carlo De Gaetano.. [The house engulfed with trees] reminds me of a fantasy book in which a new world can be entered through a hidden magic door. What is inside the house engulfed in trees? Who lives there? Is the house in ruins, or actually alive in a different way? The town evokes a known life that seems to make the main character unhappy. The house engulfed with trees offers a better place, a refuge. Is my home in the town or even in the engulfed house? Are the bushes and trees waving at me because they welcome me? Or is nature deceiving here? Is the engulfment comfortable or suffocating?

To begin testing the uses of the card deck, we invited participants to engage with the arcana through different exercises, of which an example can be seen below and a full instruction of use in Figure 4. Reflections ranged from personal reasons of attraction towards a specific visual element of the arcana to considerations on its general mood and commentary on the sensations given by the composition. It was interesting to note that often the observation of the image led to the formulation of new questions and discussion on the cli-fi tarot deck as a visual tool for reflection and a facilitator of imagining possible futures with a changing climate.

All Gone Deck: How to Use and Why

Purpose: The All Gone deck co-created by the Visual Methodologies Collective and artificial intelligence trained on Western climate-fiction novels can be used to identify strategies for living with climate change. How might a deck based on these stories help us situate ourselves in relation to the climate, and envision futures within it?

Instructions: Try out one or more of the following readings or dream up your own, recording your thoughts on the notes page in the booklet to share with friends or reflect on later.

Reading 1: mirror, mirror

Pick a card that speaks to you. Take your time to look at it and read the text. What attracts you to this card? Does it spark a particular emotion? What details set your imagination in motion? Do the words help or hinder your understanding of it?

Reading 2: one of these days

Pick a card that makes you think about a day in the future. Which elements of the card make you travel in time? Is it a future you’d like to live in or that you fear? What is happening? Does it make you think of something that is happening now (events, politics, news, etc.) that leads to that future? Or do you maybe feel related to that scenario through some personal experience? How does the text help or obstruct these feelings?

Figure 4.

All Gone Tarot Deck and Instructions of use. The All Gone deck co-created by the Visual Methodologies Collective and artificial intelligence trained on Western climate fiction novels can be used to identify strategies for living with climate change.

Conclusions

Inspired by Ghosh’s warning of a ‘crisis of imagination’ (2016), we developed an artistic research project that included experiments with machine learning and a set of climate fiction novels. The goal of this human-machine collaboration has been to produce ‘evocative objects’ that foster new climate future imaginaries. The first two iterations resulted in diary entries, postcards from the post-Anthropocene, in print and as a podcast and a tarot deck. Each card in the deck features a collage of machine-generated images based on our cli-fi stories. We have begun using these outcomes as probes in workshops to facilitate reflection about climate change and imagining possible futures.

These small experiments and interventions so far have had some promising results that imply the artistic use of AI can contribute to sparking new climate future imaginaries. The experiments have since then led to collaborations with choreographers, who are developing a performance based on the recorded stories, the 3D knitting field lab in collaboration with a fashion school, with whom we have created an All Gone wall tapestry, and an art and technology lab, with whom we have worked on an AI-driven interactive light sculpture, and various educational projects, where students create workshops for their peers to depict and discuss desirable and realistic climate futures. ⁶ The afterlives of these small experiments will themselves lead to evocative artistic work that can push our imagination of a changing climate even further. Operating from such an artistic research framework involves exploring and expressing one’s vision or argument through tools that are not available to traditional academic research.

The Climate Futures project has not been about outsourcing creative processes to machines. It tries to develop a critical technical practice in which a collaboration with AI leads to tools for reflection, both about the technology one uses as the craft it participates in. The creative process comes with many glitches. The stories rendered by the trained algorithms are strange, sometimes funny and can be hard to relate to. They need human editing and a human voice for us to be able to follow its narrative and explore its landscapes, impacted by climate change. We turned to machines as our collaborators, to extend our own limited scope and help open up and defamiliarise familiar descriptions of a future with a changing climate. Collaborations with machines for writing fiction fit into a longer tradition of experimentation in storytelling, poetry and commercial applications of artificial intelligence often used in attempts to mimic or even emulate human creativity. Especially with the launch of GPT-3 and DALL-E2, it has become hard to discern machine-authored text and images from those created by humans.

Training algorithms and then letting the machine generate new stories could be construed as what philosopher Slavoj Zizek and others have called interpassivity, where something (an object) is active (or interactive) in your place, turning us into interpassive subjects. An example is that of the canned laughter in a tv comedy show, where the sense of ‘being entertained’ is outsourced to the show itself. ⁷ As we have pointed out, we are not striving e the creative process to the machine, leaving us interpassive, but rather to be co-creating, writing and making images with algorithms that we make our own by way of curation, editing, giving it a human voice and a tangible and relatable outcome.

Lastly, the small experiments discussed here are based on a dataset that considers mostly Western and best-selling climate fiction novels, where the results have un-familiarised or otherwise activated these common tropes. An important next step will be to ‘pluralize’ and diversify our collection of climate fiction and move beyond Western and Anthropocentric perspectives on climate change (Ingwersen, 2018; Wakefield, 2020). Such an expanded collection would include a diversity of climate imaginaries that blend the topic of climate and nature through fiction and symbolic practices; genres such as indigenous futurism and solarpunk present distinct ways of knowing, imagining and performing possible futures. Which kind of futures will the machines learn and propose when trained with more diverse cultural perspectives? Another way of expanding the scope is by including many media and involve music, sound, illustrations, moving images, performance, theatre and more.