Abstract
Digital recreations of the past, and of the deceased, are part of the Internet’s present. They circulate within social networks where logics of connection and connectivity underpin increasingly performative memory work. In this article we explore these developments through a case study of the MyHeritage deep learning feature, Deep Nostalgia. Our analysis is informed by a close critical study of Deep Nostalgia creations, and discourses circulating around them, shared on Twitter during the two-week period following its launch, February 2021 (n.6935). We examine how memory is evoked, framed, re-worked and distorted through algorithmic processes, and within social networks in particular, and explore what this tells us about peoples' need to connect with their pasts. First, we analyse how the shift from photo to video ‘revives’ the dead via a process that we have termed ‘remediated memory’. Second, we explore the affective dimensions and resonances of Deep Nostalgia creations. In doing so, we introduce the concept of ‘algorithmic nostalgia’ to describe the ways nostalgia is generated, organised and exploited through Deep Nostalgia’s automated and recursive algorithmic mechanisms. Third, we interrogate the ways social media logics shape the use and influence of these outputs. Our study’s scholarly contribution is at the intersection of memory, automation, and algorithms. We highlight the importance of studying the ambivalence of emerging media at their nexus with memory studies and, critically, of attending to the ways corporate interests increasingly shape – and assimilate – these activities.
Keywords
Introduction
On 25th February 2021, the MyHeritage genealogy service posted a tweet introducing its new Deep Nostalgia feature to nearly 80,000 Twitter followers. Deep Nostalgia uses deep learning technology to animate faces from historical photos, creating a short video of the result which can be easily and freely shared in social networks. By 10th March 2021, 33 million people had used the feature to animate images. 1
Deep Nostalgia enhances old photographs by increasing their resolution, before animating them so that the subject can be seen smiling, blinking and turning their head. These gestures are modelled from blueprint (‘driver’) videos sourced from recordings of real humans, mostly employees at MyHeritage. 2
MyHeritage was founded in 2003 and provides genealogical services that are underpinned by billions of historical records and other data points, including those submitted by users. 3 The specific technology used in Deep Nostalgia is licensed by MyHeritage from D-ID, a private company specialising in generative AI technology, including the production of synthetic media outputs such as live portraits. 4 The framing of Deep Nostalgia within the orbit of genealogical services is important to note here as it no doubt has significant implications for perceptions of its credibility. 5 In this article however, we study Deep Nostalgia as a socio-technological phenomenon that has implications beyond its use in genealogy communities.
At the time of the Deep Nostalgia launch, MyHeritage said the following: ‘Some people love the Deep Nostalgia™ feature and consider it magical, while others find it creepy and dislike it. Indeed, the results can be controversial and it’s hard to stay indifferent to this technology. We invite you to create videos using this feature and share them on social media to see what your friends and family think’.
6
No doubt a marketing ploy to attract new users to its services, a technology like Deep Nostalgia also provokes searching questions about our interactions with our own and others’ pasts, and the uses we make of our increasingly complex personal archival apparatuses, not least within social networks: [How] Does the shift from photo to video reshape memory and memorialisation practices? What are the affective dimensions and resonances of these algorithmic creations, and just how deep is any nostalgia elicited? And how might we begin to understand the intense virality of a capability like Deep Nostalgia? Our study operationalises these prompts as research questions to explore Deep Nostalgia creations including how they circulated in social media at the time of the launch.
Our analysis in this article is underpinned throughout by an acute critical awareness of Deep Nostalgia’s ethical ramifications and a desire to understand its broader societal implications. The ethical issues we foreground are oriented around the extent to which animations can be considered exploitative; of both those who have died, and those they leave behind. This includes considering the possible social consequences of emotional exploitation - in particular where it gives the living false impressions and false connections. These are especially important considerations where they underscore extractive commercial enterprises (MyHeritage’s business model is built on the collation and connection of personal data), and where they also, through the circulation of outputs, interface with a suite of ethical concerns about social networks, as is the case with Deep Nostalgia.
Our study’s scholarly contribution to (digital) memory studies is at the intersection of memory, automation and algorithms – the latter understood as ‘sets of defined steps structured to process instructions/data to produce an output’ (Kitchin 2017: 14). The majority of previous studies at this intersection address the ways in which social media platforms rework users’ data into timely memory packages (look back videos for example) through automated algorithmic processes. Our study goes further, exploring a technology that not only brings the past into people’s social media feeds, but animates it, ‘reviving’ the dead via a process that we have termed ‘remediated memory’. We connect this with the concept of ‘algorithmic nostalgia’, which we introduce here to refer to the ways nostalgia – the longing for aspects of one’s past (Routledge 2016; Becker 2018) – is generated or organised (and perhaps exploited) through automated and recursive algorithmic mechanisms.
As manifold and intersecting socio-technical infrastructures increasingly underpin memory practices – evoking, enabling, connecting, distorting, exploiting – we make the case for critically attending to both remediated memory and algorithmic nostalgia, as two distinct but overlapping processes that are particularly relevant in understanding the affordances of Deep Nostalgia. We do so following Airoldi’s prompt to ask a new set of questions about algorithmic practices which challenge dichotomies (real/fake, human/machine for example) and disrupt disciplinary assumptions (Airoldi 2022). We also highlight the ambivalence of the technology here, through which we explore different (and often conflicting) interests and impacts for stakeholders; from the extractivist datafication practices of social media platforms and MyHeritage, to the ways users might operate, appropriate or challenge the technology through creative, productive and even subversive practices. 7
In the next section, we set out the key ideas and debates informing our analysis, before introducing our approach (Approach to the Study). In ‘This Is the First Time I’ve Seen Him Smile’: Remediated Memory, Algorithmic Nostalgia and the Ambivalent Logics of Social Media we detail and discuss our findings, demonstrating how the theoretical issues we identify intertwine in the orbit of Deep Nostalgia. In Conclusion we draw together a series of concluding points.
Remembering the dead in an algorithmic present
Here, we follow three main avenues of theoretical inquiry relevant to our research questions. First, we introduce a set of dynamic trans-disciplinary concerns related to representing and remembering the deceased in digital contexts, specifically when characterised by remediation. In Algorithmic Practices and Memory Work, we explore the character of peoples’ interactions with algorithmic systems, especially where those have a memorative and nostalgic quality. For the purpose of this article, we see memories as the result of complex interactions ‘between brain, material objects, and the cultural matrix from which they arise’ (van Dijck, 2007: 28). In Social Media Memory ‘Work’, we examine the role played by socio-technical infrastructures, and social networks in particular. This theoretical framework then structures our empirical findings in Approach to the Study.
Remediating the dead, digitally
In disciplines such as journalism, anthropology, conservation and museum studies, there has been an expanded debate about the conditions under which we might work with, represent, or dispose of, human remains. Although unresolved, and typically ethnocentric, those discussions have led to a suite of ethical codes, professional guidelines and much scholarship. We are only beginning to consider whether and how those resources might translate for the digital environment in relation to a person’s own born digital human remains (a Facebook profile or files in the cloud for example), or digital remains that might be created on their behalf posthumously by other people. As we shift from a notion of self which is grounded in atoms to one ‘made of data’ (Cheney-Lippold, 2017), identities can continue to evolve, even after death.
Digital environments offer a multitude of ways for users to remember or even interact with the dead, to the extent that we might now say ‘technology is where the dead live’ (Kasket, 2019:7) or even, the ‘digitally undead’ (Sisto, 2020: 3). The deceased are, according to Kasket ‘everywhere you go online, their images on your screens, their voices in your ears – accessible anytime, in the palms of our hands’ (2019:33). 8 The digital environment demands then that we re-appraise our understandings of loss and connection, a need that some have noted became more acute during the COVID-19 pandemic as death and grief in so many cases moved more comprehensively online (Beaunoyer and Guitton 2020; Harrington, 2020).
One of the assumptions inherent in the naming and framing of Deep Nostalgia centres on the capacity of its animations to induce nostalgia for those who have died, and the pasts they might suggest, even digitally. If we accept this claim at face value for a moment, Deep Nostalgia might be considered an example of a thriving ‘nostalgia culture’ (Sayers, 2020:2) which is now pervasive. Academic interest in nostalgia continues to grow, although it too remains an unsettled quality. Some (such as Chrostowska, 2010) have proposed that nostalgia tends to pare down the heterogeneity of the past, simplifying and distorting our world view, and Lizardi argues that nostalgia constructs pasts that are ‘devoid of any critical or radical difference’ (2016). Other scholars are more positive in their appraisal of nostalgia’s functions however (Smith and Campbell, 2017); Davis has proposed that nostalgia is a ‘deeply social emotion’ (Davis, 1979), and Wilson (2005) has argued that it can be progressive and productive, with the potential to galvanise people or give them a grounded and coherent sense of identity. To Sayers, it is clear that ‘nostalgia is often the container of conflicted desires’ (Sayers, 2020:196) and Niemeyer proposes that it would be helpful to talk instead about ‘nostalgias’ and how they interact (2014:6). This is an approach we embrace in this study, exploring interactions between differing readings and uses of nostalgia, including within social networks which, some suggest, are becoming key sites for nostalgia-fication (Lizardi, 2016). We also centre interactions between nostalgia and emotion which are made more visceral – and in turn economically productive – in the age of marketing (Routledge, 2016).
It has been suggested that ‘it is via new technologies and new media that nostalgia is given its most poignant expression’ (Sayers, 2020: 27). Sisto has explored this proposal in relation to emergent technologies such as chatbots and holograms, concluding that such media challenge our capacity to orient ourselves toward the future, trapping us instead in what he calls a ‘backward-looking melancholy’ (2020: 83). We might note that such concerns have been expressed throughout history, particularly since the invention and growth of photography in the nineteenth century (Raun, 2018). It is notable that the photos animated through Deep Nostalgia largely pre-date our own era of pervasive and seemingly limitless photo production, and most will have been produced for the purpose of memorisation and conservation rather than for sharing (Esposito, 2022). They will have been stored in personal physical archives and photo albums, until such time as they were digitised, perhaps for the first time, in order to be used in Deep Nostalgia. We can consider the automation and animation of a photo in this way as a process of ‘remediation’ (Bolter and Grusin, 1999) which itself is remarkable, featuring as it does the representation of one medium (the photo) within another (the short, animated video). This shift to what we term in this paper ‘remediated memory’ suggests new possibilities, including for remembrance, nostalgia, immediacy and intimacy, which are in turn both amplified and distorted by the frenetic social media sharing which occurs around these creations. We explore this further in Social Media Memory ‘Work’, after a closer analysis of the implications of algorithmic practices for memory work, given that a Deep Nostalgia recreation is driven from its core through computational processes.
Algorithmic practices and memory work
Uses of artificial intelligence within arts and culture (Murphy and Villaespesa, 2020; Jin, 2021; McCosker, 2022) and as they impact activities in social networks (Vaccari and Chadwick, 2020) are now receiving significant scholarly attention. Developments at the intersection of archival practices and deep learning techniques have also been examined (Kidd and Rees, 2021; Lee, 2020; Villaespesa and Murphy, 2021). Discussions about the ethics of deep learning technologies – in particular deepfakes (Fletcher, 2019; Chesney and Citron, 2019) – have often centred concerns about authenticity and debates about disinformation, such as in Paris’ assessment of them as ‘false performances’ (2021) and Rini’s concern with the ‘epistemic chaos’ that might flow from their circulation (2019:12). Natale makes the case that what he calls ‘banal deception’ now underscores our interactions with AI to the extent that it subtly and pervasively ‘disappears into the fabric of our daily lives’ (2021:129). Dealing in such deception comes with ethical responsibilities, and Natale encourages developers to reflect on how they might design ways to make deceit apparent, and in-so-doing, help users to ‘better navigate the barriers between banal and deliberate deception’ (2021:131).
Natale’s work is demonstrative of a broader trend in studies of algorithms which treat them as ‘more than cold mathematical objects’ (Airoldi, 2022:xii), recognising the ways they intervene within and are shaped by society and culture, including how they mediate our processes of memory and memorisation. Esposito, for example, asks ‘How can we deal with a social memory driven by algorithms? How can we ensure both the preservation of the past and the openness of the future, when the agents that manage data move in an eternal present, without remembering and without forgetting?’ (2022:71). One data type Esposito considers at some length is the photograph, noting the proliferation of photo-taking practices brought about by smartphones, and the trend toward ‘social multiplication’ as photos enter ‘the fluid circuit of images on the web’ (2022:80), and as they encounter algorithms. To Esposito, this demonstrates a ‘shift from memorization to communication as the primary use of photography in the digital age’ (ibid.). We have been able to explore this proposition in our own study, particularly as Deep Nostalgia was marketed not just for mnemonic purposes, but also as a source of shareable creations.
In an examination of the standardised automated videos which are produced from our photo and video resources by online platforms like Facebook, Lambert et al. (2018) introduce the notion of the ‘algorithmic afterlife’. Such systems use algorithms to ‘filter, merge, rearrange, reshape and render visual data to produce novel digital artefacts’ (Facebook Look Back videos for example), and the outputs are ‘explicitly mnemonic’ (2018: 157; see also Jacobsen and Beer, 2021b). Their algorithmic processes operate at the nexus of everyday remembering, biography, social and public memory, as well as practices of documentation and archiving, but they remind us too of the ethical dimensions of remembering and forgetting. Lambert et al. (2018) reflect upon some of the harms these outputs can do, not least for those who are grieving. The lack of these systems’ capacity for ‘personalised emotional labour’ means the videos produced often do a poor job of ‘death work’ in their assessment (see also Sisto, 2021). Their brutality occurs in the way they appear unannounced – and persistently – in the social media feeds of grieving users, and they raise questions about digital legacy which are difficult to resolve. This notion of ‘algorithmic afterlife’ is intriguing given the focus of our own analysis, and one we will return to. We build on it to propose the concept of ‘algorithmic nostalgia’, by which we suggest the generation and organisation (through automated and recursive algorithmic mechanisms) of particular memories that are aimed at eliciting nostalgic effects and affects in users of these technologies, including for exploitative purposes.
Algorithmic nostalgia differs from concepts such as ‘quantified nostalgia’ (Jacobsen and Beer, 2021a) or ‘automated memory’ (Jacobsen and Beer, 2021b) which describe automated approaches that classify, rank and sort the past. Algorithmic nostalgia focuses instead on how computational processes are inserted (or insert themselves) into the very ‘fabric’ of personal or institutional archival assets, and what the implications are for memory. 9 Applied more generally, it suggests the ways nostalgia can be crafted, charged or contained, as deep learning mechanics are sedimented into our representations and memories. It is also a call to critically examine the ethical ramifications of these algorithms, including issues around consent, potentials for exploitation of the bereaved, and possibilities for manipulation or deception, as we will go on to explore.
In recent years, artificial intelligence has been somewhat mainstreamed through, for example, SnapChat filters and FaceApp, which Vaccari and Chadwick refer to as ‘common, banal forms of this technology’ (2020:7). Such uses are not inconsequential however, as they seed new norms when it comes to the adoption of these technologies and behaviours around them. The technologies (as other technologies) are in and of themselves materially ambiguous and ambivalent, although of course not neutral, until we decide to put them to use. In the case of Deep Nostalgia, there are clearly different motivations for using the technology – from MyHeritage’s marketing and extractivism to users’ mnemonic purposes and connection. It is critically important then that we ‘consider the attendant ethical and policy questions’ that our uses of such techniques raise (Paris, 2021:1, see also Hepp et al., 2022), and in the case of Deep Nostalgia, that we do so in relation to social networks also.
Social media memory ‘work’
Lambert et al. note that ‘As death becomes increasingly mediated by digital technologies, the relationship between the dead and the living becomes more complex and dynamic’. They propose that ‘part of this dynamism comes from the influence of multiple actors in networked and public spaces’ (Lambert et al., 2018:158). In this section, we explore that dynamism further.
Hoskins has written about the consequences for memory of what he has called ‘the connective turn’ (2011), detailing how the immediacy, volume and pervasiveness of contemporary digital media and archival resources, impact our sense of what the past is, and our understanding of what work it might do in the present. Hoskins proposes that living with an abundance of media in a period of post-scarcity means living also with an abundance of past-ness which can be more disconcerting than reassuring. In the days following the launch of Deep Nostalgia, millions of people encountered multiple animated representations of the past in their social media feeds, which may well have been unsettling at first. To Jelen (2021), changes in cultural and social frameworks – and no doubt media also – mean that pasts we had forgotten reappear ‘and acquire new symbolic or political presence’. This can prompt a re-examination of their meaning, or indeed allow new meanings to be negotiated. In social networks, it can also precipitate differing or conflicting pasts being brought into ‘dialogic relation to each other’ (Bond and Rapson, 2014:19) in what can be powerful mnemonic networks (Liebermann 2021).
In a study of Facebook as collective memory (2017), van Dijck reminds us that connective practices within social networks are themselves underscored by algorithms that construct and take advantage of users, rather than simply enabling connections; social networks are ambivalent at best, and exploitative by design. In the context of memory work, these are of course not simply technical observations, but ethical ones too as we have begun to observe (see also Rumsey, 2016). Smit et al. argue that ‘highly dynamic’ memory practices within social networks (in their case Facebook) are underpinned by operational logics which help to popularise and stabilise particular narratives, representations and mnemonic discourses, ‘driven by a commercial strategy revolving around visibility and “the new”’ (2018: 3122). Over time, it is entirely possible that, as Jacobsen and Beer argue, platforms and their logics of selection and prioritisation begin to distort our relationship with and understanding of our own biography (Jacobsen and Beer, 2021b).
As becomes apparent here, the socio-technical infrastructures of social networks are profoundly implicated in digital memory work. Calling attention to this as ‘work’, as we have done in this section (following Smit, 2020), highlights the continually emergent nature of digital memory, not only for individuals or groups, but also in relation to technologies and objects (Esteve Del Valle and Smit, 2021). Prey and Smit observe this adroitly when they recognise that although Facebook is a platform through which ‘we compose ourselves’, it is, at the same time ‘composing us’, in large part through its ‘proprietary algorithms’ (2018: 218). Jacobsen and Beer (2021a) push these observations further, proposing that the metricisation processes core to social media sites’ operational logics shape what and how we remember, as well as how we feel about our memories. So, for example, knowing the number of likes or comments a posted ‘memory’ has received from our friends or followers might impact how we subsequently feel about that memory. This notion of ‘quantified nostalgia’ is useful in its suggestion that metricisation and quantification in social networks impact memory, but it is too limited for our purposes as it doesn’t account for the recursive and propelled qualities of algorithmic nostalgia.
Vaccari and Chadwick (2020:7) have written about how banal applications of machine learning such as FaceApp are underscored by data collection practices that ‘pull users in’ in a seemingly innocuous way, but ultimately to the benefit of the technology companies. According to Sujon and Dyer, datafication processes ‘are linked to the emptying out of social connection’ within digital environments, leaving only ‘data accumulation, vanity metrics, algorithmic bias and surveillance capitalism’ (2020:1126). Data-driven practices, especially where they are perceived as opaque, feed concerns about surveillance, inequalities, security and governance, as well as in relation to data colonialism and extractivism. 10 These in turn amplify unsettlement about possible future uses of this technology, and about the implications for individual and social mnemonic practices. To Grønning, it is imperative that studies of mediated memory are grounded critically and remain mindful of ‘socio-technical developments and the affordances of the digital modes of communication’ (2021:744). In our study, as is demonstrated in this section and in the concepts of ‘remediated memory’ and ‘algorithmic nostalgia’, we have responded to this prompt. In Approach to the Study, we introduce our approach.
Approach to the study
This study is oriented around the launch of MyHeritage’s Deep Nostalgia feature, 25th Feb 2021. Its discussions are informed by a thematic analysis of Twitter data on the hashtag #DeepNostalgia from the following two-week period. The Twitter data were sourced from Vicinitas. 11 The resulting dataset (n.6935) was quantitatively analysed using available metadata from Twitter, before a randomised five percent sample was extensively and qualitatively analysed (n.347, reduced to n.316 once tweets that were no longer available were excluded). 12 This smaller sample was a ‘thickening strategy’ for the analysis in recognition that ‘data abundance’ is not in itself an indicator of research quality or insightfulness (Latzko-Toh et al. 2016) and following Manca’s assessment that there is ‘a need to use mixed-method approaches that combine quantitative tools and qualitative instruments’ (2021:14).
The quantitative analysis of manifest data included date, language, type of media, type of tweet and number of replies, retweets and quoted tweets. During the inductive and qualitative analysis of the smaller sample, we coded for themes in relation to emotional resonances, attitudes toward the animation of bodies in Deep Nostalgia (was it ‘creepy’ or ‘amazing’ for example), assessments of the ethical and technological ramifications of this practice, and the tone of tweets. We also carried out an analysis of any visual materials in the post. This more nuanced manual qualitative analysis meant we were able to remain attentive to each tweet’s context and the juxtaposition of elements it contained. So, for example, tweets containing combinations of text, videos (mostly shares from the Deep Nostalgia feature) and emoji could be considered in their entirety, or if a post was a response tweet we could refer back for context, as well as exploring any comments or discussion that followed.
We acknowledge the limitations of any study of data of this type, and in this way. Our analysis needs to be mindful that a ‘Twitter public’ is not by any means a representative societal cross-section, and that choosing this network rather than any other will have skewed the population in our study. It also no doubt impacts the ways the issues were debated, given that Twitter tends to be disproportionately white, male and managerial/professional in occupation (Sloan, 2017). Indeed, those who work or research in the technology sector tend to be over-represented on Twitter – as was the case in our sample – and this too may have impacted the character of the discussions. 13 There were practical reasons for our choice of social network however. Chief amongst these was the fact that MyHeritage was using Twitter as a way of launching the feature, but acquiring a dataset was also more straightforward than it would have been for Facebook, the only other network Deep Nostalgia users could share their creations through. 14
These data shed light on the confluence of public and professional discourses circulating around the launch of Deep Nostalgia. In the following section, this exploration grounds our discussion.
‘This is the first time I’ve seen him smile’: Remediated memory, algorithmic nostalgia and the ambivalent logics of social media
Our findings unfold across three sections corresponding with the research questions introduced in Introduction, and analysed and discussed within the theoretical framework set out in Remembering the Dead In an Algorithmic Present. In Deep Nostalgia’s Remediated Memory, we explore the concept of remediated memory, that is, how the shift from photo to video in Deep Nostalgia reshapes memory and memorialisation practices. In Deep Nostalgia’s Algorithmic Nostalgia, we focus on the affective dimensions and resonances of the creations, exploring how they might be understood as algorithmic nostalgia. Finally, in The Logics of Socio-Technical Infrastructures, we discuss how social network logics shape users’ interactions with Deep Nostalgia, and how they underscored its intense virality at the time of the launch.
Across all sections, we give particular attention to ethical questions raised by these practices. There was concern expressed in wider media coverage of Deep Nostalgia about the data collection processes which underpin its functionality, both within the specific context of MyHeritage, and the broader spaces of social networks. However, in our sample of tweets, ethical concerns were raised in only 6% of posts, with most of those raising issues being researchers, scholars, journalists, and writers, often with an interest in technology (as per their user biographies): ‘@xxxx People are keen to feed the #DeepNostalgia #MyHeritage engines. Soon these phantasms will have voices and will speak to us, with words supplied by the likes of Cambridge Analytica. Very bad psyops incoming.’
15
Nonetheless, we noted that there were other, more subtle, ways in which we could interpret some of the users’ comments and practices in our sample in relation to ethical concerns – from coding for user’s reaction to the fidelity of animations, to noting what kinds of content were the most popular. In 4.1, there is a subsidiary focus on the authenticity of animations framed within current debates about deepfakes. In 4.2, we analyse the ethics of reviving the dead, and in 4.3, we highlight the datafication of users by Deep Nostalgia compounded by its reliance on social media, in this particular case, on Twitter.
Deep Nostalgia’s remediated memory
Deep Nostalgia is powered by a deep learning algorithm created by D-ID to match each uploaded photo with a compatible ‘driver’ video to support that photo’s animation most convincingly. It does so by analysing key features in the photo such as head orientation, and matching those features to a suitable blueprint video. This means that, like other algorithmic systems, Deep Nostalgia operates best at a degree of abstraction; in the design of such a system, a programmer will exclude more atypical or ‘chaotic’ data inputs in a bid to secure more predictable and persuasive outputs (Markham, 2020:10). This underpinning logic leads to a degree of uniformity and conformity during Deep Nostalgia’s process of remediation. These are not the subject’s own movements: they have to all intents and purposes been directed by a programmer. El-Hadi captures this well when he calls the videos ‘digital frankensteins’ (2021); creations of people with gestures that they perhaps never intended or meant.
This shift, from photograph to video (now with driver video sedimented within it), is clearly significant in its attempt to maximise immediacy. As the subject in the photograph is set in motion, so too is the user’s line of vision, a potentially more involved and intimate engagement. This is what we conceptualise as a remediated memory of that person. The short, looped video refashions other media (the photograph, the blueprint video), and any memory attached to the photo comes to be represented, echoed or distorted, within a new medium. The process could even be said to put those memories into conversation with other(s’) memories in that it relies on movements and expressions that are borrowed from another; the coming together of two datafied ‘bodies’ (as ‘frankensteins’ to echo El-Hadi above).
Remediation operates in this case in the ways in which the new medium – powered by artificial intelligence – intersects with the ‘old’ medium of analogue or digital photography; not superseding it, but refashioning it for mennonic purposes. This form of algorithmically remediated memory oscillates between two strategies. The first attempts to eliminate the medium, making the user ‘believe that he [sic] is in the presence of the objects of representation’ (Bolter and Grusin, 1999: 272–73) – in this case, their deceased relative. The second draws attention to the medium (and materiality) given that it is the user who actively triggers the remediation of the photograph – an example of what Bolter and Grusin term ‘hypermediacy’ (1999; 272). In suggesting the concept of remediated memory, we pick up on van Dijck’s prompt to consider the process of remediation when thinking about ‘mediated memory’ (2007: 48–49). We do so in order to highlight the ambivalent, iterative and hypermediated quality of memories that have undergone remediation, especially where they are co-constructed through algorithms and/or networked with (in) social media, as is the case here. Of course, remediation has significant implications for memory work both individually and collectively where it unsettles or recasts our connections with our past(s). There are profound ethical considerations related to broader debates about memory modification and enhancement, as well as the externalisation of memory, through new and emerging technologies (Kourken and Sutton, 2017).
This process of mnenomic remediation was an important theme in the tweets in our sample. We found evidence in comments on the technology, the movement of bodies, or the authenticity of the resulting animations. We present results from our analysis in the following table (Table 1) and discuss them below:
Deep Nostalgia’s remediated memory is intriguing to consider in light of broader scholarship about the ambiguity of digital memory practices, which Garde-Hansen et al. (2009) propose destabilise the boundaries between life and non-life, and organic and inorganic things. In Algorithmic Practices and Memory Work we situated Deep Nostalgia within the context of work on ‘algorithmic afterlife (Lambert et al., 2018) given that the qualities of ‘synthetic resurrection’ (Ajder, 2019) are evident in its animations. Here, we focus on reactions of users to the embodied qualities of the animations, a recurring theme in 61% of tweets in our sample.
16
For example: ‘Animei a foto do meu avô no #DeepNostalgia e mandei pra minha mãe no whatsapp. A véia ficou tão emocionada que chorou: “Como vc conseguiu isso? Chegou me da nervoso, parecia ele com sorriso mesmo”’. [I animated my grandfather's photo on #DeepNostalgia and sent it to my mom on whatsapp. The old woman was so moved that she cried: “How did you do that? It made me nervous, he looked like he was smiling.”]
We identified two key mnemonic responses to Deep Nostalgia’s practices of remediation in our sample, sometimes in isolation, and in other instances, in combination (in keeping with the ambivalence and the double logic of remediation we record in this article). In the first response, remediation solidifies or even amplifies an image’s memorative significance; the video is understood to restore the person in the image and further solidifies any extant mnemonic attachments. Here, the emphasis is on the ways artificial intelligence underscores a kind of fixity or eternalisation, resurrecting people (however incompletely) through animation, and even supporting the creation of new posthumous memories, of the smile for example. In the second response, the process of remediation seems somehow to undermine an image’s memorative clout. The meaning of the image, and perhaps its associated mnemonic qualities, become(s) less certain or fixed through the hypermediacy of remediation. For example, Bolter and Grusin (1999:28) suggest that in a still photograph it is the light reflecting off a person which provides the invaluable contact point between that person and their representation in the resultant image. In Deep Nostalgia, creations that contact point is obscured by, or perhaps finds itself in antagonism with, the algorithmic process; it begins to bend and flex in a way that is not wholly convincing or comforting.
Many of those in our Twitter sample who commented on the nature of the embodiment on offer through Deep Nostalgia demonstrated a kind of ambivalence in response to these gestures, displaying conflicting responses to it simultaneously: ‘I’m both in awe and creeped out. Still super cool‘. The video outputs amount to forms of ‘virtual creepiness’ (Anderson, 2019: unpaged) for many, and there is a cumulative sense of the uncanny in our dataset, with users referring to the animated images as ‘creepy’, ‘weird’, ‘freaky’ or in similar terms in 11% of tweets our sample: ‘Ok so some of the #DeepNostalgia stuff is so uncanny it’s terrifying, but I took a photo of my grandad from the archive of family photos I made a while ago and ran it through and honestly - the result is quite neat!’
This notion of the uncanny is interesting, connecting with debates within robotics and artificial intelligence about feelings of unsettlement provoked by representations that are human, but not quite human enough. The uncanny, according to Arnold-de-Simine, can be both enabling and dangerous, allowing people to ‘hold potentially conflicting reactions (disturbing/comforting) in suspension’ (2019:92), as we saw in the ambivalence expressed by many in our Twitter sample. This notion of ambivalence will be a recurring one in the sections that follow, demonstrating the unsettled quality of our responses to these technologies, and of the technologies themselves.
Exploring ‘remediated memory’ in the dataset (n. 316).
Deep Nostalgia’s algorithmic nostalgia
Here, we explore Deep Nostalgia’s claim that it ‘bring [s] beloved ancestors back to life’ for nostalgic purposes. We offer an overview of our data sample, analysed against the concept of algorithmic nostalgia. We explore how the technology as operationalised by the users in our sample crafts, charges, or contains differing versions of nostalgia, particularly through its affective impacts.
In our study, we were therefore attentive to ‘the emotional landscape’ (Powell, 2018:13) produced by Deep Nostalgia’s creations. MyHeritage noted proudly in a tweet on Mar 3 that ‘#DeepNostalgia is bringing MyHeritage users to tears!’ and this emotional resonance was in evidence in 60% of our sample. People were willing to overtly perform their (overwhelmingly positive) emotional responses to these moving images within social networks:
19
‘Thank You to @MyHeritage for animating this photo of my Grandfather. He tragically died at the hands of another in 1966, years before I was born. This is the first time I’ve seen him smile. #DeepNostalgia #RootsTechConnect’
As noted in Remembering the Dead In an Algorithmic Present, interest in our digitally mediated emotional lives has been increasing in recent years, not least for commercial reasons (McStay, 2018:1). The imperative to make people feel – and to encourage expression or performance of feeling through, for example, the use of emoji – is now strong, whether it be excitement, pride, anger or despair that we are experiencing. Advertisers and brands, as well as political and cultural actors, have paid increased attention to expressed or implied ‘sentiment’ (Puschmann and Powell, 2018), and emoticons and animated gifs have become part of the day-to-day vernacular of digital communications (Miltner and Highfield, 2017). This is worthy of note in relation to Deep Nostalgia given the ambitions of the corporate actors here: MyHeritage aims to promote and increase users for its genealogy services through exposure, and social media companies (including Twitter) create economic value through connectivity and the collection of data. The likelihood of both these ambitions being met is significantly increased where users’ emotions can be mobilised. This can be understood as ethically jarring; the calculations made as Deep Nostalgia processes an image are computational and abstract, but the impacts they produce are emotional and psychological.
An important part of the success of the affective dimension of Deep Nostalgia can perhaps be attributed to a kind of ‘biomediation’ (Garde-Hansen et al., 2009: 12) where technology interfaces with the human and communicates through processes that are more than semiotic and cognitive (Angel and Gibbs, 2006). We saw in our sample that it is often movement in the images that triggers emotional resonances. This is unsurprising given that most of the animated images feature deceased relatives, many of whom would not (or could not) have been captured through moving imagery in their lifetime.
Content of image animated with Deep Nostalgia.
Interestingly, across the top 10 tweets on all attention measures in our sample (likes, retweets, comments and quoted tweets) all but one featured historical figures, and half were paintings or sculptures, like a popular recreation of a Neanderthal model shared by the Natural History Museum, London. This suggests people felt more inclined to like, share or comment on posts where the animated subject was evidently not somebody’s deceased relative. This begins to suggest the ethical considerations that are thrown into sharp relief by D-ID, MyHeritage, Deep Nostalgia, and its creations. Of particular concern is the question of how a deceased person might be considered to have consented to the uploading of their image. Although it may be true that the dead cannot object, and there are no legal obstructions that prevent a relative from animating an image, there are clearly ethical and moral considerations here where a person’s right to be forgotten meets another person’s sense of a duty to remember, and the agency that they might feel comes with that.
Deep Nostalgia recreations are about more than their deceased subjects however, to which they have more than an indexical relationship. They become symbolic mnemonic objects that mediate users’ own identities and not just those of their deceased relatives. The chance Deep Nostalgia gives to users to reanimate photographs from their personal archives can be read as filling in gaps in familial histories (although clearly not straightforwardly), and another aspect of self-construction. The use of imagery more broadly within social media has become an important aspect of identity construction and self-representation, and how a person’s photos intermingle with the photos of others is an important aspect of those processes. Deep Nostalgia might contribute to the co-creation of networked identities as ‘users contribute to the stories of each other’ (Leaver, 2018), in this case including those already dead.
Susan Sontag understands photographs as ‘incitements to reverie’ (2008:16), and we understand our dataset to be nostalgic in that it constitutes a psychosocial mobilisation of emotion (as we saw above), more often than not, galvanised around past-ness; pasts that are constructed as silent and smiling in the Deep Nostalgia videos. We questioned whether this amounted to an excess of ‘historical musing’ which is ‘myopic, perpetual and ultimately destructive’, or whether it could instead be understood as a ‘reflexive, collective or adaptive view of history’ (Lizardi, 2016). Our research suggests that the history on offer in Deep Nostalgia is more persistent than adaptive, and quite literally perpetual in that it plays on a loop. These mechanisms are coded by programmers in ways which are no doubt context and culturally dependent, and the nostalgia assembled is (over)determined and situated as a result. Affective remediated memories underscore a pervasive algorithmic nostalgia in our dataset then; outputs which tend toward uniformity and conformity (as we noted in 4.1) feed a generalised nostalgia in our data which is generated and organised through automated and recursive mechanisms. 21 The Deep Nostalgia interface makes it possible for users to easily onboard, upload and animate their photographs, and, crucially, share the resulting video on social media, effectively nostalgia-ficating what is otherwise a marketing tool.
It is too easy however to dismiss Deep Nostalgia creations and the algorithmic nostalgia they elicit as sentiment and myopia. As Sayers notes, ‘nostalgia is not always, or only, a sign of stuck-ness’ (2020:190). In some moments we saw uses of Deep Nostalgia that accorded well with the general sense we can have in social networks, and in society more generally (according to Lipovetsky 2005), of living in a perpetual present; distanced from the past and feeling insecure about the future. But in other moments, there was interest in the past which might be understood as productive or generative. Significantly, we found uses of the technology aimed at revindicating a collective memory of people that experience racism in ways that seemed more productive than myopic, for example in the frequent animation of the Black abolitionist Frederick Douglass. 22 Again, the ambivalence of the technology is at play here as Deep Nostalgia ‘conflates the desire to honor the past with an impulse to appropriate it’ (El-Hadi, 2021). The multiplicity of meanings attached to the process of animating, remembering and sharing these images are negotiated by users in ways that collapse and exceed the intended uses of the technology, while ethical considerations remain – for example, the disproportionately harmful effect that these technologies might have on Black people (El-Hadi, 2021).
There was curiosity evident in our sample also – about what future for remembering these videos might anticipate or set in motion, and what possibilities might flow from that. Deep Nostalgia creations are then about the past, the present and the future, a paradox symptomatic of the ambivalence of this technology, and of responses to it in our dataset. This is unsurprising given that, as Routledge contends, nostalgia is a ‘complex emotional experience’ (2016: 44), that can be both past and future-oriented in its adaptive qualities (FioRito and Routledge, 2020) and which can itself be felt as ambivalent.
Algorithmic nostalgia is clearly of interest in relation to the technology’s memorative effects and affects, as well as how these are achieved through processes of automation and remediation. It is also intriguing to examine in the context of social networks’ algorithmic and datafication practices, where monetising connectivity and user data emerge as considerations. In the next section, we explore these in more detail.
The logics of socio-technical infrastructures
In this section, we respond to research question three by exploring the ways social media’s connective and attention logics shaped how users interacted with Deep Nostalgia. Our analysis reveals that the outcome of using Deep Nostalgia is not always zero-sum a memory (and by extension a memorialist). It resulted in dialogic practices that were communicative, performative and extractive too.
Sharing via social media is clearly an integral part of the logic of Deep Nostalgia, blurring the boundaries between private and public memory. 70% of tweets in our sample shared animated images, a figure which increased in the members of the public category, where 88% of people had used Deep Nostalgia and shared the results. We found that people often used (and shared) Deep Nostalgia more than once, posting threads of multiple animated videos: ‘I’m still obsessed with #DeepNostalgia’.
Nearly a quarter of tweets featured the default text accompanying animated images when shared directly from the MyHeritage app or website; ‘I love the way MyHeritage brought the people in my photo to life! Try it too and be amazed. #DeepNostalgia’. These users therefore acted as inadvertent marketeers for MyHeritage, given that sharing an animated photograph was inextricably linked with promoting the technology, whether actively encouraging others to use it or not. When looking at the comments, the success of this strategy can be seen as we found many users asking how to use the feature and where to find it. Animated images linked to the MyHeritage website in most cases, where to use Deep Nostalgia, users were asked to sign up and relinquish, at the very least, a name and email address.
For MyHeritage, the sharing of videos performed an invaluable social advertising function where promotional messages were blended with users’ sense of identity and belonging (van Dijck, 2017). In terms of public endorsement and brand image, such an approach is invaluable for a company like MyHeritage; their product (genealogical services) is turned into a story that can be excitedly shared. According to van Dijck (2017), such practices do not constitute social or collective memory however so much as mere ‘connectivity’ in service of the social media platforms and their business models. The animated image is merely a(nother) ‘transaction in a data network’ (Dewdney, 2022:24). Echoing Bory’s observations, elements of the spectacle are in evidence here, employed by an AI company as a way of advocating for its products (Bory, 2019). This is noted by some users who criticise or ridicule the technology for these very reasons: ‘Deepfake technology is used to bring dead relatives back to life. I’m sure marketers are already strategizing this for marketing plans down the road.#deepnostalgia #deepfake’
We found in the tweets, as one would expect, many of the vernacular elements of Twitter as a platform. The playfulness of social media interactions appeared in 13 % of tweets in our sample. These frequently featured gifs as a framing device. Several tweets also made connections to other popular posthumous animations, for example, in the Harry Potter universe, offering a critical reading of the purportedly innovative nature of the technology. While not prevalent in our sample, such responses are a helpful reminder of the importance of the visual in social media cultures (e.g. Leaver et al., 2019) and the extent to which the entire Deep Nostalgia campaign was enmeshed within and shaped by the ‘internet’s visual turn’ (Vaccari and Chadwick, 2020). Here we are reminded that the adoption or ‘domestication’ of these technologies is not always linear; users find alternative ways of employing them, including those that are subversive or whimsical (Kitchin, 2017: 19).
Social media companies are navigating the emergence of deep learning technologies in real time and are clearly disinclined to restrict activities that are gaining traction and attention, not least given the importance of ‘compulsive connectivity’ (van Dijck, 2017) to their business models. The response of social media companies to deepfake technologies has thus been slow and, so far, mixed; TikTok has now banned what it calls ‘synthetic or manipulated content’, Facebook has banned deepfakes although with exceptions for parody and satire, and Twitter’s policy is that tweets featuring deepfakes should be labelled as such, only removing them if they are likely to cause harm. Our data suggests that most users were not troubled by these ethical concerns however or were at least able to put them to one side to participate in what had become a shared mnemonic experience within peoples’ social media feeds.
The viral nature of this case study is worth reflecting on here, alongside a broader look at the ways AI is discussed in society. According to Nguyen et al. (2021) AI tends to be framed within discourses about technological trends, economic potentials, data risks and questions of governance, but Deep Nostalgia also has cultural and historical dimensions which are less common, and a strong personal/familial aspect too. This unusual emphasis may explain Deep Nostalgia’s rapid (albeit fleeting) popularity within social networks; a chance to partake in an activity that was somehow dissonant or not normative, yet also familiar given its characteristics. The easy-to-use capability for sharing creations on Twitter and Facebook, 23 coupled with the perceived novelty of the technology and the combination of emotional resonance and playful elements, no doubt contributed to the viral success of Deep Nostalgia in the weeks following its launch. To some degree, the connective nature of genealogy communities online might also bear on its popularity, given that ‘genealogists use technology to research their family, but they also use technology like social media to connect with other family researchers and to share ideas and information’ (Kennedy-Eden and Gretzel, 2021). Deep Nostalgia did not trend for long however and was ultimately unable to sustain itself as a phenomenon. Nevertheless, its work for MyHeritage was done.
Following Natale, we might note this as the point when Deep Nostalgia’s deception became ‘banal’ and disappeared into the fabric of our daily lives (2021); the fact that it so quickly became predictable and ordinary contributed to its recuperation. The next time we see forms of algorithmic nostalgia being promoted – such as in Amazon’s recent announcement that its Alexa will be able to channel the voices of dead people in a bid to ‘make memories last’ (Paúl, 2022) – there will likely be less surprise, friction, and most troublingly perhaps, even less attentiveness to its ethical ramifications. As Bory points out, it falls to us then as media and communications scholars to identify and understand ‘how corporate narratives are driving the symbolic and cultural integration of new intelligent systems in society’ (Bory, 2019).
Conclusion
In this article, we have explored Deep Nostalgia, and the mnemonic and socio-technical concerns that it brings sharply into focus. In doing so, we have been careful not to treat the creations as mere ‘technological curiosities’ (Vaccari and Chadwick, 2020:10), instead seeking to understand their richness and ambiguity. Our focus has been on Deep Nostalgia’s algorithmic and automated deep learning technology and its effects and impacts beyond genealogy communities. The success of MyHeritage’s strategy of reaching out to a broader public is evident in user profiles and the content of tweets in our sample. 24
Through this study, we have explored a technology that not only brings the past into people’s social media feeds, but animates it, ‘reviving’ the dead as what we have termed ‘remediated memory’. Deep Nostalgia’s shift from photo to video reshapes and remediates memory such that it becomes both real and unreal, immediate and remote, comforting and disturbing. We considered concerns about the authenticity of the animations in relation to deepfakes, and beyond the context of a genealogy service that trades on reliability and credibility. We also introduced the concept of ‘algorithmic nostalgia’ to refer to the ways nostalgia itself can be generated or organised through automated and recursive algorithmic mechanisms, including in Deep Nostalgia. Algorithmic nostalgia was revealed as at once myopic and productive, static and generative, as well as past, present and even future-oriented. We have seen that mechanisms for remediating memories such as Deep Nostalgia have the potential to disrupt established dichotomies and demonstrate varied ambivalences, and we have interrogated the ways social media logics shape the use and influence of these outputs. Our study starts to unpack the social implications of this technology, demonstrating an enduring individual and collective need to connect with our past(s), and a desire to test and extend our memories and recollections through increasingly intense and proximate new media formats.
Remediated memories raise multiple ethical and theoretical questions which we have begun to unpack. Nascent tensions around exploitation, extractivism and manipulation underpin debates about these technologies, but we have demonstrated that for the majority of users sharing their creations on Twitter, these concerns were overlooked. Cheney-Lippold makes the case that ‘we lack the vocabulary needed to enact a politics around our algorithmic identities’ and we would extend that point to the activities recounted in this article (2017: 30). Surfacing these considerations is especially important as mainstream uses of these technologies increase, as is bridging where appropriate to critiques of business interests which thrive on the collation and connection of personal data (in our case here, MyHeritage and Twitter).
As we have demonstrated, there are social consequences and implications for personal and collective memory-making where automation, algorithms and ambivalence become banal. Continuing to draw attention to the ethical dimensions of these technologies is critical – whether in relation to the contextual and situated nature of deep learning methods (including the social inequalities they perpetuate); or the effects of the nostalgia-fication of memory and the manipulative mobilisation of grief. We must relentlessly and perceptively chart the past’s adaptive and pervasive qualities, not least where those intersect with corporate interests and narratives.
Footnotes
Acknowledgements
The authors thank the anonymous reviewers for their valuable comments.
