Abstract
This article examines discourses about mixed reality as a data-rich sensing technology – specifically, engaging with discourses of time as framed by developers, engineers and in corporate PR and marketing in a range of public facing materials. We focus on four main settings in which mixed reality is imagined to be used, and in which time was a dominant discursive theme – (1) the development of mixed reality by big tech companies, (2) the use of mixed reality for defence, (3) mixed reality as a technology for control of populations in civil society and (4) mixed reality as a technology used in workplace settings. Across these settings, the broad narrative is that mixed reality technologies afford overwhelmingly positive benefits like efficiency and security through their capture, relay and rendition of data (about the environment, about the body etc.) – affording a form of anticipatory power to the user. The framing of temporality, we argue, is underlain by social and political values, which represent certain interests, but leave others out in the imagination of mixed reality's technological advance.
Introduction
Mixed reality (MR) – an emerging class of head-mounted and spatially aware computing devices – is a domain of increasingly intense focus for major technology companies. According to Milgram and Kishino’s (1994) foundational definition, MR refers to a spectrum of technologies – such as virtual reality (VR) and augmented reality (AR) – which to varying degrees incorporate the spatial dimensions of the digital into the physical and vice versa. More recent work by Heemsbergen et al. (2021) has sought to focus less on understanding MR in terms of formal qualities – particularly, around virtual-real dichotomies and defining features – and more on mediation: what these media do, what sorts of sensory experiences they elicit in their subjects. MR, in this sense, ought to be understood not in terms of the kind of ‘image’ it produces, but in terms of how these images – where the digital is part of our vision of the world – establish new configurations of perception and agency through the interplay between digital and physical space. As emerging accounts of MR have shown (Egliston and Carter, 2021c), these capacities for mediation are subtended by technologies and techniques of data capture and processing as a means to translate the embodied inputs and rhythms of the user and their environment into a machine-readable format for the MR system.
Like many other technologies that require or generate data in their use, MR is increasingly characterised by what Fourcade and Healey (2017) call the ‘data imperative’, that is, the widespread institutional and cultural imperative to collect and analyse information about users and uses of technology. In this article, we pay attention to the some of the emerging political, social and economic aspects of MR as a data rich technology. Rather than exploring the use and experience of currently existing and adopted MR technologies (cf. Carter and Egliston, 2020; Saker and Frith, 2020), this article explores the imagination of emerging forms of MR and the futures they promise to instantiate by virtue of their extensive data processing, or what we elsewhere refer to as a ‘fantasy of perfect data’ (Carter and Egliston, 2021) – a fantasy that MR's intensive sensors provide complete, and therefore more objective or trustworthy, insights about the world.
Our analysis looked at visions of MR ‘futured’ through advertising and promotional materials, industry and developer conference talks, developer interviews with trade presses and patent documents. Indeed, discourse around MR technologies is anything but incidental – technologies on the MR spectrum have existed largely within a discursive register for the last several decades, and these discourses whether industrial (Chesher, 1994; Evans, 2019; Harley, 2020; Liao, 2016) or academic (Heemsbergen et al., 2021) have shaped what MR-spectrum technologies look like today.
Our analysis was guided by two main questions. First, on what grounds do MR companies create and sell a coherent and positive vision of the future? We suggest that MR is framed as a technology offering security, prosperity and efficiency – things that are presented as unquestionably positive. The theme of temporality manifests in these future visions, specifically, the idea of the subjective apprehension of mediated time through MR interfaces, which offer a pre-emptive, anticipatory capacity to users. In applications we surveyed, MR generated perceptions of a ‘now’ moment through the ‘cramming’ of an instantly retrievable past, that shapes people's anticipation of the future. MR is framed as responsive to changing and contingent circumstances through the constant capture and processing of data, often rendered as a visual overlay of bodies or environments.
Second, following from writers like Andrejevic et al. (2020), we take stock of MR's temporal politics, that is, how the pre-emptive or anticipatory experiences it promises to instantiate enact a vision of power that works in potentially asymmetrical and harmful ways. As such, we ask what values and agencies get realised through such imaginaries of MR experience, and what get left out? What can this tell us about the potential inequalities and unevenness of an imagined MR future? Of course, in our analysis we do not take the claims to a particular vision of the future at face value (cf. Kinsley, 2010; Vinsel, 2021), many of which are premised on mostly impossible or vaguely defined logics, futures that may very well never come to materialise, and may well be proliferated to drum up speculative investment capital. Yet, we believe they are useful to engage nonetheless in that they reveal the kinds of logics, aims, values and biases held by these companies, those which guide MR's future trajectory.
To explore how the MR imaginary is charged with political valence, intertwined with the interests and values of powerful actors developing, proliferating and harnessing these technologies, this article focuses on four case studies – settings in which MR is imagined to be adopted. These are the use of MR by (1) big tech firms (focusing on Facebook's Oculus technologies), 1 (2) the military, (3) the state (as a mechanism for governing civil society) and (4) the workplace.
To guide our analysis of MR's temporal politics, we draw principally from the ethico-political philosophy of Bernard Stiegler, particularly his thought on mediated (temporal) perception and power (1998, 2009, 2011a), one that has been productively used in studying the perceptual shaping mediated through data-based technologies such as videogames (Ash, 2015), spatial computing (Kinsley, 2015) and machine vision (Azar et al., 2021). To summarise briefly here, Stiegler's wager is that mediation is always a component of perception, and that this mediation increasingly (in the moment of networked digital media) serves the (largely economic) ends of those who create and distribute those media. Stiegler's account of media temporality and psychopolitics (2010, 2011a), particularly of the technical shaping of expectation, the capacity for projective, forward-leaning thought – offers both an explanatory framework in describing how MR is commonly framed as pre-emptive and shaping our anticipation (on the basis of its retention of information about users, their bodies, the spaces they inhabit), but also a lens through which to critically think about the stakes of these new technologies for everyday life.
This analysis, following Stiegler's philosophy, reveals the way that the imagination of MR futures tends to frame the technology as diminishing the capacity of users to perceive the present on the basis of accumulated knowledge of the past, in turn shaping anticipation of the future (cf. Stiegler, 2010: 37–44). Rather, the present ‘now’ operates on the basis of futural anticipations and memories of the past collapsed into the present through an interface that captures (or retains) and relays large amounts of data – rendering individuals as more profitable consumers, workers or subjects for those controlling the destiny of MR; a future that benefits some, and serves some interests, more than others. While Stiegler himself does not substantively invoke questions of justice or (in)equity, we see in his emphasis on the entanglements of technologies and lives, on an ethics of being and becoming, and moreover, mediation, a powerful complementary perspective to those borne out in fields like critical data studies, which brings a critical sociotechnical lens to data-based technologies. In drawing from Stiegler and contesting MR futures at the level of discourse, the goal of advancing this more critical research agenda is not solely to give more nuanced attention to aspects of MR which receive minimal and uncritical attention at present. Much in the same vein as existing ‘anticipatory’ approaches to studying emerging technology, from literatures in responsible innovation (Stilgoe et al., 2013) and STS (Jasanoff, 2003), the broader normative goal of asking these kinds of ‘what if?’ questions about futures to come – and evaluating the basis on which these visions are constructed – is to provide the means for both critique and to articulate the possibility of more equitable and socially beneficial futures of MR.
It is our hope that such reflections can catalyse conversations in academia – particularly in orienting conversations about MR in critical data studies (Dalton et al., 2016) and data justice (Taylor, 2017). Presently, these fields encompass most existing data-gathering things ‘smart’ and ‘sensing’, from homes to cities, to lightbulbs – yet we lack, and urgently need consideration of MR devices as new, impactful forms of sensor technology. Questioning the ‘positive’ visions of this technology is as such a first step– here, to critically evaluate the promotion of these futures and to think about what more equitable futures might look like. It is through Stiegler's call to think critically about the impacts of technology on human experience and perception – things which threaten both psychic and material violence – that we argue, following Stiegler, for a rational, reflective view toward these imaginaries, to take these technologies as instantiating regimes of control over action and cognition.
This article proceeds as follows. We first introduce existing literature covering mixed reality spectrum technologies and power, particularly those focusing on data. We then outline our approach to the study material. From there we introduce our theoretical framework, Stiegler's political and philosophical project of mediated time. From there we discuss four main areas in which MR is imagined as a data-intensive device, and something that is discursively framed as beneficial to certain social and economic interests through its temporal affordance of anticipation. The article concludes with a brief discussion of how we might imagine and practice MR futures that move away from violence and control.
Prior work: VR, AR, data and power
A range of work to date has addressed questions of data and power to do with technologies on the MR spectrum (AR and VR). In a way that dovetails with Srnicek’s (2017) attention to common surveillance-focused modes of capital accumulation in software today, much of this literature focuses on how AR/VR technology companies control the flow of data emerging from their software, feeding it back into the optimisation of that software. An adjacent set of concerns emerge around questions of privacy – focusing on the geospatial or embodied nature of MR data – chiming with broader concerns, expressed in this journal, with the ‘spatial’ nature of big data (see Dalton and Thatcher, 2015).
Much existing work on AR and data is focused on mobile (smartphone) AR, primarily about the AR games Pokémon Go and Ingress from developer Niantic. As Moore (2013) writes, AR games are data rich technologies. They generate and require a great deal of data to fulfil their most basic operations – with data the means through which AR games generate (often location specific) images perceptible to and manipulable by the human user, and how they turn the continuous movements of human bodies through space into a format legible to the game. How we experience a mundane activity like capturing an Ingress portal or controlling a Pokémon Go gym, is something that happens through data as a (scarcely perceptible) layer of mediation. Work to date has variously explored questions of power to do with the data-rich nature of AR gaming – covering questions such as the use of AR data to mediate commercially desirable activities in physical space (Whitson and French, 2021; Zuboff, 2019). Beyond mobile AR, wearable AR has emerged as a point of focus around the short-lived Google Glass, rolled out in 2013–2014. Writers have addressed both privacy concerns (Mann and Ferenbok, 2013), and impacts for policy and regulation, conflicting with privacy laws (Meese, 2014; Wassom, 2014).
Work on VR has begun to address similar issues surrounding data extraction. Indeed, VR has historically been conceived as a natural or intuitive embodied technology, thus reliant on sensing and processing both embodied inputs and the environments in which the technology is being used (see Coyne, 1994). Data was a central component to the industrial discourses of VR in the 1990s, with companies like VPL developing wearables that were sensitive to discrete movements of the hand and fingers (see Chesher, 1994). More recent commentaries have begun emphasising the way big tech companies (like Facebook with their Oculus acquisition) aim to use VR for the capture of a new form of data (see Egliston and Carter, 2020; Evans, 2019). Recent empirical work in HCI (Pfeuffer et al., 2019) and computer science (Miller et al., 2020) has pointed out that data collected by consumer VR devices is so intimate with the user that it is biometric. Miller et al.'s study highlights that 5 min of VR data (with all personally identifiable information stripped) correctly identified 95.3% of a 511-participant sample. Beyond the body, other recent scholarship has discussed the way VR devices sense the built environment. Saker and Frith’s (2020) concept of coextensive space captures the way that VR devices incorporate the physical world in their experience. Subsequent work has focused on the surveillant potential (and platform-economic benefit) of this coextensive relation, of VR in tracking the body of the user and the environments surrounding them (Egliston and Carter, 2021a, 2021c.
While this literature focuses on how MR-spectrum technologies (particularly, VR and smartphone AR) threaten civil liberties like privacy, what we lack at present is a sense of the potential data borne harm and violence of emerging wearable MR (that is, those that directly mediate our vision through head worn devices) and the contexts in which it is imagined to be adopted.
Approach
Our focus on four sites of MR ‘futures’ was determined by a thematic review of literature on MR applications and a survey of popular uses (published in Carter and Egliston, 2020). In the present article, our discussion emerges from a further grounded study that incorporated discourse analysis of documents to do with the development and use of MR technology (such as patents, promotional material presented at developer conferences and events, press interviews with developers, and technical documentation). In attending to discourse we were interested in how particular knowledges and power relations are constructed through public comments or statements (Foucault, 1970) by those developing and propagating MR technology.
MR – in these documents – was a term used to describe AR and VR based technologies, which variously incorporated the digital into the physical and vice versa. For example, the Microsoft HoloLens, an interface that annotates physical spaces with digital markers is framed as a ‘mixed reality device’. Likewise, the Oculus Quest – which principally renders fully virtual environments – increasingly mediates encounters with space through AR-like features (see Egliston and Carter, 2021c Saker and Frith, 2020), and is increasingly described by Oculus as a ‘mixed’ (or ‘extended’) rather than virtual reality. While many wearable technologies augment the user's embodied or perceptual capacities in a similar way to those described in this article (e.g. wearable fitness devices and health data shaping how people think about and arrange their daily lives, see Lupton, 2019), our analysis and study material focused specifically on head-worn devices that ‘mix’ reality through the affordance of mediated vision.
Our analysis drew from Critical Discourse Analysis (CDA) (Fairclough, 1995) – an approach oriented toward the intersections of discourse, power and social structure. Our analysis of the material followed Corrigan's approach to discourse analysis in political economy of communication (2018), where a crucial step is to immerse oneself in the data as to develop sensitivity to key meanings and to spot and scrutinise inauthentic meanings and texts (2018: 2764). While analysing documents, Egliston composed memos about the various topics and time-based values that emerged and coded them into emergent categories. Initial themes identified by Egliston were subject to a process of review and discussion with Carter, sensitised by a review of the critical cultural and social issues previously identified in our wider literature review.
Stiegler and the politics of mediated time
To frame our discussion of MR's temporal politics, we draw from the work of Bernard Stiegler. We find resource in two (distinct but related) parts of Stiegler's philosophy – his constative account of mediated time, and the prescriptive, political dimension of his philosophy.
Inspired by a perceived lack of attention to the foundational role of technics in human culture and society in Western philosophical thought, Stiegler (1998) argues that the condition for human existence and individuation (i.e. the human subject's always ongoing becoming, or self-constitution) is neither the result of some innate biological force nor determined by the presence of technological supplements. Rather, he argues, that individuation is a process of mutual and ongoing refinement between humans and technics, a condition Stiegler terms ‘technicity’.
In claiming that human individuation always has an equipmental horizon, Stiegler is focused squarely on the reference point of time as the milieu of human experience, building on Husserl's phenomenology of time. 2 Following on from his mentor Jacques Derrida, who had previously developed a critique of Western metaphysics of presence by way of Husserl, Stiegler draws on and re-reads Husserl's concept of the ‘temporal object’ – which refers to how an object of our perception unfolds, and is constituted, in time (see Stiegler, 2011a: 8–34). Husserlian phenomenology centres the ‘noetic’ relation, that is, where the perceiving subject takes up an intentional relation to some subject or aspect of the world. It is as such that Husserl's theorisation of time privileges the ‘purity’ of the living present of perception – the perception of the ‘now’, and forward-leaning anticipation of the future, based on our ‘just past’ (where perception essentially works as a constant folding of the recent past back into itself). He calls this primary retention. Distinct, and always separate, he writes, is secondary retention – memory, recall or imagination of the temporal object (see Stiegler, 2009: 188–243).
Stiegler's wager is that these two forms of memory are distinct yet related (contra Husserl). Our subjective memory, he suggests colours our primary retention. Additionally, the relation between memory and perception is always mediated – with Stiegler arguing that the question of technics is neglected in Husserl (see Stiegler, 2011a: 8–34). For Stiegler, human temporality is constituted out of an anticipatory (or what he calls ‘protentional’) projection of the past, made possible through the medium of technics. Desire, agency, action, imagination – any capacity to project toward the future – are temporal phenomena that are only ever possible through technics, of a past recalled through the ‘trace’ of artefactual exteriority (such as through systems of inscription, such as orthography, which are exteriorisations of mental memory, or through objects which imply modes of thought or action. For instance, the tool as an exteriorisation of gestural memory or practical knowledge). Adding to Husserl's schema of time and memorialization, Stiegler calls this mediated substrate of memory ‘tertiary retention’. In this sense, while Stiegler is by no means offering or engaging with ‘mixed’ reality, the constative aspect of his philosophy could be essentially summed up as arguing that human ‘reality’ is originarily ‘mixed’ with artefactual exteriority.
Relevant to our analysis of MR, Stiegler's account of tertiary memory and technicity has an ethico-political orientation (from Stiegler, 2009 onwards). While technicity is the condition for care, attention, intellection and imagination, technics are – in the form of modern, globalised, ‘hyperindustrial’ and digital technics – the condition for social and psychic violence and control, destitution and deindividuation. Taking this point at its face, Stiegler might be seen as just following some of the more critical aspects of key philosophers he reads in developing his account of technics – such as Heidegger (1977), who sees an antagonism between the human and technical arising from technological advancement (with the latter imposing upon the former). Yet crucially we must understand, recalling Stiegler's thesis of originary technics (1998), that he is not arguing that technological advancement is of concern because it imposes upon life. The ways technics change up modes of experience and perception (as temporal processes) – what he refers to as a series of ‘disorientations’ (Stiegler, 2009) – are a necessary part of how we constitute ourselves. These disorientations are, however, increasingly harmful he argues, because they tend to represent a distinctive and definitive change in the dynamics of individuation, specifically, undermining the quality of individuation itself. Modern technics which capture and relay information at speed and that operate at great scale constantly inundate us with new kinds of tertiary retentions (or what he refers to as ‘digital tertiary retentions’ in Stiegler, 2016) which discorrelate the circuits between external, tertiary memory and our experiential memories, perceptions and anticipations. Digital tertiary retentions are, Stiegler argues, taken directly into experience and perception, leaving little room for careful, critical thought and reflection – rendering us as existing in a more deterministic rather than co-constitutive relation with technology (that is, undermining the technicity that makes us human).
We might think here of data driven technologies and the regimes of ‘algorithmic governmentality’ they institute, as Stiegler does in Automatic Society (2016) or Age of Disruption (2019) – in which he argues life is increasingly circumscribed through the realtime capture and relay of data. It is here Stiegler gets to grips with how (digital) tertiary retentions increasingly operate outside consciousness, becoming directly integrated into our memories and perceptions. The secondary literature on Stiegler focusing on data-based digital media highlights further instances of the deleterious effects and affects of contemporary technical shaping of memory. For writers like Ash (2015) on videogames and Kinsley (2015) and Wilson (2015) on spatial computing, Crogan (2016, 2020) on drones, or Azar et al. (2021) on machine vision, data-based systems as a mode of technological memorialisation have negative societal effects by virtue of closing the ‘gap’ between consciousness and the technical operation of these digital media, ushering in states of cognition without consciousness, so to speak. In this secondary literature, this technological enframing through retentional and protentional instruments has effects such as amplifying consumerist impulses through perceptual shaping (such as by capturing or ‘retaining’ data about consumer activity), to the diminishing of capacities for care, justice and intellection (that form through repeated engagement and reflection with exteriority). However, important to note is that for Stiegler, despite this bleak view, technics cannot completely determine how we think, feel and act – and it is for this reason that he remains optimistic for a different, better future on the basis of critical, creative, or thoughtful, engagement with our own technicity, even if this alternative is not always easy to imagine. It is in this sense, Stiegler writes, that technology can be characterised in terms of ‘pharmacology’ (2013) – as something which is at once ‘therapeutic’ in sustaining the conditions for individuation, but at the same time, something that possess a destructive and toxic tendency (with the goal of Stiegler's own activist efforts being to strengthen the former against the latter).
In what follows, we focus on how the imagination of MR technologies – which are framed as generating and capturing data about the world around the user, and about the user themselves – might be understood in terms of temporality and tertiary retention. These devices, we argue, are often framed in terms of a temporal affordance – around narratives of smartness, and a capacity to ‘anticipate’ or pre-empt by virtue of their data-richness, overlaying the user's vision with data informed interfaces or by feeding data from MR use into other databases (subsequently informing decision making processes). In keeping with Stiegler's ethicopolitical philosophy, and moreover with sociotechnical critiques of data in fields like critical data studies, we ask what these narratives ignore or overlook; narratives which themselves we might call in Stiegler's terms, ‘stupid’ (2015). 3 By this, Stiegler means without recourse to rational, reflective knowledge – and in the case of MR, without properly considering the potential implications of MR (and the forms of data-based profiling and tracking baked into it) on society. In doing this, we draw on Stiegler's theorization of mediation and time to think about the potentially harmful implications of MR mediation for shaping the temporal perception of its imagined subject. What might MR's data richness mean for the user of commercial MR, or what might it mean for those using MR in emerging law enforcement or military contexts, or in the workplace? How might MR change up how practices are enacted? But following Stiegler's broader claims about technics, and the ways that digital technics fundamentally lack the projective aspect through which new retentional and collective horizons are opened (2011b: 113), we also consider what the pre-emptive, anticipatory temporality stressed in MR imaginaries might mean for collective visions of care, knowledge and justice. How might they simply bolster currently existing (techno)cultures that enable harm, exploitation and inequality?
In so doing, we hope to not only address the inequalities that inhere within these visions of MR futures, but rather to participate in part of the call, advocated by writers like Hoffmann (2020), to challenge the credibility and authority of popular understandings of data driven technologies and the classificatory regimes they enact – in particular, the kinds of discursive violence done by normalising the ‘violent or oppressive presuppositions’ (2020: 6) of many imagined data futures. In Stiegler's terms, such critique of the potentially deleterious effects of MR as a data rich technology (on experience, perception and individuation more broadly), might be seen as a key part of how we might better ‘care’ for social life and its technological entailments. If, following Stiegler, a condition for individuation is an interplay between our individual and collective negotiations with technology, then a ‘polemical process’, reliant on ‘conflict and dissent in the encounter of communal and individual difference’ is vital to ‘remain open to … possible futures’ (Olma, 2016: 217). The goal of our work then is to ‘pay close attention to the disjunctions between popular claims about the social values and uses of technologies … in order to decenter dominant discourses about technology and their influence’ (Hoffmann, 2020: 6).
Case studies
In what follows, we critically examine four areas in which MR was imagined to be incorporated. In all cases, MR is framed as convening the protentions of its users, or those administering MR's use, through MR's data richness. These areas were in the use of MR by (1) big technology firms (focusing on Facebook's Oculus technologies), (2) the military, (3) by the state in the governance of civil society (specifically, in urban settings) and (4) workplaces. Our analysis is interested in pointing out the unevenness, inequality and disadvantage, inhering within these MR futures and the way they centre prediction, pre-emption or anticipation as key affordances.
Project Aria, LiveMaps and Facebook's mixed reality gambit
MR has become a major focus for Facebook – a company who has been subject to both public and academic scrutiny regarding its data practices (such as its profit from data-driven advertising models, see Srivansen, 2018). In 2014, following Facebook's acquisition of VR hardware company Oculus, Oculus inventor and former CEO Palmer Luckey claimed that the future of Oculus would be a merging of AR and VR into one wearable and portable device. At Facebook's annual F8 conference in 2016, Mark Zuckerberg first adorned a pair of smart glasses. With the hope that one day these glasses might be capable of both virtual and augmented reality, Zuckerberg introduced AR as technology that ‘gives you the ability to see the world, but also deliver overlays on top of the real world’ (Wong, 2016) – an interface that mediates our perception of the space around us. Framing VR as a ‘stepping stone’ to MR (Zuckerberg, quoted in Wagner, 2017), Facebook has ardently pursued its goal of a MR future. The company's acquisition of Oculus marked only the beginning of an ongoing process of acquiring a range of start-ups (ranging from room mapping to computer vision) to form what is now called ‘Facebook Reality Labs’ (FRL) – Facebook's expansive AR and VR research and development wing (with 17% of Facebook employees now working on FRL projects, see Hamilton, 2021), currently an $18.5bn a year investment for Facebook (Rubin, 2021). As Facebook put it, it is through FRL that it sees MR as “the interface of the future” (Tech@Facebook, 2021: n.p.).
Increasingly, Facebook's VR suite of Oculus technologies operate as MR rather than simply VR – with the most recent ‘Quest’ consoles including a feature for moving between digital and physical environments, and for layering persistent digital interfaces over physical space (see Egliston and Carter, 2021c; Saker and Frith, 2020). It is here that the company has imagined new advances in MR computing, such as a virtual desktop that overlays one's physical environment (see Figure 1), facilitating work from home during the COVID-19 pandemic (Oculus, 2020). As we have identified elsewhere (Egliston and Carter, 2021a), these kinds of shifts toward MR are concerning given that Oculus’ software license agreements are sufficiently broad to enable data extraction, including of the user's room dimensions and layout. In this sense, the user's embodied engagement with the interface of an Oculus device is retained in Facebook's data centres, with potential to be used as a mechanism for powering its advertising arm, that is, to use its data to develop probabilistic ad targeting, an intensification of Facebook's already existing methods of mediating protention (as Stiegler, 2016 would have it). 4
Infinite office as presented at Facebook Connect, 2020.
The company has elsewhere made more ambitious claims on the future of MR, with the September 2019 announcement of their ‘LiveMaps’ technology concept, and its renaming and expansion as Project Aria in September 2020. In summary, Aria is a research and development project for an imagined set of portable, MR smart glasses which provide contextually specific annotations of the environments the user is in (e.g. scanning the environment to locate where your keys are in the room, and notifying you that you have forgotten them before leaving the house, or providing information about a restaurant you are passing by. See Figure 2). In this sense, the interface shapes our ongoing engagement with the world as a kind of tertiary retention in a way that is framed as affording convenience or knowledge.
Project Aria, as presented at Facebook Connect 2020.
As FRL has noted during the Facebook Connect developer conference in 2020, due to the company wanting to create a lightweight device, the computing power of the device will be relatively limited, and as such is reliant upon data that Oculus has already captured about space around the user (as opposed to using algorithmic odometry techniques as it does with the Quest, which require substantial computing power, see Hesch et al., 2019). As an alternative, the company will reportedly use data sourced by Facebook employees, who are currently using prototype smart glasses throughout the United States. As FRL note, ‘the data they gather will support the development’ (Tech@Facebook, 2020: n.p.) of the digital map of the world upon which Aria will be built. It is in this way, that Applin and Flick (2021) suggest that Aria represents a means for Facebook to ‘follow us more deeply into the Commons’ – that is, into what is collective and public; capturing and using it as a mechanism in the capitalisation of attention, for creating these circuits of protention and retention that afford us Aria's imagined conveniences.
While Aria promises everyday conveniences, it is hard to disentangle this vision from the more concrete reality of Facebook, where the vast wealth accumulation of its executives is the result of its extractivist business practices (and particularly, considered in light of the company's already existing statements about Oculus’ data and its potential for use in advertising, see Egliston and Carter, 2021a). In a way that intensifies how geolocative data already operates as a valuable asset in mediating consumer behaviour (cf. Wilken, 2014) it is easy to imagine visions of the future where Aria's same affordances of spatial annotation are used for advertising, to “market to you” (as Facebook have it). As documents emerging from the 2021 ‘Facebook Files’ leaked by whistle-blower Frances Haugen would suggest, the company's ambitions for a MR-enabled ‘Metaverse’ are motivated by the monetisation (and indeed, platformisation) of all aspects of life through the medium of all-day, embodied computing, promising to ‘generate significantly more ARPU [average revenue per user] than other social graphs’ (see Rodriguez, 2021: n.p.). While Facebook frames Aria around knowledge and convenience, it is easy to think of visions of similar futures depicted in artworks like Keiichi Matsuda's Hyper Reality (2016), where MR interfaces inundate the user with a deluge of personally and contextually targeted advertising; new (and potent, due to their appearance in our field of vision) forms of algorithmic targeting that lie outside of yet have the potential to shape human thought and action.
War and the military industrial pipeline
Across the study material, MR was widely framed as something to exert control and power over people – tracking, identifying and ordering information about people and things in order to shape decision-making capacities, or ways of acting in the future. Here, we focus on MR's supposed benefits for the military (specifically MR's utility in extending the sovereign power of the United States as a warfighting tool), furthering recent Western military doctrines of pre-empting threat through mediated vision.
Entanglements between MR development and the military are not new and can be traced to mid-century research and development in the United States. 5 That relationship continues today. The US Department for Defence, for example, solidifies this in their recent plans for an Integrated Visual Augmentation System (IVAS), essentially a research and development program focused on MR for warfighting by the US Army (see Egliston and Carter, 2021b). As a US Army document on IVAS reads, ‘soldier lethality will be vastly improved through cognitive training and advanced sensors, enabling squads to be the first to detect, decide and engage’ (US Army Augmented Reality IVAS Statement of Objectives, 2019: n.p).
We see this increasing military demand being met through companies like Kopin – who presently develops gunsights and MR displays for military and air force pilots. Likewise, Elbit (who also develops for the commercial market) currently develop MR interfaces for infantry and pilots. Perhaps most notable in this space is Microsoft, who in 2019 secured a US$480 M contract with the US military (see Hollister, 2019) to develop a MR system, in the style of its commercial HoloLens headset, for use for military combat and training (receiving a subsequent US$22BN to further develop and supply these headsets over the next two years (see Egliston and Carter, 2021b). 6
As shown in Figure 3 – Microsoft's patent for a military-grade MR headset imagines how wearable MR technologies will enable real time capture and relay of information between soldiers, but also between soldiers and reconnaissance technologies such as drones – rendered within the headset's interface, enabling more efficient targeting, tracking and killing (or ‘lethality’). Elsewhere, we are presented with an image that encapsulates the United States’ imperialist ambitions of conquest in the Middle East. We see an Arab man being tracked and read as a threat through HoloLens’ iris recognition – rendered as such based on information stored in databases, transferred to the headset at light speed – a ‘system of memory’, as Kinsley (2015) would have it, that feeds into and shapes the protentional capacity of the device's user in realtime. The imagined technicity here, via Stiegler, is one characterised by systematic coordination of soldiers, computer-based systems, and the sensing capacities of the device. How soldiers engage conflict unfolds through the artefactual exteriority of MR interfaces which capture information about external environments and the organic and inorganic things in them. These are framed as forms of digital tertiary retention which minimise the need for soldiers to make split-second decisions in combat. Decisions for using lethal force are distributed between the soldier and the data centres feeding information into the MR device's head up display.
Facial and iris recognition capabilities for military grade augmented reality (AR) imagined in Microsoft's patent.
This imagined ‘distributed’ nature of military warfare is framed as enhancing accuracy and accountability (as a US Army press release has it, see Siter, 2019). This promise has clear parallels to the wider history of militarised technics and sensing propagated by the West (and perhaps most clearly exemplified in the drone) – one based on mediated techniques and “anticipatory logics of developing a pre-emptive mastery of the territory and its potential threats” (Crogan, 2016: 655), sensing and modelling – through digital technics – perceived enemy potentiality. What is being advocated – much as with the military-industrial framing of the drone – is the strategic necessity of adopting the deployment of data driven systems that amount to a less considered mobilisation of lethal force in favour of a military doctrine of pre-emption and automated classification of threat.
It is particularly important to note that these discourses surrounding military adoptions of MR are not incidental. These are discourses that have been taken up in the context of attempts to think MR within the context of international policymaking about war and counterterrorism, such as in a 2021 panel convened by the United Nations Office of Counterterrorism. 7 The goal of this panel was to raise awareness of how MR technologies might militate against threats of terrorism and violent extremism, rather than focus on the harms inherent within these technologies and the institutions that employ them for purposes of counterterrorism. The ‘stupidity’ of MR boosterism – inasmuch that it both imagines ‘automatic’ subjects determined by the tertiary retentions of data and its rendition in the MR interface, and in that it does not take seriously the wider sociotechnical questions and issues inherent in its own use – becomes enfolded into dialogues in policymaking.
MR and the militarisation of everyday life
Beyond the technicity of shaping the anticipatory awareness of defence personnel in the context of the ‘war on terror’, we see a similar technicity around tracking, sorting, and profiling, framed as benefiting the governance of civil society – specifically, in mundane, public urban spaces. In this sense, MR becomes just another part of the modern city as a surveillant assemblage (Shapiro, 2020) – another mechanism for governing the rhythms of everyday life.
Already, MR is imagined as installing hard-line forms of control in its mobilisation by police forces. One champion of MR-enabled policing is American firm Vuzix. While Vuzix – like many examples discussed thus far – develop largely for a commercial market, they also develop for “governmental entity customers that primarily provide security and defence services, including police, fire fighters, EMTs, other first responders, and homeland and border security” (Vuzix, 2020). While a number of companies – such as American firm SIX-15 – have contracts with local police forces, Vuzix is a particularly striking example given their clear ambition to develop facial recognition enabled MR for law enforcement. In 2020, it was reported that facial recognition platform ClearView AI (which is already being licensed to law enforcement agencies in countries like the US and Australia) – a platform which reportedly draws on datasets of more than three-billion images from websites like Facebook and Instagram – has tested their software with Vuzix's hardware (Haskins et al., 2020). Vuzix has elsewhere partnered with UAE based facial recognition tech firm NNTC.
In a press release for NNTC, the (curious) comparison is made between the software (and Vuzix's hardware) and Paul Verhoeven's 1987 film RoboCop: ‘While the dystopian society envisaged in Robocop is nothing to emulate, the innovative tools its main character uses to protect those in need is within reach’. Vuzix seemingly miss Verhoeven's subtext of RoboCop as social satire and commentary on the corporatisation of urban governance in light of Reagan-era privatization (and moreover, like Stiegler, that our current trajectory of technological becoming is unsustainable). It seems lost on Vuzix that as purveyors of technologies of violence and control to police forces, they play a similar role to Omni Consumer Products – the nefarious corporation and antagonist in the film – a company that facilitates and seeks to profit from the chaos of a dystopian Detroit through provision of extremely harmful weapons to police forces. Indeed, given the many faults in the present with facial recognition's ‘pre-emptive’ capacities (see Crawford and Paglen, 2019), it's not hard to imagine the ways that equipping police with MR-based facial recognition would reinforce existing biases, or result in the misidentification and harm of those it surveils.
One NNTC video shows a mocked-up interface, where a target is identified and profiled in real time. What we see played out here is one of Stiegler's key idea that digital tertiary retentions deterministically shape perception (2016: 27). Instead of thinking about what they’re doing before, and what they should do in future, based on a mediated preindividual reserve, Stiegler argues that thought becomes short-circuited. We see this in the video, as the individual instantly responds to the data-driven readout of the target, alerting them to a threat perceived by the MR device. Much like a wider lineage of policing technologies, the focus is on prediction and anticipation rather than using data for explanatory purposes (cf. Brayne, 2017). While this is framed around faster thinking and communication in service of public safety, we must of course, following Stiegler, think about the harms that the delegation of thought and experience to machines may further exacerbate.
While NNTC imagines a radical changing up of the practices and processes of policing, it is important to consider that it is not the object in and of itself that has the potential for harm here, but its integration into a wider milieu of policing. As writers like Andrejevic et al. (2020) have suggested, data-driven technologies used in law enforcement need to be understood as existing within the institutional and social settings and histories of police forces as institutions entrenched in systemic racism. This is clear through recent cases where police have manipulated the data generated by policing software to give grounds for unjust arrest and prosecution (see e.g. Feathers, 2021). While these MR technologies are framed as bringing about new ways of ‘doing’ policing, perhaps then their potential for harm also lies in that they may well further calcify the systemic cultures, the ‘long circuits’ as Stiegler would have it, of racism characterising law enforcement.
Augmenting exploitation: MR in the workplace
Technologies on the MR spectrum are also expected to be adopted within the workplace – framed to offer an augmented worker, something which benefits an augmented boss – who can further oversee, control and profit from their employees. The act of work – generally, but not always, manual labour (such as manufacturing or logistics) – is mediated through a visual interface (often providing instructions), framed as a means to enhance worker autonomy in performing tasks. At the same time these tools provide greater autonomy and control for employers over their employees.
Currently, there is much imagined promise of an augmented worker in of manufacturing and logistics, with the emergence of technologies by companies like Google and Vuzix. Such technologies are envisaged as enhancing the productivity of largely manual labour in factories or logistical work. In one testimonial for the Google Glass Enterprise Edition, by American aerospace company Boeing, MR is framed as appending the limited capacities of the human worker (specifically, those installing electrical wiring on aircraft) – offering workers ‘real-time, hands-free, interactive 3D wiring diagrams – right before their eyes’ (Boeing, 2018: n.p.). In so doing, the need to remember instructions in one's head are exteriorised and instantly retrievable through the interface of the Glass.
These statements are relatively (and deliberately) innocuous. MR, we are told, is simply a tool which makes an employee's work easier. It supposedly provides the worker with more autonomy to complete their tasks.
Notably, there's a focus on framing this as augmentation – as just another tool for the worker, rather than something that determines the worker's actions, perhaps as to downplay increasing anxieties about the role of automation and human inferiority (for example, the Glass ‘intuitively fits into your workflow’, a device for ‘smarter and faster hands-on work’). Of course, it should be noted that the emphasis on augmentation and working ‘smarter’ is something that benefits the employer. MR supports a kind of protention that results in forms of thought and action amenable to an employer's bottom line.
MR as an assistive technology – while providing autonomy to the worker – is also framed as further empowering the employer by offering a mechanism for worker surveillance and control. Recently, both Microsoft and Google announced ‘remote assistance’ features for the HoloLens and Glass, respectively, through which employers can view worker activity from an employee's perspective. Video feeds – from the visual overlays of these devices – make the worker's very gaze a site of surveillance, a form of visual data that even in its most benign imagination (as a means for employees to be assisted by their bosses) provides a further degree of surveillant control to employer. Such concerns are especially warranted in light of the fact that companies like Amazon have recently patented (Madan et al., 2017) MR wearables to be used in their warehouses (Figure 4) – presenting the worker with visual prompts that point them left and right through the maze of shelves, up and down to locate the object, a further way to metricise and discipline the worker's body.
2018 Amazon patent for augmented reality glasses for warehouse workers.
Currently existing MR technology, rolled out in the context of corporate ‘soft skills’ training by the Dutch ‘MR-based learning company’ TaleSpin exemplifies this potential for the data-veillance of the MR user. TaleSpin's technology is positioned in its promotional material as providing an effective delivery mechanism for anti-bias and empathy training in a corporate setting. By rendering visually realistic and immersive simulations of workplace tasks, these MR interfaces promise to generate traces that can be operationalised as measures of things like user ‘sentiment’ (and moreover, a means to create, through repeated use and engagement, forms of ‘emotional muscle memory’). A central focus for TaleSpin has been the use MR to develop training to combat racial bias, doing so by capturing the inputs made in a particular MR simulation (such as in an interview situation with a Black employee). On one hand, these could well be productive in identifying individual biases which go unrecognised. On the other, however, it is to misrecognise how racial bias is systemic – something that cannot deprogrammed by focusing on the level of the individual; retentions are needed at scale to inscribe explicitly anti-racist forms of psychosocial individuation. More generally, applications like TaleSpin represent a mechanism for making ‘soft skills’ quantifiable – “allow[ing] Talespin customers to track employee engagement and development” (TaleSpin, 2020). While recent social critiques of workplace data-veillance (including our critique of VR-based tools, Carter and Egliston, 2021) have advocated for a greater degree of transparency (and in particular, auditing of their rollout in workplaces), we found no discussion of transparency in the company's promotional materials.
By affording employers a further degree of insight into their employees, and the ability to make organisational decisions around data about their employee's performance – the potential to further widen power imbalances between employers and employees is not hard to imagine. The practices and processes of work – whether in the context of manufacturing or logistics, or in the context of the white-collar office job – are intensified sites of surveillance when your employer can access a video feed of what you currently see or can make decisions about your employment status based upon analytics to do with your engagement with MR training software.
Conclusions
We have focused in this study on four main ways that MR is imagined to be adopted: (1) the development of MR by big tech companies, (2) the use of MR for defence, (3) MR as a technology for control of populations in civil society and (4) MR as a technology used in workplace settings. Our analysis has focused on how these technologies all broadly promise convenience, security, or insight. In affording these promises, MR is cast as protentional, providing a capacity to anticipate life in advance of itself. This is based on the device's accumulation of data (from the user, or about other individuals in one's environment), or through combination of MR with data derived from other surveillance databases (e.g. for facial recognition). While these visions are speculative, and may never materialise as they are imagined today, they reveal a great deal about the logics, aims and biases held by these companies, and moreover, appeal and contribute to the wider enactment of surveillant futures.
Following Stiegler, such visions of an MR future present perception as the outcome of a technological collapsing of future and past into the present – rendering individuals as more profitable consumers, productive workers, or subjects to be disciplined by the state. MR is not yet fully materialised in the contexts we discuss – and so our work is not to identify and call for the regulation of currently existing harms. Rather, it is to identify how these imaginaries are predicated on the basis of installing regimes of classification, verification and identification – threatening to cause direct harm but also indirectly via faulty biases, logics etc., which normalise existing forms of (dis)advantage (cf. Hoffmann, 2020). In this way, we believe that MR is not just important to consider as an emerging data rich technology, but a major terrain in the struggle over social justice – something worth challenging in its nascency, rather than following its potentially widespread adoption. In Stieglerian terms, we challenge MR futures at the level of discourse, at the level of the semiotic artefacts which form and are formed by existing social, economic and technological contexts – a worthwhile project if we turn to the many existing studies of the power of the social shaping of technology (see e.g. Jasanoff, 2015). To contest futures at the level of discourse is an attempt to materialise the kind of positive pharmacological potential in technics, and MR specifically.
On the basis of our analysis, we believe it is urgent that critical data studies challenge the authority and credibility of narratives about the futures these technologies promise to bring to bear – recognising and challenging the violence they would inflect upon marginalised social groups, and the kinds of systemic advantage that might be normalised and produced in their wake. In short, our account of MR futures, geared around a pre-emptive mode of temporal perception, is one that seeks to expose and push back against the hierarchical logics associated with these imaginaries, which produce advantaged and disadvantaged subjects. While this might seem quite minor, as Michelle Murphy puts it, ‘even the refusal of hegemonic concepts that capitalism does not want to live without is a generative act’, an approach rooted in a belief that we ‘can recognize both violence and the possibility of exceeding violence’ (2017: 17) – one commensurate with Stiegler's bleak yet not altogether defeatist view of the future of technoculture.
By positing ‘what if’ questions about futures to come, our goal has been to provide the means for both critique and to articulate the kinds of futures we want from MR, and to start the conversations about how we might ‘care’ (as Stiegler would have it) for MR as an emerging part of our sociotechnical infrastructure. Indeed, there have been promising cases of resistance to some of the imaginaries described in this paper, such as widespread resistance to early forms of wearable AR in the case of Google Glass during its 2013 rollout in the San Francisco Bay Area, or widespread worker pushback in 2018 at Microsoft to the company's involvement in developing military grade MR. Elsewhere, there is some promising development in the European policy space around the regulation of MR technologies – with the German national competition regulator launching in 2020 an investigation into the data-extractive nature of Oculus and how it might benefit Facebook(resulting in Oculus products being pulled from sale in Germany). 8 It is our hope that further reflections, such as those offered in this article, and in the emerging critical literature on MR and its data-centred nature, can catalyse conversations in academia, classrooms and indeed in policy debate, about MR as a data-rich technology that is increasingly coming to bear upon society.
While the point of this article has been to challenge MR's unquestioned societal benefits, as it is understood by industry boosters, it is worth noting (keeping with Stiegler) the positive pharmacological directions MR development may take, through critical or reflective engagements with the technological form of MR as a way to mediate forms of protentiality and subjectivity characterised by care. For example, Carter et al. (2020) has used Oculus headsets to develop applications for animal conservationist education. While the device itself may benefit Facebook (e.g. in terms of data capture), this project stokes long circuits of knowledge formation required in combating ecological threat. We might also think of emerging uses of smartphone-based AR in artistic projects (see Guntarik et al., n.d.), which could easily be applied to MR tech, focused on annotating unceded indigenous land in Australia with descriptions of cultural histories, long circuits, displaced through the legacy of colonialism. In short, the point is not to reject MR. It is to articulate that we need to renegotiate the terms on which we are asked to accept MR into our lives.
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship and/or publication of this article.
