Interface critique at large

Abstract

This article considers how the pursuit of problematization advocated by Agre’s concept of critical technical practice has been articulated in relation to the increasing proliferation of interfaces across everyday life. While the ethos of Agre’s work to bridge the split identity of critique and craft can readily be found in reflexive design or software art, these cases are not always situated within broader ecologies of practice that also grapple with the asymmetries and exploitative aspects of interface design. Drawing from software studies and media theoretical accounts of the interface as a fluid milieu, I provide a navigational matrix to contextualize modes of interface critique at large, namely specifying traps and enclosures, surfacing asymmetries and augmenting alternatives. I argue, finally, that these modes provide an invitation to develop new metacritical theories and common capacities, particularly through the possibilities of grappling with systems of domination otherwise built to prefigure our experiences of them.

Blending reflexive critique with the functional craft of design, Philip Agre’s (1997) concept of critical technical practice (CTP) was first conceived to ‘cultivate awareness’ and challenge the customs and inherited habits that shape computational work, especially from ‘the planning view’ of digital systems design that dominated early artificial intelligence research. Poised as a means to overcome the impasses of cognitive information processing (or ‘mentalism’) in this domain, the proposal for CTP was also animated by a sense of socio-political urgency. For Agre, there was a looming sense of danger from the tendency for computational analysis, design and implementation to impose itself and reshape, not merely represent, existing zones of everyday activity, governance and labor, to the extent that ‘computing has been constituted as a kind of imperialism; it aims to virtually reinvent every other site of practice in its own image’ (p.131). The encroaching logics of ‘the computer world’, according to Agre, produced complex borderlands in which a majority of people struggled to make sense of the technological abstractions and obfuscations they were confronted with. Today, these dynamics have arguably only become ever more insistent through the dispositions and logics of corporate platforms, where regimes of user experience design (UX) valorize an aesthetics of invisibility, targeted personalization and an ongoing experimentation on users from behind the scenes, generating conditions that the philosopher Frédéric Neyrat (2018) has referred to as ‘saturated immanence’.

This article considers how the problematization of software rationalities and norms advocated by CTP has been elaborated in relation to the proliferation of interfaces throughout everyday life. In the context of academic research, for instance, Agre’s interventions have led to efforts to bridge the split identity of critique and craft through proposals like reflexive design (Sengers, et al. 2006), humanist HCI (Bardzell and Bardzell, 2015) and theorizations of software art as interface criticism (Andersen and Pold, 2018). Acknowledging the various strengths of these initiatives, I equally take note of a gap or disconnect that exists between such inquiries and the kinds of critical work broadly entangled with more industry-based practices of interaction design as a whole (Gray, et al. 2018). Accordingly, in this piece, I discuss cases of interface critique at large, building bridges between these other domains influenced by CTP, while also situating and rethinking some of their key tenets. Assisted by media theories of the interface, I conceptually map several important ways these domains navigate the fluid convolutions of interfaces to generate public understandings of their operativity, including specifying traps and enclosures, surfacing asymmetries and augmenting alternative experiences; what might be considered as a navigational matrix for exoteric interfacing. I argue, finally, that a consideration of these wider ecologies of practice might facilitate not only a deeper articulation of interface ethics (Dourish, 2019), but also new potentials to collectively grapple with extractive systems otherwise built to prefigure our individual experiences of them.

HCI, software art and beyond

Agre’s notion of CTP has been influential across several fields in ways worth briefly unpacking. Within HCI, while inspiring new understandings of the discipline's historical evolution through critiques of dominant paradigms over time (Harrison, et al. 2007), it has in general provoked a profound recognition that digital technologies both embody and unsettle taken-for-granted sociocultural values (Nissenbaum, 2001). This has, consequently, led to the elaboration of new forms of critical design, tactical intervention and activism over the past two decades. Such engagements have been further reinforced through initiatives gathered up by the term ‘humanistic HCI’, which have advanced the interdisciplinarity of CTP along interpretive and hermeneutic lines; that is, ‘research or practice that deploys humanistic epistemologies (e.g. theories and conceptual systems) and methodologies (e.g. critical analysis of designs, processes, and implementations; historical genealogies; conceptual analysis; emancipatory criticism) in service of HCI processes, theories, methods, agenda-setting, and practices’ (Bardzell and Bardzell, 2015: p. 3). These frameworks essentially follow the ethos of CTP by adding further impetus to questioning norms and presumptions that have shaped various subfields of HCI research.

CTP, moreover, has led to various proposals for transdisciplinary and ‘second-order’ reflexive initiatives. As the predominance of computer science and cognitive psychology doxa has been challenged and transformed through the application of perspectives from science and technology studies (STS), artistic practice and cultural studies, metacritical positions such as reflective HCI emerged shortly after the turn of the century (Dourish, et al. 2004). Following the appeal to import new epistemological and methodological perspectives, reflective design aimed to take such developments into account by offering ‘a systematic approach to folding critical reflection into the practice of technology design’ (Sengers, et al. 2004, p. 49). Drawing from traditions of Marxism, feminism, critical race theory, media studies and psychoanalysis, the overarching goal was not only to advocate these lineages as a resource for HCI researchers, but to design systems for users themselves to become involved with critique, so that ‘reflection itself should be a core technology design outcome’ (Sengers, et al. 2005, p. 50). While converging with many closely related frameworks, including participatory design, ludic design, critical design, value-sensitive design and reflection-in-practice, developments such as reflective design ultimately take inspiration from critical theory, particularly with a commitment to ‘unveil’ the unconscious thoughts and habits of actors, bringing them to consciousness in pursuit of new knowledge and experiences. This strand of HCI, moreover, has frequently enlists aesthetic experimentation to pursue these ends, partly to navigate how designing for experience frequently risks implementing reductive frameworks based on usability, rather than experimenting with more radical suspensions of prescribed meaning, behavior and knowledge between users, designers and systems. At stake were processes of abstraction and data extraction whereby emotional states might be rendered as merely informational units, or approaches that treat ‘experience as something to be poured into passive users’ (p. 1). By contrast, interpretative attitudes might arise through the deployment of aesthetic strategies like ambiguity, exaggeration and defamiliarization, an approach influentially deployed in ‘design probes’ (Gaver et al., 2003), which would pave the way for later formations like speculative and critical design (Dunne and Raby, 2013). This metacritical trajectory can take off into realms somewhat removed from Agre’s concerns, yet remain troubled by the ambiguities of designing for autonomy and the proper role of the designer (Höök, 2004).

Distinct from these lineages, another idea of interface critique has been elaborated at the intersections of software art and media theory in Europe. This version arose through net.art and tactical media in the late 1990s and early 2000s as fostered by the V2_ Institute in Rotterdam, Furtherfield in London, Transmediale in Berlin and the Readme travelling software art festival, which has further migrated into more recent developments like the Interface Politics conferences at HANGER Barcelona and the Interface Critique publication series run by Florian Hadler, including texts like Olia Lialina’s (2021) important series of essays on ‘resisting alienation’ in HCI written during the 2010s. Christian Ulrik Andersen and Søren Bro Pold’s (2018, 2021) work offers one notable representative survey and theorization of this domain. Seeing computers as accumulations of mental labor, Pold and Andersen have argued that contemporary interface design encourages the user to forget the abstraction involved in programming along with the material conditions of digital signal processing. Their account foregrounds the politics of a specifically artistic style of interface critique found in works by Lialina, UBERMORGEN.COM, The Critical Engineering Working Group, Ben Grosser, Joana Moll and others. Central to their account is Benjamin’s conception of the capacity for an artwork to reflect on its own conditions of production to reveal historical ‘tendencies’ within relations of production. Critique becomes a dialectical exploration of the role of the apparatus in shaping labor, sense and perception, organization and thought. Following the integrative dynamics of cloud computing services, platform corporations, smartphones and ambient sensing, moreover, this apparatus shifts toward a more ubiquitous ‘metainterface’ state. Transformative artistic techniques, accordingly, aim to avoid ‘supplying’ the metainterface through extractive data profiling by intervening to redirect subjectivation toward a new common capacities. While covering some similar grounds as reflexive design, accounts like Andersen and Pold’s are also unique by providing an alternative vision of what CTP might mean beyond the auspices of HCI; the works they discuss often utilize distinct cultural resources, like avant-gardism and the entwined political-artistic traits of tactical media, to embed themselves directly within digital infrastructures via appropriative means and gestures.

While taking stock of such theories, projects, cross-disciplinary exchanges and discussions that have emerged since Agre’s provocations, in this article, I want to consider whether interface critique might be productively considered in even broader and more diverse terms. What can be gained, for instance, by widening the scope of the term beyond the intricacies of academic HCI debates or theories of software art towards a horizon of what might be called critical technical cultures? The approaches above, I suggest, are concerned with constructing systems that mediate specific understandings of critique – often calling on aesthetics to mediate these re-deployments – but they do not always consider how these dynamics already play out across today’s expanded borderlands with ‘the computer world’. Indeed, in the case of critical HCI, many of these frameworks arguably risk performing a ‘repackaging of design within design’ (Fry, 2009: p. 3) by retaining the latter’s unspoken authority as an academic and professional territory. While building for autonomy is a worthy aspiration, how do we unmake interfaces of domination and exploitation already such a part of our everyday lives? Similarly, how might the veracity of Benjaminian interface aesthetics connect to, contaminate and travel along other circuits of informational production within ecologies of practice? Here, an additional influence comes from the pragmatic sociology of critique, which focuses on where, how and why actors practice critique throughout social realities, aiming to revive critical theory by returning to things themselves, in this case, the emergence of disputes. Of central importance is the principle task “to explain, clarify and, where possible, model the methods employed in the social world to make and break bonds” (Boltanski, 2011: p. 25). While concerns have been raised around the extent to which digital technologies circumvent such forms of liberal critique altogether (Davis, 2020), I turn to milieus of design practitioner-based critique in which interfaces as formats of capture themselves become the focus of socio-technical examination. In doing so, I draw from media theoretical and software studies approaches to clarify and situate critical competencies of practitioner-based design, particularly as they tackle the kinds of spatio-temporal involutions of control and subjectivation associated with interfacing. My approach thus aims to clarify a range of diverse, yet related cases that might not otherwise be considered in the same context. It builds bridges between theories of interface critique and technological dependency towards an analysis of those tools, techniques and methods that can expand common capacities to identify and act against exploitative regimes of UX.

A vortex of agencies

Challenging computational customs and inherited habits, as Agre's CTP envisioned, has only intensified as cloud-based platforms and data-intensive infrastructures have become increasingly enmeshed with the everyday. Today’s digital infrastructures reach into pre-individual processes to generate global cognitive assemblages (Hayles, 2017), but they also institute mechanisms of capture supported by micro-temporal sensing and information processing (Hansen, 2015). Interfaces play a central, yet enigmatic role in these regimes: they govern sense, perception and cognition, while appearing to disappear into the background. As dynamic thresholds, they complicate conditions of possibility for reflexivity. As prefigured environments for optimized performance, they trouble conceptions of performativity. Interfaces are, moreover, not reducible to a single relation with human users, but are enacted through diverse milieus of hardware and software (Fuller and Cramer, 2008); they are central to the operations of infrastructures and platforms, whilst mediating between computational systems and non-computational entities broadly speaking (Distelmeyer, 2019). In the following section, I consider these challenges by positing the interface as the governance of a fluid milieu consisting of flows, troughs and turbulence; in doing so, I foreground the significance of design techniques in the prefiguration of this fluid-like operativity, and consider what this means for practices and theories of interface critique.

From a media theoretical perspective, the interface is typically considered less a thing or object, than a dynamic, systematized relation. Over two decades ago, in an exchange for V2_ Institute, Siegfried Zielinski, 2019 (1997/2019, p. 50) put it this way: ‘[the interface] separates and connects media-people and media-machines... it is the borderline where the medium takes its shape’. Alexander Galloway’s (2012) account of ‘the interface effect’, similarly, describes the generativity of the interface as a threshold, while Joanna Drucker (2013) calls it ‘a space of affordances and possibilities structured into organization for use’. In these accounts, there is a sense of dynamic performativity co-existent with a form of channeling, sequestering or partitioning. Media theories of the interface, moreover, often take inspiration from the spatial connotations of the term which are genealogically tied to the control of fluid environments, coinciding with the use of interface as both a noun and verb (Wirth, 2016). Such deeper associations are charted in Branden Hookway’s (2014) conception of the interface as a historically distinct form of relating to technology or form of relation based on a twofold setting of turbulent flow. Here, the 19th century engineer and physicist James Thomson – who first coined the term ‘interface’ – takes on a prominent role, particularly a key conceptual image in his investigations of fluid dynamics involving the distribution of unequal energy between two expansive bodies of water (2014, pp. 67–75). Cast partly as a thought experiment, the scenario involved speculating on the interaction between one completely still body of water and another which was rapidly moving. Separated by a frictionless barrier, Thomson’s original theorization of the interface, as Hookway recounts, was centered on the eventfulness that would suddenly unfold with the removal of this obstacle as a third boundary condition would form through laminar turbulence; an interaction that was not entirely chaotic, but constitutive of internal and external pressures arising between the two bodies. As other commentators have observed, Thomson’s investigation was core to 19th century concepts of dynamic form in industry and natural philosophy, particularly as they intersected with the development of vortex turbines, electric telegraphy and steam engines (Schaefer, 2011). The interface as a concept, accordingly, directly contributed to understandings of ‘system’ in thermodynamics and shaped discussions of entropy that would eventually inspire nascent ideas of programmability via the conceptual figure of Maxwell’s demon (Terranova, 2004). Following this lineage into 20th century information theory and cybernetics, therefore, the interface can be understood as part of a becoming fluidity of machines or liquification of media, specifically a transmutation and delimitation of this dynamic occasion for the extraction of labor. As Melody Jue (2014, p. 92) has recently suggested, these genealogies suggest a shift from focusing on questions of medium-specificity to the struggle of being submerged in a milieu-specificity or ‘an aquatic paradigm of informatic “flow” programmed to influence human behavior’. Where Jue’s contribution enacts a conceptual displacement of ‘terrestrial bias’ in media theory by thinking through seawater, such a turn to the oceanic conversely also implies a transfiguration of fluid mechanics in the design of interfaces via technical reasoning and control.

The notion of the fluid-like interface as a form of relation results in convolutions of agency that require care in applying theories of distributed agencies or entanglement. From actor-network-theory to agential realism, such thinking influentially comprehends agencies as an achievement of mutually constituted and contingent configurations of the human and nonhuman. A well-known contribution of this line of thinking within HCI, for instance, has been to erode away the presumption of interaction taking place between discrete and self-contained entities – that is, human and computer – by foregrounding a multiplicity of enacted relations as socio-technical assemblages. Engaging with fluid interfaces, however, I suggest also requires some further consideration of the deliberate saturation of boundaries and internal dissolution of fixed partitions, alongside the propagation and dissipation of emergent form. Through prefiguration, testing and modeling, design techniques seek out points of ‘friction’ or ‘pressures’ to be enlisted into a fluid operativity. The achievement of an interface within phenomena occurs as both boundary and condition, involving processes of both separation and fusion. Such agential ambiguity arguably lingers also as a problematic in ‘the phenomenological matrix’ of so-called third-paradigm HCI (Dourish, 2001), which might be further explored with Jue’s (2020) suggestion of less ‘grounded’ traditions of experiential thought from Gaston Bachelard’s reflections on imagination and water to Luce Irigaray’s feminist philosophies of fluid dynamics. To complicate matters further, as a system of augmentation, the fluidity of the interface delivers mixtures of autonomy and constraint that grant new possibilities of control on an externalized environment. This is something recognized in early software studies, where Manovich’s genealogy of the split-subject of the ‘dioptric arts’ culminates with speculative statements on future mobile devices like ‘we will carry our prisons with us - not in order to blissfully confuse representations and perceptions (as in cinema) - but to always “be in touch,” always connected, always “plugged-in”’ (2002, p. 113). The interface, in other words, grants novel capacities for action, but simultaneously blurs its division of the user-as-subject through a cascading or submersive momentum. The disassociative dynamics of the milieu, in addition, complicate classical grounds for critique, particularly by obscuring asymmetries that are constitutive of and perpetuated by these systems, from the extraction of labor to the elevation of design practice and expertise itself, as Lucy Suchman has notably observed (2007, pp. 268–271).

The significance of these theories can be illustrated with reference to debates around social media interfaces designed to channel and capture the agency of users. Recalling the McLuhanesque allegory of a fish oblivious to the surrounding environment of water (Zylinska, 2021), critical accounts of social media interfaces tend to emphasize how platforms are designed to efface their logics by heightening immersion toward habitual use and addictive ends (Chun, 2016). Relevant to these discussions are considerations of how the psychology of flow and other persuasive behavioral paradigms like captology (aka ‘computers as persuasive technologies’) are embedded into interfaces (Seaver, 2019; Soderman, 2021). Such frameworks mesh with a broader emphasis on ‘simplicity’ and ‘convenience’ in user experience design through instant sensemaking, immersion and invisibility (Hadler and Irrgang, 2015). Their synthesis results in accelerated currents of activity, what Silvio Lorusso (2021) calls a ‘speedrun’ aesthetic, that streamlines experience to discourage any dwelling within the interface, thereby circumventing the potential for inventiveness or poesis. A parallel can be observed with Natasha Dow Schüll’s (2014, p.175) discussion of ‘perfect contingency’ in machine gambling, whereby the intensified coincidence of action and response draws players into a ‘machine zone’ or ‘a state in which alterity and agency recede’. This captivating flow is a predominant disposition of social media (Easterling, 2021), an effect which arises by programming and controlling a specific fluid form of relation.

Critical attention has been brought to bear on the processes of subjectivation bound up with this intensified flow of social media interfaces. As Drucker (2020) notes, social media interface dispositions are further exacerbated by the ‘unmarked’ traits of many designed commercial systems, where elements, layouts and affordances are presented as ‘transparent’ and ‘objective’ to users. In the terms of STS, they follow the logic of the ‘neutral scientific instrument’ whereby authorship is erased as a ‘matter of fact’ (Suchman, 2007: p. 214). Drawing from Benveniste’s theories of enunciation, however, Drucker argues that a dominant outcome of these logics is a system of speaking where the user is already unknowingly spoken for: ‘the speaker of the system, the “l” embodied in the design, allows “you” certain choices (and of course, by design and by the mere fact of the limits of design, not others)’ (p.109). Artist and critic Lialina (2021), moreover, situates this slippery self-identification in the history of the web, where the DIY style of ‘my personal webpage’ enabled through open protocols have been subsumed by platform corporations in an enunciative move toward the social media profile of Me: ‘where My was dangerous, Me was perfect. Me is cheap, Me is easy to control, Me is easy to channel, Me is slave of its own reflection, Me is a slave of the platforms that make the reflection glossy. Me is data. Me is data closest to metadata. This makes Me just perfect to satisfy advertisers and to sate neural networks’. The fluid disposition of the interface delivers this ‘inherently compromised’ sense of agency whereby subjects become expressed beyond their will or authorship through an exposure to new forms of experimentation (Hookway, 2020). As Andersen and Pold (2021) have recently added, users are ‘encased’ like characters in a ‘big data drama’ that obfuscates distinctions between who is reading and who is writing, and who is being written and who is being read. In their assessment, this culminates in a user persona that most closely resembles a zombie, a characterization echoed by the figure of the sleepwalker (Sampson, 2020). Accelerated dispositions, therefore, pivot on regimes that plot out patterns of communication on the one hand, while capturing and processing behavioral data on the other. Critical interface theory, therefore, assists with elucidating the spiral of forces that reproduce, for instance, the extractive realities of surveillance capitalism (Zuboff, 2019), yet also suggests particular experiential vorticities that complicates a capacity to establish a solid point of view.

However, despite these agitations of sense and perception, the uncertainties of interfacing should not be underestimated. From errors and mistakes to internal modes of control that displace forms of data capture and algorithmic processing, the interface can quickly turn into a zone of struggle and dispute. In some theories, an emphasis has been placed on the epistemological gap between the interface interior and exterior as a site of contestation and resistance; for instance, in John Cheney-Lippold’s re-imagining of the conditional programming statement “else” to conceptualize some “wiggle room” or respite that ‘offers a possibility for deviance, mobility, and ultimately unincorporability according to the conceptual space between algorithms and us’ (2017, p.193). In a similar way, Olga Goriunova (2019) uses the term distance to negotiate the relationality and difference between the self and the digital subject as an entity composed by data. In such accounts, radical contingencies can be mobilized to disturb the realities of testing, modeling and performance. Indeed, ambiguities and incongruencies are routinely generated with complex machinic systems as automated mechanisms fail to achieve particular expected symbolic or representational correspondences. Accordingly, while yielding augmentation by submerging the user into a programmable milieu – that is, tacit knowledge, embodied skills, new capacities – interfaces as active thresholds always retain the status of being untimely and affording various modes of evaluation and critique.

These uncertainties can be further illustrated by returning to the initiation and surfacing of kinetic form in Hookway’s (2014) genealogy, particularly the external modes of control for designing and maintaining the interface. Here, the conditions setting up Thomson’s thought experiment on fluid dynamics are key in the formation of a barrier between the bodies of water and the timing of its removal. This is an indirect mode of control externally imposed by design as a ‘proximate cause of the event’ (p. 71). The barrier, accordingly, marks an approximate territory; a prefiguration of the interface as an event. Through this foreshadowing, moreover, the barrier also conversely signals the potential of transforming the interface into a surface. Surfacing allows for particular interpretations and understandings of the activity that takes place within the interface; for instance, through sensing variations in pressure which are recorded as data, so that ‘aspects of the interface that would only be available from within the system are here rendered available, at least in part, as a surface knowledge, to being known from outside that system’ (p. 71). External control can, in other words, surface data from the fluid-like states of the interface via testing for purposes of monitoring and modeling its performance, yet the output always remains approximate knowledge of internal conditions. Theories of control that develop around the interface, accordingly, seek ‘to iteratively decrease the proximate distance between the model, barrier, or surface and the event or interface, so as to control a system by shaping and influencing the emergence of interfaces within it’ (p. 71). In addition to the initial partitioning of the barrier – recalling what Karen Barad (2007) described as ‘agential cuts’ – interface design unfolds over time through recursive arcs of activity like steering and nudging that strive to coordinate with this gap – what might be described as ‘agential curls’. Fluctuating as kinetic form, an unfathomable distance nevertheless remains between the external control of an interface and the experience of augmentation that plays out within.

In what follows, I take cues from such media theories – in particular, milieu-specific analysis and the becoming fluid of machines – to situate and contextualize modes of interface critique at large. Since UX can often encourage an ‘aesthetic flattening’ that focuses attention on screens alone (Diehm and Choi, 2021), these theories are effective in bringing into view the full magnitude of the interface-as-event along with an understanding of how critique can unfold across four dimensions. This also means going beyond the user by making inquiries into the infrastructural agencies and micro-temporal processing of computational media. Within Hookway’s genealogy, prefiguration, testing and augmentation provide distinct, yet overlapping spatio-temporal orientations toward interface dispositions. They follow the transposition of fluid dynamics into a programmable system of control and offer proximate forms of knowledge of an expansive socio-technical milieu. In the sections that follow, I illustrate these orientations with several examples, laying the foundations for a different metacritical conception of interface exploitation and domination that highlights traps and enclosures, mapping convergent interests and the struggle to provide experiential alternatives.

Orientations of interface critique

Diverse modes of interface critique circulate widely throughout digital culture, even if they are not always recognized as such. Sizing up the interface from different orientations can assist with situating these modes, including locating cases of reflexive design and software art in relation to wider ecologies of practice. Importantly, these orientations do not suggest positions of indifference, but intentional trajectories that follow methods, techniques and tools that support design practice or design-abilities (Stolterman, et al. 2018). A useful parallel can be drawn with Simondonian-inspired accounts of software-making, where the expressivity of algorithmic cultural techniques perpetually concretizes an environment of technicity, which in turn diffuses new transformative individuations and potentials (Rieder, 2020). Interface design-abilities similarly redistribute agencies with implications that are at once discursive and practical, delimiting voice and authority, while enabling and constraining action (Cohn et al., 2010). Making use of heterogeneous sources and materials, design techniques might include graphic arts like sketching wireframes, developing mental models and personas, working with software development kits, monitoring user metrics or advocating for new terminology, among others. Studies of interface design practice emphasize trends toward an adaptive, appropriative use of such techniques in situ, often appearing at odds with the highly formalized or objective descriptions of methods found in academic domains of HCI. There is, in other words, a particular tolerance for flexibility in the application of such techniques, influenced by any number of situated practicalities, which resembles by contrast a conception of design labor as bricolage (Gray, 2016).

In their ordinary course, however, design-abilities readily correspond with today’s regimes of UX. The latter is admittedly a somewhat nebulous term at times attached to third-paradigm HCI concerns with affect and emotions, non-instrumental understandings of interaction, and situated contexts of use (Hassenzahl and Tractinsky, 2006), but which has mainly risen to prominence through a branding of services following an industry-led ‘aestheticization of information tools’ (Manovich, 2012). Figures like Don Norman and Jakob Nielsen (2006), in particular, have promoted UX as a ‘broader concept’ than usability, referring to ‘all aspects of the end-user’s interaction with the company, its services, and its products’. In this commercial context, design-abilities are often tasked with the creation of dispositions toward a positive experience of brands supported by an ‘aesthetics of invisibility’ with regard to technology (Masure, 2019). UX can thus be mobilized for smoothing over or saturating any number of contradictions that might otherwise be experienced through the interface; reconfirming ‘symbolic forms’ and ‘states of affairs’ to put this in the language of the sociology of critique (Boltanski, 2011: p. 103). While possibly a topic for another article, my interest here lies with how UX design-abilities are transformed to unsettle the realities of such dispositions. This might be considered as moments where those actors caught up with the design of exploitative and asymmetric systems get a grip on what is happening by turning their expertise towards critique. Conceptually, it involves a shift from design-abilities that reproduce the fluidity of interfacing toward a ‘metapragmatic register’ marked by increasing reflexivity toward these systems along descriptive and normative dimensions (pp. 67–68). Based on prefiguration, critical diagnostics and augmentation, therefore, I outline a navigational matrix for three orientations of interface critique, which includes specifying traps and enclosures, surfacing asymmetries and, finally, augmenting alternatives.

Prefigurative critique: specifying traps and enclosures

Prefiguring an interface involves techniques, tools, diagrams, texts and artifacts that posit a spatio-temporal approximation of its performance. Like the barrier in Thomson’s thought experiment, they work to demarcate the position of a potential interface by anticipating forces and pressures that can give rise to a particular form of relation (Hookway, 2014: p. 148). Such positioning, in other words, seeks out points of resistance, couplings and conduits of energetic exchange to eventuate interfacing, from ‘symbolic handles’ (Fuller and Cramer, 2008) like buttons, sliders and check-boxes – what Marianne van den Boomen (2014) called ‘depresentational’ icons – to keyboards, controllers and trackpads as hardware affordances. From a wider infrastructural perspective, protocols, formats and other technical standards operate as zones of anticipated action for more-than-human forms of interfacing. Laid into sequences of action, prefiguration takes up models and heuristics to plot propensities, which themselves draw from past configurations to project future enactments. Prefiguration thereby works through recursive patterns that contribute to the stabilization of a fluid-like interface and its performance. Critical engagements that experimentally explore the material and cultural significance of such positioning are well-known in media arts, from the bureau-aesthetics of classic net.art pieces like Alexei Shulgin’s Form Art (1997) to post-digital works such as Aram Bartholl’s Map (2006–2019), yet such projects are not often considered within wider areas of relevant critique that similarly interrogate compositional apprehensions.

Reflecting on iterative and changing formats used for positioning, practitioner-based approaches can readily assist with collectively naming the otherwise unnamed features of everyday interfaces. The widely used framework of design patterns, for instance, is especially relevant in this context. First proposed by informational architects Christopher Alexander, Sara Ishikawa and Murray Silverstein in the 1970s, a design pattern refers to ‘a problem that occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice’ (Alexander et al., 1977, p. x). In constrast to earlier proposals for design methods in the 1960s, the notion of patterns suggested a flexibility in applying already established norms to built environments. Typically presented as a catalog of entries or library, design patterns follow a format of being individually named and described with a scenario of use that always requires interpretation and adaptability. Originally imported into software development through object-oriented programming, before becoming used in the first Wiki for documenting user interface patterns, the concept’s architectural links have encouraged a more ‘spatially embodied’ understanding of software development (Alt, 2011). Contributing to the concretization of techniques for software-making, design patterns foreground modes of regularity and predictability in complex systems and infrastructures (Fuller and Goffey, 2017). As Galloway (2020) observes, they provide a ‘road map’ for otherwise featureless systems: ‘design patterns are basically an atlas of digitality as a whole’. Within the context of user interfaces, in particular, design patterns might assist with identifying arrangements of information architecture like ‘Feature, Search and Browse’ or ‘Step-By-Step’ navigation flows; they might alternatively assist with describing layouts of lists like ‘Two-Panel Selector’ or ‘Alpha/Numeric Scroller’, among others (Tidwell, et al. 2020). Indeed, design patterns can be productively blended with media theoretical concepts such as affordances or framing to consider recurring configurations of agency, or connected to larger questions of power as they are enfolded into settings like secure financial access and exchange (Dieter and Tkacz, 2020).

Within practitioner-based contexts, design patterns have been developed in different directions for purposes of interface criticism. The term ‘anti-patterns’, for instance, has been coined for frequently used solutions that inevitably create more problems, particularly when alternative, more effective approaches already exist (Koenig, 2005). More recently, the designer Harry Brignull has pushed the framework into the register of critique through the concept of ‘dark patterns’ to describe the deceptive prefiguration and positioning of interfaces. While tending toward struggles around consumer rights, dark patterns also invites a consideration of the complex issues that emerge at the intersection of behavioral economics and interface design, particularly the governmental ambition to instill conducts that are ‘freely’ embraced, yet deliberately planned and controlled. Other than providing a taxonomy of deceitful traps, this mode of critique regularly sheds light on areas of contestation in the concretization of patterns. For instance, such limits are apparent in the promotion of the ‘forced minimalism’ of corporate flat and material design languages against more singular or bespoke styles. App interface design, in particular, is positioned by the high degrees of systematization in patterns and elements from platform owners like Apple and Google. Indeed, trends toward automating the implementation of patterns can be observed in the rise of design systems, which not only represent a further encroachment of control, but additionally exacerbate asymmetries between front-end and back-end development, or the graphic and engineering domains of interface positioning. Finally, prefigurative critique might draw attention to restrictions in utilizing specific design patterns through the power of patents from Apple’s ownership of ‘Slide to Unlock’ to Facebook’s ‘Like’ or the Microsoft’s Windows startup grid (Tang, 2019). This projective hold curtails and encloses the potential of patterns in general, since corporations claim ownership over models that may never be actualized, but are retained to control the innovation space of competitors and pursue litigation if profitable to do so. With such examples in mind, the critique of prefiguration it could be said focuses on contradictions that arise through design as a concretization of forethought, opening up areas of ethico-aesthetic inquiry that connects with thinkers of captivation like Alfred Gell, Vilém Flusser and Peter Sloterdijk (Singleton, 2014). It concentrates on conditions where actors subtly impose their will through contraptions and new enclosures to exploit anticipated flows of agency, and suggests in turn a tactics of sensing and cataloguing, or a collective inventory of plots.

Critical diagnostics: Surfacing asymmetries

Interfaces are not black boxes with sovereign power inside. As an eventful, programmable and submersive milieu, the interface is a vortex of agencies, which maintains multiple facings, contrasting interior and exterior states, and relational pressures that sustain its threshold condition. Alongside prefiguration and positioning, there exist a range of tests to evaluate and reinforce the emergent performance of an interface. This second mode involves critical diagnostics which aims to surface activity from the event of the interface; that is, it applies tools, techniques and methods to materialize the fluidity of the interface as a surface that can support close reading or analytics. There is a notable tension here: while surfaces tend to be associated with interpreting obscure or ‘hidden’ deeper meanings, interfaces complicate this idea through their constitution by bringing diverse entities into fluid-like relations. As Hookway notes, the interface as a producer of surfaces follows a particular mode of inquiry which involves maintaining or holding constant specific relations between entities bound up with this form of relation. In doing so, mutually shaped behaviors can be steadily inscribed as a surface, but only by recapitulating ‘the binding together of those entities into relation’ (2014, p.15). Diagnostics, in other words, should not be taken as discovering the essential properties of the interface as an object, but tracing the interplay between heterogenous, dynamically related entities. The significance of this distinction becomes immediately apparent when considering, for instance, the performativity of walkthrough methods, a framework widely used in HCI, practitioner-based and vernacular modes of interface criticism, and more recently introduced as an approach in digital media studies. Here, a researcher ‘mimics’ the everyday use of software to render ‘salient’ and ‘visible’ an ‘environment of expected use’ for purposes of critical analysis (Light, et al., 2018). Although relatively unremarked upon, the involvement of the researcher in these processes of surfacing suggests a need to develop ‘personas’ when navigating the fluidity of the interface, especially when dealing with mechanisms of profiling (Dieter, et al., 2019). The submersive qualities of the interface, moreover, raise interesting challenges in dealing with scenarios of obfuscated algorithmic personalization, idioms of voice, natural gestures and branching conversational design. In short, any dynamic analysis must grapple with undercurrents of often obscure socio-technical interactions that mobilize heterogeneous actors and agencies, and at once separate and augment the operator through the threshold of the interface.

Another side of critical diagnostics, in this respect, opens onto domains of information processing, particularly the logistical space of dataflows and micro-temporalities of digital infrastructures. Intersecting with UX via performance optimization, these design-abilities focus on the technical timings of interfaces in relation to network connections – or what the media theorist Wolfgang Ernst (2016) describes as time critical (Zeitkritische) signal processing. From across global infrastructures, for instance, techno-phenomenological and political economic pressures can converge on an interface, particularly as loading times are modulated to manage traffic from distant servers, ad-tech systems and other content-delivery-networks (CDNs), while still rendering such transactions undetectable to an ordinary user. Poor performance can immediately and dramatically impact on user conversion and engagement rates, essentially dissipating the submersive flow of the interface. Here, developer tools to optimize performance become crucial, like waterfall analysis of calls to application programming interfaces (APIs), the timings of resources being transferred over networks or how long it takes for images to render. Large repositories like the HTTP Archive, moreover, host web performance data over time to assist with the identification of trends, key transformations and exemplary cases of such ‘chrono-design’ over time. Interestingly, surfacing these machinic processes is only possible in the sense of applying one interface to the operations of another, recalling sociological discussions of ‘interface methods’ as new techniques are elaborated through the agential ambiguities within this form of relation (Marres and Gerlitz, 2016). A key illustrative example of how these techniques shift into critique is the recent controversy around Google’s Accelerated Mobile Pages (AMP), where the initiative to ostensibly deliver faster mobile UX was disputed as a corporate incursion into open web standards. Diagnostics, in this case, were utilized to evidence instances of Google artificially ‘throttling’ the speed of its advertising network to encourage widespread adoption (Gooding, 2021). Here, critical diagnostics can resemble a kind of ‘reality testing’ cast into a machinic ensemble of agencies to surface what is and what should not be (Boltanski, 2011).

Other cases of critical diagnostics can assist with examining asymmetries across ad-tech systems. For instance, modified optimization techniques support novel forms of critique that surface behavior between front-end and back-end processes across multiple time-scales to identify elaborate tracking systems that accompany seemingly retro-lightweight forms of web design (Dieter and Gauthier, 2019). While surfacing cookies and scripts or mapping data transfers are well-established forms of digital methods, the measurement of micro-temporal dimensions connects to the interface as a fluid-like form of relation within the political economy of UX. These modes of critique, moreover, are important since the vortex of agencies environmentally enacted through the interface can at once blur and obscure the significance of these actors in techno-phenomenological terms. Such experiments can be considered as a contribution to exoteric interfacing, moreover, since the repurposing of performance optimization in these ways can play an essential role in the inauguration of new publics (McKelvey, 2014).

Augmented critique: Inventing alternative experiences

A final orientation based on interface augmentation can briefly be mentioned. Rather than interrogating the performance of an ordinary interface through diagnostics as noted above, critique arises via design modifications to support the emergence of extraordinary interfaces. This mode arises through design-abilities like hacking, modding, the development of plugins or other alternative applications facilitated through software extensibility that alter an expected scenario of use. Extensibility was first driven by large-scale programming, client customization and the sustainable evolution of software over time, before gaining further traction with the notion of modularity in object-oriented programming. Today, it has been thoroughly absorbed into the economic, infrastructural and governance logics of platforms. Within this context, extensibility enables third-party software development, while securing proprietary infrastructural resources through g'at the use of APIs. Indeed, from the perspective of platform economies, this is understood as promoting generativity through flexible and open affordances that drive the development of platforms and their interrelations as an ecosystem (Schwarz, 2017). Uncertainty, nevertheless, remains as features, apps or entirely new platforms are built on top or alongside existing software, even while these processes also remain highly controlled through multiple dimensions of governance. In this context, augmented critique negotiates and interrogates the social, political and cultural issues attached to platform power by operating from a secondary or auxiliary stance. From the reality of interface dispositions, these projects indicate what might be otherwise by letting a world of alternatives flood in.

Browser extensibility, in particular, is one obvious area where this mode of critique has developed in a variety of ways. While web platforms have been a major ongoing topic in digital media studies, browsers have also long existed as platforms; not only in their diverse roles rendering network protocols and web standards but also as extensible systems branching into heterogeneous milieux of critical technical practice (Schiller, 2021). After the release of Internet Explorer 4.0 in 1999, add-ons have been formally supported by major browsers, including through toolbars and menus, which have supported diverse aberrant expressions. Userscripts, in particular, typically involving lightweight JavaScript have been supported through managers like Greasemonkey (2005), or closely related spin-offs such as Tampermonkey (2010) and Violentmonkey (2013). While appropriated as a source of free labor and innovation, userscripts have been a source of aberrant critique from automating fake logins for site registration to making visible hidden form fields utilized for tracking. Ad-blockers, in general, can be understood as originally a kind of augmented interface critique. Plugin projects like Mozilla’s Lightbeam (2018), first known as Collusion (2011), are also notable for mapping network trackers in real time with stylistic visualizations. These interventions disrupt undercurrents of datafication, challenging regimes of UX in service of surveillance capitalism. Other variations include ‘user style sheets’ implemented by the browser through the rendering of cascading style sheets (CSS) to change the intended appearance of a site, along with style manager extensions, which have been utilized to support and promote accessibility, thus disputing the administration of a presumed universal user. Artworks that rely on plugins to, for instance, perform reductionist aesthetics also make sense when situated in this milieu, including pieces like Rafaël Rozendaal’s Abstract Browsing (2014) or Ben Grosser’s Facebook Demetricator (2012) which leverage ‘code to investigate and critique code… to ask “what if we had a different script?”’ (O’Dwyer, 2021). At its most inventive, therefore, augmented critique opens up an existential challenge to UX by shaking users out of their habits, and interestingly re-establishes both reflexive design and software art within a collective milieu of evolving practices suggestive of a tinkering critical technical republic (Harwood, 2019). Predictably, this kind of extensibility has also come under considerable scrutiny due to its subversive potential. Consider ‘script kiddies’, for instance, as a derogatory term referring to the apparent dangerous accessibility of augmented programs. It is telling that with the ascendency of the platform economy, modes of augmented critique have become increasingly reigned in and tightly policed.

Conclusion

Starting from Agre’s original proposal of CTP and its various influences, this article has ultimately sought to track down and make a case for interface critique at large. Taking a media theoretical approach, I have suggested that such cases are not often recognized together due to the difficulties comprehending the interface as a fluid-like milieu or vortex of heterogeneous agencies. In response, I have sketched out a navigational matrix of three orientations based on – prefiguration, diagnostics and augmentation, and illustrated how modes of critique operate uniquely within these zones. My brief discussion here has tended to focus on interface design-abilities for apps, platforms and the web; no doubt this could be taken further through different examples or refined via empirical research into these modes of critique, including the circulation and reception of their apparent achievements. Nevertheless, I want to emphasize that they offer invaluable, under-appreciated resources for furthering critical knowledge and pursuing progressive ends regarding often esoteric, yet everyday technical subjects. Alongside software art and reflexive design, they exemplify an inventive search for new criteria for critique and, in so doing, provide means to construct new metacritical theories and common capacities from within conditions of saturated immanence.