Sensing data: Encountering data sonifications,materializations,and interactives as knowledge objects

Abstract

Digital and networked media are characterized by a fundamental disconnection between the modes of operating of these media, the human sensorium and knowledge. In order for humans to consciously participate in this expanded domain of sensibility, additional mediation is required to ‘presentify’ what is not accessible to human perception. This situation has motivated the creation of manifold ‘data objects’ in the form of visualizations, soundscapes and sonifications, 3D materializations, and data-driven interactives in the neighboring domains of art, design, science, and humanities scholarship. These new knowledge objects, as we propose to call them, both produce and mediate knowledge in the process of making data experienceable. They turn that what essentially does not correlate to human sensory capacities into something that the human sensorium is capable of engaging with. As such, they can play a crucial role in providing access, and thereby modalities of critical and creative relating, to what Hansen (2015) describes as the expanded domain of sensibility. Yet, the effects and implications of the sensory specificities of these knowledge objects remain underacknowledged, as are the potentials of these modes of presentifying data. Here, we explore and compare a diversity of such knowledge objects and look at their different media modalities and different experiential qualities, and how these afford ways of knowing. By approaching several knowledge objects as ‘theoretical objects’ (Bal (2013); Damisch in Bois et al. (1998), we investigate which and how experimental sensory techniques are used for data presentification. With this, we want to draw attention to the onto-epistemological implications of the design of these knowledge objects, the main implications being relationality and performativity. Understanding the potential and implications of this performativity would benefit from what we call a creative humanities approach that combines insights from the critical and digital humanities with those from the fields of creative arts and design.

In Feed Forward: On the Future of Twenty-First Media, Mark B. Hansen observes that digital and networked media are characterized by a fundamental disconnection between the modes of operating of these media and the human sensorium. He is referring to (among others) micro-sensors, data processors, smart technologies, search engines and how they detect intensities, differences, fluctuations, and patterns inaccessible to human sense perception. Their modes of operating do not directly correlate to human sensory capacities the way 19th- and 20th-century media like photography, cinema, or sound recording do. Rather, they ‘open up an expanded domain of sensibility that can enhance human experience’. To access this domain of sensibility, ‘humans must rely on technologies to perform operations to which they have absolutely no direct access whatsoever and that correlate to no already existent human faculty or capacity’ (Hansen, 2015: 4–5). In order for humans to consciously participate in this expanded domain of sensibility, therefore, additional mediation is required to ‘presentify’ what is not accessible to human perception.

The inaccessibility of (digital) data has motivated the creation of manifold ‘data objects’ that make that what is inaccessible to human perception experienceable in the form of visualizations, soundscapes and sonifications, 3D materializations, and data-driven ‘critical interactives’, following the definition of Cooley and Buell (2012). In the process of making data experienceable, these knowledge objects, as we propose to call them, mediate and produce knowledge. Referring to them as knowledge objects, we want to draw attention to the relationship between the sensory affordances of these objects and how they become part of practices of knowing: how the sensory engagement they afford is a condition for the production of knowledge. They turn what essentially does not correlate to human sensory capacities into something that the human sensorium is capable of engaging with. As such, knowledge objects can play a crucial role in providing access, and thereby afford modalities of critical and creative relating. They are, as such, performative in how they bring about rather than represent these ways of knowing and how they effectuate that what they bring about as a result of how they engage their users. Understanding the potential and implications of this performativity would benefit from what we call a creative humanities approach to data that brings together insights from the digital and critical humanities with those from the creative arts and design.

In the following, we elaborate such a creative humanities approach to knowledge objects as interfaces that are relational, situational, and performative. This approach comprises a combination of three perspectives: a dramaturgical perspective in order to theorize these objects as spatiotemporal design of emergent relations, a curatorial perspective which considers how the principles of selection and framing construes meaning, and a perspective on knowledge objects as dispositifs which brings to the fore how this design constructs spectatorial subjectivity, various forms of agency, and types of knowledge. ¹ We explore and compare a diversity of knowledge objects and look at their different mediating modalities and experiential qualities, and how these afford ways of knowing.

With this integration of perspectives of dramaturgy, curation, and dispositif, we wish to sketch the outlines of a comparative approach. We frame our examples, here, with labels that denote not so much a classification or genre of design but point to the various sensory translations of data and the engagements that they afford. We follow a route via visualization, sonification, materialization, and interactivity, and their respective visual, oral, haptic, and playful affordances. This route aims to not only develop insights in shared or divergent characteristics of various data-based knowledge objects, but also in how creative and artistic, experimental design can play a role in a creative humanities approach to critical data studies that researches through their design how affordances for sensory engagement contribute to the production of sensible) knowledge. To this end, we will approach these knowledge objects as ‘theoretical objects’ (Bal, 2013; Damisch in Bois et al., 1998). To take an object as a theoretical object involves an approach that does not use the object to demonstrate what is the case, but rather, to allow the object to guide us in finding out and theorizing what is the case. What kind of objects are these knowledge objects and how do they mediate in presentifying what is not accessible to the human sensorium? How do these knowledge objects address and engage their human users? How do they activate different sensory modalities? How do they perform as knowledge objects and mediate in ways of knowing? How does this offer new perspectives on the role of sensory perception in our engagement with data sets and the practices of knowing resulting from such engagement? We will show how approached as theoretical objects, these knowledge objects are not simple tools or even artifacts but ‘respond, (“speak back”) to the look cast onto them’ and ‘entice viewers to theorize’ (Bal 2013: 51–52). Especially this latter claim speaks to their affective implications, which are sometimes systematically disregarded (e.g. Crowley and McDonald, 2015), not just in the Digital Humanities discourse with its semi-automated tools affording a seemingly ‘detached’ viewpoint as implied in concepts like distant reading (Moretti, 2013).

We will argue that approaching these knowledge objects as theoretical objects draws attention to the onto-epistemological implications of their design: how objects of knowledge and knowing subjects, and the relationship between them, are effectuated in the intra-actions afforded by the design of these knowledge objects. The design of data-based visualizations, sonifications, materializations, interactives, and applications exemplify such relational understanding in how, rather than (providing the illusion of) presenting a transparent window to aspects of the world previously inaccessible to humans, they are apparatuses that set the stage for intra-actions that engage in and effectuate ways of knowing.

Looking at knowledge objects

A well-known experimental data visualization project from the field of the arts is the website Synchronous Objects created by Norah Zunigah Shaw and Maria Palazzi from Ohio State University and choreographer William Forsythe. ² The website shows 22 different visualizations of what is referred to as ‘systems of organization in the choreography of William Forsythe’s creation One Flat Thing, Reproduced (2000)’. ³ These visualizations are based on spatial data collected by tracking a single point on each dancer in two recordings of this work by Forsythe (one frontal and one from the top), in combination with attribute data derived from the information given by the dancers, for example, about giving and taking cues. Some of these visualizations draw attention to structural characteristics of the dance by means of drawings on a recording and thus present an image of the dance that is quite close to how humans visually perceive the dance. Like, for example, the CueAnnotations, in which lines drawn on video recordings help to recognize what movements performed by one dancer serve as cues for other dancers. Or the AlignmentAnnotations, in which colorful lines drawn on video recordings mark short moments of synchronization between the dancers in which their movements share some, but not necessarily all attributes. The drawings support human ways of looking at the dance and direct attention to what otherwise may go unnoticed.

Other visualizations, however, are quite different from an annotated recording of a human-like perception of the dance, like the MotionVolumes, in which the outer edges of the dancer’s motions are transformed into volumes that emerge, transform, and dissolve in space. Or the CenterSketch that visualizes the accumulation of the dancer’s movement over time. The website explains what is visualized here as follows:

[T]he center of motion of the dance is located, marked with a dot, and then animated in a compressed time frame. Each dot represents the center of movement for all dancers at that moment in the source video. The color shifts over the duration of the animation as the dots accumulate, revealing the passage of time in the change from red to blue. ⁴

These visualizations do not even remotely look like the dance. It is actually quite hard to relate them to the dance as seen with human eyes. These visualizations thus alert us to the difference between human ways of seeing and experiencing the ‘physical’ dance and computer readings of the ‘data’ dance.

The ways of sensing and interpreting of these media technologies expand the sensible and how things can be known in ways different from human perception and understanding. This raises the question of how humans can relate to how technology senses ‘dance’. This distinction illustrates the transformations brought about by what Hansen terms 21st-century media and how these ‘impose a new form of resolutely non-prosthetic technical mediation’ (p. 4, italics in original). The moving visualizations of Synchronous Objects are attempts at mediating such relating. On the Synchronous Objects website, the makers explain that ‘Our objects are not a substitute for the live stage performance of One Flat Thing, Reproduced, but offer alternative sites for understanding Forsythe’s work and seeing its choreographic structures unfold’ (italics added). ⁵ Their project demonstrates how the computer readings of a data dance allow to discover structural patterns in the physical dance that would otherwise go unnoticed, and may even be imperceptible to humans, and how as a result the choreography can be known in new ways.

The project also demonstrates how these readings require visualizations that allow humans to relate to the patterns discovered by, and in, the computer reading, and how these visualizations as knowledge objects mediate in ways of knowing the choreography. The visualizations of the computer readings turn the dance into 22 synchronous knowledge objects that each mediate in coming to know different structural aspects of the choreography. These objects, the makers observe, ‘are a way of sharing ideas: about the dance, about visualizing complexity, about interpreting works of art in unconventional ways’. ⁶ That is, these visualizations are not simply demonstrations of ‘how things are’ with this choreography but experimental ways of affording humans to relate to computer-generated readings of data captured from the choreography.

Describing their visualizations as sites for understanding Forsythe’s work and seeing its choreographic structures ‘unfold’, the makers of Synchronous Objects do not present them as visual representations of aspects of Forsythe’s choreography so much, but rather as sites where understanding may be generated; ‘places’ of encounter where understanding of the work may come about as a result of how they engage those looking at them. In Synchronous Objects, these sites are moving visualizations seen on screen. These visualizations unfold in time and thus take the viewer along in the unfolding of the relationships captured by data processing. They do so in different ways that suggest different ways of understanding the choreographic patterns of Forsythe’s creation and different ways in which we are engaged as a viewer. The CenterSketch invites an understanding of the choreography in terms of an accumulation of traces that remain and that are overwritten time and again, in some places more densely than others. The accumulation of traces mediates in acts of knowing that suggest a position of the agent of knowing as being in a position of overview over time passed so far and, due to the continuous transformation of the color of the traces from red to blue, also of the time still to come. The MotionVolumes, on the other hand, visualizes choreographic structures as constantly shapeshifting volumes occupying space. They invite an understanding of the choreography in terms of the unfolding of 3D volumes that sometimes merge and at other times split up, and that exist only in their unfolding, to leave no traces. In this case, the visualization gives no clue as to the relative position of the transformations observed at any moment in time in relation to the totality of the choreographic structure and thus denies the viewer a position of overview.

Placing 22 of them side by side, the website Synchronous Objects draws attention to their respective differences, and to how the ways in which they are constructed invites different ways of relating and knowing. This is what makes them so interesting as what Hubert Damisch and Mieke Bal have termed ‘theoretical objects’. A theoretical object, Damisch argues:

[…] obliges you to do theory but also furnishes you with the means of doing it. Thus, if you agree to accept it on theoretical terms, it will produce effects around itself…[and] forces us to ask ourselves what theory is. It is posed in theoretical terms; it produces theory; and it necessitates a reflection on theory. (Bois et al., 1998: 8)

Although this description still leaves many questions as to what exactly theoretical objects are, it suggests what theoretical objects can do – what they perform and what they produce. A theoretical object obliges us to respond to it by theorizing. It obliges to further elaborate or rethink theoretical insights and it inspires new theoretical insights. It furnishes us with the means to do so by offering ways to engage with it, if we agree to accept the object on its theoretical terms. That is, if we accept the object to intervene in our thinking and to offer suggestions that may guide us to new insights. Whether an object can function as a theoretical object, therefore, is also a matter of a particular attitude toward it, of allowing it to play this role in our analysis of it. Some objects are better equipped to play this role than others.

For Mieke Bal, the possibility of objects to oblige us to do theory and to furnish us with the means of doing is what makes an approach to objects as theoretical objects such a useful and important alternative to the more common understanding of objects as case studies. Evoking Karen Barad (2007) diffractively, ‘objects’, Bal observes,

are active participants in the performance of analysis in that they enable reflection and speculation; they can contradict projections and wrong-headed interpretations (if the analyst lets them!), and thus constitute a theoretical object with philosophical relevance, whether materially embodied or not. (Bal, 2013: 53)

Accepting these synchronous objects on theoretical terms draws attention to the modes of operating of these objects as sites for understanding and their active participation in analysis (via Bal) and knowledge production (as per Barad). As simultaneous a ‘zone of encounter’ (Hookway, 2014: 12) and a ‘mediating environment’ or ‘critical zone that constitutes a user experience’ (Drucker, 2011: 10), they can be understood relationally and situationally as interfaces. As interfaces, they provide the contours for a sensory encounter with data and through enabling this experiential form of relating and engagement with data, they performatively produce sensible knowledge through and about these data.

In the following, we will take a closer look at a selection of other examples that investigate the possibility of other means to presentify what is (otherwise) imperceptible to humans and see how these objects may guide us in theorizing how they activate different sensory modalities and how as a result they effectuate different ways of knowing. Afforded by the specificities of their design, that is, by their material and structural specificities the objects set the stage for perceiving intra-actions that otherwise stay imperceptible. Such perceptions are the starting point of how approaching them as theoretical objects can give us insight in the ways in which knowledge objects can yield knowledge in the encounter.

Listening to data

Sonification is a way to break with the predominance of visualization as a mode of access to data by using sound for the presentation of otherwise-inaccessible data. More precisely, it is ‘the technique of rendering sound in response to data and interactions’ (Hermann et al., 2011: 1). Rendering such responses in very different ways, sonifications afford different ways of relating to data thus producing very different kinds of knowledge objects. One example is the Music by Oceans project developed by oceanographers Erik van Sebille and Will de Ruijter and composer Stef Veldhuis. Their project takes the data collected by ocean probes for use in weather forecasts and climate models as basis for musical compositions. Ocean probes drift on underwater currents in oceans around the world. They surface every 10 days to transmit via satellite the data they have collected about, among others, currents, temperature, and salinity levels. ‘The sensors collect a huge amount of data’, van Sebille explains,

but at the moment, we can only use nice illustrations and infographics to make sense of it. But is it possible to interpret the data using our other senses as well? Can we perhaps hear something that we cannot observe with our eyes? ⁷

Veldhuis made several compositions on the basis of these data. From a dramaturgical perspective, we may understand these compositions as spatiotemporal designs of emergent relations between sounds produced by musical instruments. They are transitory objects that can only be experienced in their unfolding and in the perception of relationships and differences between simultaneous and successively performed sounds. Veldhuis made his compositions for a string quartet. The choice for these instruments inscribes these knowledge objects in a history of composing, and listening to, Western classical and modern music. This history is also part of the way in which these knowledge objects as dispositifs construct spectatorial subjectivity. This history and the setting in which the audience encounters these objects – as live audience listening to the performances of a string quartet – positions them as a ‘musical audience’ that listens to the creative expression of a musical artist.

As knowledge object, it comes about and is effective in the zone of encounter, the mediating environment in which the audience situationally experiences the composition. This composition is, as the composer describes it in collaborative terms, 50% his and 50% that of the ocean. ⁸ For example, for one composition he composed eight short phrases of ‘cells’ for each instrument and related these to temperatures and salinity levels. The data captured then determined the order of these cells and thus determine the composition of the piece. However, notwithstanding the fact that he sought ways to give the ocean a role in composing the pieces, the composer himself played a key role in what from a curatorial perspective can be observed regarding the principles of selection and framing and their role in the construction of meaning. Although some pieces started from experimenting with direct translations of data in sound, this principle was soon abandoned because the result was not considered musical and Veldhuis did not want his compositions to sound like musical diagram but to appeal as music. To this end, he started looking for ways to translate more emotional and associative responses to the ocean, its temperature and movements into music. In the end, his aim was to create music that would work musically. Using the data to influence the composition of the pieces allowed him to open up the sometimes-rigid musical logic of inherited traditions and compose in new and different ways. This he describes as the main gain of working with the data: how they inspired new ways of working and innovate his ways of composing.

The electroencephalogram (EEG) sonification tools for exploratory analysis of data from psycholinguistic experiments developed by Thomas Hermann, head of the Ambient Intelligence Group at Center of Excellence in Cognitive Interaction Technology at Bielefeld University and colleagues, take a very different approach to sonification than a musical one, even if techniques focus on something that is also addressed in oceanographer van Sebille’s question (quoted above): ‘Can we perhaps hear something that we cannot observe with our eyes?’. The primary application of these tools is to diagnose patients with epileptic symptoms. These EEG sonification tools allow users to listen, observe, and also to interpret patterns in EEG data. While EEG data can also be visualized to afford interpretation, the advantage of sonifying these data is in how they offer alternative ways of making data accessible to perception and interpretation. Human beings, Hermann et al. observe,

…are equipped with a complex and powerful listening system. The act of identifying sound sources, spoken words, and melodies, even under noisy conditions, is a supreme pattern recognition task that most modern computers are incapable of reproducing. The fact that it appears to work so effortlessly is perhaps the main reason that we are not aware of the incredible performance that our auditory system demonstrates every moment of the day, even when we are asleep! Thus, the benefits of using the auditory system as a primary interface for data transmission are derived from its complexity, power, and flexibility. (2011: 2)

Furthermore, following sonification, the users of these systems may now interpret the EEG data while looking at something else, for example, at the patients they are treating.

The knowledge object, here, is not an artistic response to data but an example of what is called ‘auditory display’:

Auditory Display encompasses all aspects of a human-machine interaction system, including the setup, speakers or headphones, modes of interaction with the display system, and any technical solution for the gathering, processing, and computing necessary to obtain sound in response to the data. (Hermann et al., 2011: 1)

The construction of the EEG sonification tools makes the EEG data accessible to the human listening system with its ‘supreme pattern recognition tasks’. The key question for auditory display, in accordance with our analysis, is how that, what is not itself auditory, can be made available to human hearing capabilities in ways that allow humans to make sense of this perception.

From a dramaturgical perspective, the object, here, is an apparatus that sets up relationships between data, sounds, and human listeners. This knowledge object as site of encounter affords interactions between humans and these sounds. This apparatus as dispositif positions the user as an active listener listening for patterns in simultaneous and successive sounds. From a curatorial perspective, it can be observed that principles of selection and framing are guided by the need for detecting relevant patterns in the data. Relevant patterns are, for example, those that help to recognize the rhythmic activity of epileptic seizures as measured by human EEG. Baier et al. explain how epileptic seizures manifest themselves as spontaneous or induced disorder-order transitions in human EEG:

The irregular disordered pattern of normal (ongoing) brain activity turns into a globally ordered rhythmic pattern in the case of the so-called generalized seizures. (…) On the neural level this corresponds to a dramatic change in the degree of synchronization of the firing between neighboring and widely separated neurons. As a consequence, the EEG time series display sharp changes in (temporal) autocorrelations and (spatial) cross-correlations. (2006: n.p.)

Sonifications allow to detect these rhythmic patterns in real time. In their text ‘Sonified Epileptic Rhythms’, Baier et al. describe the choices they made with regard to what events from the EEG data to sonify and also how optimizing the detection of these patterns by the human ear required choices with regard to the volume, duration, and number of harmonics of the tones used. They also observe:

For a real-time application as e.g. clinical EEG monitoring, one fixed set of sonification parameters might not be enough. Rather, one would have the opportunity to adjust the sonification interactively according to e.g. the state of the patient, the pathologic features searched for, the amount of artifacts, etc. (2006: n.p.)

This knowledge object as site for understanding thus includes the technical apparatus that affords adapting the ways of relating through sonification to otherwise imperceptible data and demonstrates Barad’s observations about the inseparability of object of knowledge and agency of observation (scientific apparatus and human listener).

The Music by Oceans project and the EEG sonification tools are two very different examples of the use of sonification to afford ways of relating to data. The Music by Oceans project provides a site for interaction with an artistic response to data. What audiences listen to is a composition inspired by ‘oceanic data’. The encounter with the composition provides access to the composer’s creative ways of relating to the data and his way of turning them into sound. What this may tell us about the possibility to interpret the data using our other senses and whether we can perhaps learn something from that we cannot observe with our eyes remains to be seen. The project is, as van Sebille also observes, exploratory. It is a trying out of what happens when a composer engages with these data, and what this engagement may bring. ⁹

The EEG tools, on the other hand, use sonification to afford interaction with data. The transformation of human EEG into sound makes them accessible to human listeners and their capacity for pattern recognition and the apparatus that is the knowledge object affords them to use this capacity to engage with the data. Addressing the ‘supreme pattern recognition tasks’ of human listening, this knowledge object subverts the ocularcentric bias in conceptions of knowing and knowledge. The object of knowing here is an auditory object that is transitory and relational. The apparatus as dispositif sets the stage for this object of knowledge to emerge from the interactions between the sonifications of the data and the human listener. The capacities for pattern recognition of the listener are co-constitutive of the object of knowledge emerging from the interaction with the data. Such capacities are not the same for everyone and they are not merely given. They can be trained, and they may develop in particular directions as a result of expertise and knowledge. A doctor trained in using the EEG tool will probably develop advanced capacities for recognizing patterns in the sonified data, just as experienced doctors using a stethoscope will be better capable of making sense of what they hear than those just learning to use this instrument. This also raises the question whether, for example, a composer or a musician might be capable of detecting patterns in sonified data that other listeners would not detect and whether this might lead to new observations and interpretation of data, for example, those collected by the ocean probes.

Materializing data

Data materialization, also called data physicalization or in more artistic terms data sculpture, is another domain in which creation, engineering, and research collaborate. While for some, the ubiquitous presence and material impact of datafication of culture is a source of inspiration and critical engagement – for others, the development of new technologies and methods for dynamically displaying and possibly touching data are the focus of investigation. For various scientific fields, physical data objects may enable researchers to demonstrate, analyze, and interpret digital data in radically new ways. The making tangible of data in and about public spaces, however, addresses different questions.

Data sculpture as data-inspired 3D art entails, according to Zhao and Vande Moere (2008), the making of ‘a data-based physical artifact, possessing both artistic and functional qualities, that aims to augment a nearby audience’s understanding of data insights and any socially relevant issues that underlie it’ (p. 343). Here, we understand this augmentation as a form of materialization, giving three-, and in some cases also four-dimensionality to static, numeric, digital data. This means that objects, flows, or processes that are first measured, framed, and translated into fixed numbers can be perceived by humans in their dynamic, material situatedness again. This is enabled by their rematerialization into form, texture and perhaps most importantly, time-based shifting of shape. The aim, then, of data sculpture seems to be, according to Zhao and Vande Moere, to contribute to understanding the fact and impact of data, or perhaps in terms most relevant to our perspective, the presence and performance of data.

Artist and media architect Refik Anadol aims to make, in his words, ‘the invisible visible’ with what he calls parametric data sculptures. He uses trompe l’oeil techniques to make his screen-based installations to be perceived as 3D, moving architectural surfaces. One of his data sculptures is a video wall for the 350 Mission Building, Virtual Depictions: San Francisco that can be seen through the glass facade of the 350 Mission Building (Figure 1). As the artist attests, the work is part of his project to ‘define new poetics of space through media arts and architecture and to create a unique parametric data sculpture that has an intelligence, memory and culture’. ¹⁰

Figure 1.

Refik Anado, Virtual Depictions: San Francisco. Image: Refik Anadol.

As a material object, it is firstly a screen surface that wraps around the corner of the architectural wall of a building, and which has the visual effect of a thick mass. Secondly, the work displays a series of 12 fluidly changing abstract vistas – sometimes colorful, sometimes black-and-white – that, with special optical effects, visually materialize and animate static numeric, digital data from various sources. Anadol uses publicly available, ‘frozen’ data DataSF and Twitter’s real-time API service and combines them with architectural algorithms to create his site-specific work. These animated visualization techniques do not make the data legible as such; it does not invite a distillation of information, but rather awe from these spectacular and also enigmatic visuals. ¹¹

The imagery is abstract and is not accompanied by a legend, scale table, or other tools for interpreting the data materials used for its mediation. For example, the media wall might display real-time geographical information about online Tweets, but not in a familiar map-like, readable image. Instead, the data sets are translated into a gripping visual spectacle of fluid architecture. The trompe l’oeil effect produces the kinetic and haptic appearance of the otherwise flat and fixed surface. The movement of the images makes it seem as though the visual material protrudes from and almost spills out of its frame. The moving surface dynamically expands and transforms its architectural surroundings, and its visual suggestion of material fluidity brings life into the static surface of the façade. As 3D optical illusion, it alludes to a sense of touch and an intimate haptic visual relationship with the image. More than an illusion, this relationality is produced by the dispositif of the installation. As Laura Marks has argued, haptic visuality is not only a quality of a way of looking but can also be attributed to the object of the gaze. This intra-active relationship between image and spectator is constituted, then, by the reciprocity of the hapticity, as the haptic quality of specific imagery invites a specific embodied relationship with the work on screen:

The term haptic visuality emphasizes the viewer’s inclination to perceive haptically, but a work itself may offer haptic images. Haptic images do not invite identification with a figure so much as they encourage a bodily relationship between the viewer and the image. Thus it is more appropriate to speak of the object of a haptic look than to speak of a dynamic subjectivity between looker and image. (Marks, 2002: 3, emphasis in text)

The artist addresses the specific spectatorial positing and his ambition to create a spectatorial experience that is perhaps the opposite of distant and rational interpretation:

[The] main motivation with this seminal media architecture approach is to frame this experience [of the parametric data sculpture] with a meticulously abstract and cinematic site-specific data-driven narration. As a result, this media wall turns into a spectacular public event making direct and phantasmagorical connections to its surroundings through simultaneous juxtapositions. ¹²

Indeed, the origin of the data is not traceable, and its curation is obscured. However, the work’s dramaturgy produces a situational and relational co-presence of both data and subject, through the experience of its temporal unfolding – its performance in the here-and-now.

Between cinema and architecture, the work presentifies originally geo-locative, now-digital data and re-infuses it with materiality and temporality. It makes numeric difference between the discrete data perceptible for a spectator in the form of a spatiotemporal flux, sustained by the optical illusion of three-dimensionality and thickness of the image/surface. Anadol frames this as his architectural intervention in response to the presence of data in and about public space:

Traditionally, architecture cannot produce buildings that transform themselves in response to a[n] environmental data feed. The architecture of the future, however, is enticingly malleable and increasingly collaborative, gathering architects with media artists, designers, programmers and engineers. ¹³

This suggests that for working with digital data, the architect works with new materials and needs new tools. It only underscores how data are about space and time: its origin, its presence, and its performance in public space.

Melting Memories is another project that consists of a series of data sculptures by Anadol, which similarly depicts digital data in 3D animations on a flat media wall (Figure 2). A protruding box frame and abstract animations enhance the illusion of depth and texture. ¹⁴ For this series, however, Anadol experiments with technologies developed by Neuroscape Laboratory at the University of California, San Francisco for neuroscientific research, gathering data sets of the neural mechanisms of cognitive control from an EEG that measures changes in brain wave activity and, as such, can be used to provide evidence of how the brain functions over time. The software reads, analyses, and visualizes the data collected from Alzheimer’s disease research. These data sets are used for the algorithms that feed the 3D visual images that, like Virtual Depictions, suggest dynamic texture and – emphasized by the theme and name of Melting Memories – a fugitive temporality.

Figure 2.

Refik Anadol, Melting Memories. Image: Refik Anadol.

In both installations, while speaking directly to our haptic senses, we cannot touch the light and screen-based images. Movement and transformation also make this object too elusive for detailed exploration and investigation, even if it simultaneously suggests proximity and availability. The origin of the data and its curation – whether in on- or offline spaces, or in the body – is made invisible. The data are presented to reflect upon its presence, rather than to analyze its content. What is primarily relayed in its imagined materialization is data’s fundamental dynamicity and its situatedness. Time, as well as space, is inherent in all data. Moreover, as argued by Loukissas (2019), data are fundamentally ‘local’ as it is created by humans, their machines, in specific places, at specific times, before the data are recontextualized by interfaces, and become indexes to local – and hence, spatiotemporal and situated – knowledge. To use Baradian terminology, this situating is a ‘spacetimemattering’ that we can see going in multiple directions, mediating between past and present and in its unfolding presentifying a future. As Karen Barad (2007) has argued, matter performatively materializes time and space rather that unfolding within them. In this sense, as knowledge objects, the dispositif of both installations do make the spacetimingmattering of data experienceable and, as such yield a form of embodied knowledge about the specific presence of data in public space and in our bodies.

Materialization of data can, obviously, also be found elsewhere, within multiple fields, and for different purposes. The projects of the Mediated Matter Group at the MIT Media lab, for example, include the use of material 3D printing techniques to translate complex 3D data sets into fixed, physical, material objects, that contrary to Anadol’s works can be touched, manipulated, and investigated from various angles (Figure 3). ¹⁵

Figure 3.

Compilation of a variety of 3D printed physicalizations. Image: The Mediated Matter Group at https://www.media.mit.edu/projects/making-data-matter/overview.

The research group proposes that the materiality of these data physicalizations allows for a specific kind of knowledge, enabling what they frame as a ‘comprehensive’ and ‘inherently intuitive’ mode of understanding. As they put it:

Although conventional screen-based media visualizations are known to be effective, it has been argued that physical manifestations of data sets can leverage active and spatial perception skills, enabling a more comprehensive understanding of presented information in an inherently intuitive manner. (Bader et al., 2018: 4–5)

Haptic perception, here, is relying on a physically active subject and can yield what is suggested here, a complete and direct, intuitive, if not intimate, form of knowledge through visual scrutiny. This description suggests a transparency of the medium and direct presentation of information. With the technique of voxel, or 3D-pixel printing, the loss of information that on-screen 3D imagery would entail is minimized, according to the developers. The results of the printing technique are radically new objects combining transparent and colored materials. The printing technique allows to create objects with and inside transparent material and enables a physical visualization of compact time-based manifolds such as unconnected point cloud data, lines and curves, open surfaces, and volumetric data. The transparent material is actually necessary for perceiving and getting insight in the volume and shape of the object. It contains the data object, making it on the one hand possible to hold and manipulate, in order to watch it from all sides, on the other is encapsulates the object. Touch, then, will in this case necessarily always remain on the surface.

These different techniques for, and forms of materialization – the installations screen-based and the other case print-based – work in very different ways. While both invoke touch, as folded into haptic visuality in the encounter between image and spectator, touch in each case appeals to different experiences and forms of knowledge. The curatorial gesture of both ‘use’ data as design material. Anadol’s artworks frame this as an eventful experience of the moment of encounter, the Mediated Matter Group frames their objects with the possibility of interactive investigation. One perhaps contributes most to an experiential understanding of the emergent, spacetimemattering quality of data, the other to the possibility and necessity of movement of the subject to gain insight in the volumetric properties of the data object itself.

Playing data

In another category of knowledge objects, we find a foregrounding of interactivity that characteristically affords play. A playful disposition, especially in the sense of ‘cognitive spontaneity’ (Lieberman, 2014: 84), ‘social spontaneity’, and ‘physical spontaneity’ (both 91), is already implicitly present in several of the other knowledge objects introduced above, yet perhaps not as an explicit and integral design principle. Different from vision, hearing, or touch, ‘play’ is not a genuine physiological sensory capacity but a distinct mode of acting for humans in order to experience and make sense of the world. Play scholar Bernie de Koven speaks of the ‘sense of play’ (De Koven, 2014: 149) to indicate that we play with as many things as we smell, touch, and hear, and that it ‘helps us understand how things work’ (p. 149) independently from language as an intermediary. Thus, we consider play a mode of deliberately juxtaposing sensory inputs to generate new knowledge or new insights into seemingly familiar phenomena.

Few games so far have been designed to function specifically as knowledge objects and vice versa. Yet, it is useful to investigate how critical interactives, that is, applications that ‘mobilize ludic methods in order to engage participants in socially and politically sensitive […] subject matter’ (Buell and Rae Cooley, 2012: 489) can become sites of playful engagement with data. One of the few data objects explicitly designed as a game is Salubrious Nation, created by Diakopoulos et al. (2011) (Figure 4). The authors compare a static infographic visualizing US health parameters with two interactive versions: one based on a ‘guessing game mechanic’ (p. 1720) and the other on eliminating contiguous swathes of the same color (p. 1721); the former has received considerably more attention online and will be central for the argument at hand.

Figure 4.

Salubrious Nation. Image: Nicholas Diakopoulos, Funda Kivran-Swaine, and Mor Naaman (http://www.salubriousnation.com).

Compared to the more distinctly sensory affordances of the knowledge objects above, functional and interface ‘affordances’ (Stanfill, 2015: 1060) in critical interactives combine to explicitly afford playfulness. Most basically, randomizing the counties and data players need to guess, coupled with short play sessions that last no more than a few minutes, promotes curiosity. Moreover, the points metric makes players’ progress observable and comparable, providing feedback to refine one’s hypotheses while playing. However, the ‘productive power in [the] design’ (Stanfill, 2015: 1061) of the interactive also shapes the players’ experience of the underlying data.

The curatorial perspective, that is, the selection and framing of data chosen to describe the eponymous ‘nation’, inevitably unfolds in a more dialogical manner between game rules and player input. For instance, basic demographic factors (e.g. age or average life expectancy) are provided for each county while more complex behavioral or environmental factors (e.g. adult smoking rate or number of fast-food restaurants per capita) have to be inferred to win the game. That selection of data may imply correlations (e.g. between unemployment or poverty rate and ‘binge drinking’), but these have to be instantiated by the player acting on them while honing their guessing strategies.

From a dramaturgical perspective, emergent relations between the data are derived by combining the sensory modalities of vision and touch. The color-coded map is visually reminiscent of established static visualizations, blending aspects of region maps and heatmaps, ¹⁶ but is not static. The counties are dynamically recolored to reflect how they compare to the currently selected slider value. Due to the affordance of real-time input, players often move the slider left and right repeatedly without knowing exactly what to look for; Kirsh and Maglio (1992) already observed similar patterns in their analyses of Tetris players, who would rotate pieces seemingly without purpose because it would prove faster to understand and process their rotations haptically than cognitively (i.e. purely in the player’s mind). This suggests a quasi-tactile access to the data, while the perceptive focus shifts from the individual data to larger geographic areas that change in synchronicity. Yet by evoking tactility, the interface – following Hansen – effectively masks the fact that the underlying data are too vast and heterogeneous to be meaningfully processed by a human user. Instead, the feature contributes to a dispositif, in which the ‘player’ as an experimenter creates multiple consecutive before–after constellations in an attempt to still perform (rather than ‘discover’) correlations in the data.

From a discursive interface analysis (Stanfill, 2015: 1062) perspective, marking the US average on the primary input device, the slider, emphasizes normalization (Hall and Link, 2004), that is, calibrating one’s interpretation of demographic data with reference to (national) averages. In that regard, it notably differs from the experience of most aforementioned knowledge objects. Statistical averages are not just represented as points on a scale but also ‘transcoded’ (using Lev Manovich’s terminology) into a different visual experience, color. Specifically, since the individual demographic parameters in the visualization are difficult to systematically ‘read’ include as clues, players often intuitively choose a value that produces an average shade of orange across the map. Even more, while the threshold of ‘normal’ values in data visualizations is implied, for example, through the scale factor, in Salubrious Nation normalization is part of the ongoing performative engagement with the data. It even applies to the high scores, that is, the data collected from earlier play sessions, and thus applies to the stratification of the players themselves.

As suggested above, the affective dimension of working with data is often strategically disregarded, yet some of the knowledge objects above like Veldhuis’ music composed based on oceanographic data have noticeable affective qualities. This particularly applies to games, as even a ‘serious’ game-like Salubrious Nation incentivizes re-playing and can elicit strong emotional responses. To acknowledge this aspect of knowledge objects, Sherry Turkle’s notion of evocative objects, which emphasizes ‘the inseparability of thought and feeling in our relationship to things’ (Turkle, 2007: 5), can be of use. Knowledge objects, and games in particular, are evocative because, through repeated use, they contain ‘traces’ of their users and become ‘companions to our emotional lives [but also] provocations to thought’ (p. 5). In Salubrious Nation, specific data and data relations can become affectively significant depending on the gameplay context players encounter them in, for example, as part of a question they were able to answer surprising well or based on associations accumulated through subsequent questions.

As indicated, this section did not address a different sensory mode but proposes to reconsider the knowledge objects above through the lens of play to address facets that might otherwise be easily missed. Games themselves are not (yet) established as knowledge objects to make data experienceable, also because they often reflect the primacy of visuality rather than seeking to afford multi-sensory engagement. For instance, senior data visualization engineer at Netflix Elijah Meeks (2018) calls for developing ‘analytical applications that can be informed by game design and mechanics’ based on the contentious premise ‘that video games and data visualization are related in that they both show information visually’. Instead, it would be more productive to focus, for example, on how rules of play, not unlike entangled objects, ‘lack an independent, self-contained existence’ (Barad, 2007: ix), that is, how we understand games as well as the subjectivity of players as created through play rather than pre-existing.

After-encounter

In this essay, we have proposed an understanding of data visualizations, soundscapes and sonifications, 3D materializations, and interactives as knowledge objects. Moreover, we have shown how approaching them as ‘theoretical objects’ and accepting them on theoretical terms can illuminate how they operate as sites of encounter with data: how their design and the specificities of their materiality set the stage for intra-actions that effectuate ways of knowing. Karen Barad (2007) introduces the term intra-actions (as an alternative to ‘interactions’) as part of a radical relational approach to knowing in being in which relata are understood to emerge from relationships rather than to precede them. From such radical relational perspective, objects of knowledge cannot be separated from the apparatus within which they are produced, nor can the agencies of the acts of knowing. ‘Objects of knowledge’ being the (temporary) outcome of a process of knowing in being, ‘knowledge objects’ must be situated right in the middle of that process. They are objects but actively participating in the process of making themselves, and entangled aspects of the world, known. Objects of knowledge conceal access to the ongoing intra-active processes that the human knower is also participating in. Knowledge objects, on the other hand, put these processes center stage. The sensorium is the domain where this can take place.

Rather than (providing the illusion of) presenting a transparent window to aspects of the world previously inaccessible to humans, the objects of knowledge discussed above are apparatuses that mediate in relating to that what is perceptually inaccessible to humans. From a combination of a dramaturgical perspective, a curatorial perspective, and a perspective on knowledge objects as dispositifs, we have investigated the relationships between the construction of these knowledge objects, their modes of operating and their performativity: how they effectuate rather than represent ways of knowing. These knowledge objects, indeed, are performative because of how they bring about rather than represent ways of knowing. Understanding how these ways of knowing are brought about requires what Barad terms a posthumanist performative approach.

Defining knowledge objects as ‘sites of encounter’ that activate a combination of different sensory modalities emphasizes the performativity of knowledge production, and the potential of, and for, digital humanities research taking place within these sites. Analyzing and comparing different sites of encounter with data from an expanded perspective that includes the sciences and the (digital) humanities, but also the creative fields of performance, art, and design – a perspective of creative humanities – is an important step in developing our data literacy and, perhaps most importantly, our sense of data. Additionally, this adds nuance to the common dichotomy between ‘digital tools’ and ‘digital artifacts’ (Ramsay and Rockwell, 2012: 77). We see our proposal as related to Joanna Drucker’s call for a humanities approach to interface theory. In her 2011 essay with the same name, she states: ‘Interface theory has to take into account the user/viewer, as a situated and embodied subject, and the affordances of a graphical environment that mediates intellectual and cognitive activities’ (2011: 16). Elsewhere, she calls for a ‘humanistic interface design’ that ‘applies’ a theory of performative materiality (2013: 1). Much in line with her double focus on analysis and design, we argue that scholarship and design practice share much of the same goals and we see the meeting of both domains as part of a creative humanities perspective that we work with, here.